# Tokenizer Class

      template <
            class TokenizerFunc = char_delimiters_separator<char>,
            class Iterator = std::string::const_iterator,
            class Type = std::string
      >
      class tokenizer

The tokenizer class provides a container view of a series of tokens contained in a sequence. You set the sequence to parse and the TokenizerFunction to use to parse the sequence either upon construction or using the assign member function. Note: No parsing is actually done upon construction. Parsing is done on demand as the tokens are accessed via the iterator provided by begin.

## Example

    // simple_example_1.cpp
    #include<iostream>
    #include<boost/tokenizer.hpp>
    #include<string>

    int main(){
       using namespace std;
       using namespace boost;
       string s = "This is,  a test";
       tokenizer<> tok(s);
       for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
           cout << *beg << "\n";
       }
    }

The output from simple_example_1 is:

> `This
> is
> a
> test`

### Template Parameters

Parameter | Description
--- | ---
`TokenizerFunc` | The TokenizerFunction used to parse the sequence.
`Iterator` | The type of the iterator the specifies the sequence.
`Type` | The type of the token, typically string.

## Related Types

Type | Remarks
iterator | The type returned by begin and end. Note: the category of iterator will be at most ForwardIterator. It will be InputIterator if the Iterator template parameter is an InputIterator. For any other category, it will be ForwardIterator.
const_iterator | Same type as iterator.
value_type | Same type as the template parameter Type
reference | Same type as value_type&
const_reference | Same type as const reference
pointer | Same type as value_type*
const_pointer | Same type as const pointer
size_type | void
difference_type | void

## Construction and Member Functions

    tokenizer(Iterator first, Iterator last,const TokenizerFunc& f = TokenizerFunc())

    template<class Container>
    tokenizer(const Container& c,const TokenizerFunc& f = TokenizerFunc())

    void assign(Iterator first, Iterator last)

    void assign(Iterator first, Iterator last, const TokenizerFunc& f)

    template<class Container>
    void assign(const Container& c)

    template<class Container>
    void assign(const Container& c, const TokenizerFunc& f)

    iterator begin() const

    iterator end() const

Parameter | Description
c | A container that contains the sequence to parse. Note: c.begin() and c.end() must be convertible to the template parameter Iterator.
f | A functor that is a model of TokenizerFunction that will be used to parse the sequence.
first | The iterator that represents the beginning position in the sequence to be parsed.
last | The iterator that represents the past the end position in the sequence to be parsed.

Revised 16 February, 2008

Copyright © 2001 John R. Bandela

Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)