ModErn Text Analysis
META Enumerates Textual Applications
Public Member Functions | List of all members
meta::analyzers::token_stream Class Referenceabstract

Base class that represents a stream of tokens that have been extracted from a document. More...

#include <token_stream.h>

Public Member Functions

virtual std::string next ()=0
 Obtains the next token in the sequence.
 
virtual operator bool () const =0
 Determines whether there are more tokens available in the stream.
 
virtual void set_content (std::string &&content)=0
 Sets the content for the stream. More...
 
virtual ~token_stream ()=default
 Destructor.
 
virtual std::unique_ptr< token_streamclone () const =0
 Clones the given token stream. More...
 

Detailed Description

Base class that represents a stream of tokens that have been extracted from a document.

These tokens may be raw tokens (in the case of a tokenizer class) or filtered tokens (from the filter classes).

Member Function Documentation

virtual void meta::analyzers::token_stream::set_content ( std::string &&  content)
pure virtual

Sets the content for the stream.

Parameters
contentThe string content to set
virtual std::unique_ptr<token_stream> meta::analyzers::token_stream::clone ( ) const
pure virtual

Clones the given token stream.

Returns
a unique_ptr to copy this object

The documentation for this class was generated from the following file: