Class Tokenizer

Namespace

RandomizerCore.StringParsing

Assembly

RandomizerCore.dll

Converts a raw input string to a Token stream.

public class Tokenizer

Inheritance

object

Tokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

Tokenizer(IOperatorProvider, string, char?)

Constructs a tokenizer

public Tokenizer(IOperatorProvider operatorProvider, string input, char? stringDelimiter)

Parameters

operatorProvider IOperatorProvider

The operator definition provider to use when tokenizing

input string

The input string

stringDelimiter char?

If the grammar supports strings, the delimiting character that appears around strings otherwise null.

Methods

Tokenize()

Tokenizes the input

public List<Token> Tokenize()

Returns

List<Token>

Exceptions

TokenizingException

When an unrecoverable tokenization error occurs