edu.ucsb.cs56.pconrad.parsing.tokenizer

Class Tokenizer

java.lang.Object

edu.ucsb.cs56.pconrad.parsing.tokenizer.Tokenizer

public class Tokenizer
extends Object

A simple tokenizer based on a Finite State Automaton

Author:

Phill Conrad

See Also:

FiniteStateAutomaton

Constructor Summary

Constructors

Constructor and Description
Tokenizer(String input)

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args) main method that can be used for interactive testing of the tokenizer.
static FiniteStateAutomaton	makeFSA() Creates a specific instance of FiniteStateAutomaton with the states and transitions needed to parse the language implemented by this arithmetic expression evaluator.
ArrayList<Token>	tokenize() Convert the input for this tokenizer into a sequence of tokens
static Token[]	tokenizeToArray(String input) Convenience method for use in testing that takes input as a string, and returns a plain old java array of Token objects

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(String input)

Method Detail

makeFSA

public static FiniteStateAutomaton makeFSA()

Creates a specific instance of FiniteStateAutomaton with the states and transitions needed to parse the language implemented by this arithmetic expression evaluator.

Returns:

an instance of FiniteStateAutomaton that can be used to tokenize a string

tokenize

public ArrayList<Token> tokenize()

Convert the input for this tokenizer into a sequence of tokens

Returns:

list of tokens

tokenizeToArray

public static Token[] tokenizeToArray(String input)

Convenience method for use in testing that takes input as a string, and returns a plain old java array of Token objects

Parameters:

input - string

Returns:

array of tokens

main

public static void main(String[] args)

main method that can be used for interactive testing of the tokenizer. Tokenizes hard coded string "2+2" unless a command line argument is passed, in which case it parses that.

Parameters:

args - Command line arguments, where arg[0] is the string to parse, if present