hlt.language.io

Class StreamTokenizer

java.lang.Object

hlt.language.io.AbstractStreamTokenizer

hlt.language.io.StreamTokenizer

public class StreamTokenizer
extends AbstractStreamTokenizer

This provides the exact same interface as that of the standard Java Core API's java.io.StreamTokenizer but with a simpler (and IMHO sounder) implementation, as well as a more complete and more general set of methods (see AbstractStreamTokenizer).

Version:

Last modified on Mon Nov 26 18:30:39 2012 by hak

Author:

Hassan Aït-Kaci

See Also:

AbstractStreamTokenizer, LAStreamTokenizer

Copyright:

© by the author

Field Summary

Fields inherited from class hlt.language.io.AbstractStreamTokenizer

COMMENT, isInteger, NUMERIC, nval, ORDINARY, QUOTE, SPECIAL, sval, TT_EOF, TT_EOL, TT_NOTHING, TT_NUMBER, TT_SPECIAL, TT_WORD, ttype, WHITESPACE, WORD

Constructor Summary

Constructors

Constructor and Description
StreamTokenizer(java.io.Reader rd) Creates a StreamTokenizer for the specified Reader and sets the default syntax.

Method Summary

Methods

Modifier and Type	Method and Description
void	commentChar(int c) Makes the specified character a comment start character exclusively.
void	disableComments()
void	enableComments()
boolean	isCommentChar(int c) Checks whether this character's type is comment, among other types.
int	nextToken() Reads a token and returns its type, which is also the value of ttype.
void	setEscapeChar(int c) Sets the default escape character.
void	setTypePrecedence(int first, int second, int third, int forth, int fifth) Changes the precedence ordering among the character types to the order of the specified parameters.
void	slashSlashComments(boolean flag) C++-style comments will be skipped iff the argument is true.
void	slashStarComments(boolean flag) C-style comments will be skipped iff the argument is true.

Methods inherited from class hlt.language.io.AbstractStreamTokenizer

currentChar, eolIsSignificant, getFile, getLineNumber, ignoreNonDecimals, isNumericChar, isOrdinaryChar, isQuoteChar, isWhitespaceChar, isWordChar, lineno, location, ordinaryChar, ordinaryChars, ordinaryChars, parseDigitsOnly, parseNonDecimals, parseNonDecimals, parseNumbers, parseNumbers, parsingDigitsOnly, peek, peekBack, pushBack, quoteChar, quoteChar, quotePair, quotePair, resetSyntax, setDefaultSyntax, setOrdinaryChar, setOrdinaryChars, setQuoteChar, setQuoteChar, setQuotePair, setType, setType, setType, setWhitespaceChar, setWhitespaceChars, setWordChar, setWordChars, skipChar, skipChar, spaceIsSignificant, tokenEnd, tokenStart, toString, unsetType, unsetType, unsetType, whitespaceChar, whitespaceChars, whitespaceChars, wordChar, wordChars, wordChars

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

StreamTokenizer

public StreamTokenizer(java.io.Reader rd)

Creates a StreamTokenizer for the specified Reader and sets the default syntax.

Method Detail

enableComments

public final void enableComments()

Specified by:

enableComments in class AbstractStreamTokenizer

disableComments

public final void disableComments()

Specified by:

disableComments in class AbstractStreamTokenizer

commentChar

public final void commentChar(int c)

Makes the specified character a comment start character exclusively.

Specified by:

commentChar in class AbstractStreamTokenizer

slashSlashComments

public final void slashSlashComments(boolean flag)

C++-style comments will be skipped iff the argument is true.

Specified by:

slashSlashComments in class AbstractStreamTokenizer

slashStarComments

public final void slashStarComments(boolean flag)

C-style comments will be skipped iff the argument is true.

Specified by:

slashStarComments in class AbstractStreamTokenizer

nextToken

public final int nextToken()
                    throws java.io.IOException

Reads a token and returns its type, which is also the value of ttype. The possible token types are:

TT_NOTHING

if no token has been read yet;

TT_EOF

if the end of file has been read;

TT_EOL

if the end of line is significant and has been read;

TT_WORD

if a word has been read;

TT_NUMBER

if number parsing is effective and a number has been read;

a quote character

if a quoted word has been read;

a white space

if white spaces are significant and one has been read;

an ordinary character

otherwise.

If the type is TT_WORD or a quote character, the value of the word read is in sval. If the type is TT_NUMBER, the value of the number is in nval. If the number read is an integer, the flag isInteger is set to true.

Specified by:

nextToken in class AbstractStreamTokenizer

Throws:

java.io.IOException

isCommentChar

public final boolean isCommentChar(int c)

Checks whether this character's type is comment, among other types.

setEscapeChar

public final void setEscapeChar(int c)

Sets the default escape character.

setTypePrecedence

public void setTypePrecedence(int first,
                     int second,
                     int third,
                     int forth,
                     int fifth)
                       throws java.lang.Exception

Changes the precedence ordering among the character types to the order of the specified parameters. For example, the default ordering is set by:

setTypePrecedence(WORD,WHITESPACE,COMMENT,QUOTE,ORDINARY);

and this means that the WORD type is considered in priority, then WHITESPACE, etc...

The NUMERIC type is not among those that can be ordered because it is relevant only when numbers are parsed. If they are, then the full Java number format is recognized independently of the character syntax setting. This means that NUMERIC has always highest precedence if number parsing is effective, and ignored otherwise.

Throws:

java.lang.Exception