AbstractStreamTokenizer (API documentation of hlt.language packages)

java.lang.Object
- hlt.language.io.AbstractStreamTokenizer

Direct Known Subclasses:

LAStreamTokenizer, StreamTokenizer
```
public abstract class AbstractStreamTokenizer
extends java.lang.Object
```
There are two concrete subclasses of this abstract class: hlt.language.io.StreamTokenizer and hlt.language.io.LAStreamTokenizer. They provide the exact same interface as that of the standard Java Core API's java.io.StreamTokenizer but with a simpler (and IMHO sounder) implementation. One may use both subclasses instead of java.io.StreamTokenizer without change. However, they support a more complete and more general set of methods. In addition, hlt.language.io.LAStreamTokenizer provides a method for arbitrary lookaheads:
```
 public final String lookahead (int n) throws IOException
 
```
which returns the string of n characters in the input stream including and starting with the current character.
The syntax configuration used as the default setting of the tokenizer is such that:
- the type precedence ordering is set to: WORD, WHITESPACE, COMMENT, QUOTE, and ORDINARY;
- control characters and space are WHITESPACE characters;
- letters and numerals are WORD characters;
- _ and . are WORD characters;
- single quote and double quote are quote characters;
- end-of-line is not significant;
- white spaces are not significant;
- all other characters are ordinary;
- parsing numbers is effective;
- C-style and C++-style comments are recognized and skipped.
This default configuration is reinstated with setDefaultSyntax().
See Also:
StreamTokenizer, LAStreamTokenizer

Field Summary

Fields
Modifier and Type	Field and Description
`static byte`	`COMMENT`
`boolean`	`isInteger` This is set to true whenever an integer number has been read.
`static byte`	`NUMERIC`
`double`	`nval` This contains the token value whenever a number has been read.
`static byte`	`ORDINARY` The following are public constants denoting recognized token types.
`static byte`	`QUOTE`
`static byte`	`SPECIAL`
`java.lang.String`	`sval` This contains the token string when a word, or a quoted word, has been read.
`static int`	`TT_EOF` This value indicates that the end of file has been read.
`static int`	`TT_EOL` This value indicates that an end of line has been read.
`static int`	`TT_NOTHING` This value indicates that no token has been read yet.
`static int`	`TT_NUMBER` This value indicates that a number has been read.
`static int`	`TT_SPECIAL` This value indicates that a special token has been returned.
`static int`	`TT_WORD` This value indicates that a word has been read.
`int`	`ttype` This always contains the type of token just read.
`static byte`	`WHITESPACE`
`static byte`	`WORD`

Constructor Summary

Constructors
Constructor and Description

AbstractStreamTokenizer()

Constructors
Constructor and Description
`AbstractStreamTokenizer()`

Method Summary

Methods
Modifier and Type	Method and Description
`abstract void`	`commentChar(int c)` Makes the specified character a comment start character exclusively.
`int`	`currentChar()` returns the current character.
`abstract void`	`disableComments()`
`abstract void`	`enableComments()`
`void`	`eolIsSignificant(boolean flag)` The end-of-line is recognized as a token iff the argument is `true`.
`java.lang.String`	`getFile()` Returns the current file if any, or `null`.
`int`	`getLineNumber()` Returns the current line number.
`void`	`ignoreNonDecimals()` Specifies that octal and hexadecimal numbers should not be parsed.
`boolean`	`isNumericChar(int c)` Checks whether this character's type is numeric, among other types.
`boolean`	`isOrdinaryChar(int c)` Checks whether this character's type is ordinary, among other types.
`boolean`	`isQuoteChar(int c)` Checks whether this character's type is quote, among other types.
`boolean`	`isWhitespaceChar(int c)` Checks whether this character's type is whitespace, among other types.
`boolean`	`isWordChar(int c)` Checks whether this character's type is word, among other types.
`int`	`lineno()` Returns the current line number.
`java.lang.String`	`location()`
`abstract int`	`nextToken()` Reads a token and returns its type, which is also the value of `ttype`.
`void`	`ordinaryChar(int c)` Makes the specified character ordinary exclusively.
`void`	`ordinaryChars(int from, int to)` Makes all characters between the first and second arguments inclusive ordinary exclusively.
`void`	`ordinaryChars(java.lang.String chars)` Sets all characters in this string to be ordinary characters, exclusively.
`void`	`parseDigitsOnly()` Sets this tokenizer to parse numbers, but only whole numbers made out of maximal sequences of numerical digits, considering as non-numeric characters the `'.'`, `'+'`, `'-'`, as well as letters (i.e., `'e'`, `'E'`, `'f'`, `'F'`, `'l'`, `'L'`, `'d'`, or `'D'`))
`void`	`parseNonDecimals()` Specifies that octal and hexadecimal numbers should be parsed.
`void`	`parseNonDecimals(boolean flag)`
`void`	`parseNumbers()` Specifies that numbers should be parsed.
`void`	`parseNumbers(boolean flag)`
`boolean`	`parsingDigitsOnly()`
`int`	`peek()` Returns the next character to be read.
`int`	`peekBack()` Returns the previous character that was read.
`void`	`pushBack()` If called, the next call to `NextToken()` will return the same token again.
`void`	`quoteChar(int c)` Makes the specified character a quote character exclusively.
`void`	`quoteChar(int c, int e)` Makes the specified character a quote character exclusively, and specifies its escape character.
`void`	`quotePair(int left, int right)` Same as above with implicit default escape character.
`void`	`quotePair(int left, int right, int escape)` Defines a pair of characters as a pair of left and right quotes.
`void`	`resetSyntax()` Make all characters ordinary, do not parse numbers, and do not recognize comments.
`void`	`setDefaultSyntax()` Resets the syntax configuration of the tokenizer to the default setting.
`void`	`setOrdinaryChar(int c)` Sets this character to be an ordinary character, non-exclusively.
`void`	`setOrdinaryChars(java.lang.String chars)` Sets all characters in this string to be ordinary characters, non-exclusively.
`void`	`setQuoteChar(int c)` Same as above with implicit default escape character.
`void`	`setQuoteChar(int c, int e)` Sets this character to be a quote character, non-exclusively, and specifies its escape character.
`void`	`setQuotePair(int left, int right, int escape)` Same as `quotePair` but `left`'s `QUOTE_TYPE` is non-exclusive.
`void`	`setType(int c, int type)` Set the type of this character to be the one specified, non-exclusively.
`void`	`setType(int low, int high, int type)` Set the type of all characters in this range to be the one specified, non-exclusively.
`void`	`setType(java.lang.String chars, int type)` Set the type of all characters in this string to be the one specified, non-exclusively.
`void`	`setWhitespaceChar(int c)` Sets this character to be a whitespace character, non-exclusively.
`void`	`setWhitespaceChars(java.lang.String chars)` Sets all characters in this string to be whitespace characters, non-exclusively.
`void`	`setWordChar(int c)` Sets this character to be a word character, non-exclusively.
`void`	`setWordChars(java.lang.String chars)` Sets all characters in this string to be word characters, non-exclusively.
`void`	`skipChar()` A public method for skipping the current character in the input.
`void`	`skipChar(boolean flag)` Same as `skipChar()`, but if the specified flag is `false` (resp., `true`) resets the start (resp., end) location of the current token.
`abstract void`	`slashSlashComments(boolean flag)` C++-style comments will be skipped iff the argument is `true`.
`abstract void`	`slashStarComments(boolean flag)` C-style comments will be skipped iff the argument is `true`.
`void`	`spaceIsSignificant(boolean flag)` White spaces are recognized as tokens iff the argument is `true`.
`Location`	`tokenEnd()` Returns the location of the end of the token at hand.
`Location`	`tokenStart()` Returns the location of the start of the token at hand.
`java.lang.String`	`toString()` Returns a printable value of the current state of this `StreamTokenizer`.
`void`	`unsetType(int c, int type)` Unset the specified type, but no other one, for this character.
`void`	`unsetType(int low, int high, int type)` Unset the specified type, but no other one, for all characters in this range.
`void`	`unsetType(java.lang.String chars, int type)` Unset the specified type, but no other one, for all characters in this string.
`void`	`whitespaceChar(int c)` Sets this character to be a whitespace character, exclusively.
`void`	`whitespaceChars(int from, int to)` Makes all characters between the first and second arguments inclusive white space characters exclusively.
`void`	`whitespaceChars(java.lang.String chars)` Sets all characters in this string to be whitespace characters, exclusively.
`void`	`wordChar(int c)` Sets this character to be a word character, exclusively.
`void`	`wordChars(int from, int to)` Makes all characters between the first and second arguments inclusive word characters exclusively.
`void`	`wordChars(java.lang.String chars)` Sets all characters in this string to be word characters, exclusively.

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - ORDINARY
```
public static final byte ORDINARY
```
    The following are public constants denoting recognized token types.
    
    See Also:
    Constant Field Values
  - WHITESPACE
```
public static final byte WHITESPACE
```
    See Also:
    Constant Field Values
  - QUOTE
```
public static final byte QUOTE
```
    See Also:
    Constant Field Values
  - WORD
```
public static final byte WORD
```
    See Also:
    Constant Field Values
  - NUMERIC
```
public static final byte NUMERIC
```
    See Also:
    Constant Field Values
  - COMMENT
```
public static final byte COMMENT
```
    See Also:
    Constant Field Values
  - SPECIAL
```
public static final byte SPECIAL
```
    See Also:
    Constant Field Values
  - TT_EOF
```
public static final int TT_EOF
```
    This value indicates that the end of file has been read.
    
    See Also:
    Constant Field Values
  - TT_EOL
```
public static final int TT_EOL
```
    This value indicates that an end of line has been read. This happens iff eolIsSignificant has been set to true.
    
    See Also:
    Constant Field Values
  - TT_WORD
```
public static final int TT_WORD
```
    This value indicates that a word has been read. The field sval contains the string. All alphabetical letters are word characters by default. So are the numerals '0' to '9' and the characters '_', and '.'.
    
    See Also:
    Constant Field Values
  - TT_NUMBER
```
public static final int TT_NUMBER
```
    This value indicates that a number has been read. The field sval contains the number as a double.
    This happens whenever parseNumbers() (or parseNumbers(true)) has been called (default) and parseNumbers(false) has not yet been called. Note that the characters '0' to '9' and the characters '-', '+', and '.' are always considered numeric when parsing numbers is on.
    The format of numbers recognized is that of Java, including octal (0...) and hexadecimal (0x... or 0X...) integers, and complete floating point format. Type letter suffixes (i.e., 'l' or 'L' for integers, and 'f', 'F', 'd', 'D' for floating point numbers, are also recognized, but they are ignored since the value returned is always a double.
    See Also: Constant Field Values
  TT_NOTHING public static final int TT_NOTHING This value indicates that no token has been read yet. See Also: Constant Field Values TT_SPECIAL public static final int TT_SPECIAL This value indicates that a special token has been returned. See Also: Constant Field Values ttype public int ttype This always contains the type of token just read. It is equal to: TT_NOTHING, if nothing has been read yet. TT_EOF, if the end of file has been read. TT_EOL, if the end of line is significant and has been read. TT_NUMBER, if numbers are parsed and a number has been read. TT_WORD, is a word has been read. Any character can be declared to be a word character with wordChar(...). a quote character, if a quoted word has been read. The word value is then in sval. The single quote character and the double quote character are quote characters by default. Any character can be declared to be a quote character with quoteChar(...). Distinct left/right quotes can also be defined with quotePair(...). An ordinary character, otherwise. Any character can be declared to be an ordinary character with ordinaryChar(...). TT_SPECIAL, if nothing else above applies. sval public java.lang.String sval This contains the token string when a word, or a quoted word, has been read. nval public double nval This contains the token value whenever a number has been read. isInteger public boolean isInteger This is set to true whenever an integer number has been read.
Constructor Detail AbstractStreamTokenizer public AbstractStreamTokenizer() Method Detail commentChar public abstract void commentChar(int c) Makes the specified character a comment start character exclusively. slashSlashComments public abstract void slashSlashComments(boolean flag) C++-style comments will be skipped iff the argument is true. slashStarComments public abstract void slashStarComments(boolean flag) C-style comments will be skipped iff the argument is true. nextToken public abstract int nextToken() throws java.io.IOException Reads a token and returns its type, which is also the value of ttype. Throws: java.io.IOException enableComments public abstract void enableComments() disableComments public abstract void disableComments() tokenStart public Location tokenStart() Returns the location of the start of the token at hand. tokenEnd public Location tokenEnd() Returns the location of the end of the token at hand. getLineNumber public final int getLineNumber() Returns the current line number. lineno public final int lineno() Returns the current line number. For compatibility with java.io.StreamTokenizer. getFile public final java.lang.String getFile() Returns the current file if any, or null. location public java.lang.String location() ordinaryChars public final void ordinaryChars(int from, int to) Makes all characters between the first and second arguments inclusive ordinary exclusively. wordChars public final void wordChars(int from, int to) Makes all characters between the first and second arguments inclusive word characters exclusively. whitespaceChars public final void whitespaceChars(int from, int to) Makes all characters between the first and second arguments inclusive white space characters exclusively. quoteChar public final void quoteChar(int c) Makes the specified character a quote character exclusively. quoteChar public final void quoteChar(int c, int e) Makes the specified character a quote character exclusively, and specifies its escape character. ordinaryChar public final void ordinaryChar(int c) Makes the specified character ordinary exclusively. eolIsSignificant public final void eolIsSignificant(boolean flag) The end-of-line is recognized as a token iff the argument is true. spaceIsSignificant public final void spaceIsSignificant(boolean flag) White spaces are recognized as tokens iff the argument is true. parseNumbers public final void parseNumbers() Specifies that numbers should be parsed. parseNonDecimals public final void parseNonDecimals() Specifies that octal and hexadecimal numbers should be parsed. ignoreNonDecimals public final void ignoreNonDecimals() Specifies that octal and hexadecimal numbers should not be parsed. parseDigitsOnly public final void parseDigitsOnly() Sets this tokenizer to parse numbers, but only whole numbers made out of maximal sequences of numerical digits, considering as non-numeric characters the '.', '+', '-', as well as letters (i.e., 'e', 'E', 'f', 'F', 'l', 'L', 'd', or 'D')) parsingDigitsOnly public final boolean parsingDigitsOnly() parseNumbers public final void parseNumbers(boolean flag) parseNonDecimals public final void parseNonDecimals(boolean flag) pushBack public final void pushBack() If called, the next call to NextToken() will return the same token again. toString public java.lang.String toString() Returns a printable value of the current state of this StreamTokenizer. Overrides: toString in class java.lang.Object quotePair public final void quotePair(int left, int right, int escape) Defines a pair of characters as a pair of left and right quotes. This allows tokenizing words quoted by two different characters. Note that only the left quote is categorized as QUOTE_TYPE. The third argument specifies the escape character: if 0, no escape is allowed within these quotes. quotePair public final void quotePair(int left, int right) Same as above with implicit default escape character. setQuotePair public final void setQuotePair(int left, int right, int escape) Same as quotePair but left's QUOTE_TYPE is non-exclusive. setQuoteChar public final void setQuoteChar(int c) Same as above with implicit default escape character. setQuoteChar public final void setQuoteChar(int c, int e) Sets this character to be a quote character, non-exclusively, and specifies its escape character. setOrdinaryChar public final void setOrdinaryChar(int c) Sets this character to be an ordinary character, non-exclusively. wordChar public final void wordChar(int c) Sets this character to be a word character, exclusively. setWordChar public final void setWordChar(int c) Sets this character to be a word character, non-exclusively. wordChars public final void wordChars(java.lang.String chars) Sets all characters in this string to be word characters, exclusively. setWordChars public final void setWordChars(java.lang.String chars) Sets all characters in this string to be word characters, non-exclusively. whitespaceChar public final void whitespaceChar(int c) Sets this character to be a whitespace character, exclusively. setWhitespaceChar public final void setWhitespaceChar(int c) Sets this character to be a whitespace character, non-exclusively. whitespaceChars public final void whitespaceChars(java.lang.String chars) Sets all characters in this string to be whitespace characters, exclusively. setWhitespaceChars public final void setWhitespaceChars(java.lang.String chars) Sets all characters in this string to be whitespace characters, non-exclusively. ordinaryChars public final void ordinaryChars(java.lang.String chars) Sets all characters in this string to be ordinary characters, exclusively. setOrdinaryChars public final void setOrdinaryChars(java.lang.String chars) Sets all characters in this string to be ordinary characters, non-exclusively. isOrdinaryChar public final boolean isOrdinaryChar(int c) Checks whether this character's type is ordinary, among other types. isNumericChar public final boolean isNumericChar(int c) Checks whether this character's type is numeric, among other types. isWordChar public final boolean isWordChar(int c) Checks whether this character's type is word, among other types. isWhitespaceChar public final boolean isWhitespaceChar(int c) Checks whether this character's type is whitespace, among other types. isQuoteChar public final boolean isQuoteChar(int c) Checks whether this character's type is quote, among other types. peek public final int peek() Returns the next character to be read. peekBack public final int peekBack() Returns the previous character that was read. skipChar public final void skipChar() throws java.io.IOException A public method for skipping the current character in the input. Throws: java.io.IOException skipChar public final void skipChar(boolean flag) throws java.io.IOException Same as skipChar(), but if the specified flag is false (resp., true) resets the start (resp., end) location of the current token. Throws: java.io.IOException setType public final void setType(int c, int type) Set the type of this character to be the one specified, non-exclusively. setType public final void setType(int low, int high, int type) Set the type of all characters in this range to be the one specified, non-exclusively. setType public final void setType(java.lang.String chars, int type) Set the type of all characters in this string to be the one specified, non-exclusively. unsetType public final void unsetType(int c, int type) Unset the specified type, but no other one, for this character. unsetType public final void unsetType(int low, int high, int type) Unset the specified type, but no other one, for all characters in this range. unsetType public final void unsetType(java.lang.String chars, int type) Unset the specified type, but no other one, for all characters in this string. currentChar public final int currentChar() returns the current character. resetSyntax public final void resetSyntax() Make all characters ordinary, do not parse numbers, and do not recognize comments. setDefaultSyntax public final void setDefaultSyntax() Resets the syntax configuration of the tokenizer to the default setting. Namely: set the type precedence ordering to: WORD, WHITESPACE, COMMENT, QUOTE, and ORDINARY; make all control characters and space WHITESPACE characters; make all letters and numerals WORD characters; make _ and . WORD characters; make ' and " quote characters (' for characters and character octal codes and unicode, and " for strings); make end-of-line not significant; make white spaces not significant; make all other characters ordinary; make parsing numbers effective; make skipping C-style and C++-style comments effective.

Class AbstractStreamTokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

ORDINARY

WHITESPACE

QUOTE

WORD

NUMERIC

COMMENT

SPECIAL

TT_EOF

TT_EOL

TT_WORD

TT_NUMBER

TT_NOTHING

TT_SPECIAL

ttype

sval

nval

isInteger

Constructor Detail

AbstractStreamTokenizer

Method Detail

commentChar

slashSlashComments

slashStarComments

nextToken

enableComments

disableComments

tokenStart

tokenEnd

getLineNumber

lineno

getFile

location

ordinaryChars

wordChars

whitespaceChars

quoteChar

quoteChar

ordinaryChar

eolIsSignificant

spaceIsSignificant

parseNumbers

parseNonDecimals

ignoreNonDecimals

parseDigitsOnly

parsingDigitsOnly

parseNumbers

parseNonDecimals

pushBack

toString

quotePair

quotePair

setQuotePair

setQuoteChar

setQuoteChar

setOrdinaryChar

wordChar

setWordChar

wordChars

setWordChars

whitespaceChar

setWhitespaceChar

whitespaceChars

setWhitespaceChars

ordinaryChars

setOrdinaryChars

isOrdinaryChar

isNumericChar

isWordChar

isWhitespaceChar

isQuoteChar

peek

peekBack

skipChar

skipChar