ANTLR3_COMMON_TOKEN_struct man page

ANTLR3_COMMON_TOKEN_struct — The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.

Synopsis

#include <antlr3commontoken.h>

Data Fields

ANTLR3_UINT32 channel
The virtual channel that this token exists in.
ANTLR3_INT32 charPosition
The character position in the line that this token was derived from.
void * custom
Pointer to a custom element that the ANTLR3 programmer may define and install.
ANTLR3_BOOLEAN factoryMade
Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it.
void(* freeCustom )(void *custom)
Pointer to a function that knows how to free the custom structure when the token is destroyed.
ANTLR3_UINT32(* getChannel )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels.
ANTLR3_INT32(* getCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the offset in the line where this token exists.
ANTLR3_UINT32(* getLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the 'line' number where this token resides.
ANTLR3_MARKER(* getStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the start index in the input stream for this token.
ANTLR3_MARKER(* getStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the stop index in the input stream for this token.
pANTLR3_STRING(* getText )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token.
ANTLR3_MARKER(* getTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns an index 0...n-1 of the token in the token input stream.
ANTLR3_UINT32(* getType )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns the token type of this token.
ANTLR3_MARKER index
What the index of this token is, 0, 1, .., n-2, n-1 tokens.
pANTLR3_INPUT_STREAM input
Pointer to the input stream that this token originated in.
ANTLR3_UINT32 line
The line number in the input stream where this token was derived from.
void * lineStart
The offset into the input stream that the line in which this token resides starts.
void(* setChannel )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)
Pointer to a function that sets the channel that this token should belong to.
void(* setCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)
Pointer to a function that sets the offset in the line where this token exists.
void(* setLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)
Pointer to a function that sets the 'line' number where this token reside.
void(* setStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
Pointer to a function that sets the start index in the input stream for this token.
void(* setStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
Pointer to a function that sets the stop index in the input stream for this token.
void(* setText )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)
Pointer to a function that 'might' be able to set the text associated with a token.
void(* setText8 )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)
Pointer to a function that 'might' be able to set the text associated with a token.
void(* setTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)
Pointer to a function that can set the token index of this token in the token input stream.
void(* setType )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)
Pointer to a function that sets the type of this token.
ANTLR3_MARKER start
The character offset in the input stream where the text for this token starts.
ANTLR3_MARKER stop
The character offset in the input stream where the text for this token stops.
pANTLR3_STRING_FACTORY strFactory
A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING.
ANTLR3_UINT32 textState
Indicates whether this token is carrying:
union {

pANTLR3_UCHAR chars
Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar.
pANTLR3_STRING text
Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL.
} tokText

pANTLR3_STRING(* toString )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as
replaced with the printable sequence '\\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations. "
ANTLR3_UINT32 type
The actual type of this token.
ANTLR3_UINT32 user1
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
ANTLR3_UINT32 user2
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
ANTLR3_UINT32 user3
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.

Detailed Description

The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.

Remarks:

Token streams are in essence provided by lexers or other programs that serve as lexers.

Field Documentation

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::channel

The virtual channel that this token exists in.

Referenced by emit(), getChannel(), setChannel(), skipOffTokenChannels(), and skipOffTokenChannelsReverse().

ANTLR3_INT32 ANTLR3_COMMON_TOKEN_struct::charPosition

The character position in the line that this token was derived from.

Referenced by emit(), getCharPositionInLine(), and setCharPositionInLine().

pANTLR3_UCHAR ANTLR3_COMMON_TOKEN_struct::chars

Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar. No sense in constructing a pANTLR3_STRING just for that, as mostly the text will not be accessed - if it is, then we will build a pANTLR3_STRING for it a that point.

Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), getText(), newDownNode(), newUpNode(), and setText8().

void* ANTLR3_COMMON_TOKEN_struct::custom

Pointer to a custom element that the ANTLR3 programmer may define and install.

Referenced by antlr3LexerNew(), emit(), factoryClose(), and getMissingSymbol().

ANTLR3_BOOLEAN ANTLR3_COMMON_TOKEN_struct::factoryMade

Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it.

Referenced by antlr3LexerNew(), antlr3TokenFactoryNew(), newPoolToken(), newToken(), and nextTokenStr().

void(* ANTLR3_COMMON_TOKEN_struct::freeCustom) (void *custom)

Pointer to a function that knows how to free the custom structure when the token is destroyed.

Referenced by factoryClose().

ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getChannel) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), fillBuffer(), serializeToken(), and toString().

ANTLR3_INT32(* ANTLR3_COMMON_TOKEN_struct::getCharPositionInLine) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that gets the offset in the line where this token exists.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getCharPositionInLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().

ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getLine) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that gets the 'line' number where this token resides.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().

ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStartIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that gets the start index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), getText(), and toString().

ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStopIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that gets the stop index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), getText(), and toString().

pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::getText) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), serializeToken(), toString(), and toStringSS().

ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getTokenIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that returns an index 0...n-1 of the token in the token input stream.

Referenced by antlr3SetTokenAPI(), createNodeTok(), createTokenFromToken(), dbgSetTokenBoundaries(), serializeToken(), setTokenBoundaries(), toString(), and toStringTT().

ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getType) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that returns the token type of this token.

Referenced by _LA(), antlr3SetTokenAPI(), createTokenFromToken(), fillBuffer(), getMissingSymbol(), getTokensSet(), getType(), and serializeToken().

ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::index

What the index of this token is, 0, 1, .., n-2, n-1 tokens.

Referenced by getTokenIndex(), and setTokenIndex().

pANTLR3_INPUT_STREAM ANTLR3_COMMON_TOKEN_struct::input

Pointer to the input stream that this token originated in.

Referenced by antlr3RecognitionExceptionNew(), createToken(), getMissingSymbol(), getStartIndex(), getText(), newPoolToken(), and setInputStream().

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::line

The line number in the input stream where this token was derived from.

Referenced by emit(), getLine(), and setLine().

void* ANTLR3_COMMON_TOKEN_struct::lineStart

The offset into the input stream that the line in which this token resides starts.

Referenced by emit(), and getMissingSymbol().

void(* ANTLR3_COMMON_TOKEN_struct::setChannel) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)

Pointer to a function that sets the channel that this token should belong to.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), fillBuffer(), and getMissingSymbol().

void(* ANTLR3_COMMON_TOKEN_struct::setCharPositionInLine) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)

Pointer to a function that sets the offset in the line where this token exists.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and getMissingSymbol().

void(* ANTLR3_COMMON_TOKEN_struct::setLine) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)

Pointer to a function that sets the 'line' number where this token reside.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), and nextTokenStr().

void(* ANTLR3_COMMON_TOKEN_struct::setStartIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)

Pointer to a function that sets the start index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), newPoolToken(), nextTokenStr(), and tokLT().

void(* ANTLR3_COMMON_TOKEN_struct::setStopIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)

Pointer to a function that sets the stop index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), nextTokenStr(), and tokLT().

void(* ANTLR3_COMMON_TOKEN_struct::setText) (struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)

Pointer to a function that 'might' be able to set the text associated with a token. Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actaully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically.

Referenced by antlr3SetTokenAPI().

void(* ANTLR3_COMMON_TOKEN_struct::setText8) (struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)

Pointer to a function that 'might' be able to set the text associated with a token. Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically.

Referenced by antlr3SetTokenAPI(), createTypeTokenText(), and getMissingSymbol().

void(* ANTLR3_COMMON_TOKEN_struct::setTokenIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)

Pointer to a function that can set the token index of this token in the token input stream.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and fillBuffer().

void(* ANTLR3_COMMON_TOKEN_struct::setType) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)

Pointer to a function that sets the type of this token.

Referenced by antlr3CommonTokenNew(), antlr3LexerNew(), antlr3SetTokenAPI(), createToken(), createTokenFromToken(), createTypeToken(), createTypeTokenText(), and getMissingSymbol().

ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::start

The character offset in the input stream where the text for this token starts.

Referenced by emit(), getStartIndex(), and setStartIndex().

ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::stop

The character offset in the input stream where the text for this token stops.

Referenced by emit(), getStopIndex(), and setStopIndex().

pANTLR3_STRING_FACTORY ANTLR3_COMMON_TOKEN_struct::strFactory

A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING.

Referenced by ANTLR3_TREE_ADAPTORNew(), antlr3CommonTreeNodeStreamNew(), antlr3CommonTreeNodeStreamNewStream(), antlr3LexerNew(), createToken(), getText(), newPoolToken(), setCharStream(), and setInputStream().

pANTLR3_STRING ANTLR3_COMMON_TOKEN_struct::text

Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL.

Referenced by createTokenFromToken(), emit(), getText(), setText(), and setText8().

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::textState

Indicates whether this token is carrying: State | Meaning ------+-------------------------------------- 0 | Nothing (neither rewrite text, nor setText) 1 | char * to user supplied rewrite text 2 | pANTLR3_STRING because of setText or similar action

Affects the union structure tokText below (uses 32 bit so alignment is always good)

Referenced by antlr3CommonTreeNodeStreamNew(), antlr3LexerNew(), createToken(), createTokenFromToken(), emit(), getText(), newDownNode(), newUpNode(), setText(), and setText8().

union { ... }

ANTLR3_COMMON_TOKEN_struct::tokText"

Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), emit(), getText(), newDownNode(), newUpNode(), setText(), and setText8().

pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::toString) (struct ANTLR3_COMMON_TOKEN_struct *token)

Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as
replaced with the printable sequence '\\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and displayRecognitionError().

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::type

The actual type of this token.

Referenced by displayRecognitionError(), emit(), fillBuffer(), getText(), getType(), nextToken(), setType(), and toString().

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user1

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.

Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user2

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.

Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user3

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.

Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().

Author

Generated automatically by Doxygen for ANTLR3C from the source code.

Referenced By

antlr3-charPosition(3), antlr3-chars(3), antlr3-custom(3), antlr3-factoryMade(3), antlr3-freeCustom(3), antlr3-index(3), antlr3-input(3), antlr3-line(3), antlr3-lineStart(3), antlr3-start(3), antlr3-stop(3), antlr3-text(3), antlr3-textState(3), antlr3-tokText(3), antlr3-user1(3), antlr3-user2(3) and antlr3-user3(3) are aliases of ANTLR3_COMMON_TOKEN_struct(3).

Tue Oct 4 2016 Version 3.3.1 ANTLR3C