antlr
Class TokenStreamRewriteEngine

java.lang.Object
  extended byantlr.TokenStreamRewriteEngine
All Implemented Interfaces:
TokenStream

public class TokenStreamRewriteEngine
extends java.lang.Object
implements TokenStream

This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser. This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1 You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2")); You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer. If you don't use named rewrite streams, a "default" stream is used. Terence Parr, parrt@cs.usfca.edu University of San Francisco February 2004


Field Summary
static java.lang.String DEFAULT_PROGRAM_NAME
           
static int MIN_TOKEN_INDEX
           
static int PROGRAM_INIT_SIZE
           
 
Constructor Summary
TokenStreamRewriteEngine(TokenStream upstream)
           
TokenStreamRewriteEngine(TokenStream upstream, int initialSize)
           
 
Method Summary
 void delete(int index)
           
 void delete(int from, int to)
           
 void delete(java.lang.String programName, int from, int to)
           
 void delete(java.lang.String programName, Token from, Token to)
           
 void delete(Token indexT)
           
 void delete(Token from, Token to)
           
 void deleteProgram()
           
 void deleteProgram(java.lang.String programName)
          Reset the program so that no instructions exist
 void discard(int ttype)
           
 int getLastRewriteTokenIndex()
           
 TokenWithIndex getToken(int i)
           
 int getTokenStreamSize()
           
 void insertAfter(int index, java.lang.String text)
           
 void insertAfter(java.lang.String programName, int index, java.lang.String text)
           
 void insertAfter(java.lang.String programName, Token t, java.lang.String text)
           
 void insertAfter(Token t, java.lang.String text)
           
 void insertBefore(int index, java.lang.String text)
           
 void insertBefore(java.lang.String programName, int index, java.lang.String text)
           
 void insertBefore(java.lang.String programName, Token t, java.lang.String text)
           
 void insertBefore(Token t, java.lang.String text)
           
 Token nextToken()
           
 void replace(int from, int to, java.lang.String text)
           
 void replace(int index, java.lang.String text)
           
 void replace(java.lang.String programName, int from, int to, java.lang.String text)
           
 void replace(java.lang.String programName, Token from, Token to, java.lang.String text)
           
 void replace(Token indexT, java.lang.String text)
           
 void replace(Token from, Token to, java.lang.String text)
           
 void rollback(int instructionIndex)
           
 void rollback(java.lang.String programName, int instructionIndex)
          Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.
 java.lang.String toDebugString()
           
 java.lang.String toDebugString(int start, int end)
           
 java.lang.String toOriginalString()
           
 java.lang.String toOriginalString(int start, int end)
           
 java.lang.String toString()
           
 java.lang.String toString(int start, int end)
           
 java.lang.String toString(java.lang.String programName)
           
 java.lang.String toString(java.lang.String programName, int start, int end)
           
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

MIN_TOKEN_INDEX

public static final int MIN_TOKEN_INDEX
See Also:
Constant Field Values

DEFAULT_PROGRAM_NAME

public static final java.lang.String DEFAULT_PROGRAM_NAME
See Also:
Constant Field Values

PROGRAM_INIT_SIZE

public static final int PROGRAM_INIT_SIZE
See Also:
Constant Field Values
Constructor Detail

TokenStreamRewriteEngine

public TokenStreamRewriteEngine(TokenStream upstream)

TokenStreamRewriteEngine

public TokenStreamRewriteEngine(TokenStream upstream,
                                int initialSize)
Method Detail

nextToken

public Token nextToken()
                throws TokenStreamException
Specified by:
nextToken in interface TokenStream
Throws:
TokenStreamException

rollback

public void rollback(int instructionIndex)

rollback

public void rollback(java.lang.String programName,
                     int instructionIndex)
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!


deleteProgram

public void deleteProgram()

deleteProgram

public void deleteProgram(java.lang.String programName)
Reset the program so that no instructions exist


insertAfter

public void insertAfter(Token t,
                        java.lang.String text)

insertAfter

public void insertAfter(int index,
                        java.lang.String text)

insertAfter

public void insertAfter(java.lang.String programName,
                        Token t,
                        java.lang.String text)

insertAfter

public void insertAfter(java.lang.String programName,
                        int index,
                        java.lang.String text)

insertBefore

public void insertBefore(Token t,
                         java.lang.String text)

insertBefore

public void insertBefore(int index,
                         java.lang.String text)

insertBefore

public void insertBefore(java.lang.String programName,
                         Token t,
                         java.lang.String text)

insertBefore

public void insertBefore(java.lang.String programName,
                         int index,
                         java.lang.String text)

replace

public void replace(int index,
                    java.lang.String text)

replace

public void replace(int from,
                    int to,
                    java.lang.String text)

replace

public void replace(Token indexT,
                    java.lang.String text)

replace

public void replace(Token from,
                    Token to,
                    java.lang.String text)

replace

public void replace(java.lang.String programName,
                    int from,
                    int to,
                    java.lang.String text)

replace

public void replace(java.lang.String programName,
                    Token from,
                    Token to,
                    java.lang.String text)

delete

public void delete(int index)

delete

public void delete(int from,
                   int to)

delete

public void delete(Token indexT)

delete

public void delete(Token from,
                   Token to)

delete

public void delete(java.lang.String programName,
                   int from,
                   int to)

delete

public void delete(java.lang.String programName,
                   Token from,
                   Token to)

discard

public void discard(int ttype)

getToken

public TokenWithIndex getToken(int i)

getTokenStreamSize

public int getTokenStreamSize()

toOriginalString

public java.lang.String toOriginalString()

toOriginalString

public java.lang.String toOriginalString(int start,
                                         int end)

toString

public java.lang.String toString()

toString

public java.lang.String toString(java.lang.String programName)

toString

public java.lang.String toString(int start,
                                 int end)

toString

public java.lang.String toString(java.lang.String programName,
                                 int start,
                                 int end)

toDebugString

public java.lang.String toDebugString()

toDebugString

public java.lang.String toDebugString(int start,
                                      int end)

getLastRewriteTokenIndex

public int getLastRewriteTokenIndex()