What's ANTLR

ANTLR, ANother Tool for Language Recognition, (formerly PCCTS) is a language tool that provides a framework for constructing recognizers, compilers, and translators from grammatical descriptions containing Java, C++, or C# actions [You can use PCCTS 1.xx to generate C-based parsers].

Computer language translation has become a common task. While compilers and tools for traditional computer languages (such as C or Java) are still being built, their number is dwarfed by the thousands of mini-languages for which recognizers and translators are being developed. Programmers construct translators for database formats, graphical data files (e.g., PostScript, AutoCAD), text processing files (e.g., HTML, SGML).  ANTLR is designed to handle all of your translation tasks.

Terence Parr has been working on ANTLR since 1989 and, together with his colleagues, has made a number of fundamental contributions to parsing theory and language tool construction, leading to the resurgence of LL(k)-based recognition tools.

Here is a chronological history and credit list for ANTLR/PCCTS.

See ANTLR software rights.

Download ANTLR.

ANTLR 2.7.5 release notes

ANTLR Meta-Language

• Meta-Language Vocabulary
• Header Section
• Parser Class Definitions
• Lexical Analyzer Class Definitions
• Tree-parser Class Definitions
• Options Section
• Tokens Section
• Grammar Inheritance
• Rule Definitions
• Atomic Production elements
• Simple Production elements
• Production Element Operators
• Token Classes
• Predicates
• Element Labels
• EBNF Rule Elements
• Interpretation Of Semantic Actions
• Semantic Predicates
• Syntactic Predicates
  • Fixed depth lookahead and syntactic predicates
• ANTLR Meta-Language Grammar

Lexical Analysis with ANTLR

• Lexical Rules
• Skipping characters
• Distinguishing between lexer rules
• Return values
• Predicated-LL(k) Lexing
• Keywords and literals
• Common prefixes
• Token definition files
• Character classes
• Token Attributes
• Lexical lookahead and the end-of-token symbol
• Scanning Binary Files
• Scanning Unicode Characters
• Manipulating Token Text and Objects
• Manipulating the Text of a Lexical Rule
• Heterogeneous Token Object Streams
• Filtering Input Streams
• ANTLR Masquerading as SED
• Nongreedy Subrules
• Greedy Subrules
• Nongreedy Lexer Subrules
• Limitations of Nongreedy Subrules
• Lexical States
• The End Of File Condition
• Case sensitivity
• Ignoring whitespace in the lexer
• Tracking Line Information
• Tracking Column Information
• But...We've Always Used Automata For Lexical Analysis!

ANTLR Tree Parsers

• What's a tree parser?
• What kinds of trees can be parsed?
• Tree grammar rules
  • Syntactic predicates
  • Semantic predicates
  • An Example Tree Walker
• Transformations
• An Example Tree Transformation
• Examining/Debugging ASTs

Token Streams

• Introduction
• Pass-Through Token Stream
• Token Stream Filtering
• Token Stream Splitting
  • Example
  • Filter Implementation
  • How To Use This Filter
  • Tree Construction
  • Garbage Collection Issues
  • Notes
• Token Stream Multiplexing (aka "Lexer states")
  • Multiple Lexers
  • Lexers Sharing Same Character Stream
  • Parsing Multiplexed Token Streams
  • The Effect of Lookahead Upon Multiplexed Token Streams
  • Multiple Lexers Versus Calling Another Lexer Rule
• TokenStreamRewriteEngine Easy Syntax-Directed Translation
• The Future

Token Vocabularies

• Introduction
  • How does ANTLR decide which vocabulary symbol gets what token type?
  • Why do token types start at 4?
  • What files associated with vocabulary does ANTLR generate?
  • How does ANTLR synchronize the symbol-type mappings between grammars in the same file and in different files?
• Grammar Inheritance and Vocabularies
• Recognizer Generation Order
• Tricky Vocabulary Stuff

Error Handling and Recovery

• ANTLR Exception Hierarchy
• Modifying Default Error Messages With Paraphrases
• Parser Exception Handling
• Specifying Parser Exception-Handlers
• Default Exception Handling in the Lexer

Java Runtime Model

• Programmer's Interface
• What ANTLR generates
• Multiple Lexers/Parsers With Shared Input State
• Parser Implementation
• Parser Class
• Parser Methods
• EBNF Subrules
• Production Prediction
• Production Element Recognition
• Standard Classes
• Lexer Implementation
• Lexer Form
• Creating Your Own Lexer
• Lexical Rules
• Token Objects
• Token Lookahead Buffer

C++ Runtime model

• Building the runtime
• Getting C++ output
• Changing the AST Type
• Using Heterogeneous AST types
• Extra functionality in C++ mode
• A grammar template

C# Runtime model

• Building the ANTLR C# Runtime
• Specifying Code Generation
• C#-Specific ANTLR Options
• A Template C# ANTLR Grammar File

Python Runtime model

• Building the ANTLR Python Runtime
• Specifying Code Generation
• Python-Specific ANTLR Options
• A Template Python ANTLR Grammar File

ANTLR Tree Construction

• Notation
• Controlling AST construction
• Grammar annotations for building ASTs
• Leaf nodes
• Root nodes
• Turning off standard tree construction
• Tree node construction
• AST Action Translation
• Invoking parsers that build trees
• AST Factories
• Heterogeneous ASTs
  • An Expression Tree Example
  • Describing Heterogeneous Trees With Grammars
• AST (XML) Serialization
• AST enumerations
• A few examples
• Labeled subrules
• Reference nodes
• Required AST functionality and form

Grammar Inheritance

• Introduction and motivation
• Functionality
• Where Are Those Supergrammars?
• Error Messages

Options

• File, Grammar, and Rule Options
  • Options supported in ANTLR
  • language: Setting the generated language
  • k: Setting the lookahead depth
  • importVocab: Initial Grammar Vocabulary
  • exportVocab: Naming Output Vocabulary
  • testLiterals: Generate literal-testing code
  • defaultErrorHandler: Controlling default exception-handling
  • codeGenMakeSwitchThreshold: controlling code generation
  • codeGenBitsetTestThreshold: controlling code generation
  • buildAST: Automatic AST construction
  • ASTLabelType: Setting label type
  • charVocabulary: Setting the lexer character vocabulary
  • warnWhenFollowAmbig
• Command Line Options