Chapter 6
Compiled Grammars
A compiled grammar is a grammar that has been parsed, analyzed, type-checked
and translated to bytecode, and then saved to a file, conventionally with the
extension .sxirc. Loading the file gives a parser that runs the parses of the
file's mode like the parser that was saved, without repeating that work. This
chapter defines the Go and command-line entry points, the modes and the
operations each one serves, the version rules and the checks that loading
performs. The byte layout of the file is defined in
SyntaxIR Bytecode Specification.
Scope and Terminology
A compiled grammar holds one execution-only compile result: the bytecode module of one mode and the start rule. It holds no grammar AST and no output schema. The mode is chosen when the grammar is saved and recorded in the file.
| Mode | Go | Serves |
|---|---|---|
parse (default) |
vm.CompiledParse |
Parse, ParsePrefix, ParseStream and Iterate without Recognize, WithTrace or WithProfile; sxir parse and sxir parse -stream |
recognize |
vm.CompiledRecognize |
Parse and ParsePrefix with Recognize and without WithTrace or WithProfile; sxir parse -recognize |
document |
vm.CompiledDocument |
NewDocument; Parse, ParsePrefix, ParseStream and Iterate with WithTrace or WithProfile and without Recognize; sxir trace, sxir profile and sxir explain |
A parse-mode and a document-mode file of the same grammar MAY hold the same module; a recognize-mode file holds the module of the grammar's recognizer.
Contract
Saving and Loading
| Operation | Go | Command line |
|---|---|---|
| Save in parse mode | compiler.Marshal(src, vm.CompiledParse) or Marshal(vm.CompiledParse) of the *vm.Parser |
sxir compile -g g.sxir -o g.sxirc |
| Save in another mode | compiler.Marshal(src, vm.CompiledRecognize), compiler.Marshal(src, vm.CompiledDocument) |
sxir compile -g g.sxir -mode recognize|document -o g.sxirc |
| Name a mode | CompiledMode.String() |
The value of -mode |
| Load | vm.LoadParser(data) |
-g file.sxirc on parse, trace, profile, explain and compile |
| Mode of a loaded parser | Parser.CompiledMode() returns the mode and true; false for a parser compiled from a grammar |
|
| Detect a compiled grammar | vm.IsCompiled(data) |
Used to tell source, JSON and a compiled grammar apart |
- The start rule is saved with the module.
sxir parse,sxir compileand the debugging commands use the saved start rule when-sis not given, and-soverrides it with another start rule of the grammar. - The same parser always produces the same bytes for a mode. A parser that was loaded and saved again produces the bytes it was loaded from (see Re-saving).
vm.IsCompileddistinguishes a compiled grammar from other data by its magic number.- A loaded parser is an ordinary
runtime.Parser: it is safe for concurrent use,WithStart,StartsandDefaultStartwork on it, and it runs the operations of its mode. For those operations, a parser loaded from a file returns the same tree, positions and errors as the parser compiled from the same source.
Operations of a Loaded Parser
A loaded parser runs on the bytecode backends. vm.LoadParser returns a parser
on the recursive bytecode VM, and Iterative of the *vm.Parser returns a
parser on the iterative VM; the choice does not depend on the mode.
An operation that the file's mode does not serve returns an error that names the file's mode, the mode the operation needs and the remedy; the parser does not fall back to another mode:
the compiled grammar is in parse mode; documents need document mode: compile it with -mode document (vm.CompiledDocument), or compile the grammar from source
The operations that need the grammar itself work only on a parser compiled from source:
| Operation | Loaded parser |
|---|---|
compiler.NewReferenceParser |
Not available: it is made from the grammar source, which a loaded parser does not have |
Recognize with WithTrace or WithProfile |
Error in every mode: traced or profiled recognition needs the grammar source |
Parser.Grammar() |
nil |
OutputSchema |
An error matching ErrSchemaUnavailable: the schema is exported from the grammar source |
lint.Config.Run, compiler.GenerateGo, compiler.GenerateTypeScript, sample.New |
Need Parser.Grammar(), so they do not take a loaded parser |
sxir convert, sxir gen, sxir lint, sxir sample, sxir schema |
Error: the command needs the grammar source (.sxir or .json) |
Re-Saving
- A loaded parser is saved again only in its own mode, and then reproduces the
bytes it was loaded from. Saving it in another mode is an error that asks for
the grammar to be compiled from source in that mode;
sxir compilewrites no output then. sxir compile -g file.sxircsaves in parse mode unless-modeis given, so a recognize-mode or document-mode file is saved again with its own-mode.
Versions and Compatibility
A file carries two version numbers after its magic number. They cover separate layers, so a change to the encoding of the module bumps only the instruction-set version.
| Number | Current value | Meaning |
|---|---|---|
| Format version | 8 | The outer frame of the file: its header and the order of its sections. A runtime loads only version 8 and refuses any other version |
| Instruction-set version | 38 | The encoding of the module's contents, its action code included. Every module declares it |
- A runtime refuses a file of another format version with
unsupported compiled grammar version N (want 8): compile the grammar again. - A runtime refuses a module of an earlier instruction set, whose contents it does
not decode. The error asks for the grammar to be compiled again. A file that requires a newer instruction set is refused with
unsupported instruction set version N. - A file records whether the program was compiled with source information, and the loaded program keeps it.
- A runtime with a narrower implementation
intrejects repetition bounds outside its range. - The same grammar MAY produce different bytes in a different release of the compiler. Equality of bytes is guaranteed only for one release.
- A saved file retains the left-recursion classification chosen by the compiler that wrote it. Loading a file does not rerun that analysis.
Loading
Loading checks the magic number, the format version, a CRC-32 checksum, the mode,
the instruction-set version and the structure of the module (every index is in
range, every jump goes forward except the one that closes a repetition and jumps
back, the start rule is a start rule of the module, and no bytes trail the module).
It also rejects a recovery region that encloses a streaming commitment, including
files that were made with #recover on the #stream repetition.
Loading does not type-check the grammar again and does not verify that the
bytecode uses the VM stacks correctly. It trusts the mode recorded in the file as
it trusts the module's types: it checks that the mode is known, not that the
module is the one compiled for that mode, so a file that labels a recognize-mode
module as parse mode MAY return no value and no error. Load compiled grammars
only from trusted sources. A file that was made to contain type errors
MAY make actions fail while parsing, and made-up bytecode is not guaranteed to
terminate. The checksum detects accidents, not tampering, and is not a signature.
A file from a source that is not trusted therefore cannot be verified by SyntaxIR;
authenticating it, or accepting grammar source and compiling it with compiler.Compile,
is up to the program.
Validation and Errors
| Message | Cause |
|---|---|
not a compiled SyntaxIR grammar |
The data does not start with the magic number |
unsupported compiled grammar version N (want 8): compile the grammar again |
The format version is not 8 |
unsupported instruction set version N: the file was compiled by an older sxir; compile the grammar again |
The file was compiled for an earlier instruction set |
unsupported instruction set version N |
The file requires a newer instruction set |
invalid compiled grammar: checksum mismatch |
The file is truncated or changed |
invalid compiled grammar: unknown mode N |
The mode byte names no mode |
invalid compiled grammar: ... |
The structure is wrong although the checksum matches |
the compiled grammar is in M mode; O N mode: compile it with -mode N (...), or compile the grammar from source |
An operation that mode N serves was requested of a file in mode M; O is parsing needs, recognition needs, documents need or tracing and profiling need |
the compiled grammar is in M mode; traced or profiled recognition needs the grammar source: compile the grammar from source |
Recognize was combined with WithTrace or WithProfile |
the compiled grammar is in M mode and is saved only in that mode; compile the grammar from source to save it in N mode |
Marshal(N) was called on a parser loaded in mode M |
the closure backend needs the grammar source; a compiled grammar runs on the bytecode backends |
The reference parser (sxir parse -backend closure) was requested of a loaded parser |
output schema unavailable: a compiled grammar holds no schema; export the schema from the grammar source |
OutputSchema was called on a loaded parser; the error matches ErrSchemaUnavailable |
F: a compiled grammar holds only the bytecode of one mode; C needs the grammar source (.sxir or .json) |
Command C that needs the grammar was given the compiled grammar F |
Compatibility and Support
A compiled grammar needs the SyntaxIR engine at run time. Generated parsers do not use compiled grammars; they are specified in Generated Parsers.
Conformance
The tests of the repository save, load and re-save grammars in every mode on the bytecode backends, compare the results with parsers compiled from source, and check that each mode refuses the operations of the others. The byte layout is established by SyntaxIR Bytecode Specification.