Chapter 13

Go Packages

The Go API of SyntaxIR is a set of packages of the module syntaxir.dev/sxir. This chapter defines which packages they are, what each is responsible for and where its contract is defined, how the packages depend on each other as a program sees it, and the package that compiles grammars.

Scope and Terminology

The public packages are the packages under syntaxir.dev/sxir/pkg. The other packages of the module are not part of the API: the internal packages, which Go does not let another module import, the commands under cmd, and the packages of the benchmarks, the examples and the tests of the specification. A package is named by its import path below without the prefix syntaxir.dev/sxir/.

The runtime side is the packages that run a parser: pkg/runtime, its subpackages and pkg/bytecode. The compiler side is the packages that read, check or compile a grammar: pkg/grammar, pkg/grammar/ast, pkg/compiler, pkg/lint, pkg/sample and pkg/editfuzz.

Contract

The Public Packages

Package Name Responsible for Defined in
pkg/grammar grammar Grammar sources, loading them into the grammar AST, the errors of loading, validation, formatting and conversion between grammar syntax and JSON Grammar Sources, Formatting
pkg/grammar/ast ast The grammar AST, its source positions and comments, its validation and its JSON form Grammar Files
pkg/compiler compiler Turning a grammar into something that runs: parsers on the bytecode VM, the reference parser, compiled grammar files, generated Go and TypeScript parsers and output schemas Compiling Grammars, Compiled Grammars, Generated Parsers
pkg/lint lint The checks of the linter, their findings and rewrites, and the complexity classification Grammar Lint
pkg/sample sample Generating inputs that a grammar accepts, near-miss inputs that it rejects, and coverage Input Generation
pkg/editfuzz editfuzz Checking incremental parsing against fresh parses with edit sequences, and their saved form Edit Fuzzing
pkg/bytecode bytecode The instruction set, the module and the compiled-file format, its encoding and its validation Bytecode Specification
pkg/runtime runtime The parser interface, parse options, limits and their errors, the stream iterator and the statistics of a Document Parsing, Incremental Documents, Streaming
pkg/runtime/input input Position units, locations, the text edits of a Document and the kinds of its change notifications Positions, Incremental Documents
pkg/runtime/value value Nodes, their traversal, cursors, text and JSON forms, and syntax errors, including the errors that recovery records Nodes and Values, Syntax Errors
pkg/runtime/value/impl valueimpl The node representation that the runtime and generated Go parsers share. It is not an API to call: programs use the methods of value.Node Generated Parsers
pkg/runtime/schema schema Output schemas, their JSON form, validation and comparison Output Schemas and Structural Queries
pkg/runtime/query query Structural queries: patterns, their compilation against an output schema, and matching Output Schemas and Structural Queries
pkg/runtime/instrument instrument Trace events, call sites and profiles of a parse Tracing and Profiling
pkg/runtime/vm vm Loading a compiled grammar into a parser and running it on the bytecode VM Compiled Grammars
pkg/runtime/libgo libgo The library of generated Go parsers. It is not an API to call: generated code imports it, or a copy of it Generated Parsers

The name of each package is the last element of its import path, except that pkg/runtime/value/impl is named valueimpl. No two public packages have the same name, so a program can import any of them together without aliases.

The public packages are the whole Go API: a program MUST NOT depend on a package of the module that this table does not list. Like the rest of the module at v0.x, a public package may change in any release, and the exported fields of pkg/bytecode describe the instruction set, which changes with its version.

Runtime and Compiler Sides

A package of the runtime side imports no package of the compiler side. A program that runs parsers without compiling a grammar, by loading compiled grammars with vm.LoadParser or by calling generated Go parsers, therefore does not link the grammar parser, the front end or the compiler. Generated Go parsers depend only on packages of the runtime side and on the standard library, as Generated Parsers defines.

Every parser returns the types of the runtime side, whatever made it: a parser of compiler.Compile, one that vm.LoadParser loaded and a generated Go parser give nodes as *value.Node, syntax errors as value.SyntaxErrors and limit errors as *runtime.LimitError, so code that handles a result does not depend on where its parser came from.

pkg/grammar/ast imports only the standard library, so a program can read, build and write grammar ASTs and their JSON form without the parser or the compiler.

Compiling Grammars

pkg/compiler compiles a grammar given as a grammar source or, for CompileGrammar, as a grammar AST.

Function Returns
compiler.Compile(src, opts...) A runtime.Parser that runs on the bytecode VM; it is a *vm.Parser
compiler.CompileGrammar(g, opts...) The parser of Compile for a grammar AST, such as one a program builds
compiler.NewReferenceParser(src, opts...) The reference parser, which runs the grammar's Core IR as Go closures. The other backends are tested against it. It is the only parser that inserts missing tokens, and its Documents have only the methods of runtime.Document
compiler.Marshal(src, mode, opts...) A compiled grammar in the mode mode
compiler.GenerateGo(src, config, opts...), compiler.GenerateTypeScript(src, config, opts...) The files of a generated parser
compiler.GenerateLibGo(module), compiler.GenerateTypeScriptLib() The files of the library of generated parsers alone
compiler.OutputSchema(src, opts...) The output schema of the start rule

Every function that takes a grammar takes the same options, so one grammar with one set of options is the same parser in every artifact:

Option Meaning
compiler.WithStart(name) The start rule; an empty name selects the default start rule
compiler.WithSourceInfo() Record source information
compiler.WithDefaultValue() Return default values
compiler.WithOptimize(o) The optimization setting: the profile o.Profile (compiler.Speed, compiler.Memory or compiler.Size), then the values o.Values, then the optimizations o.Off turned off. No setting changes what a parser returns. Every compile function takes it, and which backends apply it is stated in Optimization Settings. An unknown profile or optimization, or a value that an optimization does not take, is an error of every function

A grammar source that does not load gives the errors of grammar.Load, among them grammar.SyntaxErrors for grammar syntax. A grammar that loads but does not compile, such as one that refers to a rule that it does not define, gives compiler.GrammarErrors, a non-empty list of *compiler.GrammarError, each with the position Pos (an ast.Pos, the zero Pos when the error has no position in the source) and the message Msg. The functions return the list itself, which a program obtains with errors.As; the list does not unwrap to its elements. Its text is its errors, one per line, each as line:column: message when its position is known and as the message otherwise. lint and editfuzz report a grammar that does not compile with the same type.

Compatibility and Support

At v0.x, the packages, their names and their exported API may change in any release. The compiled-file format and the library protocol of generated parsers have versions of their own, which Compiled Grammars and Generated Parsers define.

Conformance

An implementation of the Go API conforms when the public packages are those of the table, with those names, when the runtime side imports no package of the compiler side, when pkg/grammar/ast imports only the standard library, and when the contracts of the chapters that the table links hold.