Ship a Parser and Keep It Up to Date
Problem. You wrote a grammar and want your program, or your library, to use it without compiling or interpreting the grammar at run time, and without a stale parser after somebody changes the grammar.
Generate Go code from the grammar with sxir gen, commit it, regenerate it with go generate, and let a test fail
when the committed file does not match the grammar. The generated package imports only the standard library and
SyntaxIR's small pkg/runtime packages (with pkg/runtime/value/impl), not the compiler or the generator.
The grammar reads versions such as 1.2.3 and 1.2.3-rc.1:
// version.sxir
type Num = node[string]
type Ident = node[string]
type Version = { Major Num; Minor Num; Patch Num; Pre *Ident }
@start
rule main Version = v:version $$ => $v
rule version Version = (
major:num "." minor:num "." patch:num pre:pre?
) => Version{Major: $major, Minor: $minor, Patch: $patch, Pre: $pre}
rule pre Ident = -"-" i:ident => $i
rule num Num = ("0" / [1-9] [0-9]*) #atomic
rule ident Ident = ([a-z0-9.]+) #atomic
The layout is a module with the grammar, the code that uses the parser, and the generated parser in a package of its own:
example.com/ship
├── go.mod
├── version.sxir
├── main.go
├── generated_test.go
└── versionparser/ created by go generate
├── parser.go
├── api.go
└── internal/libgo/
Record sxir as a tool of the module, so that go generate runs the version you pinned and nobody installs anything:
go mod init example.com/ship
go get -tool syntaxir.dev/sxir/cmd/sxir
Then the program, with the go:generate line that makes the parser (-types adds Go types and ParseValue):
// main.go
package main
import (
"fmt"
"os"
"example.com/ship/versionparser"
)
//go:generate go tool sxir gen -g version.sxir -module example.com/ship/versionparser -types -o versionparser
func main() {
v, err := versionparser.ParseValue(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("major %s, minor %s, patch %s", v.Major.Value, v.Minor.Value, v.Patch.Value)
if v.Pre != nil {
fmt.Printf(", pre-release %s", v.Pre.Value)
}
fmt.Println()
}
go generate ./...
go run . 1.2.3-rc.1
go run . 1.2.3
go run . 01.2.3
major 1, minor 2, patch 3, pre-release rc.1
major 1, minor 2, patch 3
1:2: syntax error: expected "."
A test keeps the generated files honest. It generates the package again in memory, with the options of the
go:generate line, and compares every file with the one on disk:
// generated_test.go
package main
import (
"bytes"
_ "embed"
"os"
"path/filepath"
"testing"
"syntaxir.dev/sxir/pkg/compiler"
"syntaxir.dev/sxir/pkg/grammar"
)
//go:embed version.sxir
var grammarSrc string
// The files of versionparser are what sxir generates from version.sxir. The module path and the options must be those
// of the go:generate line.
func TestGeneratedIsUpToDate(t *testing.T) {
want, err := compiler.GenerateGo(grammar.FromString("grammar.sxir", grammarSrc),
compiler.GoConfig{Module: "example.com/ship/versionparser", Types: true}, compiler.WithStart("main"))
if err != nil {
t.Fatal(err)
}
for _, f := range want { // parser.go, api.go and the library, with paths relative to the package
got, err := os.ReadFile(filepath.Join("versionparser", filepath.FromSlash(f.Path)))
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, f.Content) {
t.Errorf("versionparser/%s is out of date: run go generate ./...", f.Path)
}
}
}
If somebody changes the grammar (here, the characters of ident) and does not run go generate:
--- FAIL: TestGeneratedIsUpToDate (0.02s)
generated_test.go:33: versionparser/parser.go is out of date: run go generate ./...
FAIL
FAIL example.com/ship 0.652s
FAIL
How It Works
- A small dependency at run time.
go list -deps .for the program lists only the packages undersyntaxir.dev/sxir/pkg/runtime(includingsyntaxir.dev/sxir/pkg/runtime/value/impl) of SyntaxIR: the result and error types of the generated API. The engine and the generator are not linked into it.go.modrequires SyntaxIR for these, for the tool and for the test. - The generator is deterministic. The same grammar and options give the same file, which is what makes the
comparison in the test, and a review of a diff of
parser.go, meaningful. - Regenerate after upgrading SyntaxIR. The copy of the library in
versionparser/internal/libgocomes from the version ofsxirthat wrote it. Without the test, a new version of the tool leaves the old library in place until somebody runsgo generate; with it, the test fails. Pinning the tool withgo get -toolkeeps every developer and CI on the same version. - Without the test,
go generate ./... && git diff --exit-codein CI says the same. - The same tree. The generated
Parsereturns the same tree and errors as the engine for the same grammar, so you can develop with the engine and thesxirtools (sxir parse,sxir trace) and ship the generated code.
When the Grammar Must Stay Data
If you want the grammar to be a file you load, not code (users supply it, or it changes without a rebuild), compile it
once to a .sxirc file and embed or ship that, which skips parsing and checking the grammar at start-up:
go tool sxir compile -g version.sxir -o version.sxirc
//go:embed version.sxirc
var compiled []byte
p, err := vm.LoadParser(compiled) // then p.Parse(...) as with compiler.Compile
The file serves Parse and ParseStream on the bytecode backends; compile it with -mode recognize for
Recognize or -mode document for documents and traces. Loading is much faster than compiling: for the
Python grammar of alpha-parsers/python, on an Apple M3 Max with Go 1.27.1, compiler.Compile took
about 60 ms and vm.LoadParser of the .sxirc under 1 ms in every mode.
A generated parser needs no preparation at all, and is also the fastest at parsing.
Variations and Limits
- Several grammars: one
go:generateline and one package each; alpha-parsers/generate.go does this for the ready-made parsers. - A recognizer: add
-recognizeforRecognize, which only checks, as in Parse log lines fast. - TypeScript:
sxir gen -lang tsgenerates a module with no dependencies; see TypeScript parsers. - A parity test: compare the generated parser with the engine on a list of inputs, to catch a difference after an upgrade (see the code generation guide).
See Code generation for the generated API, what the generated parser does not support
(#stream, Document), and how typed values are produced.