Ship a Parser and Keep It Up to Date

Problem. You wrote a grammar and want your program, or your library, to use it without compiling or interpreting the grammar at run time, and without a stale parser after somebody changes the grammar.

Generate Go code from the grammar with sxir gen, commit it, regenerate it with go generate, and let a test fail when the committed file does not match the grammar. The generated package imports only the standard library and SyntaxIR's small pkg/runtime packages (with pkg/runtime/value/impl), not the compiler or the generator.

The grammar reads versions such as 1.2.3 and 1.2.3-rc.1:

// version.sxir
type Num = node[string]
type Ident = node[string]
type Version = { Major Num; Minor Num; Patch Num; Pre *Ident }

@start
rule main Version = v:version $$ => $v

rule version Version = (
    major:num "." minor:num "." patch:num pre:pre?
) => Version{Major: $major, Minor: $minor, Patch: $patch, Pre: $pre}
rule pre Ident = -"-" i:ident => $i

rule num Num = ("0" / [1-9] [0-9]*) #atomic
rule ident Ident = ([a-z0-9.]+) #atomic

The layout is a module with the grammar, the code that uses the parser, and the generated parser in a package of its own:

example.com/ship
├── go.mod
├── version.sxir
├── main.go
├── generated_test.go
└── versionparser/       created by go generate
    ├── parser.go
    ├── api.go
    └── internal/libgo/

Record sxir as a tool of the module, so that go generate runs the version you pinned and nobody installs anything:

go mod init example.com/ship
go get -tool syntaxir.dev/sxir/cmd/sxir

Then the program, with the go:generate line that makes the parser (-types adds Go types and ParseValue):

// main.go
package main

import (
	"fmt"
	"os"

	"example.com/ship/versionparser"
)

//go:generate go tool sxir gen -g version.sxir -module example.com/ship/versionparser -types -o versionparser

func main() {
	v, err := versionparser.ParseValue(os.Args[1])
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	fmt.Printf("major %s, minor %s, patch %s", v.Major.Value, v.Minor.Value, v.Patch.Value)
	if v.Pre != nil {
		fmt.Printf(", pre-release %s", v.Pre.Value)
	}
	fmt.Println()
}
go generate ./...
go run . 1.2.3-rc.1
go run . 1.2.3
go run . 01.2.3
major 1, minor 2, patch 3, pre-release rc.1
major 1, minor 2, patch 3
1:2: syntax error: expected "."

A test keeps the generated files honest. It generates the package again in memory, with the options of the go:generate line, and compares every file with the one on disk:

// generated_test.go
package main

import (
	"bytes"
	_ "embed"
	"os"
	"path/filepath"
	"testing"

	"syntaxir.dev/sxir/pkg/compiler"
	"syntaxir.dev/sxir/pkg/grammar"
)

//go:embed version.sxir
var grammarSrc string

// The files of versionparser are what sxir generates from version.sxir. The module path and the options must be those
// of the go:generate line.
func TestGeneratedIsUpToDate(t *testing.T) {
	want, err := compiler.GenerateGo(grammar.FromString("grammar.sxir", grammarSrc),
		compiler.GoConfig{Module: "example.com/ship/versionparser", Types: true}, compiler.WithStart("main"))
	if err != nil {
		t.Fatal(err)
	}
	for _, f := range want { // parser.go, api.go and the library, with paths relative to the package
		got, err := os.ReadFile(filepath.Join("versionparser", filepath.FromSlash(f.Path)))
		if err != nil {
			t.Fatal(err)
		}
		if !bytes.Equal(got, f.Content) {
			t.Errorf("versionparser/%s is out of date: run go generate ./...", f.Path)
		}
	}
}

If somebody changes the grammar (here, the characters of ident) and does not run go generate:

--- FAIL: TestGeneratedIsUpToDate (0.02s)
    generated_test.go:33: versionparser/parser.go is out of date: run go generate ./...
FAIL
FAIL	example.com/ship	0.652s
FAIL

How It Works

  • A small dependency at run time. go list -deps . for the program lists only the packages under syntaxir.dev/sxir/pkg/runtime (including syntaxir.dev/sxir/pkg/runtime/value/impl) of SyntaxIR: the result and error types of the generated API. The engine and the generator are not linked into it. go.mod requires SyntaxIR for these, for the tool and for the test.
  • The generator is deterministic. The same grammar and options give the same file, which is what makes the comparison in the test, and a review of a diff of parser.go, meaningful.
  • Regenerate after upgrading SyntaxIR. The copy of the library in versionparser/internal/libgo comes from the version of sxir that wrote it. Without the test, a new version of the tool leaves the old library in place until somebody runs go generate; with it, the test fails. Pinning the tool with go get -tool keeps every developer and CI on the same version.
  • Without the test, go generate ./... && git diff --exit-code in CI says the same.
  • The same tree. The generated Parse returns the same tree and errors as the engine for the same grammar, so you can develop with the engine and the sxir tools (sxir parse, sxir trace) and ship the generated code.

When the Grammar Must Stay Data

If you want the grammar to be a file you load, not code (users supply it, or it changes without a rebuild), compile it once to a .sxirc file and embed or ship that, which skips parsing and checking the grammar at start-up:

go tool sxir compile -g version.sxir -o version.sxirc
//go:embed version.sxirc
var compiled []byte

p, err := vm.LoadParser(compiled) // then p.Parse(...) as with compiler.Compile

The file serves Parse and ParseStream on the bytecode backends; compile it with -mode recognize for Recognize or -mode document for documents and traces. Loading is much faster than compiling: for the Python grammar of alpha-parsers/python, on an Apple M3 Max with Go 1.27.1, compiler.Compile took about 60 ms and vm.LoadParser of the .sxirc under 1 ms in every mode.

A generated parser needs no preparation at all, and is also the fastest at parsing.

Variations and Limits

  • Several grammars: one go:generate line and one package each; alpha-parsers/generate.go does this for the ready-made parsers.
  • A recognizer: add -recognize for Recognize, which only checks, as in Parse log lines fast.
  • TypeScript: sxir gen -lang ts generates a module with no dependencies; see TypeScript parsers.
  • A parity test: compare the generated parser with the engine on a list of inputs, to catch a difference after an upgrade (see the code generation guide).

See Code generation for the generated API, what the generated parser does not support (#stream, Document), and how typed values are produced.