Package sxir/pkg/compiler
import "syntaxir.dev/sxir/pkg/compiler"
Generated from the source of this version. Also on pkg.go.dev.
Overview
Package compiler makes executable artifacts from SyntaxIR grammars: parsers that run on the bytecode VM (Compile, CompileGrammar), the reference parser that differential tests compare with (NewReferenceParser), compiled grammar files (Marshal), Go and TypeScript parser source (GenerateGo, GenerateTypeScript) and output schemas (OutputSchema).
Every function takes the same Options: the start rule (WithStart), source information (WithSourceInfo), default values (WithDefaultValue), the optimization setting (WithOptimize) and the debugging setting of the compiler (WithDebug), so a grammar means the same parser whatever the artifact. The parsers implement runtime.Parser; the parser of Compile is a *vm.Parser, which also parses prefixes, iterates over streams, saves itself (Marshal) and runs on the iterative VM.
A grammar source that does not load gives the errors of grammar.Load, such as grammar.SyntaxErrors; a grammar that loads but does not compile gives GrammarErrors.
Index
Variablesfunc Compilefunc CompileGrammarfunc Marshalfunc NewReferenceParserfunc OutputSchematype Debugtype Filetype GoConfigtype GrammarErrortype GrammarErrorstype LibModetype Optimizetype Optiontype TSConfig
Variables
var (
// Speed is the fastest parse that does not cost code or memory out of proportion; the default.
Speed = Optimize{Profile: "speed"}
// Memory allocates typed values one by one: fewer bytes allocated and less live heap, with
// many more allocations.
Memory = Optimize{Profile: "memory"}
// Size makes smaller generated Go source and binaries, for a little parse time.
Size = Optimize{Profile: "size"}
)
The optimization profiles.
Functions
func Compile source
func Compile(src grammar.Source, opts ...Option) (runtime.Parser, error)
Compile compiles the grammar src to a parser that runs on the bytecode VM. The parser is a *vm.Parser.
func CompileGrammar source
func CompileGrammar(g *ast.Grammar, opts ...Option) (runtime.Parser, error)
CompileGrammar compiles a grammar given as its syntax tree, such as one a program builds, to a parser that runs on the bytecode VM. The parser is a *vm.Parser.
func Marshal source
func Marshal(src grammar.Source, mode vm.CompiledMode, opts ...Option) ([]byte, error)
Marshal compiles the grammar src and encodes it in the .sxirc format in mode: the bytecode that mode serves and the start rule. vm.LoadParser loads it without the grammar or the compiler.
func NewReferenceParser source
func NewReferenceParser(src grammar.Source, opts ...Option) (runtime.Parser, error)
NewReferenceParser compiles the grammar src to the reference parser, which runs the grammar's Core IR as Go closures. It is the parser that the other backends are tested against; it is slower than the parser of Compile, and the only one that inserts missing tokens (runtime.WithMissingTokens). Its Documents have only the methods of runtime.Document.
func OutputSchema source
func OutputSchema(src grammar.Source, opts ...Option) (*schema.Output, error)
OutputSchema exports the output schema of the grammar src: the types of the values that its start rule returns, of the rules those values contain and of the types they refer to. The parsers' OutputSchema with runtime.WithAllSchemaRules exports every rule and type.
Types
type Debug source
type Debug struct {
// PrintAfter lists the steps and passes after which the intermediate representation is written
// to Output as text: elaborate and the names of the Core passes for the Core IR; flowlower, the
// names of the Flow passes and flow-finish for Code IR at the Flow stage; execlower and the names
// of the Exec passes for Code IR at the Exec stage; or all. "sxir gen -h" lists the names.
PrintAfter []string
// PrintBefore lists the passes before whose runs the intermediate representation they receive is
// written to Output, as PrintAfter writes it, or all for every pass. The required steps
// (elaborate, flowlower, flow-finish and execlower) are not passes to print before.
PrintBefore []string
// PrintChanged writes the intermediate representation after each pass that changed it, and the
// unit that the first such pass received before its first dump. A pass that PrintAfter also
// names is written once.
PrintChanged bool
// VerifyEach verifies the intermediate representation after every pass instead of only where a
// stage ends, so that an internal error names the first pass that broke it, and reports a pass
// that claims to preserve every analysis but changed its unit. It costs compile time.
VerifyEach bool
// Stats writes, after the build, the statistics of each pass that ran: the units it ran on,
// those it changed, its own counters and the applications of its patterns.
Stats bool
// TimePasses writes, after the build, the time each pass took, summed over the units it ran on,
// and its share of the total.
TimePasses bool
// OptBisectLimit, if set, runs only the runs of optional passes numbered up to the limit, in the
// order the pipeline makes them, and skips the later ones; the required steps always run, so the
// output stays a valid parser. A run is one pass on one unit, such as one function, and each
// application of a pattern of a pass takes a number of its own. Each decision is written to
// Output with its number, the pass and the unit. Under a limit the generators generate once, so
// the output may keep rules that it would otherwise leave out. A negative limit is an error.
OptBisectLimit *int
// Output receives everything above. It is required when any of it is set.
Output io.Writer
}
Debug is the debugging setting of the compiler: it observes and checks the pass pipeline that makes an artifact, and changes the artifact only under a bisection limit. Each field is a flag of sxir gen: PrintBefore is -print-before, PrintAfter -print-after, PrintChanged -print-changed, VerifyEach -verify-each, Stats -stats, TimePasses -time-passes and OptBisectLimit -opt-bisect-limit; Output has no flag, as the command writes to standard error. The pass and counter names and the text written to Output are a debugging aid: their form is not stable and may change in any version.
type File source
type File struct {
Path string
Content []byte
}
File is a file of generated code. Path is relative to the directory of the generated package and uses slashes.
func GenerateGo source
func GenerateGo(src grammar.Source, c GoConfig, opts ...Option) ([]File, error)
GenerateGo generates the files of a Go parser for the grammar src in the package with import path c.Module: its Parse(input) parses from the start rule that WithStart selects, or the default start rule. The files are parser.go, with the code generated from the grammar, api.go, which re-exports the library names the parser's users need, with Types types/types.go, the package Module/types that declares the Go types of the grammar, and, with LibEmbed, the files of a copy of the library in internal/libgo.
func GenerateLibGo source
func GenerateLibGo(module string) ([]File, error)
GenerateLibGo generates only the embedded copy of the library for the package with import path module, to refresh a parser generated with LibEmbed without regenerating it.
func GenerateTypeScript source
func GenerateTypeScript(src grammar.Source, c TSConfig, opts ...Option) ([]File, error)
GenerateTypeScript generates the files of a TypeScript parser for the grammar src: its parse(input) parses from the start rule that WithStart selects, or the default start rule, and returns the same trees and errors as the parser of Compile. The files are
- lib.ts, the library, which does not depend on the grammar (with LibEmbed),
- parser.ts, the code generated from the grammar, which imports ./lib.ts, or the LibModule with LibImport, and
- types.ts with Types, the declarations of the grammar's types, which parser.ts imports.
Generated TypeScript parsers have no stream parsing.
func GenerateTypeScriptLib source
func GenerateTypeScriptLib() ([]File, error)
GenerateTypeScriptLib generates only the library of TypeScript parsers, lib.ts: the module that parsers generated with LibImport import, or the copy of a parser generated with LibEmbed.
type GoConfig source
type GoConfig struct {
// Module is the import path of the generated package. It is required, except with
// LibImport and a PackageName and without Types.
Module string
// PackageName is the package name of the generated code. It defaults to the last element of
// Module, without a major version suffix (example.com/m/v2 gives m).
PackageName string
// Lib selects how the generated code reaches the library (LibEmbed by default).
Lib LibMode
// Types generates declarations for the grammar's types and typed results: ParseValue for the
// start rule and ParseRuleValue for named rules. The types are declared in the package
// Module/types, so Module is required.
Types bool
// Expectations makes the generated parser record what a syntax error expects: the expectations
// at the farthest failure and the messages of @error, as runtime.WithExpectations does for a
// compiled parser. Without it the parser keeps no table of expectations, and a syntax error
// reports the farthest failure position with an empty Expected list and no Messages.
Expectations bool
// Observability emits rule tracing and profiling for batch parsers and Documents, including
// their typed entry points. Collection is selected per invocation.
Observability bool
// NoLimitControls opts out of the limit-controlled form of the parser, which is the default: the
// controlled form supports cancellation and the compute limit, re-exports the option WithContext
// and applies the default limits to a parse that sets none. With NoLimitControls the generated
// parser is smaller, faster to build and faster to run, and a parse that passes a context or a
// finite Compute limit returns an error; Limits.Input and Limits.Depth work in both forms.
NoLimitControls bool
// LogicalUnits is internal: in the default controlled form, the functions count logical compute units
// (design record 076) in place of the charges of the limit-controlled form. It does not cover
// Documents, streams, observability, left recursion, Operators Rules or recovery.
LogicalUnits bool
// LogicalVariant selects a variant of LogicalUnits for tests and measurements: "keep-guards"
// keeps every dispatch guard without the effects of the calls it skips (inexact, to measure what
// the effects cost); "event" keeps only the guards whose skip makes no call, so that each call is
// counted where it happens, as the reference of the exactness tests.
LogicalVariant string
// Recognize also generates a function Recognize, which checks the input against the start
// rule without building a tree, like runtime.Parser.Recognize.
Recognize bool
// MaxDepth is the generated parser's default nested rule-call limit. Zero selects 100,000;
// negative values fail generation. Individual invocations may override it with Limits.Depth.
MaxDepth int
// Document generates a mutable incremental Document. With Types, it also generates
// Document.ParseValue with direct incremental matching for eligible grammars and Node
// conversion otherwise.
Document bool
// Stream generates ParseStream, which reads the input from an io.Reader and passes each
// committed element of the start rule's #stream repetition to a callback, as
// runtime.Parser.ParseStream does. The start rule must have a #stream repetition. With Types it
// also generates ParseStreamValue and IterateValue, which pass the typed value of each element.
Stream bool
// TypedOnly generates only the typed entry points of a parser: ParseValue and its siblings, and
// no Parse or ParseRule. Recognize may be added. The generated package is smaller and builds
// faster. It needs Types and cannot be combined with Document or Stream, which build untyped
// Nodes.
TypedOnly bool
// RecognizeOnly generates only Recognize, which checks the input against the start rule: no
// Parse, ParseRule or typed entry point. The generated package is the smallest. It implies
// Recognize and cannot be combined with Types, TypedOnly, Document or Stream.
RecognizeOnly bool
// OmitPackageDoc leaves the package comment out of the generated parser, for a package that
// has its own documentation in another file.
OmitPackageDoc bool
}
GoConfig configures the generation of a Go parser. The zero value generates an untyped parser with an embedded library; Module must be set for it.
type GrammarError source
type GrammarError struct {
// Pos is where the error is: a 1-based line and a 1-based column counted in characters, as in
// the positions of the grammar AST. It is the zero Pos when the error has no position in the
// source, such as an error of a grammar AST that a program built without positions.
Pos ast.Pos
// Msg is the message, without the position.
Msg string
}
GrammarError is a grammar error that compilation finds in a grammar that loads: a name that is not defined, a type that does not check, an attribute with an invalid argument or target, or another rule of the language that the syntax does not decide.
func (*GrammarError) Error source
func (e *GrammarError) Error() string
Error returns the position and the message as "line:column: message", or only the message when the position is not known.
type GrammarErrors source
type GrammarErrors []*GrammarError
GrammarErrors is the list of grammar errors of a grammar that does not compile; it is never empty. The errors are in the order in which compilation reports them.
Every function of this package that compiles a grammar returns the list itself when the grammar loads but does not compile, so a program obtains it with errors.As:
var errs compiler.GrammarErrors
if errors.As(err, &errs) {
for _, e := range errs {
// e.Pos, e.Msg
}
}
A grammar source that does not load gives the errors of grammar.Load instead, such as grammar.SyntaxErrors. The list does not unwrap to its elements: errors.As always gives the whole list, never one error.
func (GrammarErrors) Error source
func (l GrammarErrors) Error() string
Error returns the errors, one per line.
type LibMode source
type LibMode int
LibMode selects how generated code reaches the library of generated parsers.
const (
// LibEmbed writes a copy of the library next to the parser: the package internal/libgo in
// Go, which depends only on the runtime packages under syntaxir.dev/sxir/pkg/runtime, and
// lib.ts in TypeScript, which depends on nothing. It is the default. GenerateLibGo and
// GenerateTypeScriptLib refresh the copy without regenerating the parser.
LibEmbed LibMode = iota
// LibImport imports the library: syntaxir.dev/sxir/pkg/runtime/libgo in Go, the module of
// TSConfig.LibModule in TypeScript.
LibImport
)
The library modes.
func (LibMode) String source
func (m LibMode) String() string
String returns the lowercase name of the mode, embed or import, and LibMode(n) for a value that names no mode.
type Optimize source
type Optimize struct {
// Profile is "speed" (the default, also for an empty name), "memory" or "size".
Profile string
// Values gives optimizations a value other than the profile's: 1 turns on one that the
// profile leaves off, and a count sets an integer one.
Values map[string]int
// Off turns the named optimizations off, after Values.
Off []string
}
Optimize is an optimization setting: a profile, then values for single optimizations, then optimizations turned off. None of them changes what a parser returns, only how it runs. The names are those of the guide to generated code and of sxir gen -opt and -no-opt.
func WithDebug source
func WithDebug(d Debug) Option
WithDebug selects the debugging setting. A name that PrintAfter or PrintBefore does not know, a negative OptBisectLimit, or a setting that writes without an Output, is an error of every function. The setting takes effect in the functions that run the pass pipeline, the generators today; for the others it is checked and has no further effect.
func WithDefaultValue source
func WithDefaultValue() Option
WithDefaultValue makes the parser drop value actions and declared rule types and return the default node tree: the value of each rule is what it would be without its action. Actions that predicates read are kept. It cannot be combined with typed generated output.
func WithOptimize source
func WithOptimize(o Optimize) Option
WithOptimize selects the optimization setting. An unknown profile or optimization, or a value that an optimization does not take, is an error of every function. The optimizations are those of the generators today: the bytecode and the reference backends have none to select, so for them the setting is checked and has no further effect.
func WithSourceInfo source
func WithSourceInfo() Option
WithSourceInfo makes the parser record the rule of each node and the names of captures in parse results.
func WithStart source
func WithStart(name string) Option
WithStart selects the start rule of the parser, which must be a start rule (a rule marked @start). An empty name, the default, selects the default start rule.
type TSConfig source
type TSConfig struct {
// Lib selects how the generated code reaches the library (LibEmbed by default).
// LibImport needs LibModule.
Lib LibMode
// LibModule is the module specifier from which a parser generated with LibImport
// imports its library, such as "@scope/sxir-lib" or "../shared/lib.ts". The library
// module is the lib.ts of GenerateTypeScriptLib (or of any parser generated with LibEmbed).
// It is an error with another mode.
LibModule string
// Types generates declarations for the grammar's types and the entry points parseValue and
// parseValueWithOptions, which convert the completed Node result into plain objects and arrays.
// Declared aliases preserve the target value type.
Types bool
// Expectations makes the generated parser record what a syntax error expects: the expectations
// at the farthest failure and the messages of @error, as runtime.WithExpectations does for a
// compiled parser. Without it the parser keeps no table of expectations, and a syntax error
// reports the farthest failure position with an empty expected list and no messages.
Expectations bool
// Observability emits rule tracing and profiling for batch parsing, including the typed entry
// points. Collection is selected per invocation.
Observability bool
// NoLimitControls opts out of the limit-controlled form of the parser, which is the default: the
// controlled form supports the signal option and the compute limit and applies the default limits
// to a parse that sets none. With NoLimitControls the generated parser is smaller and faster, and
// a parse that passes a signal or a finite compute limit returns an error; the input and depth
// limits work in both forms.
NoLimitControls bool
// Recognize also generates recognize, which checks the input against the start rule without
// building a tree, like runtime.Parser.Recognize.
Recognize bool
// Document also generates createDocument, which makes a mutable Document of the start rule:
// edit changes its text, and parse returns what parse of the whole current text returns, as
// the Go Document does. With Types, the Document also has parseValue.
Document bool
// MaxDepth is the generated parser's default nested rule-call limit. Zero selects 100,000;
// negative values and values that are not exact JavaScript integers fail generation.
// Individual invocations may override it with limits.depth.
MaxDepth int
}
TSConfig configures the generation of a TypeScript parser. The zero value generates an untyped parser with an embedded library.