Chapter 7
Generated Parsers
sxir gen translates a grammar into the source of a standalone parser, in Go or
in TypeScript. A generated parser does not load or compile a grammar at run time
and does not depend on the SyntaxIR engine. This chapter defines the generator, the
API of the generated Go and TypeScript code, the features each target supports
and where its behavior differs from the engine. Where this chapter does not state
a difference, a generated parser returns the same value, the same positions and
the same errors as the engine for the same grammar, input, start rule and
position unit.
Scope and Terminology
The engine is the runtime.Parser that parses with a compiled grammar. A
generated parser is the source that compiler.GenerateGo, compiler.GenerateTypeScript (package
syntaxir.dev/sxir/pkg/compiler) or sxir gen writes. The generation options select optional parts of the
output. The generated package is the Go package named by -pkg; the
generated module is the TypeScript file. The generated Go code uses the library
package, either imported as syntaxir.dev/sxir/pkg/runtime/libgo or as a copy in internal/libgo. The
imported package and the copy that -lib only writes are the whole library, the same for every
grammar of one generator release. The copy that -lib embed writes next to a parser is narrowed to
the features that the grammar and the generation options use: it leaves out the declarations and
code of the library that the parser cannot run (none with the optimization used-features off).
In both cases the generated code also uses the packages under syntaxir.dev/sxir/pkg/runtime, which the library depends on; the
TypeScript module uses the TypeScript library, lib.ts, imported or as a copy that is narrowed in
the same way.
Every generated Go and TypeScript file starts with the line // Code generated by sxir <version>. DO NOT EDIT.,
where <version> names the generator: a release tag (v0.1.0) for a release, a pseudo-version (v0.0.0-<time>-<commit>) for a build from one commit, and (devel) for a generator built without a module version, with the module replaced, or from a modified tree. One generator writes the same stamp every time, so regeneration is byte-identical.
Contract
The Generator
sxir gen -g <grammar> [-lang go|ts] [-module <import path>] [-pkg <package>]
[-lib embed|import|only] [-s <rule>] [-o <path>]
[-types] [-document] [-stream] [-recognize] [-expectations] [-observability]
[-source-info] [-default-value] [-nodoc] [-max-depth <n>]
[-optimize <profile>] [-opt <name[=count],...>] [-no-opt <name,...>]
[-print-before <pass,...>] [-print-after <pass,...>] [-print-changed] [-verify-each]
[-stats] [-time-passes] [-opt-bisect-limit <n>]
| Flag | Go API: a field of compiler.GoConfig / compiler.TSConfig, or as stated |
Meaning |
|---|---|---|
-lang |
compiler.GenerateGo with a GoConfig, or compiler.GenerateTypeScript with a TSConfig |
go (default) or ts |
-g |
the grammar source argument | The grammar source: .sxir or .json. A compiled grammar holds no grammar and is an error |
-module |
Module |
Required for Go (with -lib import only when -pkg is omitted); the import path of the generated package; rejected with -lang ts |
-pkg |
PackageName |
The package name; MUST be a valid Go identifier; default: the last element of the import path without a major version suffix (example.com/m/v2 gives m), and when that is not a valid identifier, generation fails and suggests -pkg; rejected with -lang ts and with -lib only |
-lib |
Lib, a compiler.LibMode: LibEmbed or LibImport |
How the parser reaches the library. embed (default) writes a copy of the library for the parser: internal/libgo in Go, lib.ts in TypeScript. import imports it: syntaxir.dev/sxir/pkg/runtime/libgo in Go, the module that -lib-module names in TypeScript. A Go parser generated with import refers to the constant ProtocolN of the library protocol version N it was generated for, so it builds only against a library that implements that protocol. A Go copy refers in the same way to the constant ProtocolN of syntaxir.dev/sxir/pkg/runtime/value/impl, so it builds only against a core that serves the node representation of its protocol. only writes just a copy of the whole library (GenerateLibGo, GenerateTypeScriptLib), which parsers of any grammar can use, and takes no grammar and no other generation flag; in Go, compiler.GenerateLibGo and compiler.GenerateTypeScriptLib |
-lib-module |
LibModule (compiler.TSConfig) |
TypeScript only, and required with -lib import: the module specifier that parser.ts and types.ts import the library from, such as ../shared/lib.ts. Rejected with -lang go and with any other -lib |
-s |
the option compiler.WithStart |
The start rule, a rule marked @start. Default: the default start rule |
-o |
For Go, the output directory (required), which the command creates; it writes parser.go, api.go, with -types types/types.go, and, with embed, the library package in internal/libgo, and removes generated files of an earlier generation that it did not write again (only writes and removes only files of internal/libgo, so it never removes parser.go, api.go or types/types.go). A file that lacks the // Code generated by sxir <version>. DO NOT EDIT. header is never replaced or removed: the command fails before it writes anything. For TypeScript, the output directory (required): it writes parser.ts, with -types types.ts, and, with embed or only, lib.ts |
|
-types |
Types |
Generate typed values (see Typed values) |
-document |
Document |
Generate the Document (createDocument in TypeScript) |
-stream |
Stream |
Generate ParseStream (Go only; the start rule MUST contain a #stream repetition) |
-recognize |
Recognize |
Generate recognition without a tree |
-typed-only |
TypedOnly |
Go only; needs -types. Generate only the typed entry points (ParseValue, ParseRuleValue and their options); there is no Parse, ParseRule or Recognize. Rejected with -recognize, -document, -stream and -lang ts. A typed entry point that converts the Node of its rule keeps the unexported Node rule functions it runs |
-expectations |
Expectations |
Let a parse record expectations: what a syntax error expects and the messages of @error. The parser then has the table of expectations and a parse option WithExpectations (Go) or expectations (TypeScript); a parse records only when it passes the option, and otherwise a syntax error has the farthest position, an empty Expected, no Messages and the message syntax error. Without the flag the parser has no table and no option. A Document fixes the choice when it is created. It applies to every entry point, including the typed, Document and stream forms |
-observability |
Observability |
Generate tracing and profiling, specified in Observability |
-source-info |
the option compiler.WithSourceInfo |
Make the parser record source information: the rule of each node and the names of captures. Without it, Node.Rule, Node.Field and Node.Fields of a parsed node are empty and the outputs omit them |
-default-value |
the option compiler.WithDefaultValue |
Make the parser return default values, as Default Values describes. Cannot be combined with -types, because the types come from the dropped actions |
-optimize |
Profile of the compiler.Optimize of the option compiler.WithOptimize |
The optimization profile: speed (default, also the empty name), memory or size. An unknown profile is an error that lists the valid names. Rejected with -lib only |
-opt |
Values of compiler.Optimize (a map from name to value) |
Gives single optimizations a value, as name or name=count. Rejected with -lib only |
-no-opt |
Off of compiler.Optimize |
Turns the named optimizations off. Rejected with -lib only |
-print-after |
PrintAfter of the compiler.Debug of the option compiler.WithDebug |
Writes the intermediate representation to standard error after the named steps and passes. Rejected with -lib only |
-print-before |
PrintBefore of compiler.Debug |
Writes the intermediate representation that the named passes receive to standard error before their runs. Rejected with -lib only |
-print-changed |
PrintChanged of compiler.Debug |
Writes the intermediate representation to standard error after each pass that changed it. Rejected with -lib only |
-verify-each |
VerifyEach of compiler.Debug |
Verifies the intermediate representation after every pass. Rejected with -lib only |
-stats |
Stats of compiler.Debug |
Writes the statistics of the passes to standard error after the build. Rejected with -lib only |
-time-passes |
TimePasses of compiler.Debug |
Writes the times of the passes to standard error after the build. Rejected with -lib only |
-opt-bisect-limit |
OptBisectLimit of compiler.Debug (nil: no limit) |
Runs only the runs of optional passes numbered up to the count, skips the later ones and writes each decision to standard error; generates once. Rejected with -lib only |
-nodoc |
OmitPackageDoc |
Omit the package comment (Go only; rejected with -lang ts) |
-max-depth |
MaxDepth |
The default nested recursive rule-call limit; zero selects 100,000, a positive value overrides it, a negative value fails generation. TypeScript also requires a safe integer |
- The output is deterministic: generating twice yields identical bytes for the same grammar, options and version of the generator. Omitted, zero and an explicit 100,000 depth default generate identical source.
- Go output is formatted as by
gofmt. Generated Go with the defaultembedlibrary depends only on the packages undersyntaxir.dev/sxir/pkg/runtime(withvalue/impl) and the standard library, and generated TypeScript has no dependencies. Generated TypeScript uses only erasable syntax and needs ES2020; it is type-checked under strict settings. - Grammar errors are reported with positions before any file is written, in the
same form as in other commands. An unknown start rule, an invalid package name,
a compiled grammar (
gen needs the grammar source),-streamwithout a#streamrepetition in the start rule and a grammar that contains#missingare errors. - A Go package may contain only one generated parser, because every generated file
defines the same names.
api.gore-exports the names the users of the parser need, as aliases of the types of the runtime packages and the library and wrappers of the library's functions. - With
-typesthe Go types of the typed values are declared in the package<module>/types(the filetypes/types.go), which the parser package imports; a module path is required. The types package imports only the library and the runtime packages, and declares a constantSXIRSchema_<digest>that the parser package refers to once (see below). The parser package refers to the types astypes.Name, and the users of the parser import<module>/typesfor them. The grammar's types therefore share no namespace withparser.goandapi.go: a grammar type may be namedNode,Parser,Walk,ParseOptionor like any other export of the library.
Optimization Settings
The optimization settings are a profile and the values of named optimizations. They belong to the compilation of a grammar, not to one generator: the settings are the same for every backend (the Go and TypeScript generators, the closure engine and the bytecode VMs), and each backend applies the optimizations that concern it. Today only the generators have optimizations to apply: the closure engine, the bytecode VMs and the reference backends accept the setting, check it and apply nothing.
The settings are not part of the grammar's meaning: an optimization changes the code a parser is generated as, and the memory and
time it needs, but not what it returns. It is either on or off, or it has a count where 0 is off.
Each optimization has a kind: pass (a transformation of the Core IR, the first intermediate
representation built from the grammar), flow pass and exec pass (a transformation of Code IR at the Flow
stage and at the Exec stage that is converted from it), code pass (a transformation of Code IR at both stages),
pattern (a local rewrite that a code pass applies, each application on its own), lowering (an option of a step that lowers one
representation toward the next), printer (a choice of what the Go and TypeScript printers emit) or
runtime (a control of the runtime structures that the printers emit). sxir gen -h lists every optimization with its kind, its default and whether it takes a
count.
- No option changes what a parser returns. For every accepted combination of
-optimize,-optand-no-opt, the generated parser accepts and rejects the same inputs and returns the same trees, typed values, syntax errors and recovered errors, and the same trace and profile events, as one generated with the defaults. Options change only the generated source, its size and the time and memory of a parse. The debugging flags (-print-before,-print-after,-print-changed,-verify-each,-stats,-time-passes) write diagnostics to standard error, or fail on an internal error, and change neither the files nor the parser.-opt-bisect-limitchanges the files: it skips optional passes and generates once, which can keep rules that are otherwise left out, and the parser behaves the same unless a skipped pass is wrong, which is what the flag finds. - Profiles. A profile is a fixed set of values and has no behavior of its own.
speedis the default and sets every optimization to its default value.memoryallocates typed values one by one instead of in slabs (fewer bytes and less live heap, more allocations).sizeproduces smaller generated Go source and binaries for a little parse time. - Order. The settings are the profile's, then the values of
-optreplace them, then-no-optturns optimizations off.-no-opttherefore wins over-optfor the same name. - Values.
-opt nameisname=1. An optimization without a count takes0or1; one with a count takes0or a positive number. A name that is not an optimization, a value that the optimization does not take, a count that is not a decimal number, an empty item and a name given twice in-optare errors, and generation fails before it writes a file. - Backends and languages. An optimization that concerns only one backend or language, such as the slabs of typed Go values, has no effect on another, and naming it is not an error.
-print-aftertakes the stepselaborate(the Core IR of each rule table as built),plan(for Go, the entry points of the program, and the route of a call of each rule and how the entry points reach it),flowlower(each function as the Code IR listing at the Flow stage, as lowered from Core IR),flow-finish(the same listing after the Flow stage of a function ends), andexeclower(each function as the Code IR listing at the Exec stage, as converted from the Flow stage), the name of any pass, orall, which selects every step and pass exceptplan. A pass that does not apply to the parser prints nothing. An unknown name is an error that lists the valid ones.- Debugging output. What the debugging flags write is not part of this contract: its form, the pass names and
the counter names may change in any version. In Go, the fields of
compiler.Debugmap to the flags one to one, andOutput, the writer that receives the output, is required when any field butVerifyEachis set; the command writes to standard error.
Generated Go API
func Parse(input string, opts ...ParseOption) (*Node, error)
func ParseRule(name, input string, opts ...ParseOption) (*Node, error)
type ParseOption func(*parseOptions)
func WithUnit(unit PositionUnit) ParseOption
func WithLimits(limits Limits) ParseOption
func WithContext(ctx context.Context) ParseOption // unless generated with -no-limits
type PositionUnit int
const (
CodePoints PositionUnit = iota // the default
Bytes
)
Parseparses the whole input with the start rule.ParseRuledoes the same with the named start rule, a rule marked@start; another rule is the errorrule <name> is not a start rule; parsing can begin only at a rule marked @start, an unknown name the errorrule <name> is not defined, the name of a hidden rule the error that the rule is hidden, and an empty name selects the default start rule of the grammar. Both require the rule to consume the whole input. Generated parsers contain entry code only for start rules.Node,SyntaxErrorandSyntaxErrorsare aliases of the types of the packagessyntaxir.dev/sxir/pkg/runtime/valueandsyntaxir.dev/sxir/pkg/runtime, defined in Nodes and Values and Parsing, and so are the position units, limits and traversal. Withimportand withembedthey are the types that the engine uses, so results and errors of a generated parser and of the engine have the same Go types. Node JSON is identical to the engine's.- Options apply in order. A final
Limits.DepthofUnlimitedreturns a configuration error before matching, actions or named-rule lookup, because generated code nests rule calls on the host stack; zero selects the generated default. A nil option panics when applied. A call with no options uses code points and the generated depth default. - A generated parser is safe for concurrent use: each call has its own state.
- Errors in actions (such as division by zero) return plain errors as in the engine, and
exceeding the nesting limit returns a
*LimitErrorof kindLimitDepth. The depth limit counts nested entries of recursive rules (Limits); a larger limit does not guarantee stack capacity. The default error isnesting too deep: more than 100000 nested recursive rule calls. - Every generated Go parser exports
WithLimits,Limits,LimitKind,LimitError,ErrLimit,LimitCompute,LimitInput,LimitDepth,LimitMemoryandUnlimited. A parser is generated with limit controls unless it opts out (-no-limits,NoLimitControls), and one generated with them also exportsWithContext. A parser generated with limit controls has the controlled form only, and a parse that sets no limit runs it with the default budget: a generated parser applies the default limits,Computeofmax(2^20, 1024 * n)among them, unless it opts out. A parser generated without them (-no-limits) holds no code that spends budget or reads a control: one form of each function; with-documentit is the form that serves a Document, without charges. Such a parser, given a context,WithComputeReportor a finiteComputeorMemorylimit, returns an error that the parser was generated without limit controls; no option is ignored. A nil context,ComputeandMemorylimits that are zero orUnlimited, anyInputand anyDepthset no control and are accepted. WithContextandWithLimitsapply the semantics of Cancellation and Compute Limits to each invocation, with counters that restart for every call. They apply to named rules, recognition, typed parsing and conversion. The last context wins andWithLimitsmerges field by field; a nil context means no cancellation; a zero limit selects the default andUnlimiteddisables it. An exhausted limit matchesErrLimit, anderrors.Asretrieves a*LimitErrorwithKind(LimitCompute,LimitInputorLimitMemory),Limit,UsedandRequested, and forLimitComputealsoHotspot, which names the rule that the profile of the parse ranks first. A finiteMemorybounds the storage of the parse as the memory limit defines; the typed values of-typescount too. An input longer thanInputfails withLimitInputbefore any matching, in every entry point, and a generated stream fails when the parser reaches a unit beyond it. An interrupted entry point returns its zero result, and recovery does not turn an interruption into a successful partial tree. Cancellation is cooperative.
Typed Values
With -types, the Go package also defines a Go type for each type of the grammar
and the functions
func ParseValue(input string, opts ...ParseOption) (T, error) // T: the Go type of the start rule's type
func ParseRuleValue(name, input string, opts ...ParseOption) (any, error)
ParseValue returns the errors of Parse. The mapping of types is:
| Grammar type | Go type |
|---|---|
int, string, bool |
The same |
Struct type T |
*T with the same fields and an embedded Span, which is input.Span{Start, End int} of package runtime/input: the range [Start, End) of the input in the parse's position unit |
node[T] |
*TNode[T'], with Span and Value of the Go type T' of T: *TNode[string] for a string node, whose Value is the text, and *TNode[any] for node[_] |
Tuple (A, B, ...) |
TupleN[A', B', ...] with the fields E0, E1, ...; Unit for () |
error (from #recover) |
*Error, a type of the types package with Span and Message |
Join U ~ A | B of node types: struct types, named node types, unnamed node types and error |
An interface U implemented by *A, *B and *Error, and by every *TNode[T] if a member is an unnamed node type. A value converts to the member its type names, or, for a node of an unnamed node type or of a named node type that is no member, to the unnamed node member of its kind (string, tuple or list) with that member's Go type; node[_] takes every other node as a *TNode[any] |
Join of node types written in place (a field, a list element, a rule value) and declared by no alias: A | B (two or more distinct members, with or without error), or A | error (one member and error) |
An interface named after its members in the order of their first spelling in the file (declarations in source order, then the fields of a type), AOrB, or AOrError for one member and error, implemented like the interface of an alias; an unnamed node member is named after its value and Node, as StringNode for node[string], ItemListNode for node[[]Item] and Node for node[_]. The same set of members in any order is one interface, and a set that an alias declares is that alias. A type of the grammar with the generated name, or two different sets with the same name, is a generation error. A join without a struct or named node type member, a join with a member that is no node type (a list, a scalar), or one with two unnamed node members of one kind has no interface and is any; an alias gives a join of unnamed node types an interface |
[]T |
A slice; an empty list is an empty slice and not nil |
*T |
*int, *string or *bool for basic types; otherwise the Go type of T, nil when absent |
| A rule whose type is not declared | The Go type of its value (an action's value maps by the rows above): *TNode[TupleN[E0, ..., En]] for a tuple of known elements, *TNode[[]E] for a list, *TNode[Unit] for the unit (); *TNode[any] when the elements are not known; a rule with a declared type, node[_] included, always has the Go type of that type |
_, joins with basic members, with unnamed node members of one kind, or with members of different shapes, a value that closes a cycle of rules |
any, holding the converted value |
Alias Alias ~ T |
A Go alias to the value type of T |
TNode[T]isstruct { Span; Value T };TupleNisstruct { E0 E0; ...; En En }andUnitisstruct{}. A missing element of a tuple isnil. A list node'sValueis a slice.- A grammar type keeps its name. The types package declares, besides the grammar's
types,
TNode,Error,SXIRHooksand, when used,Unit,TupleNandSXIRNodeTypes, and a constantSXIRSchema_<digest>; generation fails, naming the type, if a grammar type has one of these names. The digest covers the package's declarations, not its comments or formatting. The parser package refers to the constant once, so aparser.goand atypes/types.gogenerated from different schemas do not compile together; the runtime relies on this when it reads a string node as its named node type. A grammar type namedUnitis an error only when the generated code uses the empty tuple, and one namedTupleNonly when it uses a tuple of that arity. A struct field named like a field or method of the embeddedSpan(Start,End,Range) or like a method of the typed values (TName, ...) gets a trailing underscore in Go (Start_), soStartandEndalways read the span; the value still reads back under the grammar name. Every declared alias is exported. - After
#recover, an error value in a list of a join type stays in the list as*Error; where a struct type is expected it becomesnil. ParseRuleValueconsumes the whole input, applies options once before it looks up the name, and returns the selected start rule's typed value asany. Names thatParseRulerejects, with the same errors, and invalid options returnnilwith an error. A parse failure returns the zero value of the selected type.ParseRulecontinues to return*Node.- A generator MAY build typed values directly instead of converting the tree
that
Parsereturns when every value has a Go type of its own; the result is the same either way.
Typed TypeScript output (-types) exports an interface for every struct
and alias and the functions parseValue and parseValueWithOptions, which parse the
configured start rule and convert the node result. They return { value, error }:
a successful nil result has a null value and a null error; a failure has a null
value and the error; recovery keeps the converted value and SyntaxErrors. The
mapping is:
| Grammar type | TypeScript value |
|---|---|
| Struct | An interface with type (the grammar name), start, end and the fields; nullable where an absent or incompatible recovered value becomes nil |
node[T] |
TNode<T'>, with start, end and value of the TypeScript type T' of T; for a node of one kind, the interface of that kind, which extends it with that kind as its type: StringNode for a string node, TupleNode<T'> for a tuple node and ListNode<T'> for a list node, also when the node is one of a named node type; TNode<unknown> for node[_] |
Tuple (A, B, ...) |
readonly [A', B', ...] |
[]T |
A readonly array, or null; null differs from an empty array |
*T |
A nullable target |
int |
number | bigint, with signed 64-bit arithmetic and safe-number canonicalization |
string, bool |
string, boolean; an absent non-optional scalar field is "", false or 0 |
| Declared alias | The complete target type |
| Join of node types, declared by an alias or written in place | The TypeScript types of the members, RecoveredError and null, which narrow by type; a value converts to its member as in Go |
Join with a member that is no node type, _ |
TypedValue, following the conversion of any in Go |
Names collide the same way as in Go, with trailing underscores. Typed parsing by rule name is not available in TypeScript.
Recognition
With -recognize, Go gains Recognize(input string, opts ...ParseOption) error and
TypeScript gains recognize and recognizeWithOptions. They check that the whole
input matches the start rule without building a tree and return the errors that
Parse returns, including SyntaxErrors for recovered errors. Actions whose values
are not read by predicates are skipped; actions that predicates read still run and
can report runtime errors.
Incremental Documents
With -document, the Go package gains NewDocument(text string, opts ...ParseOption), which returns a
runtime.Document (Parse, Edit(start, end, replacement), Text, Unit, Stats, CacheStats and
ResetHistory); the package re-exports it as Document. With -types, NewDocument returns the package's own
*Document, which implements runtime.Document and adds ParseValue. The library's own Document type is not part of the
API.
Editreplaces a half-open range in the position unit chosen at construction. It rejects an invalid range, and a position inside a UTF-8 character inBytesmode, without changing the text.Text()returns the exact current input.- Construction options become the defaults of later calls. A call MAY set the depth, the context and the compute limits; it MUST NOT change the unit.
ParseStatscounts the rule evaluations (Evaluated) and reuses (Reused) of the last parse: the reused memo results (MemoHits) and the elements of repetitions that it resumed (ResumedElements).Parsereuses and resumes them by the rules of the engine's Reuse Model, so itsStatsequal the engine's after the same grammar, text and edits.ParseValuekeeps its own typed memo and resumes no repetitions, so theStatsafter it can differ.- A parse returns what a parse of the whole current text returns. An interruption discards memo history so that a later call can retry.
- A
Documentis for one goroutine. A returned tree can share nodes with later parses;Node.Clonemakes a detached copy that preserves cycles and repeated references. With-types,ParseValuereturns read-only values that share storage with the document and with earlier results. Caller mutation can therefore change what later results contain, as withNodevalues.CloneValue(value)returns an independent copy that shares nothing with the document or other results, and preserves aliases within the copy. Every slice in a returned value has capacity equal to its length, soappendon a result cannot write into another result. - The returned
SyntaxErrorvalues and theirExpectedandMessagesslices are owned by the caller. Parsing, diagnostic copying and typed publication share one budget. - It supports a subset of the engine's Incremental Documents: there
is no
EditBatch, noVersion, no snapshots and no change notifications.
With -document, the TypeScript parser.ts gains
createDocument(input: string | Uint8Array, options?), which returns { document, error }, and a Document
class with these methods: parse(options?), edit(start, end, replacement), text(), bytes(), stats()
and, with -types, parseValue(options?).
createDocumentcopies the input and reads the options once. They become the defaults of every parse;documentis null, with the error, when they are invalid or the input exceeds their limits.editfollows the GoEditand returns anErroror null. A string replacement is its UTF-8 encoding, with lone surrogates as U+FFFD; aUint8Arrayreplacement is taken as it is.text()returns the text as nodes hold it in the unit, andbytes()a copy of the exact input.- A parse returns what
parseof the whole current text returns, andparseValuewhatparseValuereturns. An invocation MUST NOT change the unit or the expectations. Aparseoreditmade while the Document parses, from a trace callback, returns an error. - A parse reuses the memo results of earlier parses that the edits since left valid,
by the rules of the Go
Document, but resumes no repetitions: it evaluates their elements again.stats()returns{ evaluated, reused }, the rule evaluations and the reused results of the last parse, which equal the GoStatsafterParseof the same grammar, text and edits when that parse resumed no repetition (ResumedElementsis zero). The GoParseValuekeeps its own typed memo, so the GoStatsafter it can differ. - A returned tree can share nodes with the results of later parses, so mutating it can change what they contain. A reused result whose position an edit moved is returned in copies, which leave earlier trees unchanged.
Streaming (Go Only)
With -stream, the Go package gains ParseStream(r io.Reader, emit func(*Node) error, opts ...ParseOption) error,
Iterate with StreamIterator and ErrStreamClosed, and the options
WithDiagnosticCallback and WithDiagnosticLimit with TruncatedSyntaxErrors. They
follow the contract of Streaming, with these differences: a nil
emit parses the stream without delivering elements, and the diagnostic options are
accepted only by the stream entry points (Parse, ParseRule, ParseValue,
Recognize, NewDocument and Document methods reject them). Parse treats
#stream as an ordinary repetition.
With -types as well, the package gains StreamValue, ParseStreamValue and
IterateValue with StreamValueIterator. StreamValue is the Go type of one element
of the #stream repetition, not of the start rule. Each element is converted on
its own when it is committed, so the value stays valid after the callback returns.
The conversion charges no compute units, as converting a result
does not, and observes the context.
Generated TypeScript API
export function parse(input: string | Uint8Array, unit?: Unit): ParseResult;
export function parseRule(name: string, input: string | Uint8Array, unit?: Unit): ParseResult;
export function parseWithOptions(input: string | Uint8Array, options?: ParseOptions): ParseResult;
export function parseRuleWithOptions(name: string, input: string | Uint8Array, options?: ParseOptions): ParseResult;
export interface ParseOptions {
unit?: Unit;
signal?: CancellationSignal | null; // unless generated with -no-limits
computeReport?: ComputeReport | null; // unless generated with -no-limits
limits?: Limits;
}
export interface ComputeReport { units: number } // set by the parse: see parsing.md#reporting-the-units-spent
export interface Limits {
compute?: number; // unless generated with -no-limits
memory?: number; // unless generated with -no-limits
input?: number; // works with and without -no-limits
depth?: number;
}
export const Unlimited: number; // Number.MAX_SAFE_INTEGER
export interface ParseResult { node: Node | null; error: Error | null }
// With -limits:
export interface CancellationSignal { readonly aborted: boolean; readonly reason?: unknown }
// In both forms:
export type ParseInterruptionKind = "canceled" | "compute-limit" | "memory-limit" | "input-limit" | "depth-limit";
export class ParseInterruptionError extends Error {
readonly kind: ParseInterruptionKind;
readonly limit: number;
readonly used: number;
readonly requested: number;
readonly hotspot: string; // "" unless a compute-limit with a profile
readonly cause: unknown;
}
export type Unit = 0 | 1;
export const CodePoints: Unit; // the default
export const Bytes: Unit;
export function marshal(n: Node | null): string;
- Public names. The public API of a generated TypeScript parser is what
parser.tsexports, plus the names oftypes.tswith-types. From the library it re-exportsValue,Unit,CodePoints,Bytes,NodeField,NodeJSON,Node,marshal,SyntaxError,SyntaxErrors,ParseResult,ParseOptions,Limits,ParseInterruptionKind,ParseInterruptionError,Unlimited,TraceKind,TraceEvent,TraceFunction,SkipReason,Profile,CallSite,SkipReasonCount,RuleProfileandProfileSnapshot, and, unless generated with-no-limits, alsoCancellationSignalandComputeReport. With-no-limitsthe exportedParseOptionsandLimitsare the forms withoutsignal,computeReportandlimits.compute.TraceKindis"enter","exit"or"skipped". The other exports oflib.ts, such asProfileData, are the contract between the library and the generated code. They are not public API and may change between versions. parsedoes not throw for bad input: it returns{ node, error }with a node and no error; no node and an error; or, after#recover, a node andSyntaxErrors. It throws only for defects that are not parse errors.Nodehastype,rule,start,end,text,children,fields,field(name),isTerminal(),toString()andtoJSON(), with the meaning given in Nodes and Values. A field value is aNode, a number, a bigint, a string, a boolean ornull.SyntaxErrorhaspos,line,col,expected,messagesandreason(), and itsmessageis the text of the engine's error;SyntaxErrorshaserrors.- The
SyntaxErrorvalues ofSyntaxErrors, the errors that#recoverrecovered from, carry only the header line instack; theerrorof a parse that failed keeps the stack of the point where it was made. marshal(node)returns exactly the bytes thatencoding/jsonwrites for the engine's node.JSON.stringifyyields the same keys in the same order, except that integers beyond the safe range are rounded and, inBytesmode, an invalid input byte is written as the surrogate that represents it.marshalandtoStringwrite trees of any depth.JSON.stringifyrecurses and can exhaust the stack for deep trees.limits.depthMUST be a positive safe integer or zero;Unlimited, negative values, fractions, NaN, infinity and unsafe integers return a configuration error in the result before matching, because rule calls nest on the JavaScript stack. Omitted or zero selects the generated default. A parse that nests more recursive rule calls returnsnodeasnullanderroras aParseInterruptionErrorwhosekindis"depth-limit", withlimitandusedthe depth limit andrequested1, in both forms.- A parser generated with
-no-limitsemits no code that spends budget and sets no control, so a parse charges nothing, and itsParseOptionshas nosignalorcomputeReportand itsLimitshas nocomputeormemory; it does not exportCancellationSignalorComputeReport. A call that passes a signal, a compute report or a finite nonzero compute or memory limit anyway returns an error in the result, before matching, that says the parser was generated without limit controls; no option is ignored. An absent or null signal or compute report, zero orUnlimitedcompute and memory limits, and the input and depth limits set no control and are accepted. A parser generated without-no-limits, the default, has the controlled form only, and a call that sets no limit runs it with the default budget,computeofmax(2^20, 1024 * n)among its defaults. The rest of this list describes the controlled form. signalis a structural cancellation signal: any object withabortedand an optionalreason, such as anAbortSignal.limits.compute,limits.memoryandlimits.inputMUST be finite nonnegative safe integers; zero selects the default andUnlimited(Number.MAX_SAFE_INTEGER) disables the limit; other values return a configuration error in the result. An unknown option or limit key, such aslimits.time, is a configuration error too, in both forms. A finitelimits.inputbounds the input in the position unit and is checked before any matching, in both forms; a longer input returns aParseInterruptionErrorwhosekindis"input-limit", withlimitthe input limit,usedzero andrequestedthe length of the input. The units of the limits are those of Cancellation and Compute Limits.- A finite
limits.memorybounds the storage the parse takes into use, in bytes, as the memory limit defines it, with the polls, order of checks and accounting given there. The count is an estimate from a model of the sizes of the runtime's objects in a 64-bit JavaScript engine: the nodes, child lists and fields the parse builds, memo entries, slots and hash indexes, first-call sets, the decoded input and its line table, the records of left recursion and of scan and skip sites, variable environments and frames, syntax errors, the text copied out of the input and the strings that actions concatenate, and the slots of the working stacks at their largest length at a poll. Not counted are the caller's input, the values of the typed conversion thatparseValuehands to the caller, and what the JavaScript engine adds beyond the model. A count above the limit returns aParseInterruptionErrorwhosekindis"memory-limit", messagememory-limit exceeded: limit N, used M, withlimitthe memory limit,usedthe count andrequestedzero. - An interrupted parse returns
nodeasnullanderroras aParseInterruptionErrorwhosekindis"canceled","compute-limit","memory-limit"or"input-limit". For a limit,limit,usedandrequestedgive the accounting, with the meaning that aLimitErrorreports, and for a"compute-limit"hotspotis theHotspotof that error, the empty string without a profile; for cancellation,causeis thereasonof the signal. Recovery does not turn an interruption into a partial result, and the parser keeps no reference to the signal after the call returns. Typed conversion (parseValue) and recognition report interruptions the same way. - A synchronous parse does not yield to the event loop: a signal that changes only through timers or other tasks of the same thread is not observed during the call. A signal that is already aborted is observed at the first poll, and the compute limits do not depend on the event loop.
- With
-recognize,recognizeandrecognizeWithOptionsreturnnullif the input matches and anErrorotherwise. With-observability, the batch entry points accept the tracing and profiling options of Observability. - A generated TypeScript module exports the same names for every grammar, so two parsers are imported under different namespaces. A parser keeps no state between calls.
Input and Positions in TypeScript
Positions count Unicode code points (the default) or UTF-8 bytes (Bytes), as in the
engine, and not UTF-16 code units. The parser decodes the input first.
- A string is parsed as its UTF-8 encoding would be. A lone surrogate becomes U+FFFD.
- A
Uint8Arrayis parsed as UTF-8 bytes; an invalid sequence decodes to U+FFFD, one per byte, as in Go. InBytesmode, positions count the original bytes. - In
Bytesmode, text keeps the bytes: each invalid byte b is represented by the lone surrogate U+DC00+b in node text and intext(...).marshalwrites such a byte as U+FFFD. - String operations in actions follow Go:
+concatenates, comparisons order by code point, andlencounts code points or UTF-8 bytes. Integers are signed 64-bit and wrap on overflow; division truncates toward zero and%takes the sign of the dividend. An integer is anumberwhile it is a safe integer and abigintbeyond.
TypeScript Errors and the Stack
| Error | error |
|---|---|
| The input does not match | SyntaxError |
#recover skipped input |
SyntaxErrors, and node is set |
| A runtime error in an action | Error with the message of the engine |
| More nested recursive rule calls than the limit (default 100,000) | Error: nesting too deep: more than 100000 nested recursive rule calls |
| The JavaScript stack is exhausted before the limit | Error: nesting too deep: the JavaScript stack overflowed at N nested recursive rule calls |
The host stack of a JavaScript runtime is far smaller than the stack of a Go program, so a TypeScript parser can reject nesting that the engine accepts.
Validation and Errors
| Condition | Result |
|---|---|
| Invalid package name, unknown start rule, grammar errors | Error before output |
#missing in the grammar |
Generation fails |
-stream with -lang ts |
Generation fails |
-g, -s, -pkg or a generation option with -lib only |
The command fails |
-lib embed without -module; -lib import with neither -module nor -pkg |
The command fails |
An existing parser.go, api.go, types/types.go or library file in -o that sxir did not generate |
The command fails before writing |
WithTrace, WithProfile, WithTraceFilter or WithTraceMaxDepth passed to a parser generated without -observability |
The call returns an error before parsing |
-pkg or -nodoc with -lang ts |
Generation fails |
-stream without a #stream repetition in the start rule |
Generation fails |
Negative -max-depth |
Generation fails |
Compatibility and Support
| Feature | Generated Go | Generated TypeScript |
|---|---|---|
| Rule expressions, captures, atomic and discarded expressions | Supported | Supported |
| Operators Rules, left recursion (direct and indirect) | Supported | Supported |
| Predicates, variables, lookahead, cut | Supported | Supported |
| Actions, types, built-in functions | Supported | Supported |
@error, #recover, SyntaxErrors |
Supported | Supported |
#missing |
Not supported | Not supported |
| Position units | WithUnit |
unit |
| Another start rule | ParseRule |
parseRule |
| Recognition | Recognize with -recognize |
recognize with -recognize |
| Typed values | ParseValue, ParseRuleValue with -types |
parseValue, parseValueWithOptions with -types |
| Streaming | ParseStream, Iterate with -stream |
None; #stream is an ordinary repetition |
| Incremental documents | A subset with -document |
The same subset, createDocument with -document |
| Cancellation and compute limits | WithContext and a finite Compute unless generated with -no-limits; WithLimits |
signal and limits.compute unless generated with -no-limits; limits.input; ParseInterruptionError |
| Nested recursive rule-call limit | Limits.Depth, -max-depth |
limits.depth, -max-depth; the host stack also bounds nesting |
Choice of backend, saved grammar, Parser.Grammar() |
Not applicable | Not applicable |
| Tracing and profiling | With -observability |
With -observability (batch parsers) |
A generated parser carries the library of the generator that produced it (embed) or imports it
(import), and does not follow later changes of the grammar or of the generator.
Shared TypeScript Library
With -lib import, parser.ts and types.ts import the names they use from the module that
-lib-module names. The generator does not check that module. The contract is:
- The module MUST be the
lib.tsthat the same version ofsxirwrites (-lib embed, or-lib onlyto refresh a copy). The parsers generated by one version need that version's library: the names, types and behavior that a parser imports are not stable between versions. - One
lib.tsMAY serve any number of parsers, which are generated by that version and share its classes. ANodemade by one of them is an instance of the same class as in the others. - A generated parser carries no library version and the library has none: a library of another version is not detected when the parser is generated or loaded. It fails to type-check or to load when a name is missing or has another type, and may otherwise misbehave. Regenerate the parsers and the shared library together.
- The generator imports only the exported names of the library that the parser uses. The library's other exports are not part of this contract unless Generated TypeScript API lists them.
Conformance
The repository tests every grammar in examples/ and the grammars of the engine
tests against the generated Go and TypeScript parsers, in both position units and
with strings and bytes as input, and require identical trees (as JSON) and errors.
TypeScript output is type-checked with tsc --strict and additional strict flags.