Chapter 7

Generated Parsers

sxir gen translates a grammar into the source of a standalone parser, in Go or in TypeScript. A generated parser does not load or compile a grammar at run time and does not depend on the SyntaxIR engine. This chapter defines the generator, the API of the generated Go and TypeScript code, the features each target supports and where its behavior differs from the engine. Where this chapter does not state a difference, a generated parser returns the same value, the same positions and the same errors as the engine for the same grammar, input, start rule and position unit.

Scope and Terminology

The engine is the runtime.Parser that parses with a compiled grammar. A generated parser is the source that compiler.GenerateGo, compiler.GenerateTypeScript (package syntaxir.dev/sxir/pkg/compiler) or sxir gen writes. The generation options select optional parts of the output. The generated package is the Go package named by -pkg; the generated module is the TypeScript file. The generated Go code uses the library package, either imported as syntaxir.dev/sxir/pkg/runtime/libgo or as a copy in internal/libgo. The imported package and the copy that -lib only writes are the whole library, the same for every grammar of one generator release. The copy that -lib embed writes next to a parser is narrowed to the features that the grammar and the generation options use: it leaves out the declarations and code of the library that the parser cannot run (none with the optimization used-features off). In both cases the generated code also uses the packages under syntaxir.dev/sxir/pkg/runtime, which the library depends on; the TypeScript module uses the TypeScript library, lib.ts, imported or as a copy that is narrowed in the same way. Every generated Go and TypeScript file starts with the line // Code generated by sxir <version>. DO NOT EDIT., where <version> names the generator: a release tag (v0.1.0) for a release, a pseudo-version (v0.0.0-<time>-<commit>) for a build from one commit, and (devel) for a generator built without a module version, with the module replaced, or from a modified tree. One generator writes the same stamp every time, so regeneration is byte-identical.

Contract

The Generator

sxir gen -g <grammar> [-lang go|ts] [-module <import path>] [-pkg <package>]
         [-lib embed|import|only] [-s <rule>] [-o <path>]
         [-types] [-document] [-stream] [-recognize] [-expectations] [-observability]
         [-source-info] [-default-value] [-nodoc] [-max-depth <n>]
         [-optimize <profile>] [-opt <name[=count],...>] [-no-opt <name,...>]
         [-print-before <pass,...>] [-print-after <pass,...>] [-print-changed] [-verify-each]
         [-stats] [-time-passes] [-opt-bisect-limit <n>]
Flag Go API: a field of compiler.GoConfig / compiler.TSConfig, or as stated Meaning
-lang compiler.GenerateGo with a GoConfig, or compiler.GenerateTypeScript with a TSConfig go (default) or ts
-g the grammar source argument The grammar source: .sxir or .json. A compiled grammar holds no grammar and is an error
-module Module Required for Go (with -lib import only when -pkg is omitted); the import path of the generated package; rejected with -lang ts
-pkg PackageName The package name; MUST be a valid Go identifier; default: the last element of the import path without a major version suffix (example.com/m/v2 gives m), and when that is not a valid identifier, generation fails and suggests -pkg; rejected with -lang ts and with -lib only
-lib Lib, a compiler.LibMode: LibEmbed or LibImport How the parser reaches the library. embed (default) writes a copy of the library for the parser: internal/libgo in Go, lib.ts in TypeScript. import imports it: syntaxir.dev/sxir/pkg/runtime/libgo in Go, the module that -lib-module names in TypeScript. A Go parser generated with import refers to the constant ProtocolN of the library protocol version N it was generated for, so it builds only against a library that implements that protocol. A Go copy refers in the same way to the constant ProtocolN of syntaxir.dev/sxir/pkg/runtime/value/impl, so it builds only against a core that serves the node representation of its protocol. only writes just a copy of the whole library (GenerateLibGo, GenerateTypeScriptLib), which parsers of any grammar can use, and takes no grammar and no other generation flag; in Go, compiler.GenerateLibGo and compiler.GenerateTypeScriptLib
-lib-module LibModule (compiler.TSConfig) TypeScript only, and required with -lib import: the module specifier that parser.ts and types.ts import the library from, such as ../shared/lib.ts. Rejected with -lang go and with any other -lib
-s the option compiler.WithStart The start rule, a rule marked @start. Default: the default start rule
-o For Go, the output directory (required), which the command creates; it writes parser.go, api.go, with -types types/types.go, and, with embed, the library package in internal/libgo, and removes generated files of an earlier generation that it did not write again (only writes and removes only files of internal/libgo, so it never removes parser.go, api.go or types/types.go). A file that lacks the // Code generated by sxir <version>. DO NOT EDIT. header is never replaced or removed: the command fails before it writes anything. For TypeScript, the output directory (required): it writes parser.ts, with -types types.ts, and, with embed or only, lib.ts
-types Types Generate typed values (see Typed values)
-document Document Generate the Document (createDocument in TypeScript)
-stream Stream Generate ParseStream (Go only; the start rule MUST contain a #stream repetition)
-recognize Recognize Generate recognition without a tree
-typed-only TypedOnly Go only; needs -types. Generate only the typed entry points (ParseValue, ParseRuleValue and their options); there is no Parse, ParseRule or Recognize. Rejected with -recognize, -document, -stream and -lang ts. A typed entry point that converts the Node of its rule keeps the unexported Node rule functions it runs
-expectations Expectations Let a parse record expectations: what a syntax error expects and the messages of @error. The parser then has the table of expectations and a parse option WithExpectations (Go) or expectations (TypeScript); a parse records only when it passes the option, and otherwise a syntax error has the farthest position, an empty Expected, no Messages and the message syntax error. Without the flag the parser has no table and no option. A Document fixes the choice when it is created. It applies to every entry point, including the typed, Document and stream forms
-observability Observability Generate tracing and profiling, specified in Observability
-source-info the option compiler.WithSourceInfo Make the parser record source information: the rule of each node and the names of captures. Without it, Node.Rule, Node.Field and Node.Fields of a parsed node are empty and the outputs omit them
-default-value the option compiler.WithDefaultValue Make the parser return default values, as Default Values describes. Cannot be combined with -types, because the types come from the dropped actions
-optimize Profile of the compiler.Optimize of the option compiler.WithOptimize The optimization profile: speed (default, also the empty name), memory or size. An unknown profile is an error that lists the valid names. Rejected with -lib only
-opt Values of compiler.Optimize (a map from name to value) Gives single optimizations a value, as name or name=count. Rejected with -lib only
-no-opt Off of compiler.Optimize Turns the named optimizations off. Rejected with -lib only
-print-after PrintAfter of the compiler.Debug of the option compiler.WithDebug Writes the intermediate representation to standard error after the named steps and passes. Rejected with -lib only
-print-before PrintBefore of compiler.Debug Writes the intermediate representation that the named passes receive to standard error before their runs. Rejected with -lib only
-print-changed PrintChanged of compiler.Debug Writes the intermediate representation to standard error after each pass that changed it. Rejected with -lib only
-verify-each VerifyEach of compiler.Debug Verifies the intermediate representation after every pass. Rejected with -lib only
-stats Stats of compiler.Debug Writes the statistics of the passes to standard error after the build. Rejected with -lib only
-time-passes TimePasses of compiler.Debug Writes the times of the passes to standard error after the build. Rejected with -lib only
-opt-bisect-limit OptBisectLimit of compiler.Debug (nil: no limit) Runs only the runs of optional passes numbered up to the count, skips the later ones and writes each decision to standard error; generates once. Rejected with -lib only
-nodoc OmitPackageDoc Omit the package comment (Go only; rejected with -lang ts)
-max-depth MaxDepth The default nested recursive rule-call limit; zero selects 100,000, a positive value overrides it, a negative value fails generation. TypeScript also requires a safe integer
  • The output is deterministic: generating twice yields identical bytes for the same grammar, options and version of the generator. Omitted, zero and an explicit 100,000 depth default generate identical source.
  • Go output is formatted as by gofmt. Generated Go with the default embed library depends only on the packages under syntaxir.dev/sxir/pkg/runtime (with value/impl) and the standard library, and generated TypeScript has no dependencies. Generated TypeScript uses only erasable syntax and needs ES2020; it is type-checked under strict settings.
  • Grammar errors are reported with positions before any file is written, in the same form as in other commands. An unknown start rule, an invalid package name, a compiled grammar (gen needs the grammar source), -stream without a #stream repetition in the start rule and a grammar that contains #missing are errors.
  • A Go package may contain only one generated parser, because every generated file defines the same names. api.go re-exports the names the users of the parser need, as aliases of the types of the runtime packages and the library and wrappers of the library's functions.
  • With -types the Go types of the typed values are declared in the package <module>/types (the file types/types.go), which the parser package imports; a module path is required. The types package imports only the library and the runtime packages, and declares a constant SXIRSchema_<digest> that the parser package refers to once (see below). The parser package refers to the types as types.Name, and the users of the parser import <module>/types for them. The grammar's types therefore share no namespace with parser.go and api.go: a grammar type may be named Node, Parser, Walk, ParseOption or like any other export of the library.

Optimization Settings

The optimization settings are a profile and the values of named optimizations. They belong to the compilation of a grammar, not to one generator: the settings are the same for every backend (the Go and TypeScript generators, the closure engine and the bytecode VMs), and each backend applies the optimizations that concern it. Today only the generators have optimizations to apply: the closure engine, the bytecode VMs and the reference backends accept the setting, check it and apply nothing.

The settings are not part of the grammar's meaning: an optimization changes the code a parser is generated as, and the memory and time it needs, but not what it returns. It is either on or off, or it has a count where 0 is off. Each optimization has a kind: pass (a transformation of the Core IR, the first intermediate representation built from the grammar), flow pass and exec pass (a transformation of Code IR at the Flow stage and at the Exec stage that is converted from it), code pass (a transformation of Code IR at both stages), pattern (a local rewrite that a code pass applies, each application on its own), lowering (an option of a step that lowers one representation toward the next), printer (a choice of what the Go and TypeScript printers emit) or runtime (a control of the runtime structures that the printers emit). sxir gen -h lists every optimization with its kind, its default and whether it takes a count.

  • No option changes what a parser returns. For every accepted combination of -optimize, -opt and -no-opt, the generated parser accepts and rejects the same inputs and returns the same trees, typed values, syntax errors and recovered errors, and the same trace and profile events, as one generated with the defaults. Options change only the generated source, its size and the time and memory of a parse. The debugging flags (-print-before, -print-after, -print-changed, -verify-each, -stats, -time-passes) write diagnostics to standard error, or fail on an internal error, and change neither the files nor the parser. -opt-bisect-limit changes the files: it skips optional passes and generates once, which can keep rules that are otherwise left out, and the parser behaves the same unless a skipped pass is wrong, which is what the flag finds.
  • Profiles. A profile is a fixed set of values and has no behavior of its own. speed is the default and sets every optimization to its default value. memory allocates typed values one by one instead of in slabs (fewer bytes and less live heap, more allocations). size produces smaller generated Go source and binaries for a little parse time.
  • Order. The settings are the profile's, then the values of -opt replace them, then -no-opt turns optimizations off. -no-opt therefore wins over -opt for the same name.
  • Values. -opt name is name=1. An optimization without a count takes 0 or 1; one with a count takes 0 or a positive number. A name that is not an optimization, a value that the optimization does not take, a count that is not a decimal number, an empty item and a name given twice in -opt are errors, and generation fails before it writes a file.
  • Backends and languages. An optimization that concerns only one backend or language, such as the slabs of typed Go values, has no effect on another, and naming it is not an error.
  • -print-after takes the steps elaborate (the Core IR of each rule table as built), plan (for Go, the entry points of the program, and the route of a call of each rule and how the entry points reach it), flowlower (each function as the Code IR listing at the Flow stage, as lowered from Core IR), flow-finish (the same listing after the Flow stage of a function ends), and execlower (each function as the Code IR listing at the Exec stage, as converted from the Flow stage), the name of any pass, or all, which selects every step and pass except plan. A pass that does not apply to the parser prints nothing. An unknown name is an error that lists the valid ones.
  • Debugging output. What the debugging flags write is not part of this contract: its form, the pass names and the counter names may change in any version. In Go, the fields of compiler.Debug map to the flags one to one, and Output, the writer that receives the output, is required when any field but VerifyEach is set; the command writes to standard error.

Generated Go API

func Parse(input string, opts ...ParseOption) (*Node, error)
func ParseRule(name, input string, opts ...ParseOption) (*Node, error)

type ParseOption func(*parseOptions)
func WithUnit(unit PositionUnit) ParseOption
func WithLimits(limits Limits) ParseOption
func WithContext(ctx context.Context) ParseOption       // unless generated with -no-limits

type PositionUnit int
const (
	CodePoints PositionUnit = iota // the default
	Bytes
)
  • Parse parses the whole input with the start rule. ParseRule does the same with the named start rule, a rule marked @start; another rule is the error rule <name> is not a start rule; parsing can begin only at a rule marked @start, an unknown name the error rule <name> is not defined, the name of a hidden rule the error that the rule is hidden, and an empty name selects the default start rule of the grammar. Both require the rule to consume the whole input. Generated parsers contain entry code only for start rules.
  • Node, SyntaxError and SyntaxErrors are aliases of the types of the packages syntaxir.dev/sxir/pkg/runtime/value and syntaxir.dev/sxir/pkg/runtime, defined in Nodes and Values and Parsing, and so are the position units, limits and traversal. With import and with embed they are the types that the engine uses, so results and errors of a generated parser and of the engine have the same Go types. Node JSON is identical to the engine's.
  • Options apply in order. A final Limits.Depth of Unlimited returns a configuration error before matching, actions or named-rule lookup, because generated code nests rule calls on the host stack; zero selects the generated default. A nil option panics when applied. A call with no options uses code points and the generated depth default.
  • A generated parser is safe for concurrent use: each call has its own state.
  • Errors in actions (such as division by zero) return plain errors as in the engine, and exceeding the nesting limit returns a *LimitError of kind LimitDepth. The depth limit counts nested entries of recursive rules (Limits); a larger limit does not guarantee stack capacity. The default error is nesting too deep: more than 100000 nested recursive rule calls.
  • Every generated Go parser exports WithLimits, Limits, LimitKind, LimitError, ErrLimit, LimitCompute, LimitInput, LimitDepth, LimitMemory and Unlimited. A parser is generated with limit controls unless it opts out (-no-limits, NoLimitControls), and one generated with them also exports WithContext. A parser generated with limit controls has the controlled form only, and a parse that sets no limit runs it with the default budget: a generated parser applies the default limits, Compute of max(2^20, 1024 * n) among them, unless it opts out. A parser generated without them (-no-limits) holds no code that spends budget or reads a control: one form of each function; with -document it is the form that serves a Document, without charges. Such a parser, given a context, WithComputeReport or a finite Compute or Memory limit, returns an error that the parser was generated without limit controls; no option is ignored. A nil context, Compute and Memory limits that are zero or Unlimited, any Input and any Depth set no control and are accepted.
  • WithContext and WithLimits apply the semantics of Cancellation and Compute Limits to each invocation, with counters that restart for every call. They apply to named rules, recognition, typed parsing and conversion. The last context wins and WithLimits merges field by field; a nil context means no cancellation; a zero limit selects the default and Unlimited disables it. An exhausted limit matches ErrLimit, and errors.As retrieves a *LimitError with Kind (LimitCompute, LimitInput or LimitMemory), Limit, Used and Requested, and for LimitCompute also Hotspot, which names the rule that the profile of the parse ranks first. A finite Memory bounds the storage of the parse as the memory limit defines; the typed values of -types count too. An input longer than Input fails with LimitInput before any matching, in every entry point, and a generated stream fails when the parser reaches a unit beyond it. An interrupted entry point returns its zero result, and recovery does not turn an interruption into a successful partial tree. Cancellation is cooperative.

Typed Values

With -types, the Go package also defines a Go type for each type of the grammar and the functions

func ParseValue(input string, opts ...ParseOption) (T, error) // T: the Go type of the start rule's type
func ParseRuleValue(name, input string, opts ...ParseOption) (any, error)

ParseValue returns the errors of Parse. The mapping of types is:

Grammar type Go type
int, string, bool The same
Struct type T *T with the same fields and an embedded Span, which is input.Span{Start, End int} of package runtime/input: the range [Start, End) of the input in the parse's position unit
node[T] *TNode[T'], with Span and Value of the Go type T' of T: *TNode[string] for a string node, whose Value is the text, and *TNode[any] for node[_]
Tuple (A, B, ...) TupleN[A', B', ...] with the fields E0, E1, ...; Unit for ()
error (from #recover) *Error, a type of the types package with Span and Message
Join U ~ A | B of node types: struct types, named node types, unnamed node types and error An interface U implemented by *A, *B and *Error, and by every *TNode[T] if a member is an unnamed node type. A value converts to the member its type names, or, for a node of an unnamed node type or of a named node type that is no member, to the unnamed node member of its kind (string, tuple or list) with that member's Go type; node[_] takes every other node as a *TNode[any]
Join of node types written in place (a field, a list element, a rule value) and declared by no alias: A | B (two or more distinct members, with or without error), or A | error (one member and error) An interface named after its members in the order of their first spelling in the file (declarations in source order, then the fields of a type), AOrB, or AOrError for one member and error, implemented like the interface of an alias; an unnamed node member is named after its value and Node, as StringNode for node[string], ItemListNode for node[[]Item] and Node for node[_]. The same set of members in any order is one interface, and a set that an alias declares is that alias. A type of the grammar with the generated name, or two different sets with the same name, is a generation error. A join without a struct or named node type member, a join with a member that is no node type (a list, a scalar), or one with two unnamed node members of one kind has no interface and is any; an alias gives a join of unnamed node types an interface
[]T A slice; an empty list is an empty slice and not nil
*T *int, *string or *bool for basic types; otherwise the Go type of T, nil when absent
A rule whose type is not declared The Go type of its value (an action's value maps by the rows above): *TNode[TupleN[E0, ..., En]] for a tuple of known elements, *TNode[[]E] for a list, *TNode[Unit] for the unit (); *TNode[any] when the elements are not known; a rule with a declared type, node[_] included, always has the Go type of that type
_, joins with basic members, with unnamed node members of one kind, or with members of different shapes, a value that closes a cycle of rules any, holding the converted value
Alias Alias ~ T A Go alias to the value type of T
  • TNode[T] is struct { Span; Value T }; TupleN is struct { E0 E0; ...; En En } and Unit is struct{}. A missing element of a tuple is nil. A list node's Value is a slice.
  • A grammar type keeps its name. The types package declares, besides the grammar's types, TNode, Error, SXIRHooks and, when used, Unit, TupleN and SXIRNodeTypes, and a constant SXIRSchema_<digest>; generation fails, naming the type, if a grammar type has one of these names. The digest covers the package's declarations, not its comments or formatting. The parser package refers to the constant once, so a parser.go and a types/types.go generated from different schemas do not compile together; the runtime relies on this when it reads a string node as its named node type. A grammar type named Unit is an error only when the generated code uses the empty tuple, and one named TupleN only when it uses a tuple of that arity. A struct field named like a field or method of the embedded Span (Start, End, Range) or like a method of the typed values (TName, ...) gets a trailing underscore in Go (Start_), so Start and End always read the span; the value still reads back under the grammar name. Every declared alias is exported.
  • After #recover, an error value in a list of a join type stays in the list as *Error; where a struct type is expected it becomes nil.
  • ParseRuleValue consumes the whole input, applies options once before it looks up the name, and returns the selected start rule's typed value as any. Names that ParseRule rejects, with the same errors, and invalid options return nil with an error. A parse failure returns the zero value of the selected type. ParseRule continues to return *Node.
  • A generator MAY build typed values directly instead of converting the tree that Parse returns when every value has a Go type of its own; the result is the same either way.

Typed TypeScript output (-types) exports an interface for every struct and alias and the functions parseValue and parseValueWithOptions, which parse the configured start rule and convert the node result. They return { value, error }: a successful nil result has a null value and a null error; a failure has a null value and the error; recovery keeps the converted value and SyntaxErrors. The mapping is:

Grammar type TypeScript value
Struct An interface with type (the grammar name), start, end and the fields; nullable where an absent or incompatible recovered value becomes nil
node[T] TNode<T'>, with start, end and value of the TypeScript type T' of T; for a node of one kind, the interface of that kind, which extends it with that kind as its type: StringNode for a string node, TupleNode<T'> for a tuple node and ListNode<T'> for a list node, also when the node is one of a named node type; TNode<unknown> for node[_]
Tuple (A, B, ...) readonly [A', B', ...]
[]T A readonly array, or null; null differs from an empty array
*T A nullable target
int number | bigint, with signed 64-bit arithmetic and safe-number canonicalization
string, bool string, boolean; an absent non-optional scalar field is "", false or 0
Declared alias The complete target type
Join of node types, declared by an alias or written in place The TypeScript types of the members, RecoveredError and null, which narrow by type; a value converts to its member as in Go
Join with a member that is no node type, _ TypedValue, following the conversion of any in Go

Names collide the same way as in Go, with trailing underscores. Typed parsing by rule name is not available in TypeScript.

Recognition

With -recognize, Go gains Recognize(input string, opts ...ParseOption) error and TypeScript gains recognize and recognizeWithOptions. They check that the whole input matches the start rule without building a tree and return the errors that Parse returns, including SyntaxErrors for recovered errors. Actions whose values are not read by predicates are skipped; actions that predicates read still run and can report runtime errors.

Incremental Documents

With -document, the Go package gains NewDocument(text string, opts ...ParseOption), which returns a runtime.Document (Parse, Edit(start, end, replacement), Text, Unit, Stats, CacheStats and ResetHistory); the package re-exports it as Document. With -types, NewDocument returns the package's own *Document, which implements runtime.Document and adds ParseValue. The library's own Document type is not part of the API.

  • Edit replaces a half-open range in the position unit chosen at construction. It rejects an invalid range, and a position inside a UTF-8 character in Bytes mode, without changing the text. Text() returns the exact current input.
  • Construction options become the defaults of later calls. A call MAY set the depth, the context and the compute limits; it MUST NOT change the unit.
  • ParseStats counts the rule evaluations (Evaluated) and reuses (Reused) of the last parse: the reused memo results (MemoHits) and the elements of repetitions that it resumed (ResumedElements). Parse reuses and resumes them by the rules of the engine's Reuse Model, so its Stats equal the engine's after the same grammar, text and edits. ParseValue keeps its own typed memo and resumes no repetitions, so the Stats after it can differ.
  • A parse returns what a parse of the whole current text returns. An interruption discards memo history so that a later call can retry.
  • A Document is for one goroutine. A returned tree can share nodes with later parses; Node.Clone makes a detached copy that preserves cycles and repeated references. With -types, ParseValue returns read-only values that share storage with the document and with earlier results. Caller mutation can therefore change what later results contain, as with Node values. CloneValue(value) returns an independent copy that shares nothing with the document or other results, and preserves aliases within the copy. Every slice in a returned value has capacity equal to its length, so append on a result cannot write into another result.
  • The returned SyntaxError values and their Expected and Messages slices are owned by the caller. Parsing, diagnostic copying and typed publication share one budget.
  • It supports a subset of the engine's Incremental Documents: there is no EditBatch, no Version, no snapshots and no change notifications.

With -document, the TypeScript parser.ts gains createDocument(input: string | Uint8Array, options?), which returns { document, error }, and a Document class with these methods: parse(options?), edit(start, end, replacement), text(), bytes(), stats() and, with -types, parseValue(options?).

  • createDocument copies the input and reads the options once. They become the defaults of every parse; document is null, with the error, when they are invalid or the input exceeds their limits.
  • edit follows the Go Edit and returns an Error or null. A string replacement is its UTF-8 encoding, with lone surrogates as U+FFFD; a Uint8Array replacement is taken as it is. text() returns the text as nodes hold it in the unit, and bytes() a copy of the exact input.
  • A parse returns what parse of the whole current text returns, and parseValue what parseValue returns. An invocation MUST NOT change the unit or the expectations. A parse or edit made while the Document parses, from a trace callback, returns an error.
  • A parse reuses the memo results of earlier parses that the edits since left valid, by the rules of the Go Document, but resumes no repetitions: it evaluates their elements again. stats() returns { evaluated, reused }, the rule evaluations and the reused results of the last parse, which equal the Go Stats after Parse of the same grammar, text and edits when that parse resumed no repetition (ResumedElements is zero). The Go ParseValue keeps its own typed memo, so the Go Stats after it can differ.
  • A returned tree can share nodes with the results of later parses, so mutating it can change what they contain. A reused result whose position an edit moved is returned in copies, which leave earlier trees unchanged.

Streaming (Go Only)

With -stream, the Go package gains ParseStream(r io.Reader, emit func(*Node) error, opts ...ParseOption) error, Iterate with StreamIterator and ErrStreamClosed, and the options WithDiagnosticCallback and WithDiagnosticLimit with TruncatedSyntaxErrors. They follow the contract of Streaming, with these differences: a nil emit parses the stream without delivering elements, and the diagnostic options are accepted only by the stream entry points (Parse, ParseRule, ParseValue, Recognize, NewDocument and Document methods reject them). Parse treats #stream as an ordinary repetition.

With -types as well, the package gains StreamValue, ParseStreamValue and IterateValue with StreamValueIterator. StreamValue is the Go type of one element of the #stream repetition, not of the start rule. Each element is converted on its own when it is committed, so the value stays valid after the callback returns. The conversion charges no compute units, as converting a result does not, and observes the context.

Generated TypeScript API

export function parse(input: string | Uint8Array, unit?: Unit): ParseResult;
export function parseRule(name: string, input: string | Uint8Array, unit?: Unit): ParseResult;
export function parseWithOptions(input: string | Uint8Array, options?: ParseOptions): ParseResult;
export function parseRuleWithOptions(name: string, input: string | Uint8Array, options?: ParseOptions): ParseResult;

export interface ParseOptions {
  unit?: Unit;
  signal?: CancellationSignal | null;      // unless generated with -no-limits
  computeReport?: ComputeReport | null;    // unless generated with -no-limits
  limits?: Limits;
}
export interface ComputeReport { units: number }  // set by the parse: see parsing.md#reporting-the-units-spent
export interface Limits {
  compute?: number;                       // unless generated with -no-limits
  memory?: number;                        // unless generated with -no-limits
  input?: number;                         // works with and without -no-limits
  depth?: number;
}
export const Unlimited: number;           // Number.MAX_SAFE_INTEGER
export interface ParseResult { node: Node | null; error: Error | null }
// With -limits:
export interface CancellationSignal { readonly aborted: boolean; readonly reason?: unknown }
// In both forms:
export type ParseInterruptionKind = "canceled" | "compute-limit" | "memory-limit" | "input-limit" | "depth-limit";
export class ParseInterruptionError extends Error {
  readonly kind: ParseInterruptionKind;
  readonly limit: number;
  readonly used: number;
  readonly requested: number;
  readonly hotspot: string; // "" unless a compute-limit with a profile
  readonly cause: unknown;
}
export type Unit = 0 | 1;
export const CodePoints: Unit; // the default
export const Bytes: Unit;
export function marshal(n: Node | null): string;
  • Public names. The public API of a generated TypeScript parser is what parser.ts exports, plus the names of types.ts with -types. From the library it re-exports Value, Unit, CodePoints, Bytes, NodeField, NodeJSON, Node, marshal, SyntaxError, SyntaxErrors, ParseResult, ParseOptions, Limits, ParseInterruptionKind, ParseInterruptionError, Unlimited, TraceKind, TraceEvent, TraceFunction, SkipReason, Profile, CallSite, SkipReasonCount, RuleProfile and ProfileSnapshot, and, unless generated with -no-limits, also CancellationSignal and ComputeReport. With -no-limits the exported ParseOptions and Limits are the forms without signal, computeReport and limits.compute. TraceKind is "enter", "exit" or "skipped". The other exports of lib.ts, such as ProfileData, are the contract between the library and the generated code. They are not public API and may change between versions.
  • parse does not throw for bad input: it returns { node, error } with a node and no error; no node and an error; or, after #recover, a node and SyntaxErrors. It throws only for defects that are not parse errors.
  • Node has type, rule, start, end, text, children, fields, field(name), isTerminal(), toString() and toJSON(), with the meaning given in Nodes and Values. A field value is a Node, a number, a bigint, a string, a boolean or null.
  • SyntaxError has pos, line, col, expected, messages and reason(), and its message is the text of the engine's error; SyntaxErrors has errors.
  • The SyntaxError values of SyntaxErrors, the errors that #recover recovered from, carry only the header line in stack; the error of a parse that failed keeps the stack of the point where it was made.
  • marshal(node) returns exactly the bytes that encoding/json writes for the engine's node. JSON.stringify yields the same keys in the same order, except that integers beyond the safe range are rounded and, in Bytes mode, an invalid input byte is written as the surrogate that represents it.
  • marshal and toString write trees of any depth. JSON.stringify recurses and can exhaust the stack for deep trees.
  • limits.depth MUST be a positive safe integer or zero; Unlimited, negative values, fractions, NaN, infinity and unsafe integers return a configuration error in the result before matching, because rule calls nest on the JavaScript stack. Omitted or zero selects the generated default. A parse that nests more recursive rule calls returns node as null and error as a ParseInterruptionError whose kind is "depth-limit", with limit and used the depth limit and requested 1, in both forms.
  • A parser generated with -no-limits emits no code that spends budget and sets no control, so a parse charges nothing, and its ParseOptions has no signal or computeReport and its Limits has no compute or memory; it does not export CancellationSignal or ComputeReport. A call that passes a signal, a compute report or a finite nonzero compute or memory limit anyway returns an error in the result, before matching, that says the parser was generated without limit controls; no option is ignored. An absent or null signal or compute report, zero or Unlimited compute and memory limits, and the input and depth limits set no control and are accepted. A parser generated without -no-limits, the default, has the controlled form only, and a call that sets no limit runs it with the default budget, compute of max(2^20, 1024 * n) among its defaults. The rest of this list describes the controlled form.
  • signal is a structural cancellation signal: any object with aborted and an optional reason, such as an AbortSignal. limits.compute, limits.memory and limits.input MUST be finite nonnegative safe integers; zero selects the default and Unlimited (Number.MAX_SAFE_INTEGER) disables the limit; other values return a configuration error in the result. An unknown option or limit key, such as limits.time, is a configuration error too, in both forms. A finite limits.input bounds the input in the position unit and is checked before any matching, in both forms; a longer input returns a ParseInterruptionError whose kind is "input-limit", with limit the input limit, used zero and requested the length of the input. The units of the limits are those of Cancellation and Compute Limits.
  • A finite limits.memory bounds the storage the parse takes into use, in bytes, as the memory limit defines it, with the polls, order of checks and accounting given there. The count is an estimate from a model of the sizes of the runtime's objects in a 64-bit JavaScript engine: the nodes, child lists and fields the parse builds, memo entries, slots and hash indexes, first-call sets, the decoded input and its line table, the records of left recursion and of scan and skip sites, variable environments and frames, syntax errors, the text copied out of the input and the strings that actions concatenate, and the slots of the working stacks at their largest length at a poll. Not counted are the caller's input, the values of the typed conversion that parseValue hands to the caller, and what the JavaScript engine adds beyond the model. A count above the limit returns a ParseInterruptionError whose kind is "memory-limit", message memory-limit exceeded: limit N, used M, with limit the memory limit, used the count and requested zero.
  • An interrupted parse returns node as null and error as a ParseInterruptionError whose kind is "canceled", "compute-limit", "memory-limit" or "input-limit". For a limit, limit, used and requested give the accounting, with the meaning that a LimitError reports, and for a "compute-limit" hotspot is the Hotspot of that error, the empty string without a profile; for cancellation, cause is the reason of the signal. Recovery does not turn an interruption into a partial result, and the parser keeps no reference to the signal after the call returns. Typed conversion (parseValue) and recognition report interruptions the same way.
  • A synchronous parse does not yield to the event loop: a signal that changes only through timers or other tasks of the same thread is not observed during the call. A signal that is already aborted is observed at the first poll, and the compute limits do not depend on the event loop.
  • With -recognize, recognize and recognizeWithOptions return null if the input matches and an Error otherwise. With -observability, the batch entry points accept the tracing and profiling options of Observability.
  • A generated TypeScript module exports the same names for every grammar, so two parsers are imported under different namespaces. A parser keeps no state between calls.

Input and Positions in TypeScript

Positions count Unicode code points (the default) or UTF-8 bytes (Bytes), as in the engine, and not UTF-16 code units. The parser decodes the input first.

  • A string is parsed as its UTF-8 encoding would be. A lone surrogate becomes U+FFFD.
  • A Uint8Array is parsed as UTF-8 bytes; an invalid sequence decodes to U+FFFD, one per byte, as in Go. In Bytes mode, positions count the original bytes.
  • In Bytes mode, text keeps the bytes: each invalid byte b is represented by the lone surrogate U+DC00+b in node text and in text(...). marshal writes such a byte as U+FFFD.
  • String operations in actions follow Go: + concatenates, comparisons order by code point, and len counts code points or UTF-8 bytes. Integers are signed 64-bit and wrap on overflow; division truncates toward zero and % takes the sign of the dividend. An integer is a number while it is a safe integer and a bigint beyond.

TypeScript Errors and the Stack

Error error
The input does not match SyntaxError
#recover skipped input SyntaxErrors, and node is set
A runtime error in an action Error with the message of the engine
More nested recursive rule calls than the limit (default 100,000) Error: nesting too deep: more than 100000 nested recursive rule calls
The JavaScript stack is exhausted before the limit Error: nesting too deep: the JavaScript stack overflowed at N nested recursive rule calls

The host stack of a JavaScript runtime is far smaller than the stack of a Go program, so a TypeScript parser can reject nesting that the engine accepts.

Validation and Errors

Condition Result
Invalid package name, unknown start rule, grammar errors Error before output
#missing in the grammar Generation fails
-stream with -lang ts Generation fails
-g, -s, -pkg or a generation option with -lib only The command fails
-lib embed without -module; -lib import with neither -module nor -pkg The command fails
An existing parser.go, api.go, types/types.go or library file in -o that sxir did not generate The command fails before writing
WithTrace, WithProfile, WithTraceFilter or WithTraceMaxDepth passed to a parser generated without -observability The call returns an error before parsing
-pkg or -nodoc with -lang ts Generation fails
-stream without a #stream repetition in the start rule Generation fails
Negative -max-depth Generation fails

Compatibility and Support

Feature Generated Go Generated TypeScript
Rule expressions, captures, atomic and discarded expressions Supported Supported
Operators Rules, left recursion (direct and indirect) Supported Supported
Predicates, variables, lookahead, cut Supported Supported
Actions, types, built-in functions Supported Supported
@error, #recover, SyntaxErrors Supported Supported
#missing Not supported Not supported
Position units WithUnit unit
Another start rule ParseRule parseRule
Recognition Recognize with -recognize recognize with -recognize
Typed values ParseValue, ParseRuleValue with -types parseValue, parseValueWithOptions with -types
Streaming ParseStream, Iterate with -stream None; #stream is an ordinary repetition
Incremental documents A subset with -document The same subset, createDocument with -document
Cancellation and compute limits WithContext and a finite Compute unless generated with -no-limits; WithLimits signal and limits.compute unless generated with -no-limits; limits.input; ParseInterruptionError
Nested recursive rule-call limit Limits.Depth, -max-depth limits.depth, -max-depth; the host stack also bounds nesting
Choice of backend, saved grammar, Parser.Grammar() Not applicable Not applicable
Tracing and profiling With -observability With -observability (batch parsers)

A generated parser carries the library of the generator that produced it (embed) or imports it (import), and does not follow later changes of the grammar or of the generator.

Shared TypeScript Library

With -lib import, parser.ts and types.ts import the names they use from the module that -lib-module names. The generator does not check that module. The contract is:

  • The module MUST be the lib.ts that the same version of sxir writes (-lib embed, or -lib only to refresh a copy). The parsers generated by one version need that version's library: the names, types and behavior that a parser imports are not stable between versions.
  • One lib.ts MAY serve any number of parsers, which are generated by that version and share its classes. A Node made by one of them is an instance of the same class as in the others.
  • A generated parser carries no library version and the library has none: a library of another version is not detected when the parser is generated or loaded. It fails to type-check or to load when a name is missing or has another type, and may otherwise misbehave. Regenerate the parsers and the shared library together.
  • The generator imports only the exported names of the library that the parser uses. The library's other exports are not part of this contract unless Generated TypeScript API lists them.

Conformance

The repository tests every grammar in examples/ and the grammars of the engine tests against the generated Go and TypeScript parsers, in both position units and with strings and bytes as input, and require identical trees (as JSON) and errors. TypeScript output is type-checked with tsc --strict and additional strict flags.