Incremental Documents in Generated Parsers

Generate with sxir gen -g grammar.sxir -module example.com/parser -document -o parser, or pass Document: true to compiler.GoConfig. This adds a mutable Node Document alongside the ordinary batch Parse and ParseRule functions. NewDocument returns a runtime.Document, the interface of the engine's Documents (Text, Unit, Edit, Parse, Stats, CacheStats, ResetHistory); with -types it returns the package's own *Document, which implements it and adds ParseValue. The engine's version of the same feature is described in Incremental Parsing.

Editing and Reparsing

doc, err := parser.NewDocument("first draft", parser.WithUnit(parser.CodePoints))
if err != nil {
	return err
}
tree, err := doc.Parse()
if err := doc.Edit(6, 11, "revision"); err != nil {
	return err
}
tree, err = doc.Parse()
stats := doc.Stats() // rule evaluations and reused memo results from this Parse
_ = tree
_ = stats

Edit(start, end, replacement) replaces the half-open range in the unit selected at construction. It rejects invalid ranges and positions inside a UTF-8 character in Bytes mode without changing the text. Text() returns the current text: the exact bytes in Bytes, and in CodePoints the text with each invalid UTF-8 byte of the initial text and of every replacement replaced by U+FFFD, as in the engine's Document. Construction options become defaults for later Parse calls; per-call options can change depth, context and compute limits, but cannot change the unit. An interruption during execution discards memo history so a later call can retry. Document is single-goroutine, and results can share nodes with later parses. Call tree.Clone() if an earlier tree must remain detached from the Document. It copies Node graphs, including Node values in []any and []*Node fields, while preserving cycles and repeated references to the same slice view. Overlapping slice views with different lengths or capacities are copied independently; arbitrary mutable action values such as maps remain shared. ParseStats counts the rule evaluations (Evaluated) and reuses (Reused) of the last parse; a generated Document reuses only memo results, so MemoHits equals Reused and ResumedElements is zero.

What Is Kept Between Parses

Both kinds of Document keep the memo results that examined only text before an edit where they are, and move those that examined only text after it by the edit's change in length; the others are evaluated again. A result of a rule whose actions or predicates read positions (.Start, .End), or that recovered errors, is not moved. The Node Document moves the nodes of a moved result when a parse reuses it, in copies, so a tree returned earlier keeps its positions; a result that holds an action value other than nodes and basic values is evaluated again instead. Edit applies itself at once only to the results at the text it replaced; a later parse moves any other result when it looks the result up, so an edit costs time in proportion to those results and to its distance from the previous edit, not to the size of the memo. A typed Document keeps its results the same way and reuses a moved result only when its value is a basic value, a boolean, a number or a string; every other value, such as a struct or a list, is evaluated again. The generated batch functions remain available. -document can be combined with -types: eligible grammars use direct typed matching with a private memo and fresh invocation arenas. A typed result is read-only and shared, as Node values are: later parses reuse the values of earlier results, so a result can share values with the Document and with other results. Modifying a result can change what later parses return, including through a node[_] field. Call CloneValue(value) to get a copy that shares nothing with the Document or with other results; the copy can be modified freely, and aliases within it are preserved. Every returned slice has capacity equal to its length, so appending to one result cannot write into another. Other grammars use the existing Node Document and convert its final result on each call; generic Node members may share the mutable Document graph, so call Node.Clone when a detached tree is needed. The conversion fallback applies to any and node results, struct fields requiring SyntaxIR nodes, or actions and predicates that read struct fields.

Diagnostics, Limits and Combinations

Both Document.Parse and Document.ParseValue return owned syntax diagnostics: recovered SyntaxError values and their Expected and Messages slices are copied; ordinary failures are freshly constructed. Mutating a returned error cannot affect memoized errors or a later parse. Parsing, diagnostic copying and typed publication share one budget; interruption returns the result type's zero value and clears reusable state as needed for a retry. Engine-only Document controls such as EditBatch are not generated. See the direct typed measurements and the conversion fallback measurements.

TypeScript Documents

With -lang ts, -document (or Document: true in compiler.TSConfig) adds createDocument to parser.ts:

const { document: doc, error } = createDocument("first draft", { unit: CodePoints });
if (error !== null) throw error;
let result = doc!.parse();
const editError = doc!.edit(6, 11, "revision");
result = doc!.parse();

createDocument(input, options) copies the input and reads the options once; they become the defaults of every parse, and document is null when they are invalid. edit(start, end, replacement) follows the Go contract and returns an Error or null; a string replacement is encoded as UTF-8, with lone surrogates as U+FFFD. In Bytes a Uint8Array replacement is taken as it is, so edits can join invalid bytes into characters; in CodePoints each invalid UTF-8 byte of the input and of every replacement is replaced by U+FFFD, as in the Go Document. text() returns the text as the nodes hold it, and bytes() returns a copy of the current text as UTF-8: the exact bytes in Bytes, invalid bytes included, and the normalized text in CodePoints. parse(options?) returns the ParseResult of parse, and with -types the Document also has parseValue(options?). An invocation cannot change the unit or the expectations, and a parse or edit made from a trace callback of a parse returns an error. Like the Go Document, it keeps the memoized results that the edits leave valid and reuses them in the next parse; stats() returns { evaluated, reused } for the last parse, the same counts as Go's Stats after Parse; Go's typed ParseValue keeps its own memo and can count differently. With -observability, parse takes the trace and profile options of parse: a reused result is reported as a memo hit, so a profile's evaluations and memo hits are the evaluated and reused of stats(). A returned tree can share nodes with later results, so do not mutate a tree while the Document is in use.