Chapter 10
Fixture Suites and Edit Sequences
Two JSON files carry test data between a user and the tools: the fixture suite that sxir test runs and the saved edit
sequence that sxir edit-fuzz writes and replays. This chapter defines both formats. Each is versioned; a reader
rejects a version it does not know.
Scope and Terminology
A fixture suite is a JSON object that names a grammar and lists cases: an input and the expected outcome of
parsing it. An outcome is accepted (the whole input matched without recovery), recovered (the whole input
matched after #recover) or rejected (the parse failed). A golden is an expected value recorded in the suite
(a node, a span or the locations of syntax errors). An edit sequence is the log of one run of the edit fuzzer: the
initial text, then edit, undo, redo and checkpoint operations.
Contract
Fixture Suites
A suite is a JSON object with these members. Unknown members are errors.
| Member | Meaning |
|---|---|
version |
The format version, the integer 1. Any other value is an error |
grammar |
Required. The grammar file: read as grammar JSON if its name ends in .json, and as SyntaxIR source otherwise. A relative path is relative to the suite file |
start |
The start rule. Default: the grammar's default start rule |
unit |
codepoints or bytes. Default: codepoints |
backends |
A list of default, closure, bytecode and bytecode-iterative. Default: ["default"], which is the closure backend. Two names for the same backend, such as default and closure, are an error; repeated names are ignored |
equivalence |
A boolean. When true, the backends must agree on every case (see below) |
cases |
Required, nonempty. The cases |
A case is an object with name (required, unique in the suite), input (required, a string, possibly empty),
expect (required, an expectation object) and byBackend (optional, a map from a backend name to an expectation
override).
An expectation has the members outcome, node, start, end, errors and clear. Unknown members and null
values are errors, except that "node": null is a valid golden for a parse that has no node.
| Member | Meaning |
|---|---|
outcome |
accepted, recovered or rejected. Required in expect for every backend after overrides are applied |
node |
The expected node as the JSON form of Nodes and Values. Compared as canonical JSON |
start, end |
The expected span of the root node, in the suite's unit; nonnegative integers |
errors |
The expected syntax errors in order, each {"pos": n, "line": n, "column": n} with all three members, in the suite's unit. [] asserts that there are none |
clear |
Only in a byBackend expectation: a list of node, start, end and errors whose values are not inherited from expect. A field cannot be both cleared and given |
A member that an expectation omits is not checked. The expectation for a backend is expect with the members of its
byBackend entry laid over it: a given member replaces the inherited one, and a name in clear removes it.
Each case runs on each selected backend and produces one result per pair. A runtime error is reported for that pair
and later cases continue. With equivalence, or with sxir test -equivalence, which overrides the member in either
direction, the backends must also agree for each case on the outcome, whether a runtime error occurred, the node,
the root span and the syntax-error locations; the report lists the fields that differ. The expectations and the
equivalence are judged independently.
sxir test -update writes the observed node, span and syntax-error locations of one explicitly selected backend as
goldens, in the expect of each case or, when the case has byBackend entries, in the entry of the selected backend.
It changes no input or outcome, and it refuses to write when a selected case has a runtime error or an outcome that
differs from its expectation. It cannot be combined with -equivalence.
Edit Sequences
A saved sequence is a JSON object with exactly these members.
| Member | Meaning |
|---|---|
version |
The format version, the integer 1 |
grammar_sha256 |
The SHA-256 of the exact bytes of the grammar source, as 64 lowercase hexadecimal digits |
grammar_normalization |
The string none: the hash is of the bytes as they are, so a change of whitespace changes it |
start |
The start rule; required and nonempty |
backend |
closure, bytecode or bytecode-iterative |
unit |
codepoints or bytes |
seed |
The seed of the run that made the sequence, an unsigned integer |
text_encoding |
The string base64 |
initial_base64 |
The initial text as base64 bytes, so invalid UTF-8 survives |
operations |
The list of operations |
The operations and the checks that a replay makes are defined in Edit Fuzzing. An operation is an object with kind. An edit has start and end, the range [start, end) of the text just
before the operation in the sequence's unit, and text_base64, the replacement as base64 bytes. undo restores the
text before the latest edit that is not undone. redo reapplies the latest undone edit; a new edit discards the edits
that could be redone. checkpoint parses the Document and compares the result with a parse of its current text from
scratch: values with their spans, and errors with their positions, expectations and messages. The sequence is also
compared at its start.
A replay is refused when the grammar's hash differs, and the file is rejected when the start rule, backend or unit is unknown or when the object has a missing, unknown or mistyped member. The tool never replaces a file that it saved.
Validation and Errors
A suite that has a wrong version, a missing or empty required member, an unknown member, a duplicate case name, an unknown backend or outcome or a malformed expectation is rejected before any case runs, with a message that names the file and, for a case, its name. A sequence that is not valid is rejected before any operation runs.
Compatibility and Support
Both formats are at version 1. A reader supports only the versions it names. A compiled grammar is not a
supported grammar of a suite.
Conformance
The tests of the fixture format and of the edit fuzzer cover the validation rules, the override and clear semantics, the
equivalence report, -update and the round trip of sequences.