Lesson 7 of 10
Testing and Linting a Grammar
You have been checking grammars by running them on a few inputs. This lesson turns those checks into a suite that you can rerun, and adds two tools that find mistakes without any input: the linter and the formatter.
What You Will Build
A fixture suite for the configuration grammar, a lint run that finds a real mistake, and a formatted grammar file.
Before You Start
Complete Building Typed Trees and Reporting Errors: you need config.sxir and robust.sxir.
A Fixture Suite
A suite is a JSON file next to the grammar. Each case has an input and the outcome you expect: accepted for a clean
match, rejected for a syntax error, and recovered for a match that needed #recover. Save as suite.json:
{
"version": 1,
"grammar": "config.sxir",
"backends": ["closure", "bytecode"],
"equivalence": true,
"cases": [
{"name": "empty file", "input": "", "expect": {"outcome": "accepted"}},
{"name": "typical file", "input": "host = \"localhost\"\nport = 8080\n", "expect": {"outcome": "accepted"}},
{"name": "comment only", "input": "# nothing here\n", "expect": {"outcome": "accepted"}},
{"name": "double equals", "input": "port == 8080\n", "expect": {"outcome": "rejected"}},
{"name": "unterminated string", "input": "name = \"oops", "expect": {"outcome": "rejected"}}
]
}
sxir test -f suite.json
PASS empty file [closure]
PASS empty file [bytecode]
PASS typical file [closure]
PASS typical file [bytecode]
PASS comment only [closure]
PASS comment only [bytecode]
PASS double equals [closure]
PASS double equals [bytecode]
PASS unterminated string [closure]
PASS unterminated string [bytecode]
PASS empty file [backend equivalence: closure, bytecode]
PASS typical file [backend equivalence: closure, bytecode]
PASS comment only [backend equivalence: closure, bytecode]
PASS double equals [backend equivalence: closure, bytecode]
PASS unterminated string [backend equivalence: closure, bytecode]
all fixtures passed
Each case runs on both backends, and "equivalence": true also requires that the two backends produce the same tree and
errors. A failing case prints FAIL with what differed and the command exits with status 1, so the suite works in a
continuous integration job as it is. Cases can also pin the tree, the root range and the positions of errors; sxir test -update refreshes those expectations from the observed results.
Linting
sxir lint examines a grammar without running it. Save a grammar with two mistakes as buggy.sxir:
// buggy.sxir
@start
rule main = "a" ws op ws "b" $$
rule op = "<" / "<="
rule ws = " "*
rule unused = "x"
sxir lint -g buggy.sxir
buggy.sxir:4:17: error: alternative 2 (`"<="`) can never match: alternative 1 (`"<"` at 4:11) matches first wherever it could [shadowed-alternative]
fix: move it before alternative 1
buggy.sxir:6:1: warning: rule unused is never used: main does not call it, directly or indirectly [unreachable-rule]
fix: remove the rule, or call it from a rule that main reaches
sxir: lint: 1 error, 1 warning, 0 hints
The first finding is the mistake of the previous lessons in disguise: "<" matches the start of "<=", so the
second alternative is unreachable. Each finding has a position, a severity, the name of the check in brackets and a
suggested fix. The command exits with status 1 if there is an error, and with -strict also for a warning. The grammars
of this tutorial are clean:
sxir lint -g config.sxir
sxir lint -g robust.sxir
A run with no findings prints nothing and exits with status 0.
The flag -classify also reports how the cost of a grammar grows with the input; for a super-linear verdict, Rewrite a
Grammar to Linear Cost shows the rewrite for each cause.
Formatting
sxir fmt rewrites a grammar into the canonical layout and keeps comments. -l lists the files whose formatting
differs, -w rewrites them in place and -check lists them and exits with status 1, which suits CI:
sxir fmt -l config.sxir robust.sxir
sxir fmt -w config.sxir
Next Steps
Shipping a Parser turns the grammar into code that other programs can use. The guides Testing Grammars and Linting Grammars cover properties, golden files, fuzzing and every check, and the cookbook has recipes for testing hand-written code against a grammar and linting in CI.