136. Measured Effect of the Generation Flags
- Status: Implemented
- Date: 2026-11-08
Problem and Mechanism
The optimizations of the generator are flags (design 063), and a preset (speed, memory, size) fixes their values.
A flag or a preset belongs in a profile only if a measurement shows what it does. This record is that measurement for
every flag of the registry, and the rule that turned it into the profiles.
The rule. Each flag was turned off (or set to other values), one at a time, against the speed profile, and the
generated Go parsers of the ten language grammars were measured beside an identical copy of speed (the control), whose
ratio is the noise of the machine.
- A flag is effective if it improves the parse time, or the metric it exists for, by at least 1% in the geometric mean beyond the control.
- An effective flag is on in
speedif it improves at least six of the ten parsers, costs at most 3% of the geometric mean parse time, and slows no parser by more than 6%. A flag with no measured cost in time, allocations, memory or source size is on inspeedas well. - Otherwise it is an opt-in. A flag that changes the output is an opt-in in any case and is in no profile.
- A profile is kept if it improves the metric it is named for by that rule against
speed.
Reproduction
The measurements were taken on main at f3df70413, before later changes to main, with the profile registry added to it. Go 1.27.1, Apple M3 Max, under the benchmark lock. go run ./bench/profilematrix
(design: docs/performance.md) generated each grammar once per variant with the options of
testdata/grammars/generate.txt, built a program for size and one for measuring, and ran the measuring programs in fresh
processes in rotated order for 10 rounds with -benchtime 300ms; time is the minimum over the rounds. The rows are the
ten grammars of testdata/grammars (json, csv, xml, yaml, cel, cue, golang, python, typescript,
duckdb) and two use cases of the CSV grammar, csv-incremental (an edit in the middle of 50000 records followed by a
reparse of a Document) and csv-stream (the file as a stream of records). Variants were run in four chunks of at most
nine programs each, every chunk with its own baseline and control. A variant whose generated Go parser is byte-identical to
the baseline's for a grammar was not measured for it: a flag counts only on the grammars where it changes the parser.
The control's time ratio was between 0.993 and 1.011 for the ten grammars in every chunk. The csv-incremental row is
noisier (the control reached 1.057 in one chunk), so no decision rests on it.
Results
Turning the flag off, or setting it to another value, as the ratio to speed (above 1 is larger or slower). Geometric
mean over the grammars where the parser changes (n of 10); the control in parentheses is the control's geometric mean over
the same grammars.
| Flag | n | Parse time | parser.go |
Other | Parsers faster with the flag |
|---|---|---|---|---|---|
guards |
10 | 1.091 (1.002) | 0.917 | parser.ts 0.902 |
10 |
scans |
10 | 1.029 (1.002) | 1.016 | parser.ts 1.017 |
6 |
inline-literals 0 |
10 | 1.016 (1.001) | 0.930 | binary 0.990 | 8 |
inline-literals 2 |
9 | 1.005 (1.001) | 0.988 | 3 | |
inline-literals 8 |
7 | 1.003 (1.002) | 1.018 | 1 | |
terminals |
5 | 1.007 (1.003) | 1.036 | 2 | |
guarded-classes |
6 | 1.005 (1.003) | 1.004 | 1 | |
class-runs |
5 | 1.005 (1.004) | 1.024 | 1 | |
class-tables 0 |
6 | 0.988 (0.999) | 1.034 | 0 | |
class-tables 3 |
6 | 1.012 (1.003) | 0.997 | 2 | |
class-tables 8 |
6 | 0.992 (0.999) | 1.001 | 0 | |
literal-sharing |
10 | 1.001 (1.001) | 1.024 | 0 | |
method-sharing |
7 | 0.996 (1.000) | 1.006 | parser.ts 1.406, gzip 1.216 |
0 |
typed-object-slabs |
10 | 0.999 (1.000) | 1.001 | allocations 3.31, bytes 0.946, live heap 0.946 | 3 |
typed-list-slabs |
10 | 0.997 (1.000) | 1.001 | allocations 2.81, bytes 0.913, live heap 0.913 | 2 |
typed-output-bootstrap |
10 | 1.005 (1.000) | 1.001 | allocations 0.801, bytes 1.129, live heap 1.229 (1.005) | 4 |
ts-literal-sharing on |
parser.ts 1.009, gzip 1.000 |
text-dispatch, field-compare and sparse-memo changed no generated parser of the ten grammars, in Go or TypeScript. The
class-tables rows were measured again in a second run of the six typed grammars (10 rounds): off 0.988 against a control of
0.999, with typescript at 0.963, golang at 0.982 and the others within 0.3%.
The incremental and stream rows of the CSV grammar: guards off 0.953 and 1.014, scans off 1.106 and 1.010,
inline-literals off 0.910 and 0.995 (time of one edit and reparse, then of the whole stream). The controls of the
chunks were 1.020 and 1.005 (first) and 1.057 and 1.004 (second). The incremental row moved by 5% to 11% in both directions,
by more than its control, with no pattern the other rows share, and the stream row stayed within 1.4%; no decision uses
these two rows.
Decisions
guardsandscansare effective by time (9.1% and 2.9%) and stay on.guardscosts 9% of source size, which is not a time cost.inline-literalsat 4 is effective (1.6%, eight parsers); the values 2 and 8 were not better than 4 by a margin beyond the control, so 4 stays and 6 and 12 were not tried. It costs 7.5% of source size, andyamlandgolangare 1.7% slower with it.terminals,guarded-classesandclass-runschange time by 0.4% or less beyond the control, shrink the source and cost nothing measured, so they stay on as zero-cost.class-tablescosts 1.1% of the parse time beyond the control (3.7% ontypescript), shrinks the source by 3.4% and does so for five of the six typed grammars, so it fails the six-of-ten rule and is an opt-in (-opt class-tables=5), a change from design 105. Its other values were not better.method-sharingandliteral-sharingcost nothing in time and make Go source smaller;method-sharingmakes the TypeScript source 29% smaller. They stay on.typed-object-slabsandtyped-list-slabschange no parse time. The flags cut the number of allocations to a third, and their price is 6% and 10% more bytes and live heap. They stay on, because the rule counts allocations as an effect. Thememoryprofile turns them off.typed-output-bootstrapcuts the live heap by 19% and the bytes allocated by 11%, and costs 25% more allocations.ts-literal-sharingandsparse-memoact on TypeScript parse time and memory, which the harness does not measure; the size result forts-literal-sharingis flat, and design 090 measured its time. They are unchanged.- No flag is removed.
text-dispatchandfield-compareshowed no effect because none of the ten grammars has the form they act on, so this measurement cannot show that they are effective.
Profiles
Variants of the profiles were measured the same way, in one chunk (control 1.000). The ratios are to speed, the
geometric mean over the ten language grammars; the worst parser is the one with the largest parse-time ratio.
| Candidate | Sets | Parse time | Bytes allocated | Live heap | parser.go |
Binary | Worst parser |
|---|---|---|---|---|---|---|---|
memory |
slabs off | 0.996 | 0.886 | 0.848 | 1.000 | 0.984 | +3.8% (xml) |
size |
inline-literals 0, class-tables 5 |
1.022 | 1.001 | 0.997 | 0.912 | 0.986 | +5.5% (json) |
| not registered | size and guards, scans, guarded-classes off |
1.137 | 0.997 | 1.010 | 0.846 | 0.971 | +29% (duckdb) |
memory and size meet the rule (memory cuts the bytes allocated by more than 1% on nine of ten grammars, size
shrinks parser.go on all ten; parse time stays within 3% as a geometric mean; no grammar is more than 6% slower). The
third candidate costs 14% of parse time for about 6 points more source reduction than size and is dropped. The
memory profile raises the allocation count 1.03 to 1.39 times on the typed grammars and 52 to 97 times on the JSON, CSV
and XML parses (the number of allocations per parse of those grammars is small); the geometric mean over the ten is 4.06.
The two use rows of the CSV grammar are not part of these means and no decision uses them. Time of one edit and reparse
of a Document (csv-incremental, control 0.968): memory 0.980, size 0.987. Time of the whole stream (csv-stream,
control 1.001): memory 0.996, size 0.995. The incremental row is the noisy one (see Reproduction).
Runtime Parameters
The chunk sizes of the typed slabs (objects 8 to 256, lists 64 to 1,024, lists of more than 256 on their own) were tried at
64 and 256 and at 1,024 and 4,096 in a copy of the runtime, one run each over the ten grammars. Bytes allocated and live heap
did not change by 1.5% in the geometric mean (0.987 to 1.015); the allocation count moved only for json, csv and xml
(2.1, 1.7 and 1.7 times with the smaller limits, 0.77, 0.83 and 0.85 with the larger); parse time differed by -0.7% and +1.0%
between runs that were not interleaved, within what the control allows. No gain beyond the control appeared, so no runtime
parameter is exposed as an optimization.
Limits
- The harness measures Go parsers. TypeScript was measured by generated size only.
- Time is the minimum of 10 rounds on one machine; ratios within 1% of the control are not distinguished from it.
- The live heap of a single run varies by 5% to 10% between processes on a few grammars (the control showed it); only differences above that are read.
- The numbers are for the ten language grammars and the inputs of
testdata/grammars; a grammar with other shapes can differ, which is why every flag stays selectable.