074. One Pass Pipeline per Compilation
- Status: Accepted
- Author: @ornew
- Date: 2026-11-20
Note (2026-11): The migration order is amended; the end state is unchanged. The VM switch to the Flow route, which deletes the Core route, comes before step 12. Until step 12,
FlowTableis the VM's Flow route in production; step 12 assembles the VM pipeline inCompileand deletes it.
Summary
A compilation is one pipeline of passes, and a pass is one kind of value whatever it does: a transform, a lowering, the
planning of a program and the printing of source are all a sealed Pass[I, O] of internal/pass, which takes a unit
of one kind and returns a unit of the same or another kind. The driver is the only package that assembles and runs
pipelines. It builds the pipeline of each backend from typed combinators that the Go compiler checks, down to the stage
of a Code IR module, and one manager runs it: every step, required or optional, is numbered, observed, dumped, verified,
timed and counted the same way. The manager also owns the copies: a pass may change its unit in place, and the manager
copies a unit once per segment of transforms when its start must survive, and once per pass only for an observer that
needs the state before each pass. Several backends, or several tables of one backend, share the steps whose input,
configuration and body are the same, without changing what an observer sees. Several backends can be built in one
compilation from the command line (sxir gen -lang go,ts) and from the API (compiler.Generate). Printers take only
units of the Exec stage, and the stage-less compositions module of the libraries, and are named after the primitives
they print; the run-time backends only consume the results, and the optimization settings reach every backend, the VM
and the closures included. A pipeline prints as text; reading a text back is internal (tests and reproduction): the
reader takes the build's options from the text's headers, assembles the reference pipeline for them, and checks the text
against it by kinds and by its required skeleton before anything runs.
This record makes the model of the pass pipeline the implemented structure, decides the interfaces that record 063 left to the code, and supersedes the non-goal of 063 that excluded a textual pipeline language. The interfaces below were prototyped and tested as a standalone module before the decision; the Limitations section lists what the prototype does not cover.
Motivation
The pass infrastructure has two models of a step and leaves the order of a compilation to its callers.
- Two models of a step.
internal/passhas one pipeline type for transforms and a separate stage type, with its own combinators, for lowerings and composition. A lowering gets no event before it runs, and the event after it carries no change bit, counters or time. There are two analysis mechanisms: named recomputations on the pipeline and an analysis manager with typed keys. - Callers sequence the stages. The driver exports its Core, Flow and Exec stages. The Go generator runs the Core stage, then plans the table and lowers it through the shared code generation package, which runs the Flow and Exec stages. The TypeScript printer runs the Flow stage for one function at a time while it prints, and the Exec stage from its Exec printer. Any package can call a pass function or a lowering directly.
- Plans read printed output. The Go generator prints the whole parser once to learn which rules are reachable, how a call of each rule is printed, the expectation table and the scopes of pruned rows, and takes the frame layouts of the typed table from the scopes of the printed Node table.
- The run-time backends run passes outside the manager and ignore the settings. The VM compiles Core IR to
bytecode and runs the Core passes with a zero environment, which turns every pass on. The closure backend does the
same lazily, when a parse first needs a family, under the parse's context. The Flow route of the VM exists only in
tests. Nothing carries the settings there:
compiler.WithOptimizeis checked and dropped, since the options of the compile entry have no field for them. - Printers read the Flow stage and lower. The Go and TypeScript printers print functions at the Exec stage, but key their switches on the names of Flow operations, read the Flow dialect's function attributes, types and immediates, and read the Core module, origin map and lowering options through the Flow state of the generator; the TypeScript printer runs the Flow and Exec passes itself. Their types, files and comments are named after the Flow stage, although record 073 turned every Flow operation into a primitive of the Exec stage.
- The stage boundary of Code IR is checked at run time only. Flow and Exec passes take the same module type, so a
composition that leaves out
execlowertype-checks, and the stages are compared when the pipeline runs. - No run-time identity of a pipeline. Composition is generic code only, so a pipeline cannot be printed completely, read back, or changed for a test except through per-stage hooks of the driver.
Goals
- One step type, sealed: a pass is created only by the pass constructor and run only by the manager.
- Static assembly is typed: a pipeline written in Go is a
Pipeline[I, O]built with typed combinators, and a mismatch between steps, including two stages of one Code IR module, is a Go compile error. - Text assembly may be type-erased: a pipeline text is resolved through registries and checked by kinds, by the required steps and by the declared features before any step runs, into the same step tree that the typed door builds; one executor runs both.
- The driver assembles one pipeline per backend under the compilation's settings and exports the compilation, not its stages. Every backend follows the optimization settings: the run-time backends get them through the compile entry, with the program they build.
- Every step emits the same events with the same data. Dumps, statistics, timing, bisection and test hooks are observers. Verification and the copies it needs are a policy of the manager.
- Sharing of common steps between backends and between tables is a capability of the manager that no observer can detect except through a marked replay.
- A new IR, stage, pass, route or backend is added by declaring it and composing it; the manager needs no change.
- A run is deterministic and cancellable, its parameters are declared, and the text format is versioned.
- A printer takes units of the Exec stage only, lowers nothing, and is named after the Exec primitives it prints; what
it needs from earlier stages reaches it through the Exec module and the Exec-stage units. The one other input is the
libraries' compositions module (the kind
library), which is at no stage and holds no Flow operation.
Non-Goals
- A public API for passes written outside the module. Passes and the text form stay internal.
- Text parsers for Core IR and Code IR, a failure reproducer, optimization remarks, parallel per-function execution and a compile cache: follow-ups that this design must not block.
- Changing generated output. Every migration step keeps
sxir genoutput identical, and the dumps identical except for the changes of dump names and headers listed under Migration. - The switch of the VM to bytecode selected from the Flow stage. This record describes the VM pipeline before and after that switch; the switch itself is separate work.
Design
Units and Kinds
A kind describes one unit type to the manager: how to verify, print and copy it. A kind is identified by its Go type, so one Go type is one kind, and the name is its spelling in the text form.
type IR[M any] struct {
Name string // the spelling of the kind in the text form
Verify func(m M) error // the invariants of the kind
Print func(w io.Writer, m M) // the dump; nil: a unit of the kind is never dumped
Copy func(m M) M // a deep copy; nil for a value kind
}
// RegisterIR makes a kind known to the manager and the text form. A name or a Go type registered
// twice panics. It returns ir, for package-level declarations.
func RegisterIR[M any](ir *IR[M]) *IR[M]
Code IR is one IR whose modules are at a stage (record 073); the stage is part of the Go
type of a unit, so two stages are two kinds and the compiler rejects a composition that skips the conversion between
them. The stage markers and the kinds of a module at each stage are declared together in the code package:
// Stage is a stage of a Code IR module; its values are marker types.
type Stage interface{ stage() }
// At is a module at the stage S. Only EnterFlow and EnterExec make one at a stage; a pass keeps
// the stage with With.
type At[S Stage] struct{ m *Module }
func (a At[S]) Module() *Module
func (a At[S]) With(m *Module) At[S]
// FlowStage and ExecStage are the unit kinds of a module at the Flow and the Exec stage.
var (
FlowStage = stageIR[Flow]("code@flow", ...)
ExecStage = stageIR[Exec]("code@exec", ...)
)
func EnterFlow(m *Module) At[Flow] // only flowlower calls it
func EnterExec(m *Module) At[Exec] // only execlower calls it
With panics unless the module already names the stage S, and the kind's verifier checks the same. The run-time
stage of a module (record 073) stays, since dumps and the verifier read it, but only
EnterFlow and EnterExec set it, so a pass cannot move a module between stages through With. Leaving out
execlower is a compile error that names both stages:
in call to pass.Then, type pass.Pipeline[code.At[code.Exec], code.At[code.Exec]] of
pass.PerFunc(code.ExecFuncs, toy.ExecPasses) does not match inferred type
pass.Pipeline[code.At[code.Flow], C] for pass.Pipeline[B, C]
The kinds of a compilation:
| Kind | Unit | Made by |
|---|---|---|
program |
the checked program, the common options and the options of every target | the driver |
table |
one table of a program and its configuration | a backend's split of the program |
core |
a Core IR module, with the origin map of its elaboration | elaborate |
cores |
the program and the Core module of each of its tables | the join of the table split |
plan |
the plan of the program: reachable rules, call routes, layouts | plan |
planned |
one table's plan, its Core module and origin map | the split of a plan |
table@flow, table@exec |
the planned table and its module at the stage (Table[S]) |
flowlower, execlower |
code@flow, code@exec |
a module at the stage (At[S]) |
the module of a table unit, and each function of it |
lowered@exec |
one function in its module at the Exec stage | a view of code@exec |
tables@exec |
the plan with every table of a generating backend at the Exec stage, and the expectation table numbered from the Node and recognizer tables | the join of the planned tables |
library |
a compositions module of the libraries (at no stage), the specializations requested of it, and one specialized function per request | the driver, for the Go library; a lens of tables@exec, for the TypeScript library |
vm-input, closure-input |
the program and the Core module of the one table that the backend builds | the join of the backend's table split |
output |
the result of one backend: files, bytecode or the closures' Core module | the last required step of every backend |
Names are unique across kinds, splits, lenses, views, keys, providers and slots, and a name that ends in a stage
(@flow, @exec) names that stage. Stage suffixes are only for Code IR modules at a stage: library has none, because
the compositions module is at no stage and its operations have no dialect or the Action dialect. Splits are named after
the parts they give (go-tables, ts-tables, vm-tables, closure-tables, planned-tables) and are not kinds.
The table unit Table[S], in a data package of the planning code (internal/middleend/flowplan/plandata, beside
internal/middleend/flowlower/lowerdata for the data that the lowering takes), is the one value that carries the plan,
the table's Core module, the origin map and the module to the printer, so no printer needs a lowering, the plan or the
driver to get them. The origin map is recorded by elaborate in the Core module, keyed by the stable IDs of Core nodes,
so it travels with every Core unit, a shared one included, and survives copies. Kind names are not dump names: a dump is
named after the pass instance that produced it, and the kinds without a printer (program, table, cores,
planned, tables@exec, the backend inputs and output) add no dumps. plan prints the entry points, and the route
and the reach of each rule of each table; it is dumped only when -print-after names it, not under all, so the dumps
of the IR forms stay as they are. cores has no printer because its Core IR is dumped as it is built and changed, and
the plan shows its entry points and routes.
Every backend ends in the one kind output, so the branches of a Fork have one result type whatever the backends
build. Output is an interface implemented by the result of each backend; a caller that needs the concrete result
asserts it once. The result types live in a data package of the driver that the backend packages import and that
imports no pass code.
Passes
A pass is declared once, sealed and registered:
// Decl is what a pass declares besides its types and its function.
type Decl struct {
Name string // unique among registered passes; the flag, dump and error name
Required bool // runs whenever reached; has no flag
Test bool // an optional pass without a flag, always enabled (tests)
After []string // passes that must precede it in the pipeline
Obligation string // what the pass preserves, in one sentence
Features []FeatureID // the features it reads (Feature panics on any other)
Params []string // the registry flags it reads (Ctx.Value panics on any other)
Patterns []string // the sub-pass decisions it asks Ctx.Allow about
Counters []string // the statistics it counts (Ctx.Count panics on any other), in report order
}
// Def is the typed definition of a pass, given once to New. Its Run stays in the declaring package.
type Def[I, O any] struct {
Decl
In *IR[I]
Out *IR[O]
Applies func(c *Ctx[I]) bool
Run func(c *Ctx[I], in I) (O, Preserved, error)
}
// Pass is a sealed step from units of I to units of O. Only the manager runs it.
type Pass[I, O any] struct{ d *passDesc }
// New seals and registers a pass. A name registered twice panics.
func New[I, O any](d Def[I, O]) *Pass[I, O]
The pass contract. A pass returns its result unit and the analyses it preserved. A transform may change its input in place and return it, or return a new unit: the input belongs to the segment that runs it, and the manager decides when a copy is needed (Copies, below). A lowering returns a new unit of another kind and does not change its input. A kind whose units are pointers declares a copy; a value kind needs none.
Parameters. The registry flag is the one source of a parameter's default and documentation; a pass lists the flags it reads. A read returns the instance's override from the pipeline text if there is one, else the setting. Library controls stay registry entries that no pass declares.
The context of a run is typed by the input kind:
type Ctx[M any] struct{ s *ctxState }
func (c *Ctx[M]) Label() Label // backend, table and function of the unit
func (c *Ctx[M]) Value(flag string) int // a declared parameter
func (c *Ctx[M]) Count(counter string, n int) // a declared statistic of this pass on this unit
func (c *Ctx[M]) Context() context.Context // for cancellation polling
func (c *Ctx[M]) Allow(pattern string) bool // a sub-pass decision (canonicalize)
func Feature[T, M any](c *Ctx[M], k *FeatureKey[T]) T // a declared feature that an enclosing scope provides
func Query[M, R any](c *Ctx[M], k AnalysisKey[M, R]) (R, error) // an analysis of the current unit
The context replaces the environments of the Core, Flow and Code IR passes: the Flow target, observation and the
configuration of a table become features, and the flag lookups become declared parameters. Every analysis is a typed
analysis key; the string-named analyses of the pipeline go, and the standalone query that tools use outside any pipeline
stays. The attributes of Core nodes (nullability, first characters and effects) are not an analysis but content of the
unit: elaborate outputs a Core module with its attributes computed, the Core passes read them from the nodes, a pass
that changes nodes keeps them current, and the verifier of the core kind rejects a module whose attributes are stale.
Patterns. A pass with sub-pass decisions, such as canonicalize, declares its patterns. Each pattern is a registry
flag (-opt, -no-opt), and each application asks Allow, which the manager numbers as a run of its own and the
bisection observer answers. -print-pipeline lists the patterns under the pass, and -stats counts each.
Pipelines: the Typed Door
A pipeline is an immutable step tree. The typed combinators and the text reader build the same tree, and one executor runs it.
type Pipeline[I, O any] struct{ n node }
func Of[I, O any](p *Pass[I, O]) Pipeline[I, O] // one pass
func Seq[M any](ir *IR[M], ps ...*Pass[M, M]) Pipeline[M, M] // transforms; may be empty
func Then[A, B, C any](a Pipeline[A, B], b Pipeline[B, C]) Pipeline[A, C]
func Then3[A, B, C, D any](a Pipeline[A, B], b Pipeline[B, C], c Pipeline[C, D]) Pipeline[A, D]
// PerFunc runs the passes of p on each function of a unit, pass-major: the first pass on every
// function in order, then the second. p must be a sequence of passes and slots.
func PerFunc[W, M any](f *Funcs[W, M], p Pipeline[M, M]) Pipeline[W, W]
// Each runs p on each part of a unit, in order, and joins the results.
func Each[W, U, V, R any](s *Split[W, U, V, R], p Pipeline[U, V]) Pipeline[W, R]
// Within runs p on the part of a unit that l selects; the part may change kind (P to Q),
// which changes the whole (W to V).
func Within[W, V, P, Q any](l *Lens[W, V, P, Q], p Pipeline[P, Q]) Pipeline[W, V]
func Shared[I, O any](k *Key[I], p Pipeline[I, O]) Pipeline[I, O]
func Fork[I, O any](bs ...Branch[I, O]) Pipeline[I, []O]
func B[I, O any](name string, p Pipeline[I, O]) Branch[I, O]
func WithConst[I, O, T any](k *FeatureKey[T], v T, p Pipeline[I, O]) Pipeline[I, O]
func WithDerived[I, O, T any](d *Derive[I, T], p Pipeline[I, O]) Pipeline[I, O]
func Slot[M any](k *SlotKey[M], p Pipeline[M, M]) Pipeline[M, M] // a typed extension point
func Fill[M any](k *SlotKey[M], part string, p Pipeline[M, M]) Edit
func Apply[I, O any](p Pipeline[I, O], edits ...Edit) (Pipeline[I, O], error)
The arguments that are functions are registered values with a name, so that the text form can name them: the view of a
unit as functions (Funcs[W, M], where M may differ from W, so that the target passes run on one lowered function
at a time), splits, lenses, keys, derived providers, feature keys and slot keys. Each is registered with the kinds it
reads, so the text form can be checked by kinds. A split declares the labels of the parts it may give; a part with
another label fails the split, and a slot body for a part must name one of them. A feature key has a format and a
parser, which the text form and the sharing keys use. Within routes the module of a table unit through the Flow or
Exec passes; it only routes units, so it is not a second step model. Fork returns a list kind derived from its
branches' kind, whose verifier verifies each element. Seq takes the kind explicitly so that an empty sequence is a
valid slot body.
The Manager
// Run is the manager of one compilation.
type Run struct {
Settings Settings
Observers []Observer
VerifyEach bool // a manager policy: verify after every pass
NoShare bool // run every use of a Shared pipeline (for tests that compare)
Recover func(x any) error // turns a cancellation panic of a step into an error; nil re-panics
Keep bool // the caller keeps the input of Execute: the run changes a copy
Stop bool // the first failure breaks the run: no rollback, so no rollback copy
}
// Execute checks p and runs it on in. An invalid pipeline fails before any step runs.
func Execute[I, O any](ctx context.Context, r *Run, p Pipeline[I, O], in I) (O, error)
// Needs says what an observer reads, so the manager computes only that.
type Needs struct {
Before bool // the dump of the input (Before's in)
TextChanged bool // Result.TextChanged: the text form before and after the pass compared
Units bool // Result.Unit (typed test hooks); disables sharing
Veto bool // Before may return false (bisection); disables sharing
Print bool // Result.Print (dumps); without it no step prints its unit
}
type Event struct {
Seq int // the number of a decided run of an optional pass, from 1; 0 for a required step
Step StepInfo
Label Label
Replayed bool // the event of a Shared result that was reused
}
type StepInfo struct {
Name, Instance string // instance: the name, with #k for the k-th occurrence (k > 1)
Required bool
Pattern bool // the decision of one application of a pattern (Ctx.Allow)
In, Out string
}
type Result struct {
Ran bool // false: an observer vetoed the run
Changed bool // the pass's report: !PreserveAll; the same whatever observers watch
TextChanged bool // only if an observer needs it: the text form changed
Counters map[string]int
Patterns map[string]Decisions // the applied and skipped applications of each pattern
Elapsed time.Duration
Unit any // only if an observer needs units
Print func(io.Writer) // only if an observer needs dumps; valid during After only
Err error
}
type Observer interface {
Needs() Needs
Before(e Event, in func(io.Writer)) bool
After(e Event, r *Result)
}
Execute first instantiates the tree, then checks it: every feature a pass declares is provided by an enclosing
With*, every After holds, a PerFunc body is a sequence of transforms and slots, a sharing key and a derived
provider read the kind their block starts on, and every body of a slot for a part names a label of the enclosing
Each's split and has the required skeleton of the slot's default body. Only then does a step run. The checks hold for
both doors: the typed door cannot get the kinds wrong, but its edits can fill a part. The needs of all observers are
joined once per run, each field on its own. Without observers the manager builds no result; the dump of a step is
produced exactly when an observer needs dumps and the step's output kind prints, and the text forms of units are
computed only for dumps, the input dump, the text comparison and -verify-each. A dump is valid only during the call
that receives it, since a later transform may change the unit in place.
| Setting | Implemented by |
|---|---|
-print-before, -print-after, -print-changed |
an observer: the input dump, the result dump, the text comparison |
-stats, -time-passes |
an observer summing counters and time per instance, in pipeline order |
-opt-bisect-limit N |
an observer that may veto, answers each decision with Seq <= N and reports it |
| typed test hooks | an observer that needs units |
-verify-each |
the manager: verification, the copies before every pass and the preserve-all check |
Labels. A label names the unit a step runs on: its backend, its table and its function. Each combinator sets only
the field it decides and keeps the others: a Fork branch sets the backend, an Each part the table, a PerFunc
block the function. So a Fork nested in an Each part keeps that part's table, a slot body for that part runs in
every branch, and dumps and errors inside the branches name the table. The check of slot part bodies passes the
enclosing Each's labels into the branches for the same reason, so the check and the execution agree.
Instances. A pipeline value may be used in several places, so the executor and the printer of the text form copy
the tree and name each occurrence of a pass: the k-th occurrence on one path of execution is name#k for k > 1. The
branches of a Fork and the alternative bodies of a slot are numbered apart, since no unit runs through two of them;
after a slot the count continues from the largest alternative. Settings (-opt, -no-opt) address every instance of a
name; -print-after, -print-before and the reports accept a name, which selects every instance, or an instance; dumps,
errors and reports print the instance.
Verification follows record 072 and the pass-pipeline design. The output kind's verifier runs right after every pass
that changes the kind, as its postcondition, and an error names the pass. Where a run of transforms of one kind ends,
which is the end of a stage, the unit is verified once if an optional pass of that run ran; for a PerFunc segment this
is each function at the end of the segment, so the Flow passes' segment verifies its own functions before flow-finish.
Outside -verify-each the manager therefore verifies only where the stages before it did, at the stage boundary:
required transforms are trusted, and a run of them only, such as the required steps after that segment, is not verified
after it, since verification exists to catch the optional passes, whose rollback depends on it. Under -verify-each
every pass, required or optional, is verified after it runs, and a pass that returned that it preserves everything must
leave the text form unchanged.
Changed. Changed is the pass's own report in every run, so -stats and every other reader of it give the same
values whatever other observers watch the run. When an observer needs the text comparison, the manager also compares
the text form before and after the pass into TextChanged, so -print-changed prints nothing after a pass that changed
nothing whatever it reported, and -verify-each cross-checks the report against the same comparison. The two are
separate fields because needs are joined per run: -print-changed must not change what -stats reports. In general, the value of every field that an observer reads depends only on the
run and the pipeline, never on which other observers watch.
Copies. The manager follows one policy, and no pass declares anything about copies.
- A segment is a maximal run of consecutive optional transforms of one kind on one unit: the passes of a sequence
and its slots, or the per-function passes of a
PerFuncblock. A required transform, a kind-changing step or a combinator boundary (Each,Within,PerFunc,Shared,Fork) ends it. A required transform ends the segment before it because what it did must survive a later failure: an optional pass after it starts a new segment, whose rollback point is the unit the required transform returned. - The manager copies the unit once per segment, before the first pass of the segment that runs, when the segment's
start must survive it: the unit is kept by someone else (the input of
ExecuteunderKeep, the result of aSharedbody for every use, the first included, the input of aFork, and the parts of a kept unit), or a failure must roll back to it (the segment runs an optional pass and the run continues after failures, withoutStop). A kept original is never changed, so it is itself the rollback point; otherwise the original is, and the passes change the copy. A run withStopand no kept unit takes no copy: its units are consumed. - A
PerFuncsegment copies the whole unit once and takes each function's rollback point from the original, so a table unit pays one copy per segment, not one per function. - When an observer needs the state before each pass (
-verify-each,-print-changed, unit hooks, bisection), the manager also copies before every transform, so that every unit an observer was given stays as its pass returned it. The rollback point stays the segment's, so the outputs and dumps of a run do not depend on which observers watch it. - After a rollback the unit is marked kept, so a required transform after it copies before changing it.
Normal builds pay one copy per segment, consuming runs (Stop, the run-time backends) pay none, and debugging pays one
per pass.
Failure. A failure is held by a scope: the root, each part of an Each, each branch of a Fork, and each function
of a PerFunc segment.
- A failed optional transform rolls its unit back to the start of its segment, and that unit goes on. The start is never before a required transform that ran, so a rollback undoes optional work only. The later optional passes of the scope are skipped; the required steps still run, so a printer still gets a lowered unit, and the first error is reported with the instance and the label.
- In a
PerFuncsegment, a failure on one function rolls that function back and ends its optional passes; the other functions finish the segment. The enclosing scope is then failed, so the later optional segments of that table are skipped for every function. - Under
Stopthere is no rollback point: the first failure breaks the run, in aPerFuncsegment too. The copy of a kept table unit protects its keeper, but gives no function a rollback point. - A failed lowering, a failed required step, and a failure without a rollback point break their scope: nothing after it
runs in that scope, and the enclosing
EachorForkis broken after its other parts or branches ran.Executethen returns no result and the first error in execution order. Eachparts andForkbranches start from the state their combinator was entered with: a failure in one table or backend skips nothing in another. AForkreturns the errors of all its branches, joined in branch order.
Cancellation. The manager checks the context before every step and returns a step error that names the instance
that was about to run, wrapping the context's error. A step that polls deeper (elaboration, the Core passes for the
closures) may unwind with the cancellation panic of internal/cancel; the driver sets Recover to turn it into an
error, so internal/pass keeps importing no package of the module.
Determinism. Given the same input, settings, pipeline and observers, a run produces the same output, dumps, events,
numbers and statistics apart from time. Each, PerFunc and Fork run in a fixed order, and a future parallel
executor must keep this order of events.
Cost. A run with no observers allocates no event, result, dump or counter map per step beyond the pass's own context, whose counters are allocated on the first count.
Sharing
Shared(k, p) runs p at most once per effective key within one Execute.
- Key. The effective key joins three things: the registered key's value for the input; the value of every feature
and parameter that the passes of
pdeclare, as the scope provides them at that point, each written with its key's format; and a fingerprint of the body as it would run on this unit (the canonical form ofpwith its instances and parameter overrides, and every slot resolved for the current part). So a Core pass that started reading the Flow target would stop the sharing of Core IR between Go and TypeScript instead of sharing it wrongly; a slot filled for one table, or a text that edits the body in one branch, never reuses another body's result; and two uses whose instances are numbered differently do not share, so a replay always reports the instances that would have run. The fingerprint is computed once per use from the step tree, without running anything. - Kept result. The entry keeps the result; every use, the first included, gets it as a kept unit, so a later segment that changes it copies it first. The analyses of the shared unit stay valid for the same reason and are carried with it.
- Same observation. On a reuse the manager replays the recorded events of the first run to the observers, with the
current backend and table in the label, fresh numbers in the same order, the recorded counters, no time and
Replayedset. When an observer needs dumps, a step inside aSharedbody records the dump it gives that observer, and the replay carries the recorded text where the first run's event carried a dump. So the event stream, the numbering, the dumps and the statistics are those of a run without sharing. Observers that could tell the difference turn sharing off: unit hooks and bisection, whose decisions could differ between two uses. - Errors. A reuse reports the error of the first run again, relabeled with the current backend and table, so each target reports what it would report without sharing.
Sharing pays inside one backend: the Core IR of a table depends only on whether it builds values, so the Node table and
its traced copy share elaboration and the Core passes in every Go build. Across backends it pays when several are built
in one compilation: sxir gen -lang go,ts, compiler.Generate, the tests and the benchmarks. The VM and the closure
backend are not built in one compilation today, since the engine builds each on first use, so they share nothing unless
the engine changes; the output kind lets them be built together, as differential tests do.
The Text Door
The printer turns a pipeline into a description whose string is the canonical text. The reader parses a text, and the resolver checks it against the registries and builds the step tree:
func Describe[I, O any](p Pipeline[I, O], h ...Line) *Description
func (d *Description) String() string
func Parse(text string) (*Description, error)
// Resolve checks d against the registry and returns the pipeline typed at its edges. It checks by
// kinds what the compiler checks for a typed pipeline, and that its required skeleton is that of
// ref, so a text can reorder, repeat, insert or remove optional passes only.
func Resolve[I, O any](d *Description, ref Pipeline[I, O]) (Pipeline[I, O], error)
ref is the pipeline the driver assembled for the same build. A text that drops a required transform such as
flow-finish is well typed but fails this check.
The skeleton. The resolver compares the text with ref by structure, not as a list of names: the required steps,
the blocks that hold them, in order, and every constant feature with its value must be those of ref. A text that
moves flow-finish! into the per-function block, or that says with target=ts in the Go branch, is rejected. The
skeleton of a slot is that of its default body, and every body of the slot for a part must have that same skeleton, so
a part body cannot add, drop or move a required step either.
Kinds of keys and providers. A shared key and a derived with provider read the kind of the unit their block
starts on; the resolver rejects one placed on another kind, naming it, before anything runs. The executor repeats this
check and the checks of slot parts for the typed door.
Grammar. The text is line-oriented; indentation is two spaces per level and is not significant to the reader.
text = "sxir-pipeline" SP version NL { header } "pipeline" SP kind SP "->" SP kind SP "{" NL { item } "}" NL .
version = digits "." digits .
header = word { SP key "=" value } NL . (* sxir, options, program, settings, debug, target *)
item = step NL | block .
step = pass [ "#" digits ] [ "!" ] [ "(" param { "," param } ")" ] [ "[" pattern { "," pattern } "]" ] .
param = flag "=" integer .
block = opener SP "{" NL { item } "}" { SP "for" SP label SP "{" NL { item } "}" } NL .
opener = "fork" | "branch" SP label | "each" SP split | "func" SP view | "within" SP lens
| "shared" SP key | "with" SP ( provider | feature "=" value ) | "slot" SP slot .
! must match the declaration of the pass. #k is informational: the reader numbers instances itself. for label
bodies follow only a slot inside an each and replace its body for the part with that label; the label must be one
that the split of the nearest enclosing each declares, so a misspelt label is an error instead of a body that never
runs. The pattern list is informational and must equal the declared patterns.
Headers. The headers carry every option of the build that is not a pass, so the driver can rebuild ref and the
run from the text alone (with the grammar). Each option has one scope, and each scope one header word:
sxiris the version of sxir that printed the text. Library files come with that version.optionsholds the options of the program common to every target, which the check of the grammar reads:start,source-info,default-value,left-recursion-bindings.program(run-time backends only) holds what makes a run-time program other thanoptions: the remaining options of the check (recognition,disable-memo,no-projections,eager-attribute-copies) androle, which names the program of the three that the compiler side of a run-time program makes:parser, orrecognizerorinsertion, which are derived from the parser in one place, so the parser's options and the role reproduce the program exactly. A text whosepipelineline starts atlibraryis the text of the Go library; otherwise a text without aprogramline is the text of a generation.settings(the call) holds the profile and the overrides.debug(the call) holds the debugging settings, which apply to the whole compilation: the manager's policies (verify-each, andopt-bisect-limit, since bisection numbers one sequence of decisions across the run) and the observation flags (print-after,print-changed,stats,time-passes, andprint-beforewhen it is added). One writer receives the output of every target; each dump and each report line names the backend in its label.target <label>(one line per target, in order) holds every other generation option of that target, in declaration order.
The writer of the dumps and the library file systems are the only options not written; the reader supplies them. A
reflection test lists the exported fields of the generation options and fails unless each is written by exactly one
header word or is on that exclusion list, so a new option cannot be forgotten and cannot be given two scopes; a second
reflection test does the same for the options of the check over options and program. Stop has no header: the
reader derives it from the backends, as the driver does, and Keep is the driver's choice for its own input. Values are
escaped so that a module path or a library specifier round-trips. A run-time text needs the program that the compiler
side of a run-time program makes, which the driver cannot make, so its reader lives in that package beside the
derivation of the recognizer and the insertion program, for tests and the internal development tool.
Versions. The reader rejects another major version and a newer minor version. A text printed by another sxir version is read with a warning, since its passes may behave differently; an unknown name is an error that names it.
Example. The Go pipeline of a parser without streams, as -print-pipeline prints it (header values abbreviated):
sxir-pipeline 1.0
sxir 0.9.0
options start= source-info=false default-value=false left-recursion-bindings=false
settings profile=speed opt= no-opt=
debug verify-each=false print-after= print-changed=false stats=false time-passes=false
target go typed-only=false recognize-only=false limits=false expectations=false ...
pipeline program -> []output {
fork {
branch go {
with target=go {
each go-tables {
shared core-config {
elaborate!
slot core-passes {
guards
scans
}
}
}
plan!
each planned-tables {
with table-config {
flowlower!
within module@flow {
func funcs@flow {
slot flow-passes {
terminals
guarded-classes
class-runs
class-tables
inline-literals
}
}
flow-finish!
}
execlower!
within module@exec {
func funcs@exec {
slot exec-passes {
}
}
func lowered-funcs@exec {
slot target-passes {
}
}
}
}
}
print-go!
}
}
}
}
go-tables lists the typed table, the Node table, the recognizer and the traced copies that the target's options
select, labeled as the dumps label them; its join gives cores. core-config keys Core IR on whether the table builds
values. plan is a program-level step: reachability starts from every entry point of the program, the typed ones
included, so it cannot be decided per table. planned-tables splits the plan into its tables again and joins them, at
the Exec stage, into tables@exec, which print-go and print-ts take; the join numbers the expectation table across
the Node and recognizer tables, which is likewise not a fact of one table. table-config derives the features of a table (budget controls, observation,
Document, traced) from its unit. A TypeScript target is a second branch of the same fork with target=ts, the split
ts-tables, the target passes ts-spend-runs ts-merged-records ts-tested-inits and print-ts!.
The Driver
The driver is the only package that assembles and executes pipelines.
package driver
// Program is the input of a compilation: the checked program and the options that every target
// shares (start rule, source information, default values, left-recursion bindings).
type Program struct{ ... }
// Check checks the grammar for generation.
func Check(g *ast.Grammar, o common.CompileOptions, start string) (*Program, error)
// ProgramOf returns the Program of a program checked elsewhere: the run-time programs, which are
// checked with rule flags and a cancel control. role and parser, the options the grammar's parser
// was checked with, are what the program header writes. It checks nothing again.
func ProgramOf(p *frontend.Result, role Role, parser check.Options) *Program
// Role names which of the run-time programs of a grammar a program is.
type Role uint8
const (
Parser Role = iota // the program of the grammar
Recognizer // derived for recognition
Insertion // derived for missing-token insertion
)
type Backend = result.Backend // Go, TypeScript, VM, Closure
// Target is one backend of a compilation with its options. Gen holds the options of Go and
// TypeScript. Its fields that Program holds and that the call holds (optimization, debugging) must
// be zero or equal to the call's values; otherwise Compile fails before any step, naming the target
// and the field.
type Target struct {
Backend Backend
Gen common.GenOptions
}
// Options are the settings of a compilation, common to every target.
type Options struct {
Settings pass.Settings
Debug Debug // the debugging settings, which apply to the whole call
Edits []pass.Edit // internal: typed edits of the assembled pipeline (tests)
Pipeline string // internal: a pipeline text whose headers and body replace the assembled ones
}
// Debug holds the debugging settings of the call (common.Debug): the dumps, the verification
// after every pass, the statistics and times and the bisection limit. Every target's
// output goes to its one writer, labeled with the backend.
type Debug = common.Debug
type Output = result.Output
// Compile runs one pipeline for the targets under one manager: the targets are the branches of one
// Fork, in order, and share what their keys allow. On an error it returns no output and the errors
// of every failed target, joined in target order.
func Compile(ctx context.Context, p *Program, o Options, targets ...Target) ([]Output, error)
// Describe returns the assembled pipeline of the targets as text, for -print-pipeline and tests.
func Describe(p *Program, o Options, targets ...Target) string
// GoLibrary runs the pipeline of the Go library under the same manager.
func GoLibrary(ctx context.Context, o Options) (Output, error)
func DescribeGoLibrary(o Options) string
// FlowTable builds the Node table of p at the Flow stage for the VM's Flow route, at the default
// settings and with no pruning. It is the VM's route in production until step 12 of the migration
// assembles the VM pipeline in Compile, which deletes it.
func FlowTable(ctx context.Context, p *frontend.Result) (*plantypes.Table[code.Flow], error)
// The slots of the assembled pipelines, for Edits.
var CoreSlot, FlowSlot, ExecSlot, TargetSlot = ...
Each target is a branch labeled by its backend (go, ts, vm, closure; a second target of one backend is
go#2). Generation runs with the failure rule: required steps continue and segments roll back. The run-time backends
run with Stop and consume their units, since the build of a parser fails on any internal error.
Dumps per target. The driver makes one dump observer per target that selects dumps: it needs dumps, takes the events whose label names its branch and whose instance its selection names, and writes them to that target's writer. A shared step's events are replayed in each branch with that branch's label, so each target gets the dumps it would get alone. The options of the call hold only what the manager applies to the whole run.
Several targets from the command line. sxir gen accepts several languages: -lang go,ts, each at most once.
- Output layout. With one language,
-o DIRis the package directory. With several,-o DIRis a parent and each language writes toDIR/<lang>;-go.oand-ts.ogive a language its own directory instead. The ownership rules of a generated directory apply per directory, and every check of every directory happens before the first write: a failing target writes nothing for any target. - Flags. Common flags configure the one check and every target:
-g,-s,-source-info,-default-value,-optimize,-opt,-no-opt. Target flags (-types,-document,-observability,-expectations,-no-limits,-recognize,-max-depth,-lib) apply to every listed language; the qualified form (-go.types,-ts.max-depth) applies to one language and wins over the unqualified one. Flags that name something of one language (-module,-pkg,-nodocfor Go;-lib-modulefor TypeScript) apply to that language. A flag that a listed language does not support (-stream,-typed-only,-recognize-onlywithtslisted) is an error that names its qualified form. With one language every flag means what it means without the list. - Diagnostics. The debugging flags (
-print-after,-print-changed,-verify-each,-stats,-time-passes) are common flags: they apply to the whole call, and dumps and report lines name the backend in their label. The driver writes the dumps in the order the steps run.-lib onlywrites the library of each listed language.
Several targets from the API. pkg/compiler gets one function over the existing configurations:
// Target is a language to generate: a GoConfig or a TSConfig.
type Target interface{ target() }
// Generated is the files of one target.
type Generated struct {
Target Target
Files []File
}
// Generate generates a parser for each target from one check of src, in order. The Options apply
// to every target (start rule, source information, default values, optimization, debugging, whose
// dumps and reports name the backend); each config's fields apply to its target. The targets share the steps whose
// input and settings are equal. Each result equals what GenerateGo or GenerateTypeScript returns
// for that target alone. On an error no files are returned.
func Generate(src grammar.Source, targets []Target, opts ...Option) ([]Generated, error)
GenerateGo and GenerateTypeScript become Generate with one target, and the command line's language list maps to
one call. Optimization settings are common to the targets of one call, which is what lets them share Core IR; a caller
that wants different settings per language makes two calls.
Planning. plan decides everything that a printer decides before or while printing: the functions of each table
and their forms, the reachable rules and the route of each call, the frame layouts of the typed table from the Core
rules' scopes, and for TypeScript the same routes with a printability analysis that replaces the refusals its printer
raises, failing with the same message. A planned table holds only functions that its target prints: a typed-only Go
parser, which prints the Node rule bodies only for the frame layouts they fix, plans no lines of the Node table's
Operators Rules, so its dumps show no such lines either. The refusal of flat functions is an analysis of the Flow
stage that flowlower runs on each flat function right after writing it, before any Flow pass: a refused function is
replaced by its lowered fallback in the same lowering, so a table is lowered once and no target's form is read.
The expectation table. The Go parser numbers an expectation the first time one of its functions records it, and
the Flow passes decide where that is: a guard or a dispatch records the first-character expectations of a choice
ahead of the bodies of its alternatives. Numbering the Core IR of the tables in plan order gives the same expectations
in another order for most grammars of the corpus, so the numbering is not part of plan. The join into tables@exec
numbers the table from the lowered Node and recognizer tables, in that order, function by function in plan order and
operation by operation in the order of their code, in the Exec lowering, after the fixed expectations. That is the
order in which printing every function records them, which a test checks over the corpus. A rule of the Node or
recognizer table that the plan does not reach keeps its table row, and the IDs are those of a parser that prints
every rule, so the numbering lowers these two tables a second time without leaving out unreached rules: a lowering
that no observer sees, with no dumps and no events, whose tables only the numbering reads. The planned tables that
print-go prints are the pruned ones, and printing them fails with an internal error if it records an expectation that
the numbering lacks. The typed table and the traced copies add their expectations after this table as they are
printed, since the typed table has no row of a rule it leaves out.
The fallback for planning. If planning without printing renumbers the expectation table or changes the scopes of
pruned rows, the driver keeps two runs instead: it executes the Go pipeline once with the feature reach unset, so
every rule is printed, and takes the reach facts that print-go returns with its files; then it executes the same
pipeline again with those facts as a constant reach, and returns the second result. No pass runs a pipeline. The
first run has no dumping observer and reports no events. Under a bisection limit the driver executes only the first
run, with the observers, and returns its output, which prints every rule, so that one execution numbers every decision;
the generators already generate once under a limit.
-print-pipeline prints the second run with with reach as a derived provider, whose value a reader recomputes by the
first run.
The VM pipeline before the switch is the production route, run through the manager:
pipeline program -> output {
with target=vm {
each vm-tables {
shared core-config {
elaborate!
slot core-passes {
guards
scans
}
}
}
vmlower!
}
}
vm-tables gives the one table that the VM builds, and its join gives vm-input: the program and the optimized Core
module, which vmlower needs together. So elaborate takes table in every pipeline, and vmlower has one input
kind. After the switch, the VM's table runs plan, flowlower, the Flow passes for the bytecode target and
flow-finish like a generating backend, and vmlower takes the Flow module of its table instead of the Core module; the
Core route is deleted, and a text printed before the switch then fails to resolve by kinds.
The closure backend builds one Core module for the whole program, and every family is built from it. Its pipeline
is the same front over each closure-tables, joined into closure-input, and closure-core!, which returns the
optimized Core module of the program. A closure target names no family: the engine keeps its cache of that module
(built for the first family that a parse needs, shared by every family of the program and of its metered program,
dropped after the last pending family) and keeps asking the compiler side of the program for it, which runs one
compilation with the closure target. The runtime reaches the compiler only through the interface of the parser state,
and no runtime package imports the driver. The build stays lazy and runs under the context of the parse that needs it,
so its cancellation is attributed to that parse. Building the closures of a family from the result is the backend
consuming the result, not a step.
The Core analyses poll the cancellation control that the Core module holds, so elaborate stores the control it builds
from the step's context in the module, and none when the context cannot end. The module that the engine caches must
poll no parse's control after that parse, and closure-core may run on the kept result of a Shared entry, so it
changes nothing in its input: its result is a shallow copy of the module header without a control, sharing the rules,
which the closure build only reads.
The run-time entry. The compiler side of a run-time program already holds a program checked with rule flags and a
cancel control, and derives the recognizer and the insertion program from it. It wraps each such program once with
ProgramOf and keeps the parser's settings beside it; nothing is checked a second time. The bytecode is built with the
VM target once per program, under a background context, since it has no context and is not cancelled; the closures'
Core module is built with the closure target and the parse's context wrapped as before, or a background context when
the caller passes none. So the manager's check before each step and the polls of elaborate and the Core passes see
the same context and attribution as before. The driver's Recover turns only the cancellation panic into a step error
and re-panics every other value, and the compiler side maps a step error that wraps a context error or a cancellation to
the build-cancelled error of the parse.
The settings of a run-time program. One carrier takes the optimization settings from the API to every run-time program:
-
pkg/compilerconvertsOptimizeto settings in the one function that builds the options of the compile entry, which also validates them, so every function that builds a parser goes through it. -
The options of the compile entry stop being the options of the check and become a struct with two named fields, so each caller states what it sets:
// Options configures the compilation of a grammar to a run-time program. type Options struct { Compile check.Options // what the program is: the check reads them Settings pass.Settings // how its backends build it: the VM and the closures }The zero settings are the speed profile, in which every Core pass is on, so a caller that sets none builds what it builds today.
-
The compiler side of a run-time program takes the settings when it compiles, checks or wraps a program, holds them beside the program, and copies them into the recognizer and the insertion program it derives, so the three programs of a grammar build under the parser's settings.
-
The bytecode depends on the settings, so the once that builds it and its result move from the checked program to the compiler side that holds the settings; the checked program keeps no output of a backend. A saved program holds the bytecode that its settings built, and loading it builds nothing.
The settings do not go into the options of the check: the check reads none of them, so they would have two scopes (the
options and the settings header), the common options of ProgramOf would include them, and the frontend's options
package would import internal/pass for a value that only backends read.
Settings for every backend. The VM and the closure backend follow the optimization settings. The reference evaluator
of Code IR is the oracle of the differential tests, which run each backend at the default settings and with every
optional pass off. compiler.WithOptimize applies to every backend. The Core passes are on in every profile, so the
bytecode and the closures change only when an override turns a Core pass off.
Test hooks become edits and observers: a typed hook on Core units is an observer that needs units; a replaced pipeline of one stage is a typed fill of the stage's slot for one table or every table; a broken test pass is a fill whose body ends in that pass, counted by an observer; test passes are declared as test passes, always enabled and without a flag; per-table text overrides are slot bodies for a part.
Printers Take Exec-Stage Units
A printer's input is only units of the Exec stage: print-go and print-ts take tables@exec, and within it the
printers print each function as lowered@exec. The one exception is the libraries' compositions module, the kind
library, which print-ts reads through tables@exec and print-libgo takes: it is at no stage, and it holds no Flow
operation by construction, since the compositions are written in the runtime's own operations and the Action dialect,
never through flowlower. A printer runs no lowering, no pass and no pipeline; every lowering is a step of the
pipeline. What a printer reads from the Flow stage reaches it through the Exec module, and what it reads from earlier
stages through the Exec-stage table unit:
- Primitive names. The primitives that record 073 made of the Flow operations get their
own data package,
internal/ir/code/exec/prim, which imports only the Code IR and Core IR packages. It declares a constant per primitive, and the printers switch on these instead of mapping each call back to its Flow operation.execlowerkeeps mapping a Flow operation to its primitive, and a specification test checks that every Flow operation has its primitive constant. - Immediates. The Go types of the immediate operands, with the loop-label functions and the register names, move
from the Flow dialect's package to
prim. A primitive has the immediate of its Flow operation (073), so the Flow dialect declares its operations with the types ofprim: one type, no alias. - Types. The classification of the runtime's handle types moves to
prim; it classifies the Code IR types of the Exec module's registers, handles and locals. - Function attributes.
execlowerconverts the Flow attributes of each function into Exec-stage attributes ofprim: every field except the forms, whichflow-finishconsumes. The Exec-stage verifier rejects a function whose attributes are of another type, as it rejects a Flow operation, so no Flow attribute survives the conversion. The Code IR listing prints no function attributes, so the dumps do not change. - Earlier stages. The Core rules, rule actions and Operators Rules, the expectation table, the origin map and the rules of the program come from the Core module, origin map and program that the Exec-stage table unit carries, not from a Flow state of the generator, which goes. The lowering options that the printers read are function attributes.
Lowering leaves the printers. Each call that lowers or runs passes inside a printer becomes a step: the Core IR of a
table is elaborate and the Core passes of the table's pipeline, and the Core IR that reachability and the used-features
analysis build is the input of plan; the functions of a table are listed by plan and lowered by flowlower and the
Flow passes of the table; TypeScript's Exec passes are the table's execlower! and its target passes; the action terms
are listed by plan per table, with the options the printers pass, and lowered in the table's pipeline before
execlower into Action-dialect functions of the table's module, which are legal at both stages (whether that lowering
is part of flowlower or of the redesigned action lowering is decided by that redesign; either way it is a step). The
specialization of the TypeScript library's compositions for the parser's features is a required step of the TypeScript
branch on tables@exec: the TypeScript backend's required step ts-library! sets the unit's library to the
compositions module with the requests of the TypeScript library's specification table, each keeping only the features
that the parser has as the plan's used-features analysis reports them, and within ts-library { specialize! } adds the
specialized compositions that print-ts prints.
The Go library is a route of the same pipeline. golang/libprint prints the compositions of the Exec dialect into
the Go runtime library, the engine state and the conformance bindings, from go generate. It goes through the driver and
the manager like the TypeScript library:
sxir-pipeline 1.0
sxir 0.9.0
settings profile=speed opt= no-opt=
debug verify-each=false
pipeline library -> output {
specialize!
print-libgo!
}
The two libraries enter differently because they differ: the TypeScript library is specialized per parser, for the
parser's features, inside the ts branch; the Go library is a checked-in artifact that no grammar determines. They
share the specialize pass.
- The source unit is
library: the driver makes it from the compositions module and the specializations that the library printer's specification tables list, which it exports as data, grouped by output file. specializeis the one pass that both libraries run. A request names the composition, the features and the name of the copy. The unit keeps one specialized function per request, in request order, outside the compositions module, so the module does not change and many requests of one composition with different features do not collide.print-libgois a printing pass of the Go backend package, besideprint-go, which takeslibraryand returns the files that the library printer writes. The library printer builds no module, runs no pass and writes no file; a printer that edits a function copies it first, so an edit of a documentation comment for one output file does not reach another file that prints the same composition. Thego generateprogram writes the files of the output, and the generated files do not change.- A library text has the headers
sxir,settingsanddebugonly, and itspipelineline starts atlibrary. The reader selects the reference pipeline by that source kind. The library pipeline has no slots, so an edit that names a slot fails before any step, naming the slot. A program text given to the library route, and a library text given toCompile, fail before any step, naming the source kind. - The library printer is a print package: every import rule of the print packages applies to it, and its references into the Go printer are limited to the printing API over Exec-stage functions.
- The tests that specialize compositions run the
specializepass from their test files throughExecute.
The analysis of dead stores, which the printers apply with their own purity, changes nothing, so it stays in the
printers but moves out of the pass package of Code IR into an analysis package of the Exec stage that imports no pass
code. The check whether a flat rule can run a lowered function, which reads no entry position or action context where
the printers reach and keeps no value in a variable, sits beside the analysis of dead stores and moves with it: it reads
only the function and the entry variables and variables it is given, so the plan decides with it on the Flow function
and the Go printer asserts with it, on the Exec function in the forms of its passes, that the decision still holds,
each with the attributes of its own stage. The flags that the Go printer registers move to the package that declares
the lowering that reads them, or to the
Go backend package that declares print-go, which declares them as parameters and gives their values to the printer as
data.
Names. The printers are named after the Exec primitives and functions they print: no file of a print package is
named flow*, and no identifier it declares contains Flow as a word. Helpers that only start a Flow lowering or hold
Flow state are deleted, since the pipeline elaborates and lowers and the table unit holds the functions. The error of a
construct that a printer cannot print says that the printer does not print it, not that Flow IR does not. A name that
collides with an existing one takes the prefix exec. Tests that check a primitive against the contract of its Flow
operation keep naming the Flow operation, since 073 states the contract per Flow operation.
What Makes the Wrong Shape Impossible
By construction:
- A pass's function is reachable only through the manager: a pass is opaque, the constructor keeps the run function in the step tree, and declaring packages keep their run functions unexported and export only the pass values and data types.
- A composition is a step tree made only by the combinators or the resolver; there is no stage API to call.
- A missing conversion between two Code IR stages, or between two IRs, does not compile.
Executeis the only executor.
Detected by architecture tests, because Go cannot restrict callers:
- Outside
internal/passand the packageinternal/driveritself (not its subpackages), no non-test package executes a pipeline. The subpackages and every other caller of a compilation go through the driver'sCompileandDescribe; they may importinternal/passfor settings and edits only. Packages that declare passes, kinds or analyses import it for those, and run nothing. internal/frontend/checkimports nointernal/pass, so the settings stay out of the check's options, and no non-test package underinternal/runtimeimportsinternal/driverorinternal/pass.internal/driverexports only an allowlist of identifiers: the compilation API above, the slot keys and the vocabulary of the settings (profiles, their documentation and values, the parsing of overrides and dump selections, and the check of the registry against the driver's step names). A new export fails the test. Its subpackages keep their own exports, which the import rules cover.- No package exports a pass definition or a function that a pass's run function names; lowering, planning and pass
packages export their pass values and data types only. In particular the pass package of Code IR exports no
specialization function: both libraries reach it through the
specializepass. EnterFlowis called only byflowlower,EnterExeconly byexeclower.- The print packages of every backend, the library printer included, and
internal/backend/codegen/commonimport none ofinternal/pass,internal/driver, the pass packages, the lowerings,vmloweror the planning code. They may import the result and data types: the planning data types, Code IR, Core IR and the lowered function. To make this checkable by package, the data types move into packages of their own apart from the code that computes them. - The printing passes and the last steps of the run-time backends are declared in the backend packages above the print
packages, which import
internal/pass; the print packages only receive results. The result package of the driver imports no pass, lowering or planning code. - Printers take Exec-stage units only. The print packages and
commonimport none of the Flow-stage packages (the Flow dialect,flowlower,flowpassand the planning code) nor the Core builderinternal/frontend/elaborate; they read primitives, immediates, types and function attributes fromprim. A type-checked walk of the print packages finds no use of the Flow stage marker, a module or table at the Flow stage, orEnterFlow, so no printer function takes a Flow-stage unit. Every registered pass whose output isoutputfor Go or TypeScript has an input kind whose name ends in@execor islibrary;libraryis accepted by name, since another stage-less kind could carry anything. No non-test file of the print packages is namedflow*, and no identifier they declare containsFlowas a word, with an empty allowlist. - The Exec-stage verifier rejects a function whose attributes are not the Exec-stage attributes.
A rule that holds before the code it checks exists (no print package uses the Flow stage marker before the marker
exists; no pass outputs output before the kind exists) proves nothing by passing, so each such rule lands with a
negative control: a planted violation, in a testdata package or a registry built only for the control, that the rule
must report. The kind rule's control plants a printing pass on code@flow and one on a stage-less kind other than
library.
Alternatives Considered
- Kind identity by name, checked at run time (several kinds per Go type). Simpler types, but the most important boundary of the compilation would be a run-time error again. Rejected for stage-typed units.
- A wrapper type per stage that embeds the module. Embedding exposes the module and a conversion is one literal away. Rejected for a phantom-typed handle with restricted constructors.
- Keep stage entry points on the driver and move the calls out of the printers. No single value describes the
compilation, so the text form,
-print-pipelineand sharing have nothing to work on. Rejected. - Type-erased pipelines only, or typed pipelines only. The first loses the compiler's check of the assembled pipelines; the second loses the text form and reading back. Rejected; both doors build one tree.
- A copy per pass for passes that change their unit, or a pass contract that forbids changing the input. Measured: a copy per rewriting pass costs 5.8% of generation time, against an A/A spread of 0.3%. Rejected for one copy per segment, which the manager decides from who keeps the unit.
- A per-segment keep declaration by the caller. The manager can tell from the pipeline which units are kept, so the declaration moves into one field of the run. Rejected.
- Each backend's own result kind and no common one.
Forkneeds one result type; an adapter step per backend would add a step that does nothing to every dump and text. Rejected for the one kindoutput. - Planning per table. Reachability is a fact of the program, and the expectation numbering one of its joined
tables. Rejected for a program-level
planand the numbering at the join. - Numbering the expectation table in
plan. The plan holds Core IR, whose order of expectations is not the one the Flow passes give the printed functions: it renumbers the generated tables of most grammars. Rejected for the numbering of the lowered tables at the join intotables@exec, which keeps the generated parsers. - A multi-target entry in the API only, or in the command line only. Both are thin over the driver; offering one without the other would make the command line and the API generate differently. Rejected.
- Shared results that emit no events. Simpler, but dumps and bisection numbers would depend on which backends are built together. Rejected for replay.
-verify-eachas an observer. It needs copies and turns a failed verification into a pass failure, which only the manager can do. Rejected.- A required transform inside a segment, rolled back with it, or a check that rejects a required transform between optional ones. The first undoes required work after a later optional failure, against the failure rule; the second forbids a shape that typed code can build and that costs only one more copy. Rejected for ending segments at required transforms.
- A closure target per family. Each family's compilation would elaborate the program and run the Core passes again, since the Core module is one for the program. Rejected for one closure target per program, with the engine's cache around it.
- Dumps per target and manager policies per call, or every debug option per target. The first puts the debugging settings in two places, the call and each target's configuration, so the command and the Go API need two surfaces for one concern; it only buys separate writers, which the backend in each label already gives a reader. The second gives verification and bisection, which one manager applies across shared steps, values that could differ between targets that share a step. Rejected for one set of debugging settings for the whole call, with one writer and the backend in every label.
- A second check with rule flags and a cancel control for the run-time programs. It duplicates the check that the compiler side of a run-time program already does. Rejected for wrapping the program it holds.
- The optimization settings in the options of the check. The derivation of the recognizer would carry them without a change, but the check reads none of them, so they would have two scopes and the check would import the pass package. Rejected for a settings field of the compile entry's options.
- Read-back for generation texts only. Simpler, but a VM or closure text could not reproduce its run. Rejected for
the
programheader and a reader beside the run-time programs. - A
Forkthat starts a fresh label, or a check that rejects slot part bodies under a nestedFork. The first runs no part body and drops the table from the branches' dumps; the second forbids a meaningful shape. Rejected for labels that each combinator sets one field of. Changedby text comparison for every observer once one needs it. A per-target-print-changedwould change what-statsreports. Rejected for the pass's report and a separate text comparison.- Printers that keep importing the Flow dialect for names, attributes and types, or aliases to the new types. The
printers would still depend on the Flow stage, and an alias gives one thing two names. Rejected for the primitives'
own package and Exec-stage attributes that
execlowerwrites. - The primitives' names and immediates in the Exec dialect's package. That package already declares names for the
compositions that would collide. Rejected for the subpackage
prim.
Testing
The structural tests fail on the code before the change that lands them and pass after it; a rule that holds before its subject exists is proven by its negative control instead.
- The architecture tests of the previous section, with their negative controls.
- A testdata package composes the Flow passes with the Exec passes without
execlower;go buildmust fail and name both stages. The same text fails to resolve, naming the step.Withon a module of another stage panics. - Golden
-print-pipelinetexts for Go, TypeScript, Go with TypeScript, the VM and the closure backend list every step from the program to the output; parsing a text and printing it gives the same text; building from the printed text reproduces the output and the dumps. - The resolver rejects a missing or extra required step, a required step moved to another block, a changed constant
feature, a wrong
!, an unknown name, a kind mismatch, an unprovided feature, a body for a part outside a slot, another major or a newer minor version, a required step added in a slot's body for one part, a part label that the split does not declare, and asharedkey or a derived provider on a block of another kind.Executerejects a typed fill for an undeclared part, or with a required step in a part body, before any step runs. - The headers: the reflection tests over the generation options and the options of the check; a text read back builds
the same
ref; changing a header value changes the run;Compilerejects a target that sets a program or call-wide field to another value than the call's. For each role, the VM and the closure text of a program read back builds the same bytecode and Core module as the run that printed it. - Dumps per target: Go and TypeScript with different dump selections and writers in one compilation; each writer gets its target's dumps only, equal to a run without sharing.
- One counting observer sees every step of the four backends, the lowerings,
planand the printers included. - Sharing: Go and TypeScript built together elaborate each configuration once, and their outputs and events, with non-empty dumps, replayed ones included, equal those of a run without sharing; each output equals the output of its target built alone, byte for byte. Within one Go build, the Node table and its traced copy share Core IR. A slot filled for one table and a text that edits one branch's shared body give the output of a run without sharing; a failure in a shared body gives the outputs and errors of a run without sharing, each error labeled with its own target; a pointer-valued feature with equal values shares.
- Copies: a pass that changes a shared result in place does not change it for the next use, also under
Stop; the number of copies is one per segment, none underStop, more under-verify-each. - The failure rule: a failed optional pass rolls back to its segment's start and the required steps go on; a required
transform between two optional passes keeps its effect when the later one fails; a failing test pass on one function
of one table leaves the other functions, the other tables and the other backend running, skips the later segments of
the table, runs its required steps and names the instance and the label; under
Stopthe run breaks, also in aPerFuncsegment of a kept unit. - Labels: a slot filled for a part, inside a
Forkinside anEach, runs in the branch, and the branch's events carry the table in their label. - Observers do not change each other's values:
-statsis the same with and without-print-changed, and a pass that reports a change without making one givesChangedtrue andTextChangedfalse. No unit is printed without an observer that needs dumps, and an observer that needs only dumps gets a non-empty dump on every event whose kind prints, live and replayed. - The closure target: one compilation gives the program's module, every family is built from it without elaborating
again, and the engine's cache compiles again only after it dropped the module. The VM and the closure built in one
compilation share the Core front, and
closure-coreon the kept unit leaves it unchanged and returns a module without a control. - Cancellation: a cancelled context names the step; a cancellation panic in the closure build is a cancelled build; a panic of the parse's context unwinds unchanged; a closure build without a context builds the module.
- Settings: turning a Core pass off through
compiler.WithOptimizechanges the parser's bytecode, which equals the driver's build with those settings, and the recognizer's and the insertion program's bytecode equal their direct builds; the closures follow the same settings; both agree with the reference evaluator; the derived programs hold the parser's settings. Dropping the settings anywhere on the way fails these tests. - Printers: a specification test that every Flow operation has its constant in
primwith the same immediate type, the verifier's negative control with a Flow attribute left, and identical generated output and dumps across the change that moves the printers onto the Exec units. - The Go library: a golden library text that reads back to an equal text and an equal output; an edit naming a parser
slot fails on the library route, naming the slot;
go generatethrough the driver leaves every generated file unchanged; onelibraryunit with two requests of the same composition under different features keeps two functions, and its module is unchanged; printing the Go terms file and then the engine's term file from one unit gives the engine file printed alone; a program text given to the library route, and a library text given toCompile, fail before any step, naming the source kind. - A toy kind, a lowering to it and a transform on it compose typed and as text and run with dumps,
-verify-each, statistics and bisection with no change tointernal/passorinternal/driver. sxir gen -lang go,tswritesDIR/goandDIR/tsequal to two single-language runs; a qualified flag applies to one language; an unsupported unqualified flag is an error naming the qualified form; a failing target writes nothing.- Each backend runs twice, with
GOMAXPROCS1 and N: identical outputs, dumps, events and statistics. - A run with no observers allocates nothing per step beyond the pass context.
- At every migration step the generated output over the generation corpus and the
-print-after alldumps equal those of the step before, except as listed under Migration.
Performance
The manager adds one instantiation per run and one event per step when observers watch. Copies follow the segment
policy: a generation pays one copy of each table's module per Flow and Exec segment and one Core copy per Shared entry,
for the rollback that the failure rule needs; the run-time backends pay none. The fingerprint of a Shared use is a
walk of its body, computed once per use. All of this is measured against the generation-time gate, within the A/A
spread of the base, and the run-time compile benchmark. Building Go and TypeScript in one compilation checks the grammar
once and shares the Core IR of the tables whose keys are equal; the saving is reported only after it is measured.
Migration
Each step keeps the generated output and the -print-after all dumps identical. The dumps change three times, each as
its own change: the removal of the stage names core, flow and exec from -print-after and the dump headers; the
header form that a kind's Print requires; and the attributes in the dump after elaborate. A kind prints the unit
alone, its functions included, so the dump observer writes one header per dump from the event,
-- after <instance> [<label>] --, in place of the header that each function's listing carried; the label names the
backend once branches exist, the table, and the function of a per-function step. elaborate makes a Core module with
the attributes of its nodes computed, since a kind's verifier checks the unit and never writes it, so the dump after
elaborate shows the attributes that the dumps after the Core passes show already.
- The new core of
internal/passbeside the old one: kinds, the pass constructor, the context, the step tree, the combinators, the manager with its copy policy and its events, labels that each combinator sets one field of, andChangedbesideTextChanged. No user. - Stage-typed Code IR units. Core, Flow, Exec and target passes are declared with the new constructor unchanged;
the context replaces the environments of the Core, Flow and Code IR passes, every analysis they query is an analysis
key, and the per-segment keep declarations become the manager's policy. The driver's stages run
Executeinternally and keep their signatures. Copies do not increase, measured against the generation gate. - The table unit with its lenses; the lowering of a table becomes one
Executeof a table pipeline; the origin map moves into the Core module; the planning and lowering data types move into their data packages. Until step 6 makes the compilation oneExecute, the driver runs oneExecuteper stage or table, and these share one numbering of the decided runs (Event.Seq), so a compilation numbers its runs once, in the order they run; the shared counter becomes internal to the manager at step 6. Until step 5 plans the functions that the TypeScript printer lowers one at a time, the driver runs each through the same steps as a table of its own: a plan of its functions in order beforeflowlower, and a table that no plan holds beforeexeclower. - Planning as a program-level pass, the expectation table numbered from the lowered tables where they are joined, and the Go generator prints once. Neither renumbers the expectation table nor changes the scopes of pruned rows, so the planning fallback is not needed; it stays the route if a later change makes planning differ from printing.
- TypeScript: routes in the plan; the printer calls no pipeline. The TypeScript table is planned and lowered
once, like the Go tables, with its outlined nodes chosen by the table's configuration, so its
-print-afterdumps show one run over the table instead of one run per function. The printability analysis lands with step 7, where the printers read the primitives that the TypeScript backend declares: built on those declarations, it needs no second list of the printed operations. Until then the printer's refusals stay. - The driver assembles the Go and TypeScript pipelines. The printers become passes in the backend packages; the
driver's stage API, the pass and lowering entry points of the code generators and the hooks go, and their users move
to
Compilewith edits, or toFlowTablewhere they need only the table. The architecture tests land with the rules that hold at this step: the caller rule and the export rules. - Printers on Exec-stage units, each change with identical output and dumps: the
primpackage; Exec-stage attributes with the verifier's rule, and the flat-function check beside the analysis of dead stores, which the plan and the Go printer call each on its own stage; the printers readingprimand the table unit; the lowering left in the printers as steps, with the TypeScript library's specialization and the import ban of the print packages; then the renames and the name rules, and the walk for Flow-stage units with its negative control. - The text form: the printer, the reader and the resolver with the skeleton and kind checks, the headers with their
reflection tests and per-target dumps,
-print-pipeline, and the internal reader for tests and a development tool. - Sharing,
Forkand theoutputkind,Compileover several targets, and the sharing of Core IR between a table and its traced copy. - The Go library route, with the library text door,
print-libgo, and the deletion of the exported specialization function; the library printer joins the import rules, and the printing-pass kind rule lands with its negative control. - Several targets from the command line and the API.
- The VM and the closure backend through
Compilewith the parser's settings: first the settings carrier, with unchanged outputs; thenProgramOfand the run-time builds through the driver, theprogramheader and its reader, and the settings tests, which fail before this step. Outputs are identical under every profile and change only under an override of a Core pass. - Gates and documentation: the generation-time gate, the run-time compile benchmark, the determinism and allocation tests, and the design pages updated for the parts this record implements.
The VM switch to the Flow route, which deletes the Core route, comes before step 12 as separate work and changes only
the VM pipeline: the VM selects its bytecode from the table that FlowTable builds. Step 12 then moves the VM, already
on the Flow route, onto Compile.
Follow-Ups
- Text parsers for Core IR and Code IR, and a reproducer: on an internal error the manager writes the pipeline text and the unit before the failing pass, and a tool reruns that pass on it.
- Optimization remarks through the context.
- Events for computing and invalidating analyses, so that
-time-passescan charge analysis time apart from the pass that asked for it. Until then it is charged to that pass. - Parallel execution of per-function passes, keeping the observable order.
- A compile cache keyed by the sharing keys, for the language server and watch mode.
Limitations and Open Questions
- The prototype does not cover the cancellation context, patterns, analysis queries, the semantics of the
debug,target,programandsxirheaders,ProgramOfand the run-time entry with its settings carrier, the command line and API over several targets, the planning fallback, the printers on Exec-stage units, the Go library route, or the allocation bound. Their shapes are fixed here; their behavior is specified, not demonstrated. - A generation copies the Core module of each
Sharedentry once, for the rollback that the failure rule needs, where the current driver consumes it. The generation gate decides whether this is visible. - Sharing is off under bisection and under unit hooks, so those runs do the full work.
- Go cannot restrict callers of exported functions; the rules on
Execute,EnterFlow,EnterExecand the run functions hold because the architecture tests check them. - Reading a pipeline text back stays internal: there is no public API or command-line flag for it.