Code Generation for Host Languages
Warning
This page describes the target architecture. The code may still differ from it.
The host language is the language of a generated parser. SyntaxIR generates parsers in Go and TypeScript today. Another host language is added in the same pattern, so the representations and the pass infrastructure do not change. This page explains the pattern, the path from the Exec stage to printed source, the passes that the host languages share, the passes of each of the two current instances, and the runtime library of generated parsers. The backends as a whole are in Architecture.
The Pattern
Each host language has subpackages for its passes, its printer and its runtime library.
- Passes legalize and optimize the Exec stage for the language. They run from the Exec stage to the Exec stage and
are registered in the same registry as the other passes, so each has a flag. Go's passes are in the
golang/passpackage and TypeScript's intypescript/pass. - Print prints the legalized Exec stage as it is. The printer does not carry transformations as string processing
or as regeneration of the code. Go's printer is
golang/print, and TypeScript's istypescript/print. - Library is the runtime library of generated parsers. Go's library parts are
golang/libprint, which prints the compositions of the Exec dialect into the library, andgolang/libsrc, the copy of the library that a generated module embeds. TypeScript's runtime library istypescript/lib.
The parts that all host languages share, such as options and the helpers of the host-language packages, are in the
common package beside them.
From the Exec Stage to Printed Source
flowchart LR
EX["Exec stage"] --> CP["Dialect-independent passes"] --> LP["Passes of the host language"] --> PR["Print"] --> SRC["Parser source and library"]
- The Exec stage is complete:
execlowerhas converted every Flow operation. - The dialect-independent passes run. They specialize functions for the features of the build, canonicalize, remove dead stores and share identical helpers and literal tables.
- The enabled passes of the host language run in order, as the last part of the same pipeline and under the same manager as every other pass. Their flags are in the registry, and their dumps and verification work as for the other stages. The driver assembles the whole pipeline of a generation, the printing included, and runs it; the code generators declare their passes and their printing steps and run none of them. Tests that change a pipeline give the driver typed edits of its slots or a pipeline text.
- The printer prints the legalized Exec stage. It takes units of the Exec stage only, reads the primitives, types and function attributes of the Exec stage, and runs no lowering and no pass. The generated parser and the library it uses come from the same representation, so the rules of a generated parser and the mechanisms they call agree.
Passes Shared by the Host Languages
The dialect-independent passes serve every host language: specialization and canonicalization of functions, dead store elimination, reachability after printing, and the sharing of identical helpers and literal tables. They are described in The Pass Pipeline.
The Passes of Each Host Language
The passes below are the instances for the two host languages of today. Each host language registers only the passes that its own features need.
Go
The Go instance has passes that are specific to Go:
- legalizing the scope of
gotoand of declarations, so that locals are enclosed and nogotojumps over a declaration; - grouping variable declarations;
- encoding terminal regions as tables;
- turning literal comparisons into a
switch; - comparing fields in typed Go;
- removing unused helpers;
- adjusting the shape of the loops of Operators Rules.
TypeScript
The TypeScript instance has passes that are specific to TypeScript:
- structuring with labeled blocks and
for(;;): TypeScript has nogoto, so an exit from a loop is a labeled break; - outlining large functions that hold a frame;
- its own decision on sharing literal tables, which differs from Go's;
- specializing the conversion to typed values;
- joining charges in a row into one charge of as many units (
ts-spend-runs), the end of an expectation record and the merge of its expectations into one merge in place (ts-merged-records), and the test of whether the call that anifbegins with matched into its condition (ts-tested-inits). These are the passes oftypescript/pass; the printer prints the operations they replace as well, so each can be turned off.
The Runtime Library
Each host language has a runtime library that the generated parsers call. For Go, it is the support library of generated parsers.
- The library printer prints the compositions of the Exec dialect into the generated-parser library and into the
shared engine state. For Go,
golang/libprintis this printer, andgolang/libsrcis the copy of the library thatgo:generateproduces for the embedded mode. That printing is an edge of code generation, run throughgo:generate, and a pipeline of the driver like any generation: the compositions are specialized by the pass that also specializes the TypeScript library for each parser, and the library printer only prints the result. It is not an import edge, so the runtime side does not depend on the printer. - Shared mechanisms. The mechanisms that are not compositions (the record of recovered errors, the hash index of
the memo, the value primitives of action expressions and the charges of value work, among others) are Go source that the Go
library and the shared engine state both compile. The library holds each such file, and
go:generatecopies it into the engine state with only its package clause changed; a freshness test fails when a copy differs. A shared file names only what both packages define alike: the other shared files, the state that both parser types embed, and the parser members that each package provides under the same name and contract. - Library mode. The generator either imports the runtime library as a library, or embeds a copy of it in the
generated output (
-lib importand-lib embedfor Go), as far as the packaging of the host language allows. The embedded copy compiles by itself, as the rules of the Package Layering require for generated parsers. - Freshness. A test keeps the printed library and its embedded copy identical to the printed output, so a change to the compositions cannot leave a copy behind.
- Controls. The printers and libraries register their own controls with the kind
library, from the package that owns them. See The Registry.
Related Topics
- Architecture: the backends, the package layout and the driver flow.
- The Pass Pipeline: the pass infrastructure that the host-language passes run on, and the registry.
- Core IR and Code IR: the Exec stage, which the host languages consume.
- Package Layering: the rule that generated parsers must not link the VM or the engine.
- Code Generation: the optimizations and the profiles, with their defaults.