Position Units
Every position that SyntaxIR reports is a count in one of two units. This guide shows what the unit changes, how to choose it and what invalid UTF-8 does.
The examples use pairs.sxir and a parser p compiled as in Running Parsers:
// pairs.sxir
type Pair = { Key node[string]; Value node[string] }
@start
rule main = ws ps:pair+ $$ => $ps
rule pair Pair = k:key "=" v:value ";"? ws => Pair{Key: $k, Value: $v}
rule key = [a-zあ-ん]+ #atomic
rule value = [0-9]+ #atomic
rule ws = [ \t\n]*
Positions can count Unicode code points (the default) or UTF-8 bytes. The unit affects node Start() and End(),
the Pos and Col of syntax errors, .Start and .End in actions, the len builtin, and the ranges of
Document.Edit. It never changes what matches: matching always proceeds by code point.
n, _ := p.Parse("あい=3")
fmt.Println(n.Elem(0).Field("Key").(*value.Node).End()) // 2
n, _ = p.Parse("あい=3", runtime.WithUnit(input.Bytes))
fmt.Println(n.Elem(0).Field("Key").(*value.Node).End()) // 6
Syntax errors follow the unit, too (the same あい=x input, with the error at the x):
sxir parse -g pairs.sxir -i 'あい=x'
sxir: 1:4: syntax error: expected [0-9]
sxir parse -g pairs.sxir -unit bytes -i 'あい=x'
sxir: 1:8: syntax error: expected [0-9]
Choosing:
- Use bytes when you index into the Go string (
input[n.Start():n.End()]), pass positions to other byte-oriented tools, or want to avoid converting offsets. Bytes is the natural unit for Go. - Use code points (the default) when positions are shown to people as character counts, or are exchanged with a system that counts characters (editors and language servers often count UTF-16 units, which neither unit matches).
- Invalid UTF-8 is read as one U+FFFD character per invalid byte, so positions stay well defined; the decoding rules and their effect on the text of string nodes are in Positions.
The units differ by a few percent: generated parsers run 3–8% faster with code points, which decode the input once,
while the closure engine streams faster with bytes, and code points cost about 8 bytes of memory per input character.
The position unit measurement has the figures, and the
benchmarks compare the backends in bytes.
The CLI flag is -unit codepoints|bytes.
Related Topics
- Running Parsers: compiling and calling a parser.
- Reading Trees: reading
StartandEndof a node. - Positions: the definition of units and decoding.