* For `-o:size` and below, uses the type erased approach
* For `-o:speed` and above, the inlined form is used
This is necessary because a generic `mem_copy_non_overlapping` cannot be optimized when type erasure is used, meaning in a hot path where `append_elem` is used a lot; thus `mem_copy_non_overlapping` becomes a bottleneck.
Previously one would get a nice `$ pid segmentation fault (core dumped) odin run .` message,
but this behaviour was (probably unintentionally) removed in #6848.
The matcher path (generic builders, hand-built, decode->re-encode) resolves an
instruction to its encoding form by linearly scanning the forms for its mnemonic
and operand-matching each -- the dominant cost on that path. Memoize it: pack
(mnemonic, per-operand shape) into a key (immediates folded to the smallest size
class they fit, matching imm_matches_inline) and cache key -> form so a repeated
instruction shape skips the scan.
Direct-mapped, fixed 8192-slot table (64 KB, no allocation). Each slot packs the
full 48-bit key and form index into one u64, read/written with relaxed atomics,
so concurrent encode() stays safe -- a reader sees a matching key or rescans,
never a torn entry. The scan stays the source of truth (a miss runs it and
records the result), so the cache is exact.
Lookup + scan live in a non-inlined find_form() so they don't bloat encode()'s
hot loop and slow the hint path that shares it. (Routing the matcher path through
the recipe emit was tried and dropped: it costs the hint path ~1.2-1.5 ns however
isolated -- the hot loop is too codegen-sensitive -- while the cache alone is
free for the hint path.)
Realistic generic-builder mix: matcher ~52 -> ~35 ns/inst (~1.49x); hint path
unchanged. Byte-exact across 2282 + idempotent.
Extend emit_recipe to the full ModR/M + SIB + displacement addressing (register
direct, RIP-relative, absolute [disp32], and base/index/scale/disp), mirroring
the interpreter byte-for-byte, and drop the caller's reg-direct guard so memory
operands take the fast path too. Only a label/relative immediate (a relocation)
still falls back.
Realistic immediate-heavy mix: ~20.1 -> ~12.9 ns/inst vs the pre-recipe base
(~1.55x, 50 -> 77 M/s). Byte-exact across 2282 + idempotent.
Precompute each encoding form into a flat Form_Recipe -- prefix byte, escape+
opcode blob, role->operand-index slots, ext, imm size, flags -- so the encoder
replays common forms straight-line instead of re-interpreting enc.ops/enc.enc
on every instruction (the resolve scan, escape ladder, prefix/REX selection).
encode() takes the fast path when the form is hinted, eligible, has a register
r/m and a literal immediate; everything else falls through to the existing
interpreter, which stays the byte-exact source of truth. First cut:
- reg-direct ModR/M only (memory r/m falls back)
- hint path only (matcher / generic builders fall back)
- ~33% of forms eligible (VEX/EVEX, 16-bit operand-size, x87 fixed-ModR/M,
moffs/far/rel/implicit operands are marked ineligible)
Recipes are built at startup into static storage (no heap); this moves into the
table generator (#loaded like every other table) once the shape settles.
Realistic immediate-heavy mix: ~19.0 -> ~16.3 ns/inst (52.7 -> 61.3 M/s).
Byte-exact across 2282 cases + idempotent.
Next: memory r/m addressing in the fast path, then the matcher path, then the
gen-time port.
when iterating the values of an array literal,if one of the values
is a function with multiple returns, it doesn't offset the struct
field index by the number of values.
* For `-o:size` and below, uses the type erased approach
* For `-o:speed` and above, the inlined form is used
This is necessary because a generic `mem_copy_non_overlapping` cannot be optimized when type erasure is used, meaning in a hot path where `append_elem` is used a lot; thus `mem_copy_non_overlapping` becomes a bottleneck.
Roll the encode/decode buffer-sizing helpers (added for x86 in 49787b7de) out
to every other ISA, and document them in the cross-arch naming contract.
Per arch (arm32, arm64, mips, riscv, ppc, ppc_vle, rsp, mos6502, mos65816):
- encode_max_code_size / encode_max_relocation_count now key off the
[]Instruction slice (were int counts); bodies unchanged (* MAX_INST_SIZE).
- encode_reserve(code, relocs, instructions): grows the caller's code []u8 by
length and reserves relocs by capacity; allocates no new buffers.
- decode_max_instruction_count / decode_estimate_instruction_count: exact
ceiling and typical estimate, keyed off the min/avg instruction size per
arch (fixed-4: arm64/mips/ppc/rsp; min-2: arm32/riscv/ppc_vle; min-1: mos).
- decode_reserve(instructions, inst_info, label_defs, data, exact=false).
docs/cross_arch_design.md: helpers added to the naming contract.
No behavior change to the existing size helpers (signature only). All 10 ISAs
check + test green (x86 2282, arm32 600, arm64 461, mips 281, riscv 154, ppc 31,
ppc_vle 281, rsp 70, mos6502 148, mos65816 53).
Give callers a clean way to pre-size their own buffers so the encode/decode
hot paths never allocate or resize, instead of decode() silently reserving the
caller's arrays itself (removed). The library allocates nothing -- these only
grow the caller's own dynamic arrays, and only when not already big enough
(Odin's reserve no-ops when capacity already suffices).
Size-only helpers (caller manages its own memory), keyed off the input slice:
encode_max_code_size(instructions) - exact code bytes
encode_max_relocation_count(instructions) - exact reloc upper bound
decode_max_instruction_count(data) - exact ceiling (1 byte/inst)
decode_estimate_instruction_count(data) - typical estimate (~3 B/inst)
Reserve helpers (pre-size the caller's dynamic arrays; nil to skip an array):
encode_reserve(code, relocs, instructions)
code is a [dynamic]u8 grown by LENGTH (so code[:] is a valid emit
target); relocs reserved by capacity on top of existing elements.
decode_reserve(instructions, inst_info, label_defs, data, exact=false)
reserves capacity on top of existing; exact=true for the ceiling.
Error arrays grow only on the failure path, so they are intentionally not
covered. check/test green; 2282 cases; exercised end-to-end (the [dynamic]u8
code pattern, factor-in-existing, nil args, exact ceiling, reserve no-op).
Data-oriented pass on the encode hot path. Profiling showed bounds checks
already elided by -o:speed; the cost was per-instruction loop/scan machinery
and immediates falling off the hint path.
- Gather the immediate slot in the single resolve pass and emit it straight-
line (no scan over enc.enc); likewise drive the legacy REX prefix from the
precomputed reg/mr/opr slots instead of a per-form scan.
- Fold the separate needs_66 (GPR16) and SPL/BPL/SIL/DIL operand loops into
the resolve pass, so user operands are visited exactly once. This was the
big one: mov r,r 27 -> 21 ns.
- Gate the whole legacy-prefix block on a single flags!=0 test (a legacy
prefix is almost always absent) instead of four branches per instruction.
- Make immediate forms hintable. A typed immediate builder names its width
(inst_add_r32_imm32), the matcher already keys off the operand's declared
size, so baking the form is byte-identical AND drops immediates from the
full match scan: mov r32,imm32 55.7 -> 17.8 ns (3.1x).
Floor (no-op) 14.55 -> 10.3 ns; realistic immediate-heavy typed mix
30 -> 20.5 ns/inst (~49 M inst/s). gen/builders/check/test/idempotent green;
2282 cases (typed==generic byte-identical, incl. the new immediate cases).
The broad VS-style `x86/` build-artifact rule was shadowing the re-housed
core/rexcode/isa/x86 source package (new files needed `git add -f`), and the
stale `core/rexcode/*/tables/*.bin` un-ignore no longer matched the deeper
isa/<arch>/tables path. Retarget both to isa/: allow the directory tree and
every arch's committed table blobs, while stray *.bin/*.obj/*.exe build
output inside it stays ignored.
Targeted branchless revert + the pre-matched form fast path, and a fix
for a pre-existing bug the latter surfaced.
(a) Revert the two speculative-write spots from the prior branchless pass
(legacy-prefix emission, widened displacement store, ENCODE_TAIL_SLACK)
back to predicted branches. In real streams a legacy prefix is almost
always absent and disp size is stable, so those branches are ~free and
the unconditional stores only added work. Every class got faster
(RET 19->17.5, MOV r,r 52->46.6, VADDPS 42.8->39.3 ns).
(b) Pre-matched form hint. Instruction.enc_hint (in the existing 11-byte
padding, idx+1 biased; 0 = matcher path) lets a typed builder that maps
to a single value-independent form bake the global form index, so
encode() skips the O(forms) match scan -- and, in a varied stream, its
unpredictable branches. Generated for non-immediate forms only (value-
dependent imm8/imm32 selection stays on the matcher). On a 100k mixed
typed-builder stream: 47.3 -> 30.2 ns/inst (-36%), byte-identical to the
matcher path -- ~2x the original baseline for codegen.
Repair the typed inst_/emit_ builders. They were non-functional: the
generator cast the hw-only typed enum straight to Register
(Register(GPR64.RAX) -> class 0), so every typed-builder operand was
rejected by the matcher (encode returned empty). Untested because the
suite builds via the generic constructors. Now they build through the
class-correct op_gpr64/op_xmm/... path (op_* already used by 3+ operand
builders), emit_ reuses inst_, and a new 30-case consistency suite
asserts typed == generic (llvm-verified) and hint == matcher.
gen/builders/check/test/idempotent all green; 2276 cases.
Three layers on the x86 encode/decode hot paths, all byte-exact (2246
LLVM-verified cases) and roundtrip-clean:
1. Branchless: legacy-prefix emission (speculative write + conditional
advance), REX/VEX/EVEX extension-bit accumulation (gate-and-mask),
ModRM mod/disp-size selection (cmov selects), displacement emission
(widened store + ENCODE_TAIL_SLACK); decoder REX/VEX/EVEX register
extensions (arithmetic instead of if/+=8).
2. Resolve-operands-once: the previous code re-derived each user operand
~5-10x per instruction (a fresh O(n) scan of enc.ops per emission
pass). Now resolved into a [4]^Operand map a single time.
3. Single-pass gather: fold the opcode-+rb and ModR/M slot-detection
scans into that one resolve pass (3 enc.enc passes -> 1).
Net on a 100k mixed-instruction benchmark: encode ~58 -> ~54 ns/inst
(best 52). Branchless alone was a ~7% encode regression (predicted
branches, nothing to recover); the algorithmic passes recovered it and
beat baseline.
A sibling to core:rexcode/isa for the intermediate representations (WASM,
SPIR-V, LLVM bitcode + the LLVM dialects AIR/DXIL). Holds the shared
vocabulary every IR package builds on, implements no specific IR.
Design stance (see docs/ir_design.md): keep the ISA layer's spirit, but
where IRs are structurally MORE uniform than ISAs (SSA + a type system
regularize the operand/module shape), the shared core is richer. ir/ owns:
status.odin Error/Error_Code (shape-identical to isa.Error)
refs.odin Id/Ref/Ref_Space/Symbol_Table (the label analog: structural
id references, not PC-relative byte offsets)
types.odin Type/Type_Ref/Type_Kind (the type table -- no ISA analog)
module.odin Module/Function/Block/Operation/Operand/Result/Dataflow
(the structured model; Operation = isa.Instruction + an
optional typed Result, opcode a u16 like Mnemonic)
print.odin token kinds + options + num-fmt (parallels isa.print)
Three honest concessions vs the ISA API, made explicit not inert: a
structured Module replaces the flat []Instruction; a first-class type
system; id-based entity refs replace labels. The encode/decode verbs take
a Module and drop label_defs/resolve/base_address. Dataflow hosts both the
WASM value stack and SSA; the codec is pluggable (table for WASM/SPIR-V,
bitstream for the LLVM family -- AIR/DXIL are LLVM dialects, not peers).
Package compiles; a hand-built SSA module round-trips through the types.
Move all ten ISA packages (x86, arm32, arm64, mips, riscv, ppc, ppc_vle,
rsp, mos6502, mos65816) from core/rexcode/<arch> to core/rexcode/isa/<arch>,
so the import pattern is now `import "core:rexcode/isa/x86"`. The shared
core stays at core:rexcode/isa.
Mechanical: relative `import "../isa"` / "../../isa" -> absolute
"core:rexcode/isa" (the only path that survives the move; the "../" and
"../.." self/generated imports move with their packages). build.lua now
builds paths as <root>/isa/<name>; stale `cd <arch>` hints in the verify
tools and the doc.odin paths updated.
WASM stays at core/rexcode/wasm for now -- it is an IR, not an ISA, and
will move under the forthcoming core:rexcode/ir once that layer lands.
All 10 arches gen/builders/check/test green; import core:rexcode/isa/x86
verified working; wasm still compiles.
