Asking for a VFMA with a lane operand emitted a VMLA. NEON's fused multiply-add
has no by-element form at all -- llvm-mc rejects `vfma.f32 d0, d1, d2[0]` -- and
the four rows sitting under VFMA/VFMS held VMLA/VMLS's lane encodings, which
VMLA and VMLS already own. Their data type gave it away too: `.I32` on a
fused multiply-add, which is float-only. Deleted.
They were also the reason `bits & ~mask` looked wrong on those rows: as
authored they were F2A000C0, and clearing the Vn high bit -- correct, since
bit 7 is the top of the register number -- landed them exactly on VMLA's
F2A00040.
Found by asking which (bits, mask, mode) triples more than one mnemonic
claims. That check found 18; this commit takes it to 9.
Three of the eighteen were whole mnemonics duplicating a base:
VRECPE_F, VRSQRTE_F every form already present under VRECPE / VRSQRTE,
which carry both the U32 and F32 variants.
VPADD_F its F32 form duplicated VPADD's; its F16 form was the
only thing it owned, so that moves to VPADD, where the
data type now selects it.
All three were on the list of names no assembler spells, so that count goes
from 14 to 11 -- and nine of the remaining eleven are the *_LANE group, still
waiting on register-list and lane-index modelling. The other two are
`psb csync` and `tsb csync`, which are correct as they are.
Two smoke checks asserted the VFMA/VFMS by-element forms and are gone with
them; a third moved index.
Verified against llvm-mc: vpadd.i8/.f32/.f16, vrecpe.u32/.f32 and vrsqrte.f32
all byte-exact, 1656/1656 decode sweep, every suite at baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapsing the encoding-shaped mnemonics into their assembler names merged
several form lists that already overlapped, leaving byte-identical Encoding
rows inside one mnemonic. The matcher takes the first form that matches, so
every duplicate was dead weight -- unreachable, and inflating both tables.
VMUL, VMLA and VMLS each gained two from the *_LANE merge; VSHR, VSRA and
VRSHR two each, VQSHRN, VQRSHRN and VSHLL one each.
The branch base had none, so these are mine: the collision check I ran before
each merge compared operand signatures, which is the right test for "can the
matcher tell these apart", but says nothing about two forms being wholly
identical. Comparing the full Encoding would have caught them.
1680 -> 1665 forms. Removing an unreachable exact duplicate cannot change
behaviour, and does not: the load/store, LDM/STM and NEON data-type encode
checks are byte-for-byte what they were, and the decode sweep still round-trips
every form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LDM and STM each held five A32 forms with identical operand shapes -- the four
increment/decrement orders plus a writeback variant -- distinguished only by
their fixed bits. Nothing could tell them apart, so the encoder always took the
first and six of the eight A32 encodings were unreachable: `ldmib`, `ldmda`,
`ldmdb`, `stmib`, `stmda`, `stmdb` could not be produced at all.
They are not variants of one mnemonic in the first place. An assembler spells
them `ldmib` / `ldmda` / `ldmdb`, with plain `ldm` meaning IA, so this follows
the same rule as the rest of the enum: one member per name an assembler
accepts. LDM/STM keep the IA order and the Thumb encodings; the other three
orders become their own mnemonics, and the T32 DB encodings join them.
All eight now encode, byte-exact against llvm-mc:
ldm e8900006 stm e8800006
ldmib e9900006 stmib e9800006
ldmda e8100006 stmda e8000006
ldmdb e9100006 stmdb e9000006
Six test checks referenced these forms by index; they were re-derived by
matching (bits, mask) against the rebuilt table rather than by adjusting
offsets, and every one was found -- so no form was lost in the move.
Writeback (`ldm r0!, {...}`) is still unreachable: it is a property of the base
operand, not a separate mnemonic, and there is nowhere to put it yet. That is
one form per family rather than four.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The encoder builds a word by ORing packed operand fields onto the form's
`bits`. It can only ever set a bit that way, never clear one -- so any bit
`bits` presets that an operand is supposed to drive is stuck at 1 forever.
72 forms did that, and two families show what it cost:
* The U bit (23) on the whole A32 load/store family. U selects add vs
subtract for the displacement, and the encoder derives it from the sign of
mem.disp -- but every form had it preset, so `ldr r0, [r1, #-4]` silently
encoded as `[r1, #4]`. Every negative displacement in the family was wrong.
* The Vn high bit (7) on the NEON lane-indexed forms. That bit is the top of
the register number, so presetting it meant Vn could only ever name
d16..d31; d0..d15 were unreachable.
Which bits are operand-driven was decided by llvm-mc rather than by reading
the manual: for each of the 230 bits a form preset outside its mask, take the
form's canonical word with the bit set and cleared and disassemble both. Same
mnemonic, different operands means the bit belongs to an operand (clear it);
a different mnemonic, or an undecodable word, means the bit is genuinely fixed
for that form. The split was not per-bit -- bit 7 is a register bit for
VMUL/VMLA/VFMA but distinguishes VNEG from VABS and VCMPE from VCMP, and bit
23 is the U bit for LDR but the load/store select for VCX3 -- so every form
was classified individually.
Ten test expectations asserted the old values and were corrected; each had the
bug baked in. Verified byte-exact against llvm-mc across the load/store family
including every negative-displacement form, and the 1680/1680 decode sweep and
all other suites are unchanged.
The other half of `bits & ~mask != 0` -- 130 forms where the bit really is
fixed and the MASK is merely too loose -- is deliberately not in this commit.
Widening those masks alone breaks decode: a bit that distinguishes two
mnemonics has to be added to BOTH forms' masks in the same pass, and doing
only the ones that set it made LSL swallow MOVS, CX3 swallow VADDLVA and VABAV
swallow VRMLSLDAVH. That needs each form's true mask derived empirically
(vary the operands, see which bits move) the way specgen does it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SHIFT_NAMES holds five entries, LSL..RRX, but Shift_Type has ten: the four
register-shifted-register markers (LSL_REG = 6 .. ROR_REG = 9) say the shift
count comes from an Rs register rather than an immediate. Both places that
indexed the table used the raw enum value, and the guard in front of them only
excluded NONE and RRX -- so any operand carrying a register shift indexed a
5-entry array with 6..9 and killed the printer:
printer.odin(473:45) Index 6 is out of range 0..<5
Fold the register-shifted variants back onto the table and give each spelling
its own case: `, lsl #3` for an immediate amount, `, lsl r3` when the count is
in a register (Rs index rides in shift_amt), and a bare `, rrx`, which takes no
amount. All nine now print what an assembler accepts -- verified against
llvm-mc -- where three of them previously crashed and RRX printed nothing.
The memory-operand site indexed the same table the same way and is routed
through the same helper.
Found by printing every entry in the decode table; that sweep now completes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shrinking Instruction to 48 was the wrong call, and measuring it said so.
The premise was that a sub-cache-line struct touches fewer lines. It does --
but `#packed` aligns the struct to 1, so a 48-byte stride straddles a line
boundary 75% of the time, and the heap base is not line-aligned either. The
old 64-byte packed layout was worse still: 100% straddling, getting none of
the benefit its size implied.
Measured on an i7-9750H (L1d 32K/core, L2 256K, L3 12M), best-of-5, median of
3 interleaved rounds, against the 64-byte packed layout this branch started
from:
scan encode decode
64 packed (was) 1.000x 1.000x 1.000x
48 packed 0.909x 1.040x 1.022x
64 align(64) 1.218x 1.034x 0.806x
Decode is ~19% faster aligned, and that holds at every working set including
ones that fit entirely in L1 -- so it is split-store cost at the store ports,
not cache-line fetches. Decode writes whole Instructions, and the aligned
stores are worth more than the 33% extra bytes they move. A fourth variant --
48 bytes with `#packed` removed -- was measured to rule out the obvious
confound, and tracked 48-packed within 0.5% everywhere, so the win is
alignment and not the loss of packing.
Encode is within a few percent throughout (it is compute-bound; the form scan
dominates), and the pure read traversal is slower, but that is a synthetic
loop and its regression is codegen, not cache -- it is present even at
L1-resident sizes where a standalone struct shows no such penalty.
The Operand and Memory work from the previous commit is what makes this
possible: a 45/48-byte payload now sits inside one line with room to spare,
where the original spent all 64 bytes. The 16-19 spare bytes cost nothing over
a straddling 48-byte struct and give new fields somewhere to land.
All 11 rexcode suites match baseline; arm64 is 73/73 byte-exact against
llvm-mc; arm32's 1680/1680 sweep passes and its encode spot-checks are
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
x86 keeps Instruction at 64 bytes -- one cache line -- by packing its memory
operand into a bit_field u64 rather than a struct. arm32 and arm64 both used
a 12-byte Memory struct, and since Memory sits in every Operand that width is
multiplied by four in every Instruction. Adopting x86's trick, plus two
smaller things, takes both ARM ISAs under the cache line.
Instruction ops[4] Operand Memory
x86 64 48 12 8
arm32 88 -> 48 72->40 18->10 12->8
arm64 64 -> 48 56->40 14->10 12->8
Memory -> bit_field u64, both ISAs. Field syntax and composite literals are
unchanged, so callers see nothing. Registers keep their type: an arm64
Register never exceeds 0x0C1F and an arm32 one never exceeds 0x401F, but the
arm64 NONE sentinel is 0xFFFF, so arm64 gives them the full 16 bits and arm32
15. What is left goes to `disp`: 23 bits on arm64 (worst case 65,520, from
LDR Q, [Xn, #imm12*16]) and 19 on arm32 (worst case 4,095, an A32 imm12) --
64x and 32x headroom respectively.
arm32 Operand also carried four tail bytes arm64 does not. `cond` was dead:
nine builders wrote it and nothing in the package ever read it, and
Instruction.cond already exists. shift_type/shift_amt/lane now ride inside
the union alongside the register they describe -- they only ever apply to a
register operand -- via a `using` bit_field, so op.reg, op.shift_type,
op.shift_amt and op.lane still read and write exactly as before.
arm32 Instruction packs cond, operand_count, mode, length and the two flag
bits into one 16-bit word; they need 13 bits between them and were spending
six bytes. `using` again keeps the field names, with the one exception that
inst.flags.sets_flags is now inst.sets_flags (five call sites).
Verified: every rexcode suite matches baseline; both generators stay
idempotent; arm64 is 73/73 byte-exact against llvm-mc on both encode and
decode round-trip; arm32's 1680/1680 sweep still passes and a memory/shift
encode spot-check is byte-identical to what the same code produced before
this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NEON reuses one operand shape across every element width, so `vadd.i8` and
`vadd.f32` are both DPR,DPR,DPR and only the type separates their encodings.
The type existed nowhere in the data: the encoder could reach the first form
of a shape and no other, and the printer reconstructed a suffix from the bit
pattern at print time. 489 of 1680 forms -- 29% of the table -- were
unreachable, and `inst_vadd(d0,d1,d2)` could only ever produce VFP vadd.f64.
Add `Data_Type` and carry `dt: [2]Data_Type` on Instruction, Encoding and
Decode_Entry. Two slots because the convert family names both ends
(`vcvt.s32.f32`); everything else leaves the second .NONE. In A64 the
arrangement belongs to each operand (`add v0.4s, v1.4s, v2.4s`); in A32 it
belongs to the instruction, which is why it goes here and not on Operand.
Instruction does not grow: it lands in bytes that were already padding, so
88 stays 88. Encoding and Decode_Entry go 21 -> 23, which is +3,360 B per
table, +6.7 KB in all.
The per-form type is derived from llvm-mc rather than hand-written: assemble
each form's canonical word, disassemble it, take the suffix. 942 forms carry
one, 38 carry two. (`.w` is the Thumb wide qualifier, not a type, and is
excluded.)
Effect: of 202 shape groups holding more than one form, 168 are now separated
by the type -- 429 of the 489 unreachable forms become selectable. `dt` left
at .NONE means "unspecified" and still takes the first matching form, so
every existing caller behaves exactly as before.
It also fixes printing. The old inference could only ever produce one type,
so the whole convert family printed `vcvt.f32` -- 13 forms sharing one string
that no assembler accepts. They now print `vcvt.f32.s32`, `vcvt.f64.f32`,
`vcvta.u32.f64`, and so on.
Verified: vadd.i8/i16/i32/i64/f32 encode to f2010802 / f2110802 / f2210802 /
f2310802 / f2010d02, matching llvm-mc exactly; all 11 rexcode suites are
identical to baseline.
Still unreachable, 60 forms in 34 groups: register lists (VLD2-4/VST2-4),
LDM/STM addressing modes, and a few lane-indexed and fixed-point convert
forms whose element size is not captured by the type alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of the 38 mnemonics still carrying an encoding-shaped name, 24 were simply
names no assembler accepts, and arm32's printer emits the enum name verbatim
-- so `vldrb_gather`, `vceq_z`, `vmov_q_r` and friends were the printed
output. Judged against llvm-mc in every case:
renamed (base name was free)
BFI_BR -> BFX the V8.1M Branch Future indeXed, not a bitfield
insert; llvm assembles `bfx .L, r0` to F060E001,
which is exactly the bit pattern this entry held.
VDOT_BF16 -> VDOT `vdot.bf16 d0, d1, d2`
VMMLA_BF16 -> VMMLA `vmmla.bf16 q0, q1, q2`
merged into the base mnemonic (21)
VCEQ_Z/VCGE_Z/VCGT_Z/VCLE_Z/VCLT_Z -> the compare-against-zero forms
are the same mnemonic with a literal `#0`: `vceq.i8 d0, d1, #0`.
VCVT_FIXED, VCVT_BF16 -> VCVT `vcvt.s16.f32 s0, s0, #4`
VFMA_BF16 -> VFMA
VLDR{B,H,W,D}_GATHER, VSTR{B,H,W,D}_SCATTER -> VLDR*/VSTR*: an MVE
gather is spelled `vldrb.u8 q0, [r0, q1]`; the vector offset is an
operand, not part of the mnemonic.
VMOV_Q_R, VMOV_R_Q, VMOV_2GPR_Q -> VMOV
VHCADD_SAT -> VHCADD, VCMLA_MVE -> VCMLA
kept, but printed properly (2)
PSB_CSYNC / TSB_CSYNC are written as two tokens, `psb csync`, the same
shape as arm64's DC/AT/TLBI. The underscore now prints as a space; no
other arm32 mnemonic has one.
Every merged form had an operand signature the matcher could already tell
apart from the base's, so nothing became unreachable. 631 -> 590 mnemonics,
underscore-bearing names 58 -> 14.
Test indices were re-derived by matching (bits, mask) against the rebuilt
table rather than by computing offsets -- every index the tests reference was
found, which is a check that the merge dropped no form.
Still blocked, and for the two reasons already known:
VPADD_F, VRECPE_F, VRSQRTE_F -- collide with their base because the NEON
data type (.f32 vs .i8/.u32) is not an operand.
VMOV_LANE, VLD1-4_LANE, VST1-4_LANE -- register lists and lane indices are
not modelled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arm64 pass turned up the same class of bug elsewhere: mnemonics named
after an encoding rather than after what an assembler accepts, and forms
that no caller can reach because the thing that tells them apart is not
checked.
mips
* The printer mapped every `_` to `.`, but MSA spells the sign qualifier
with an underscore and only the element size with a dot: `adds_s.b`,
`max_s.h`, `copy_u.w`. `adds.s.b` is rejected by an assembler. 91
mnemonics were printing text that would not reassemble. The name alone
cannot decide it -- MSA's ADDS_S_D and the FP convert CVT_S_D have the
same shape and want opposite treatment -- so the family is read off the
form's feature.
* `encode` now takes `features: Feature_Set = FEATURES_ALL` and skips
forms outside it, mirroring `decode`, which has had that parameter all
along. That asymmetry was the reason 12 mnemonics carried an ISA-variant
suffix: with no way to say which MIPS you were targeting, the pre-R6 and
R6 encodings of `mul` had to be two enum members. They are now one
mnemonic with two forms. Eight of the twelve did not even need the
feature filter -- pre-R6 MADD takes rs,rt while the PS2 MMI MADD takes
rd,rs,rt, so operand matching alone separates them. Verified against
llvm-mc: pre-R6 `mul` 712a4002, R6 `mul` 012a4098, `madd $t1,$t2`
712a0000. The printer's hand-written override table is gone.
arm32
* 20 `*_LANE` mnemonics folded into their base. The lane form differs from
the base in an operand TYPE already (DPR_ELEM vs DPR), so the matcher
could always tell them apart; the split only cost us the printed name,
which was the enum name verbatim -- `vqdmulh_lane`, which no assembler
takes. VMOV/VLD1-4/VST1-4 are left alone: their lane forms collide with
the base because register lists and lane indices are not modelled.
riscv
* ZEXT_H and REV8 each carry an RV32 and an RV64 encoding with identical
operands, and the forms were already tagged rv32_only / rv64_only -- the
encoder just never looked. `encode` now takes `xlen: XLEN = .RV64` and
filters, so the RV64 encodings are reachable at all: zext.h 0805c53b and
rev8 6b85d513, both confirmed against llvm-mc.
mos6502
* SAX_NMOS folded into SAX. The undocumented NMOS store-A&X and the
HuC6280 register swap share the mnemonic `sax`; one takes a memory
operand and the other takes none, so they are just two form sets.
Verified: every rexcode suite matches HEAD exactly, all 13 packages build,
and MIPS mnemonics llvm-mc does not recognise drop from 448 to 354.
Still open: arm32 has 201 form signatures no caller can select, because the
NEON data type (.i8/.i16/.f32) is not an operand -- `inst_vadd(d0,d1,d2)`
always yields the first form, and only a decoder-supplied form_id hint can
pick another. 38 arm32 mnemonics still carry encoding-shaped names
(VPADD_F, VCEQ_Z, VLDRB_GATHER, VMOV_Q_R, ...).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Internal label ids are allocation-order handles: the encoder's creation
order, or the decoder's branch-DISCOVERY order (a loop's latch names the
header before an earlier forward target). Printing labels by raw id leaked
that accident into listings — label numbers appeared out of order down the
page — and the printers' `label_names: ^map[u32]string` keyed the caller's
names by those synthesized ids, which a decode consumer cannot know without
re-deriving them (the practical result: naming "label 0" could caption a
random interior branch target).
Naming is now derived at the presentation seam, shared by every ISA
(`isa.Label_Display` in isa/print.odin):
- display numbers are assigned in ASCENDING ADDRESS order, so a listing
reads L0, L1, L2 … top to bottom regardless of id allocation;
- caller names are keyed by BYTE OFFSET (`isa.Label_Names`, with a
`distinct` Label_Offset key so an id-keyed map from the old contract
fails to compile instead of silently mis-naming);
- a named offset is guaranteed a label row even when no Label_Definition
points at it — `names[0] = "factorial"` heads a function's listing.
All ten ISA printers (x86, mips, rsp, arm32, arm64, riscv, ppc, ppc_vle,
mos6502, mos65816) drop their per-printer offset_to_label maps and
write_label helpers for the shared display; each arch re-exports
Label_Offset/Label_Names beside Label_Definition. En route this fixes an
arm32/ppc/ppc_vle bug where passing ANY names map suppressed the default
L<n> label rows for unnamed labels. Decode-side id assignment is untouched:
the reloc round-trip contract (encoder ids surviving decode) and the
sparse-id padding it relies on stay exactly as they were.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Riok9vMpkLmo78wsVKJHhz
Roll the encode/decode buffer-sizing helpers (added for x86 in 49787b7de) out
to every other ISA, and document them in the cross-arch naming contract.
Per arch (arm32, arm64, mips, riscv, ppc, ppc_vle, rsp, mos6502, mos65816):
- encode_max_code_size / encode_max_relocation_count now key off the
[]Instruction slice (were int counts); bodies unchanged (* MAX_INST_SIZE).
- encode_reserve(code, relocs, instructions): grows the caller's code []u8 by
length and reserves relocs by capacity; allocates no new buffers.
- decode_max_instruction_count / decode_estimate_instruction_count: exact
ceiling and typical estimate, keyed off the min/avg instruction size per
arch (fixed-4: arm64/mips/ppc/rsp; min-2: arm32/riscv/ppc_vle; min-1: mos).
- decode_reserve(instructions, inst_info, label_defs, data, exact=false).
docs/cross_arch_design.md: helpers added to the naming contract.
No behavior change to the existing size helpers (signature only). All 10 ISAs
check + test green (x86 2282, arm32 600, arm64 461, mips 281, riscv 154, ppc 31,
ppc_vle 281, rsp 70, mos6502 148, mos65816 53).
Move all ten ISA packages (x86, arm32, arm64, mips, riscv, ppc, ppc_vle,
rsp, mos6502, mos65816) from core/rexcode/<arch> to core/rexcode/isa/<arch>,
so the import pattern is now `import "core:rexcode/isa/x86"`. The shared
core stays at core:rexcode/isa.
Mechanical: relative `import "../isa"` / "../../isa" -> absolute
"core:rexcode/isa" (the only path that survives the move; the "../" and
"../.." self/generated imports move with their packages). build.lua now
builds paths as <root>/isa/<name>; stale `cd <arch>` hints in the verify
tools and the doc.odin paths updated.
WASM stays at core/rexcode/wasm for now -- it is an IR, not an ISA, and
will move under the forthcoming core:rexcode/ir once that layer lands.
All 10 arches gen/builders/check/test green; import core:rexcode/isa/x86
verified working; wasm still compiles.
