LDM and STM each held five A32 forms with identical operand shapes -- the four
increment/decrement orders plus a writeback variant -- distinguished only by
their fixed bits. Nothing could tell them apart, so the encoder always took the
first and six of the eight A32 encodings were unreachable: `ldmib`, `ldmda`,
`ldmdb`, `stmib`, `stmda`, `stmdb` could not be produced at all.
They are not variants of one mnemonic in the first place. An assembler spells
them `ldmib` / `ldmda` / `ldmdb`, with plain `ldm` meaning IA, so this follows
the same rule as the rest of the enum: one member per name an assembler
accepts. LDM/STM keep the IA order and the Thumb encodings; the other three
orders become their own mnemonics, and the T32 DB encodings join them.
All eight now encode, byte-exact against llvm-mc:
ldm e8900006 stm e8800006
ldmib e9900006 stmib e9800006
ldmda e8100006 stmda e8000006
ldmdb e9100006 stmdb e9000006
Six test checks referenced these forms by index; they were re-derived by
matching (bits, mask) against the rebuilt table rather than by adjusting
offsets, and every one was found -- so no form was lost in the move.
Writeback (`ldm r0!, {...}`) is still unreachable: it is a property of the base
operand, not a separate mnemonic, and there is nowhere to put it yet. That is
one form per family rather than four.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The encoder builds a word by ORing packed operand fields onto the form's
`bits`. It can only ever set a bit that way, never clear one -- so any bit
`bits` presets that an operand is supposed to drive is stuck at 1 forever.
72 forms did that, and two families show what it cost:
* The U bit (23) on the whole A32 load/store family. U selects add vs
subtract for the displacement, and the encoder derives it from the sign of
mem.disp -- but every form had it preset, so `ldr r0, [r1, #-4]` silently
encoded as `[r1, #4]`. Every negative displacement in the family was wrong.
* The Vn high bit (7) on the NEON lane-indexed forms. That bit is the top of
the register number, so presetting it meant Vn could only ever name
d16..d31; d0..d15 were unreachable.
Which bits are operand-driven was decided by llvm-mc rather than by reading
the manual: for each of the 230 bits a form preset outside its mask, take the
form's canonical word with the bit set and cleared and disassemble both. Same
mnemonic, different operands means the bit belongs to an operand (clear it);
a different mnemonic, or an undecodable word, means the bit is genuinely fixed
for that form. The split was not per-bit -- bit 7 is a register bit for
VMUL/VMLA/VFMA but distinguishes VNEG from VABS and VCMPE from VCMP, and bit
23 is the U bit for LDR but the load/store select for VCX3 -- so every form
was classified individually.
Ten test expectations asserted the old values and were corrected; each had the
bug baked in. Verified byte-exact against llvm-mc across the load/store family
including every negative-displacement form, and the 1680/1680 decode sweep and
all other suites are unchanged.
The other half of `bits & ~mask != 0` -- 130 forms where the bit really is
fixed and the MASK is merely too loose -- is deliberately not in this commit.
Widening those masks alone breaks decode: a bit that distinguishes two
mnemonics has to be added to BOTH forms' masks in the same pass, and doing
only the ones that set it made LSL swallow MOVS, CX3 swallow VADDLVA and VABAV
swallow VRMLSLDAVH. That needs each form's true mask derived empirically
(vary the operands, see which bits move) the way specgen does it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SHIFT_NAMES holds five entries, LSL..RRX, but Shift_Type has ten: the four
register-shifted-register markers (LSL_REG = 6 .. ROR_REG = 9) say the shift
count comes from an Rs register rather than an immediate. Both places that
indexed the table used the raw enum value, and the guard in front of them only
excluded NONE and RRX -- so any operand carrying a register shift indexed a
5-entry array with 6..9 and killed the printer:
printer.odin(473:45) Index 6 is out of range 0..<5
Fold the register-shifted variants back onto the table and give each spelling
its own case: `, lsl #3` for an immediate amount, `, lsl r3` when the count is
in a register (Rs index rides in shift_amt), and a bare `, rrx`, which takes no
amount. All nine now print what an assembler accepts -- verified against
llvm-mc -- where three of them previously crashed and RRX printed nothing.
The memory-operand site indexed the same table the same way and is routed
through the same helper.
Found by printing every entry in the decode table; that sweep now completes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shrinking Instruction to 48 was the wrong call, and measuring it said so.
The premise was that a sub-cache-line struct touches fewer lines. It does --
but `#packed` aligns the struct to 1, so a 48-byte stride straddles a line
boundary 75% of the time, and the heap base is not line-aligned either. The
old 64-byte packed layout was worse still: 100% straddling, getting none of
the benefit its size implied.
Measured on an i7-9750H (L1d 32K/core, L2 256K, L3 12M), best-of-5, median of
3 interleaved rounds, against the 64-byte packed layout this branch started
from:
scan encode decode
64 packed (was) 1.000x 1.000x 1.000x
48 packed 0.909x 1.040x 1.022x
64 align(64) 1.218x 1.034x 0.806x
Decode is ~19% faster aligned, and that holds at every working set including
ones that fit entirely in L1 -- so it is split-store cost at the store ports,
not cache-line fetches. Decode writes whole Instructions, and the aligned
stores are worth more than the 33% extra bytes they move. A fourth variant --
48 bytes with `#packed` removed -- was measured to rule out the obvious
confound, and tracked 48-packed within 0.5% everywhere, so the win is
alignment and not the loss of packing.
Encode is within a few percent throughout (it is compute-bound; the form scan
dominates), and the pure read traversal is slower, but that is a synthetic
loop and its regression is codegen, not cache -- it is present even at
L1-resident sizes where a standalone struct shows no such penalty.
The Operand and Memory work from the previous commit is what makes this
possible: a 45/48-byte payload now sits inside one line with room to spare,
where the original spent all 64 bytes. The 16-19 spare bytes cost nothing over
a straddling 48-byte struct and give new fields somewhere to land.
All 11 rexcode suites match baseline; arm64 is 73/73 byte-exact against
llvm-mc; arm32's 1680/1680 sweep passes and its encode spot-checks are
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
x86 keeps Instruction at 64 bytes -- one cache line -- by packing its memory
operand into a bit_field u64 rather than a struct. arm32 and arm64 both used
a 12-byte Memory struct, and since Memory sits in every Operand that width is
multiplied by four in every Instruction. Adopting x86's trick, plus two
smaller things, takes both ARM ISAs under the cache line.
Instruction ops[4] Operand Memory
x86 64 48 12 8
arm32 88 -> 48 72->40 18->10 12->8
arm64 64 -> 48 56->40 14->10 12->8
Memory -> bit_field u64, both ISAs. Field syntax and composite literals are
unchanged, so callers see nothing. Registers keep their type: an arm64
Register never exceeds 0x0C1F and an arm32 one never exceeds 0x401F, but the
arm64 NONE sentinel is 0xFFFF, so arm64 gives them the full 16 bits and arm32
15. What is left goes to `disp`: 23 bits on arm64 (worst case 65,520, from
LDR Q, [Xn, #imm12*16]) and 19 on arm32 (worst case 4,095, an A32 imm12) --
64x and 32x headroom respectively.
arm32 Operand also carried four tail bytes arm64 does not. `cond` was dead:
nine builders wrote it and nothing in the package ever read it, and
Instruction.cond already exists. shift_type/shift_amt/lane now ride inside
the union alongside the register they describe -- they only ever apply to a
register operand -- via a `using` bit_field, so op.reg, op.shift_type,
op.shift_amt and op.lane still read and write exactly as before.
arm32 Instruction packs cond, operand_count, mode, length and the two flag
bits into one 16-bit word; they need 13 bits between them and were spending
six bytes. `using` again keeps the field names, with the one exception that
inst.flags.sets_flags is now inst.sets_flags (five call sites).
Verified: every rexcode suite matches baseline; both generators stay
idempotent; arm64 is 73/73 byte-exact against llvm-mc on both encode and
decode round-trip; arm32's 1680/1680 sweep still passes and a memory/shift
encode spot-check is byte-identical to what the same code produced before
this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NEON reuses one operand shape across every element width, so `vadd.i8` and
`vadd.f32` are both DPR,DPR,DPR and only the type separates their encodings.
The type existed nowhere in the data: the encoder could reach the first form
of a shape and no other, and the printer reconstructed a suffix from the bit
pattern at print time. 489 of 1680 forms -- 29% of the table -- were
unreachable, and `inst_vadd(d0,d1,d2)` could only ever produce VFP vadd.f64.
Add `Data_Type` and carry `dt: [2]Data_Type` on Instruction, Encoding and
Decode_Entry. Two slots because the convert family names both ends
(`vcvt.s32.f32`); everything else leaves the second .NONE. In A64 the
arrangement belongs to each operand (`add v0.4s, v1.4s, v2.4s`); in A32 it
belongs to the instruction, which is why it goes here and not on Operand.
Instruction does not grow: it lands in bytes that were already padding, so
88 stays 88. Encoding and Decode_Entry go 21 -> 23, which is +3,360 B per
table, +6.7 KB in all.
The per-form type is derived from llvm-mc rather than hand-written: assemble
each form's canonical word, disassemble it, take the suffix. 942 forms carry
one, 38 carry two. (`.w` is the Thumb wide qualifier, not a type, and is
excluded.)
Effect: of 202 shape groups holding more than one form, 168 are now separated
by the type -- 429 of the 489 unreachable forms become selectable. `dt` left
at .NONE means "unspecified" and still takes the first matching form, so
every existing caller behaves exactly as before.
It also fixes printing. The old inference could only ever produce one type,
so the whole convert family printed `vcvt.f32` -- 13 forms sharing one string
that no assembler accepts. They now print `vcvt.f32.s32`, `vcvt.f64.f32`,
`vcvta.u32.f64`, and so on.
Verified: vadd.i8/i16/i32/i64/f32 encode to f2010802 / f2110802 / f2210802 /
f2310802 / f2010d02, matching llvm-mc exactly; all 11 rexcode suites are
identical to baseline.
Still unreachable, 60 forms in 34 groups: register lists (VLD2-4/VST2-4),
LDM/STM addressing modes, and a few lane-indexed and fixed-point convert
forms whose element size is not captured by the type alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of the 38 mnemonics still carrying an encoding-shaped name, 24 were simply
names no assembler accepts, and arm32's printer emits the enum name verbatim
-- so `vldrb_gather`, `vceq_z`, `vmov_q_r` and friends were the printed
output. Judged against llvm-mc in every case:
renamed (base name was free)
BFI_BR -> BFX the V8.1M Branch Future indeXed, not a bitfield
insert; llvm assembles `bfx .L, r0` to F060E001,
which is exactly the bit pattern this entry held.
VDOT_BF16 -> VDOT `vdot.bf16 d0, d1, d2`
VMMLA_BF16 -> VMMLA `vmmla.bf16 q0, q1, q2`
merged into the base mnemonic (21)
VCEQ_Z/VCGE_Z/VCGT_Z/VCLE_Z/VCLT_Z -> the compare-against-zero forms
are the same mnemonic with a literal `#0`: `vceq.i8 d0, d1, #0`.
VCVT_FIXED, VCVT_BF16 -> VCVT `vcvt.s16.f32 s0, s0, #4`
VFMA_BF16 -> VFMA
VLDR{B,H,W,D}_GATHER, VSTR{B,H,W,D}_SCATTER -> VLDR*/VSTR*: an MVE
gather is spelled `vldrb.u8 q0, [r0, q1]`; the vector offset is an
operand, not part of the mnemonic.
VMOV_Q_R, VMOV_R_Q, VMOV_2GPR_Q -> VMOV
VHCADD_SAT -> VHCADD, VCMLA_MVE -> VCMLA
kept, but printed properly (2)
PSB_CSYNC / TSB_CSYNC are written as two tokens, `psb csync`, the same
shape as arm64's DC/AT/TLBI. The underscore now prints as a space; no
other arm32 mnemonic has one.
Every merged form had an operand signature the matcher could already tell
apart from the base's, so nothing became unreachable. 631 -> 590 mnemonics,
underscore-bearing names 58 -> 14.
Test indices were re-derived by matching (bits, mask) against the rebuilt
table rather than by computing offsets -- every index the tests reference was
found, which is a check that the merge dropped no form.
Still blocked, and for the two reasons already known:
VPADD_F, VRECPE_F, VRSQRTE_F -- collide with their base because the NEON
data type (.f32 vs .i8/.u32) is not an operand.
VMOV_LANE, VLD1-4_LANE, VST1-4_LANE -- register lists and lane indices are
not modelled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arm64 pass turned up the same class of bug elsewhere: mnemonics named
after an encoding rather than after what an assembler accepts, and forms
that no caller can reach because the thing that tells them apart is not
checked.
mips
* The printer mapped every `_` to `.`, but MSA spells the sign qualifier
with an underscore and only the element size with a dot: `adds_s.b`,
`max_s.h`, `copy_u.w`. `adds.s.b` is rejected by an assembler. 91
mnemonics were printing text that would not reassemble. The name alone
cannot decide it -- MSA's ADDS_S_D and the FP convert CVT_S_D have the
same shape and want opposite treatment -- so the family is read off the
form's feature.
* `encode` now takes `features: Feature_Set = FEATURES_ALL` and skips
forms outside it, mirroring `decode`, which has had that parameter all
along. That asymmetry was the reason 12 mnemonics carried an ISA-variant
suffix: with no way to say which MIPS you were targeting, the pre-R6 and
R6 encodings of `mul` had to be two enum members. They are now one
mnemonic with two forms. Eight of the twelve did not even need the
feature filter -- pre-R6 MADD takes rs,rt while the PS2 MMI MADD takes
rd,rs,rt, so operand matching alone separates them. Verified against
llvm-mc: pre-R6 `mul` 712a4002, R6 `mul` 012a4098, `madd $t1,$t2`
712a0000. The printer's hand-written override table is gone.
arm32
* 20 `*_LANE` mnemonics folded into their base. The lane form differs from
the base in an operand TYPE already (DPR_ELEM vs DPR), so the matcher
could always tell them apart; the split only cost us the printed name,
which was the enum name verbatim -- `vqdmulh_lane`, which no assembler
takes. VMOV/VLD1-4/VST1-4 are left alone: their lane forms collide with
the base because register lists and lane indices are not modelled.
riscv
* ZEXT_H and REV8 each carry an RV32 and an RV64 encoding with identical
operands, and the forms were already tagged rv32_only / rv64_only -- the
encoder just never looked. `encode` now takes `xlen: XLEN = .RV64` and
filters, so the RV64 encodings are reachable at all: zext.h 0805c53b and
rev8 6b85d513, both confirmed against llvm-mc.
mos6502
* SAX_NMOS folded into SAX. The undocumented NMOS store-A&X and the
HuC6280 register swap share the mnemonic `sax`; one takes a memory
operand and the other takes none, so they are just two form sets.
Verified: every rexcode suite matches HEAD exactly, all 13 packages build,
and MIPS mnemonics llvm-mc does not recognise drop from 448 to 354.
Still open: arm32 has 201 form signatures no caller can select, because the
NEON data type (.i8/.i16/.f32) is not an operand -- `inst_vadd(d0,d1,d2)`
always yields the first form, and only a decoder-supplied form_id hint can
pick another. 38 arm32 mnemonics still carry encoding-shaped names
(VPADD_F, VCEQ_Z, VLDRB_GATHER, VMOV_Q_R, ...).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Encoder
* LSR/ASR by immediate used the generic IMM12 encoding, which writes bits
10-21 -- straight into the imms field the UBFM/SBFM base pattern already
fills, so immr stayed 0 and every shift encoded as #0 (`asr x0,x1,#7`
gave 9340fc20, not 9347fc20). They need immr alone, since imms is the
constant 31/63 fixed in the form: new ENC_SHIFT_IMMR.
* LDP/STP and friends borrowed the single-register addressing encodings,
which put an UNSCALED 9-bit displacement at bits 20:12 and OR a pre/post
marker into bits 11:10. The pair forms want a SCALED 7-bit value at
21:15, and bits 11:10 are part of Rt2 -- so `ldp x0,x1,[x2,#16]!` came
back with Rt2=3. New OFFSET_PAIR_4/8/16 (the scale does not follow from
the register type: LDPSW pairs X registers but loads words, STGP scales
by 16), with the addressing mode read from bits[24:23] where the
architecture keeps it. 26 forms retargeted.
Decoder
* Vector operands came back with size=4 always, so a decoded V register
lost its arrangement and disassembly printed a bare `v0` that no
assembler would take. Reconstruct it from the form's operand type.
* Vd/Vn/Vm/Va hardcoded REG_V, but SVE forms use those same slots with
Z_REG_* operands -- `add z0.d, z0.d, z0.d` decoded as a V register.
Take the class from the operand type, as every other slot already does.
Printer
* V/Z registers now print their arrangement (`add v0.4s, v1.4s, v2.4s`,
`add z0.d, ...`). Element views (op_v_elem_*) moved from 1/2/4/8 to odd
codes 1/3/5/7, because an element-D view and an 8B arrangement were both
size 8 and could not be told apart.
* MOVZ/MOVN/MOVK print the hw index as `lsl #16`, omitted when zero.
* BC_COND folds its condition into the mnemonic like B_COND already did,
instead of printing it twice.
Table (each bit pattern re-derived from llvm-mc)
* BTI_J and BTI_C had each other's encodings.
* FCMLA's mask left size bit 22 free, so .4s and .2d were indistinguishable
and .2d decoded as .4s.
* BFDOT carried the Q=0 pattern for its .4s/.8h form; PMULLB/PMULLT were
missing the size field; TLBI PAALL/PAALLOS had the wrong CRm/op2.
* RDSVL's imm6 sits at bits 10:5, not where IMM6 puts it: ENC_IMM6_LO.
Nine test expectations that asserted the wrong values were corrected.
specgen.lua
Was already dead before the mnemonic work -- it wrote to encoding_table.odin
and spliced a SPECGEN region, neither of which survived the merge into
instruction_table.odin. Retargeted, taught the canonical names, and made it
emit Form literals (Encoding + Clobber). It can no longer own whole
`.MNEM = { ... }` blocks either, since ADD now holds integer, NEON and SVE
forms together, so it MERGES: a form is added only when no (bits, mask)
match exists, and existing rows are never rewritten -- their hand-maintained
Clobber data has to survive a regeneration.
Verified: all 11 rexcode suites match HEAD exactly (arm64 461/461); the three
generator stages stay idempotent; a 73-case differential against llvm-mc is
byte-exact for both encode and decode round-trip. Over the whole decode table,
canonical-form disassembly re-assembled by llvm-mc goes from 594 byte-exact /
1818 unassemblable to 1737 / 678. Re-running specgen re-derives 1130 forms
from llvm-mc and finds every one already present, which independently confirms
those bit patterns.
Still open: multi-vector register lists ({z0.b, z1.b}) and lane indices
(v0.s[2]) are not modelled, so those forms print without them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Mnemonic enum had one member per encoding form -- ADD_IMM, ADD_SR,
ADD_ER, ADD_V for what an assembler just calls ADD; LDR, LDR_LIT, LDR_PRE,
LDR_POST, LDR_REG, LDR_V for LDR; SVE_ADD_Z / SVE_ADD_PRED / SVE_AND_P for
names SVE spells ADD and AND. The encoder never needed that: like x86, it
already resolves a mnemonic by scanning its run of forms and matching
operand types, so the split bought nothing and cost a printer that had to
strip suffixes back off at runtime -- incompletely, so ADD_V printed
"add.v", LDR_PRE "ldr.pre" and FCVT_H_S "fcvt.h.s".
Collapse the enum to the names assemblers accept: 1104 -> 785 mnemonics,
with the variants becoming forms under one name (ADD now has 13, LDR 15).
LSLV/LSRV/ASRV/RORV fold into LSL/LSR/ASR/ROR. Form order within a run is
precedence, and the original declaration order is already the order an
assembler resolves: "add w0,w1,w2" takes the shifted-register form, and
only the extended form can encode SP.
This needed one structural change. The matcher was blind to addressing
mode -- `case .MEM: return op.kind == .MEMORY` -- which is precisely why
LDR/LDR_PRE/LDR_POST/LDR_REG had to be separate mnemonics; all 20 merge
collisions were this and nothing else. Split Operand_Type.MEM into
mode-specific types (MEM_OFFSET/PRE/POST/REG/EXT plus four SVE), matching
how W_REG/W_SHIFTED/W_EXTENDED are already distinct types over one
register class. The decoder derives Address_Mode from `enc`, so it is
unaffected.
Encodings are unchanged: the multiset of (ops, enc, bits, mask, feature,
flags) over all forms is identical before and after except for two entries
deliberately dropped. NOT_V_ALIAS duplicated NOT_V byte for byte, and
MOV_V_ALIAS was wrong -- it encoded VN where the ORR-based MOV alias needs
VN_VM_DUP, so "mov v1.8b, v2.8b" would have emitted "orr v1.8b, v2.8b,
v0.8b".
AMX_* keeps its prefix: Apple's coprocessor is undocumented with no
assembler spelling, so there is no canonical name to collapse to and bare
"set"/"clr"/"ldx" would mislead. The two-token system instructions keep
theirs too and print with a space (dc zva, tlbi vae1, bti j).
Verified: all 11 rexcode suites match HEAD exactly (arm64 461/461); the
three generator stages round-trip idempotently; 754 of 785 mnemonics are
accepted by llvm-mc, the rest being AMX (24), TME (4, no +tme in this LLVM
build), B_COND/BC_COND and TBL2; and a 39-case encode/print differential
against llvm-mc matches 34, with the 5 others confirmed byte-identical at
HEAD (pre-existing LSR/ASR immediate and LDP pre-index packing bugs, and
printer gaps for vector arrangements and MOVZ/MOVK shifts).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
