The VFP fixed-point conversions were a handful of forms whose comments
named the 32-bit fixed-point variants but whose encodings all had `sx`
clear, so they were the 16-bit ones; two of them were duplicates
distinguished only by a looser mask. None of them decoded the fraction
width at all, because the operand slot had no encoding, so every one of
them printed `#0` -- and a `#0` fraction is not what any of these words
mean. The block is now the full matrix the architecture defines: three
floating-point widths by two fixed-point widths by four directions,
twenty-four forms. The fraction is the fixed-point width less the
imm4:i field, so the widest fraction is what a zero field encodes.
LSR and ASR reach a shift of 32 through a zero field -- a shift of zero
would be a MOV, so the encoding spends that value on the one amount
five bits cannot otherwise name. Both the standalone mnemonics and the
shifted operand of a data-processing instruction printed `#0`, which is
not the same instruction; PKHTB dropped its `asr #32` entirely, since a
zero amount prints as no shift at all.
USAT and USAT16 saturate to an unsigned width of zero to 31 and store
it as it stands, unlike SSAT and SSAT16, which store one less than a
signed width of one to 32. They had been sharing the signed encoding
and so read one too high.
The AES and SHA forms carry a bare element size -- `aese.8`,
`sha1c.32` -- and had no data type at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
VMLA, VMUL, VMLAL, VQDMULL, VSDOT, VCMLA and the rest of the by-scalar
family take one lane of a low D register as the multiplier -- `d0[2]`,
not `d0`. Every one of them printed the bare register, which is not only
missing information but assembles to a different instruction: the
by-vector form.
The forms already declared the operand type as DPR_ELEM; what they
lacked was an encoding that knows the register number and the lane
number share a single four-bit field. How they share it depends on the
element size, and the two splits are different enough to be separate
encodings: 16-bit takes Vm from bits 2:0, so D0..D7, and the lane from
M:bit3, giving four lanes; 32-bit takes Vm from bits 3:0, so D0..D15,
and the lane from M alone, giving two. The dot-product and complex
forms share the 32-bit split despite being 8- and 16-bit operations.
Those reaches are now checked when selecting a form, so `d8[0]` on a
16-bit form matches nothing rather than quietly encoding as `d0[0]`,
and a lane-less operand no longer matches a by-scalar form at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Three unrelated things the A32 sweep turned up, all of them in what an
instruction prints rather than what it encodes.
VTBL and VTBX read their table from a run of one to four D registers,
and the run length is a fixed pattern bit of the form rather than
something the operand encodes. They printed a bare `d0`, which is not
the syntax -- the table is a list even when it holds one register. The
length now rides in the operand encoding, the way the NEON
structure-list lengths already do, and the encoder checks it, so
`{d0}` and `{d0, d1}` select different forms instead of both landing on
whichever sorted first.
The NEON compare-with-zero forms and VCMP/VCMPE against zero take a
literal `#0` that no field encodes. Modelling it as an implicit operand
is enough for it to print; nothing is emitted for it.
VCMP left bit 7 out of its mask. That bit is the E, which is the only
thing separating VCMP from VCMPE, so VCMP matched both and VCMPE could
never be decoded at all.
Also: VPUSH and VPOP take no data type, but the printer's fallback
guessed one from the register bank whenever the table gave none, and
produced `vpop.f64 {d0}`. A leading register list now suppresses it --
the load/store-multiple forms are exactly the ones that lead with a
list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
SSAT, SSAT16, USAT and USAT16 encode their saturate position minus one,
and SBFX and UBFX their width minus one, so a field of zero means one.
All of them printed the raw field, which is not a value the instruction
can take -- `ssat r0, #0, r0` is not assemblable.
BFI and BFC are different again: their field is the top bit's position,
and the width the syntax wants is msb - lsb + 1. They shared an encoding
with SBFX and UBFX, which need the opposite arithmetic, so they now have
their own. Packing an msb needs the lsb from a sibling operand, so the
packer takes the instruction rather than one operand in isolation.
Also worth recording: the sweep had been running llvm-mc with
`-mattr=+all`, which that target does not recognise and silently
ignores, so every CRC32, FP16, v8.1a and dot-product entry looked like a
reserved encoding. With the features actually enabled, 815 of 1139 A32
entries round-trip, and 38 rather than 143 are genuinely reserved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Forty-one groups of forms differed only in bits their own mask left
free, so they all matched each other's words and whichever sorted first
answered for the lot. That is where the NEON element sizes went:
`vabs.s8`, `.s16` and `.s32` are three patterns differing at bits 19:18,
and with those bits outside the mask the first one spoke for all three.
121 rows.
Two things that exposed:
- LDM and STM differ only in the writeback bit, and nothing in the
operand model recorded it, so the two forms printed identically.
Instruction carries it now and the base register wears its `!`.
- The T32 shifts left the S bit free, and their patterns sort ahead of
MOV's, so `movs.w r1, r2` came back as `lsl r1, r2, #0`. There is no
LSLS form in the table for it to have become instead; pinning the
bit at least stops it answering for a word that is not its own.
Also: r0 could not be used as an index register. The printer took a
register-offset operand to be immediate-only unless the index was
non-zero, when the class alone already says whether there is one --
`[r0, -r0]` printed as `[r0]`. And a displacement of zero lost its U
bit, which llvm-mc writes as `#-0` because it is a different word.
Against llvm-mc, A32 entries whose disassembly assembles back to the
same word: 760 of 1139, from 683. Wrong: 81, from 158.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
arm32 had only ever been checked against itself. Its sweep round-trips
our own encode and decode, which agree with each other whether or not
either is right -- exactly how the arm64 bugs stayed hidden for months.
Decoding each entry and handing the text to llvm-mc says something
different: 683 of 1139 A32 entries byte-exact, 166 wrong, 290 that no
assembler would take.
Three real faults out of that first pass:
- The halfword and dual load/stores split their 8-bit offset around
the opcode at bits 7:4, and their pre/post-indexed forms were using
the 12-bit decoder. It swallowed the opcode nibble, so `strht r0,
[r0]` came back as `[r0], #-176` -- 0xb0 being the halfword opcode
read as an offset. 16 forms.
- LDRD and STRD name a register pair and we printed one half of it.
The second is always the first plus one and occupies no bits, so it
needed an encoding that reads Rt and adds one.
- An indexed form printed no displacement when it was zero, which
turns `[r0], #0` into `[r0]` -- a different instruction, since the
writeback is the whole point of the form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
A tsz of zero has no set bit, so it names no element size at all; the
architecture reserves it and llvm-mc calls the word invalid. We decoded
it as `xar z0.b, z0.b, z0.b, #16` -- a shift of 16 on a byte, which is
not a thing.
No single mask says "tsz is not zero", but four do, the same way the
cset aliases exclude AL and NV: 1xxx, 01xx, 001x, 0001 cover every
non-zero value between them and nothing else.
Splitting it that way broke encode at first, because the four forms have
the same operand types and the matcher took whichever came first, ORing
its fixed tsz bit onto a value meant for a different element size. Each
pattern pins exactly the bit its own element size sets, though, so
typing the operands by size makes the matcher pick the right one and the
OR consistent.
SVE/SME2 against llvm-mc: 707 of 707 byte-exact, nothing mismatched and
nothing it cannot assemble.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
TBL2 is gone: SVE2's two-table lookup is spelled `tbl`, and the reason
it could not merge before was that the arity is the caller's choice, so
the matcher needs it in the operand type rather than the encoding. It
has that now, and both forms stay reachable. TBX, which takes no braces,
was briefly given them and is back to plain.
ZERO's operand is a mask with one bit per .d tile, and it is written as
the list of the largest tiles that exactly cover it -- a .s tile being
two .d tiles four apart, a .h tile four two apart. A mask of every bit
is just `{za}`. It had been printing the raw number.
ZA array vectors (`ldr za[w12, 0], [x0]`) are their own operand shape,
with no tile and no element size; they printed as a bare immediate.
The pair and quad register fields were read a bit too wide. A pair
starts on an even register and a quad on a multiple of four, and ZIP and
UZP tell themselves apart with the bits below that -- so every UZP
decoded as naming the register one or two above the one it writes.
SVE/SME2 against llvm-mc: 703 byte-exact of 704, with nothing
mismatched. The one left is a reserved encoding -- XAR with a tsz of
zero names no element size, and llvm-mc calls it invalid too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
SME's outer products take five -- `smopa za0.s, p0/m, p1/m, z0.b, z1.b`
-- and Instruction held four, so the 14 forms in that family could not
be represented at all, let alone printed. They named one vector where
the instruction multiplies two.
Five Operands is 55 bytes, which is odd, so Mnemonic's alignment costs
one more; three bytes of padding still land Instruction on exactly one
64-byte cache line, as before. Encoding and Decode_Entry each grow by
two.
The tile number was wrong as well: it sits in the low bits, as wide as
the element size leaves room for -- three for .d down to none for .b --
not at bits 23:22 where the encoding read it. Every MOP form named a
tile it was not writing.
Note for anyone regenerating: tablegen re-emits tables.odin from a
template inside gen.odin, so a struct change there has to go in the
template, and the package has to compile before tablegen can run at all.
Same for the builders.
SVE/SME2 against llvm-mc: 697 byte-exact and 0 mismatched, of 704.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
`sqincb x0, w0, pow2, mul #1` writes one register under both names: the
64-bit destination and the 32-bit value it saturates into. Sixteen forms
had only the W half, so they printed `sqincb w0, ...`, which no
assembler takes. The unsigned ones really do take a single W and were
already right.
SVE LDR/STR and PFIRST/PNEXT lost their unsized operands again when the
predicate sizing was re-run over the merged rows; a predicate that is
moved whole, or that governs, carries no element size.
SVE/SME2 against llvm-mc: 683 byte-exact and 0 mismatched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
The merge brought in the SQINCP/UQINCP/INCP/DECP family reading its
predicate from bits 20:16, where the architecture has it at 8:5, so
every one of them encoded a different register than it named. Their
predicate also carries an element size -- from the form's Z operand
where it has one, and from bits 23:22 for the scalar destinations.
SVE writes an element count as a pattern by name and a multiplier as
`mul #N`: `cntb x0, pow2, mul #1`. The pattern printed as a bare number
and the multiplier did not print at all -- IMM_MUL4 had an encoder but
no decoder, so it came back as an empty operand and left a trailing
comma. That covers 64 and 62 forms respectively.
CNTP's source predicate, and eight more the merge added, wanted the same
element sizes as the rest of that family.
SVE/SME2 against llvm-mc: 663 byte-exact and 0 mismatched, from 450 and
18 when the merge landed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Adopting the upstream table formatting in 4683ce0fd took the upstream
rows wholesale, which reverted a run of encoding fixes that had been
verified against llvm-mc. 78 rows came back with older operand types,
encodings or masks, and 37 were dropped outright -- the WHILE family's
four element sizes and INDEX's, which had been split from one
size-generic form each.
Restored as they were: the WHILE and INDEX size splits; the scatter
forms, whose .s variants had gone back to carrying the .d encoding so
each pair had identical bits again; the gather and scatter index
addressing; the predicate element sizes, and the predicate-mask
operations that must stay .b; SVE LDR/STR's unsized operands.
Against llvm-mc this puts SVE/SME2 back at 559 byte-exact from 450.
18 mismatches remain, all of them in rows the merge brought in -- the
SQINCP/UQINCP/INCP family reads its predicate from the wrong field --
and are dealt with separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
The tests said one thing and the table another: WHILELT and the three SVE
scatter stores. The tests were right and had been right for a while -- the
table edit that went with them was never committed, and the branch looked
green only because the tests read a stale generated blob that still carried
the fix. Regenerating the blobs (which merging upstream forced, since its 35
new mnemonics overflowed the old ones) made the disagreement visible.
WHILELT mask 0xFF20FC10 -> 0xFFE0FC10
ST1B scatter Zt.S 0xE4008000 -> 0xE4408000, mask 0xFFA0E000 -> 0xFFE0A000
ST1W scatter Zt.S 0xE5008000 -> 0xE5408000, mask 0xFFA0E000 -> 0xFFE0A000
ST1D scatter Zt.D bits unchanged, mask 0xFFA0E000 -> 0xFFE0A000
ST1B and ST1W carried the SAME bits for their Zt.S and Zt.D forms, so the
32-bit scatter was encoding as the 64-bit one; the S rows now differ from the
D rows beside them, which is what tells them apart.
Values taken from the committed test expectations and independently
corroborated by the pre-regeneration blob -- two sources that agree.
arm64: 330 passed 0 failed, 133 passed 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Riok9vMpkLmo78wsVKJHhz
The instruction table was the only conflict, and none of it was a real
disagreement: upstream c9b28d2ec reformatted every row, our six arm64 commits
edited rows under the old formatting, and the two sets are disjoint -- upstream
touched 40 mnemonics, we touched 160, intersection empty. The previous commit
put our rows in their format so the merge had one style to reason about; what
remained merged with all 124 of their new/changed rows and all 399 of ours
present, and nothing dropped.
The arm64 blobs are regenerated here because the merge makes it mandatory, not
optional: upstream added 35 mnemonics, so the enum reached 856 while the
committed encode_runs blob held 821, and ENCODE_RUNS[u16(m)] indexed past its
end -- the test binary aborted with "Index 821 is out of range 0..<821" before
running. That staleness was already on both branches; upstream's tip fails 28
tests and crashes for the same reason, and is green once regenerated.
KNOWN FAILING, PRE-EXISTING AND OURS: 326 passed, 4 failed --
WHILELT 64 table 0xFF20FC10 tests want 0xFFE0FC10
SVE_ST1B_SCATTER_S table 0xE4008000/0xFFA0E000 want 0xE4408000/0xFFE0A000
SVE_ST1W_SCATTER_S table 0xE5008000/0xFFA0E000 want 0xE5408000/0xFFE0A000
SVE_ST1D_SCATTER_D table 0xE5808000/0xFFA0E000 want 0xE5808000/0xFFE0A000
These are not the merge's doing and not upstream's. Our arm64 commits updated
the test expectations to the corrected SVE encodings but the matching
instruction_table.odin rows were never committed -- the row reads 0xFF20FC10 at
every commit including the merge base. The branch looked green only because the
tests read the stale blob, which does carry the fix. Our own pre-merge tree,
regenerated from its own committed table, fails these same four.
So the source is behind the artifact, which is the same class of defect as the
x86 enc_hint staleness, in the opposite direction. The intended values are
recoverable: the old blob and the test expectations agree on them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Riok9vMpkLmo78wsVKJHhz
Upstream c9b28d2ec reformatted the whole instruction table: every row's
clobber literal gained an explicit `Clobber` type name, the clobber column
stopped being aligned in favour of a per-block target with a one-space
minimum, and the mnemonic column was normalised to 17 for the 30 rows that
were still at 16.
Our six arm64 commits edited that table under the old formatting, so the two
collided on nearly every line -- 39 conflict hunks, not one of them a real
disagreement: upstream touched 40 mnemonics, we touched 160, and the
intersection is empty.
This applies their formatting to our rows so the merge has one style to
reason about. Purely cosmetic: the file is byte-identical to its previous
content once whitespace and the `Clobber` token are normalised away, and the
rules were derived from their file rather than guessed -- reformatting the
merge base with them reproduces their file exactly but for one `.MSR` row,
whose operand sub-column they realigned by hand.
arm64 pipeline: 133 passed, 0 failed, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Riok9vMpkLmo78wsVKJHhz
A ZA tile printed as the bare number it is encoded as -- `addha #0,
p0/m, p0/m, z0.s` -- and a tile slice printed as one too, where the
syntax is `za0h.b[w12, 0]`: a tile, taken along its rows or columns,
addressed by one of W12..W15 plus an offset. Neither was anything an
assembler would take.
Tiles are a register class now (ZA0..ZA15, viewed at an element size),
so they print through the same path as every other register. A slice is
its own operand kind holding the four things it is made of, rather than
one packed immediate that only the encoder understood.
Getting that right needed the field layout, and the layout is not what
the encoding table implied: the tile number and the offset share the low
nibble, and how it splits follows the element size -- a byte tile has no
tile bits at all and four of offset, while a quadword tile is all tile
and none. Reading a fixed four bits as the tile made every byte slice
come back as tile 4.
LD1Q/ST1Q scale their index by 16, which the mnemonic's last letter
does not spell the way B/H/W/D do; they were left unscaled when the
other 38 forms were fixed. That also wanted a .q element shape, which
nothing had needed before.
SVE/SME2 decode entries against llvm-mc: 559 byte-exact and 0
mismatched, from 211 and 78 at the start of the session.
What is left is mostly one structural limit: SME's outer products take
five operands (`umopa za0.s, p0/m, p1/m, z0.b, z1.b`) and Instruction
holds four. Widening it would fit -- five Operands is 55 bytes of the 64
-- but it reaches through Encoding, the table, both codecs and the
builders, so it wants its own change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Seventeen predicate operations were printing an element size they do not
have. AND/ORR/NAND/BRKA and the rest of that family treat a predicate as
a bit mask rather than as elements, so they are always .b -- bits 23:22
are opcode there, not a size, and deriving the size from them produced
`ands p0.h, p0/z, p0.h, p0.h`, which no assembler takes. PNEXT and PTRUE
really are sized and keep it.
SVE2's AES are destructive: the destination is also the first source and
is written twice, so `aese z0.b, z0.b, z1.b` has one operand more than
the NEON form. All four were modelled with the NEON operand count, which
made them unassemblable regardless of what else was right.
PFIRST and PNEXT had their governing predicate sized as well; it is
written bare, like every other governing predicate.
SVE/SME2 decode entries against llvm-mc: 545 byte-exact and 0
mismatched, from 211 and 78 at the start of the session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Predicates are written with an element size wherever they are an operand
rather than the governing mask -- `zip1 p0.b, p1.b, p2.b`, and `and
p0.b, p1/z, p2.b, p3.b`, where the same instruction has both. 59 source
operands printed bare. Two paths were missing it outright: .PN never
picked up the size at all, and the WHILE family had one form standing in
for all four element sizes, so `whilelt` could only ever print `p0`.
SVE gather and scatter name the element size of their vector index and
how the base extends it (`[x0, z0.s, uxtw]`); none of that was printed.
The two lay their fields out differently and the difference is not
cosmetic: a gather has the extend at bit 22 and takes the index width
from its opcode, while a scatter has the width at 22 and the extend at
14. Reading them the same way meant the .s scatters carried the .d
encoding -- ST1B, ST1H and ST1W all had two forms with identical bits,
so one of each pair was dead.
Memory had no room left for the index's element size (16 + 16 + 23 + 3 +
3 + 3 is exactly 64; the "1 bit spare" comment was stale), so it travels
in the operand's own size field, which memory operands do not otherwise
use.
INDEX had its two operands in each other's slots: SVE_IMM5 is bits
20:16, which is the *second* operand, so the first was written to the
wrong field and the second was not written at all -- `index z0.b, #0,`
with a trailing empty operand. It was also .b-only, and its register
operands are W below .d rather than X.
SVE LDR/STR move a whole register and take no element size; an earlier
pass had given their operands one.
SVE/SME2 decode entries against llvm-mc: 525 byte-exact and 0
mismatched, from 211 and 78 at the start of the session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Two defects, one shape: something the decoder must decide was never stated, so
it fell out of where an entry landed in an unstably sorted table.
CANONICITY. Several mnemonics name one encoding -- SHL and SAL are both /4, JE
and JZ are both 0x74. Decoding a representative of each and diffing against
llvm-mc: 74 patterns, 22 agreed, 52 did not (0F 84 -> JZ where llvm-mc says JE,
A4 -> MOVS vs MOVSB, DB E2 -> FCLEX vs FNCLEX). aliases.odin now declares 57
{alias, canonical} rows, each canonical name MEASURED from llvm-mc rather than
picked, and gen.odin drops an aliased entry as it collects decode entries -- so
the name never reaches the tables. The drop is conditional on the canonical name
covering the byte-identical encoding, which is what lets MOV be the alias at the
A0-A3/B8 moffs forms while staying the only name for 88/89. Aliases stay fully
encodable; only decoding narrows. 119 of 1350 legacy entries dropped, 74 of 74
now agree.
This also fixes the four SAL/SHL failures left by the previous commit, at their
root rather than by extending a hand-written list.
ADDRESS SIZE. 0xE3's mnemonic is chosen by which counter register it tests --
the 67h axis -- and REX.W does not affect address size at all. Modelling JRCXZ
as force_rex_w made its encoding 48 E3 cb where llvm-mc emits a bare E3 cb, gave
JECXZ the bare encoding that is really JRCXZ in long mode, and left three
indistinguishable entries. Encoding_Flags gains addr_size (2 of 6 spare bits);
the matcher refuses a form the mode cannot express, the encoder emits 67h when
it differs from the default, the decoder selects on it before the operand-size
pass. All ten cases match llvm-mc, refusals included.
The tests keep no second copy of the alias table: mnemonics_eq compares
canonical_mnemonic, and run_alias_table_test asserts the stronger property the
generator guarantees -- zero encodings spelled by two surviving mnemonics. It
caught the E3 ambiguity on its own.
Also fixed: gen.odin's write_flags enumerates Encoding_Flags by hand and
silently dropped addr_size from the generated tables -- the flag read back as
its zero value and the instruction was quietly mis-modelled.
271 passed. Bites: dropping {.SAL, .SHL} returns the prime_sieve failure and the
guard names every shift-group opcode; restoring force_rex_w decodes E3 00 as
JCXZ and encodes JRCXZ to three bytes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Riok9vMpkLmo78wsVKJHhz
mnemonic_builders.odin bakes `global_index + 1` into Instruction.enc_hint so the
encoder can skip the O(forms) match scan. The index is GLOBAL, so any edit to
the encoding table that inserts or removes a form shifts every index after it
and leaves the builders naming another instruction's encoding -- and nothing
noticed: the encoder took the form, emitted its bytes and returned success.
It has happened twice. 6e17e7a2d left 2130 of 3671 builders wrong; 36af73834
regenerated both halves and cleared it; baae2636b (adding `in`/`out`) re-broke
37; 9ae9a9bf9 shifted an early mnemonic and broke 3393 of 3802 -- including
CALL, whose r/m64 builder then encoded 0F 8A (JPE) instead of FF /2, turning
every indirect call into a conditional jump. The symptom was a segfault in a
JIT'd program, arbitrarily far from the cause.
Regenerated: 3820 of 3820 builders now correct. Three guards so it cannot
return silently:
- the encoder checks, under ODIN_DEBUG, that the hinted form lies inside its
own mnemonic's ENCODE_RUNS entry. Release keeps the byte-for-byte fast path;
this is a regeneration-time mistake and only has to be caught once by
anyone running tests.
- the generator stamps BUILDER_TABLE_FINGERPRINT, an FNV-1a hash of every
ENCODE_RUNS (start, count), into the file it emits.
- run_builder_generation_test recomputes it from the loaded tables and fails
with the command to regenerate.
Known-failing at this commit: four SAL/SHL cases, a separate defect in the
alias table, fixed in the commit that follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Riok9vMpkLmo78wsVKJHhz
An entry whose `bits` set a bit its `mask` does not cover can never
match anything -- `word & mask == bits` is unsatisfiable -- so those
instructions were absent from the decoder entirely. There were 48, and
LDXR/LDAXR were among them: a plain `ldxr w0, [x1]` disassembled to
nothing.
They divide cleanly. Most are the SVE predicated *unary* ops
(ABS/CLS/CLZ/CNT/FABS/FNEG/FSQRT/NEG), which keep their opcode at bits
20:16 -- the same field that was wrong for the binary ops, except here
the stray bits sat in `bits` rather than being left free, so the entry
was dead instead of over-matching. The LDXR family has Rs = 11111 in the
same place. FCADD, SETE, SETM and UZP each had one fixed bit outside
their mask. Every one of them was verified against llvm-mc before
widening: the patterns were right, only the masks were too narrow.
Four needed more than a wider mask:
- BTI was modelled as taking a hint immediate, but its variants are
already their own mnemonics (BTI_C/BTI_J/BTI_JC) with exact
patterns. The bare row is plain `bti` and never took an operand.
- LUTI2/LUTI4 named a Z pair where the architecture has ZT0, SME2's
lookup table, and put a register at bits 20:16 where the table index
lives. ZT0 is modelled now -- one register, no bits -- and the index
is a real lane index at bits 16:15. Both now cover all three element
sizes rather than one.
- MOVA's mask missed bit 17.
Two more the newly-assemblable output exposed: ST1D's scalar+scalar form
carried the .q encoding, and LD1SH's was labelled .s while holding the
.d pattern, with the .s form missing outright.
SVE scalar+scalar addressing scales its index by the access size and an
assembler wants that spelled (`[x0, x0, lsl #2]`), which 38 forms did
not print. The amount is fixed by the form -- and by the *access* size,
not the destination, so LD1SB's is 0 even though it writes halfwords.
SVE/SME2 decode entries against llvm-mc: 438 byte-exact and 0
mismatched, from 211 and 78 at the start of the session. Nothing decodes
to `invalid` any more.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
I had this wrong. XAR is fully determined by the word: `tszh:tszl:imm3`
holds `2*esize - shift`, and since the shift is in [1, esize] the four
element sizes occupy disjoint ranges (.b 8-15, .h 16-31, .s 32-63,
.d 64-127), so the highest set bit names the size. Nothing is ambiguous;
the table just could not say "the element size is in an operand field".
So it was four identical (bits, mask) entries standing in for one. Encode
already worked -- the matcher picked among them by the operand -- but
decode had four entries tied on every sort key, and which one won was
whatever the unstable sort left. It answered `.s`, the fall-through,
because reg_size_for_type had no case at all for them.
Now there is one form. The Z operands' element size is read out of the
word (VD_TSZ/VN_TSZ), and the shift's is taken from the register the
caller passed rather than from the form -- which means the packer needs
to see the instruction, not just its operand, since this field genuinely
couples two of them. The tsz rule itself lived twice, as arithmetic in
the encoder and a threshold ladder in the decoder; it is one helper now.
Downstream, and much larger: every SVE element size collapsed onto a
single builder, exactly as the NEON arrangements did. 264 of the 470
forms with a Z operand were unreachable -- `inst_xar_z_z_z_i` could only
ever make .b. The size is in the name now (inst_add_zb_zb_zb and
friends), which took the builders from 1847 to 2141.
The sweep that verified it found two more:
- A predicate that is the destination is written with an element size,
not a qualifier: `cmpge p0.b, p1/z, ...`. 106 forms printed a bare
`p0`. The two suffixes are told apart by encoding -- PD against PG --
and by disjoint size codes.
- SVE's indexed FMLA/FMLS printed their lane as `, #1` rather than
`[1]`; those encodings were simply missing from the lane-index set.
The decode sort is now total (it ties on table order last), so which
entry wins can no longer vary between builds.
SVE/SME2 decode entries against llvm-mc: 370 byte-exact and 0
mismatched, from 211 and 78 at the start of the session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Chasing the SME2 gap turned up that the thing blocking it was much
larger than SME2. Every predicated SVE instruction printed its predicate
bare -- `p0` where the syntax needs `p0/z` or `p0/m` -- and an assembler
rejects that outright. 357 forms carried one.
A predicate's governing qualifier is fixed by the form, so it comes from
the operand type and rides in the operand as a marker the printer reads.
Predicated SVE loads and stores also write their vector as a list, so
those 42 forms go through the same one-register list path the NEON work
added: `ld1b { z0.b }, p0/z, [x0, x0]`.
SME2 then needed three things it did not have. A predicate-as-counter
register class -- SME2 governs with pn8..pn15, numbered from 8, sharing
the field but not the register bank. An element size on the pair and
quad operands, which cannot ride on the encoding the way the list length
does, because it is what separates LD1B from LD1H. And the list length
itself, which the pair/quad encodings now carry.
The sweep that verified this found two real encoding bugs behind it:
- 97 SVE predicated binary ops read Zm from bits 20:16, where the
architecture has the opcode. `add z0.b, p0/m, z0.b, z1.b` encoded
0x04010000, which is SUB. Their masks left that opcode field free
too, so each mnemonic's pattern also matched its siblings'.
- AND/ORR/EOR/BIC predicated had one form apiece, labelled .d but
encoding .b, since the size field at bits 23:22 was never in the
pattern. Split into the four sizes.
SVE/SME2 decode entries against llvm-mc: 312 byte-exact and 1
mismatched, from 211 and 78. The one left is XAR, whose four forms are
legitimately bit-identical -- the element size shares a field with the
shift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
No assembler spells these `vld1_lane` or `vmov_lane`; they are `vld1`
and `vmov` with a lane-indexed operand. All nine are now their base
mnemonic. Merging them meant fixing what the separate names had been
hiding.
VLD1_LANE and VST1_LANE were byte-for-byte duplicates of forms VLD1 and
VST1 already had, with a looser mask, and neither encoded the lane --
both used .VD_D, which has no lane field. `vld1.8 {d0[3]}, [r0]`
disassembled as `vld1.8 d0, [r0]`. VMOV had the same shape: its own
DPR_ELEM form dropped the index, while VMOV_LANE's three forms carried
the per-size encodings that actually work. The lane-dropping forms are
gone and the working ones now sit under the base mnemonic.
VLD2-4/VST2-4's single-lane forms left bits 9:8 free, and that field is
what separates VLD1/2/3/4 -- so VLD4's encoding matched VLD2's entry and
disassembled as the wrong instruction.
Two more things the lane could not survive. A lane index of 0 printed
nothing, because 0 doubled as "no lane" -- `vmov.32 d0[0], r0` came out
as `vmov.32 d0, r0`, a different instruction. And a lane inside a
register list was dropped entirely, so `{d0[1], d1[1]}` printed as
`{d0, d1}`.
The whole set is byte-exact against llvm-mc: 27 single-lane forms across
VLD1-4/VST1-4 and all three VMOV element sizes. arm32 is 1649/1649 on
the sweep and 436/436 on the table checks, with the stale expectations
for the forms this corrected updated to the verified values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
`vpush {d8, d9}` came out as `vpush {r2}`. VFP list operands kept only
the 8-bit count from the encoding and stored it in a REG_LIST operand,
whose immediate the printer reads as a GPR bitmask -- so the register
bank was wrong, the start register was gone entirely, and the count was
read as a bitmask. `vldm r0, {d1, d2, d3}` printed `{r1-r2}`.
A list is now a register plus a shape: how many, and what the run steps
by. VLDM/VSTM/VPUSH/VPOP recover their start register from Vd and their
count from imm8. A GPR list stays a bitmask, since `{r4, lr}` is not a
run at all.
The NEON structure loads were wrong in a second way. VLD1-4/ST1-4 encode
their register count in the type field at bits 11:8, not in imm8, so
writing a count into the low byte overwrote size, alignment and Rm. Rm
was left at 0, which is a register post-increment, where the plain form
needs 0b1111; every one of those 23 forms encoded a writeback nobody
asked for. The count now comes from the form, the encoding writes only
Vd, and Rm is fixed in the pattern.
Deriving those counts from llvm-mc rather than by hand turned up a
dimension that was not modelled at all: the spaced forms step two
registers at a time (`vld2.8 {d2, d4}`), and VLD1 has a to-all-lanes
form written `{d2[]}`.
All 31 A32 list forms are now byte-exact against llvm-mc, from 8. The
arm32 sweep holds at 1656/1656.
Still open here: those forms only exist for .8 data, because the size
field at bits 7:6 is unencoded, and the writeback variants
(`[r0]!`, `[r0], r1`) are not modelled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
LD1-4/ST1-4 write their registers as a list -- `ld2 {v0.16b, v1.16b},
[x1]` -- and none of that was modelled. LD2/LD3/LD4 named a single
register where the syntax names two, three or four, so every one of
their forms disassembled to something no assembler would take.
How many registers the list holds is fixed by the instruction form, not
chosen by the caller: LD2 always names two. So it rides on the encoding
(VD_LIST1..4, VN_LIST1..4, which pack exactly like VD/VN) rather than on
the operand type. Putting it in the type would have meant a type per
count per arrangement -- 32 of them -- and would have made the matcher
check something the caller cannot vary.
The operand carries the count, and the printer walks the run from the
first register, wrapping at v31. That replaces the V_LIST_16B one-off
added with the TBL/TBX fix, which could only ever express a
single-register list; TBL/TBX now go through the same path.
Operand grows a byte for the count, which comes out of the padding
Instruction already had -- it is still exactly one 64-byte cache line,
still aligned to one.
The arrangement codes were bare numbers repeated across four files, and
the generated builders would have grown more of them, so they are now
named constants (VSHAPE_16B and friends).
Verified against llvm-mc: 50 whole-register list forms byte-exact, and
all 51 lane-indexed forms byte-exact -- `ld2 {v0.b, v1.b}[1], [x1]` and
`ld1 {v0.16b}, [x1]` included. Before this and the lane-index change,
every one of those 101 printed something that would not assemble. The
vector sweep holds at 809 byte-exact with nothing unassemblable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
The three remaining shapes llvm-mc would not accept back, now that
making every NEON arrangement reachable had exposed them.
PMULL/PMULL2 wrote their destination as .2d where the architecture says
.1q. That arrangement had no operand type because the size marker is
lanes*elem-bytes and 1*16 collides with 16B, so V_1Q takes the next free
multiple of 8 instead. The encodings were already right; only the label
was wrong.
TBL/TBX write their table register as a list, `{v1.16b}`. The braces
belong to the operand rather than the mnemonic, so V_LIST_16B carries
them and the printer stays generic. Only one-register lists are modelled
-- LD1-4/ST1-4 need a count, which is still open.
SM3TT1A/1B/2A/2B were missing their lane index entirely. The mask
already left imm2 free at bits 13:12; the operand simply was not in the
table, so every one of them decoded as index 0 and printed `v2.s` with
no index at all.
Printing that index needed the piece that was never there: a lane index
is its own immediate operand, so it printed as a separate `#2` rather
than glued to the register it indexes. It now carries a marker and the
printer writes `v2.s[3]`. The marker is set from the ENCODING, not the
operand type -- EXT shares .VEC_INDEX for a byte index that really is
written `#3`.
That last part reaches further than these ten: of the 51 lane-indexed
forms, all 51 used to print an index that no assembler would take. 15
are now byte-exact against llvm-mc (SM3TT, DUP, INS) and the other 36
are LD1-4/ST1-4, which additionally need the register-list braces.
The vector sweep is now 809 byte-exact with nothing mismatched and
nothing llvm cannot assemble, from 803/10 before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
inst_add(X0, X1, X2) encoded `add v0.16b, v1.16b, v2.16b`. It built the
operand as op_v_16b(u8(reg_hw(dst))), which throws away the register's
class and rebuilds a V register from the bare number -- so an X register
became a V register before the matcher, whose whole job is to reject
that, ever saw it. It encoded, round-tripped, and printed cleanly. Seven
mnemonics with both a scalar and a vector three-register form were
affected: add, and, bic, eor, orn, orr, sub.
The vector constructors now take the register the caller actually has
and put the arrangement in op.size, so the class survives. A wrong class
matches no form and encode reports it; the right class picks the right
form, which is what the matcher was always supposed to do.
The same laundering hid a second bug. Because every arrangement built
the same Odin signature, all of a mnemonic's arrangements collapsed onto
one builder name and only the first survived -- ADD has seven NEON forms
and six were unreachable. 229 mnemonics were in that state. The
arrangement is now part of the builder name (inst_add_v8b_v8b_v8b), so
they are all reachable: 992 builders becomes 1847. The overload group is
unchanged, since it still dedups by Odin signature, so inst_add(V0, V1,
V2) still means .16b as before.
Making them reachable exposed two pre-existing bugs, both fixed here:
CMLE/CMLT/FCMLE/FCMLT compare against zero and the zero is part of the
syntax rather than an encoded operand, so their disassembly was missing
the trailing `#0`/`#0.0` and no assembler would take it; and BFCVTN was
typed .8h at the destination where the architecture says .4h (BFCVTN2 is
the .8h one, and was already right).
Verified against llvm-mc: of the 874 all-register vector builders, 803
are byte-exact and none disagree. The 10 that llvm cannot assemble are
the already-known modelling gaps -- SM3TT lane indices, TBL/TBX register
lists, and PMULL's .1q destination, which the arrangement encoding
cannot represent since 1 lane * 16 bytes collides with 16B. The other 61
are the harness passing V registers where a scalar B/H/S/D/Q view is
required, which is the class check doing its job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
They were plain i64 constants handed to op_imm, so any integer typed as
one and `inst_mrs(X0, 999999)` compiled fine. Worse, the printer could
not tell a system register from an immediate and had to recover the
distinction by mnemonic and slot -- MSR's other form holds a PSTATE
field selector in the same position, so it keyed off whether operand 1
was a register.
System_Register is now its own type with its own Operand_Kind, union
member and op_sysreg constructor, exactly as Cond is. The printer's slot
logic is gone: the operand knows what it is, so naming it is a case in
the same switch that prints every other operand kind. MSR's PSTATE
selector is typed PSTATE_FIELD, which is what it always was.
It cannot join `Register` itself: that is a u16 with the class in its
high byte, leaving 8 bits for the number, and a system register needs
15. Widening it would break `Memory`, which packs two registers plus a
displacement and a mode into exactly 64 bits.
The constants are also reorganised. They had accreted into overlapping
sections -- two "ID registers" groups, three cache groups, a "Batch 5:
comprehensive sysreg sweep" banner, and a "hmm let me recompute" note
left in a comment. All 231 are now grouped by architectural function
(18 groups, alphabetical within each) with their five fields aligned.
Verified unchanged against llvm-mc: 222 registers byte-exact through
MRS, 8 write-only through MSR, and the PSTATE form still decodes as an
immediate rather than a register.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Printing MRS/MSR operands by name made these visible: sweeping all 230
constants through encode -> decode -> print -> llvm-mc, 89 of them
assembled to bytes we did not produce. TCR_EL1 encoded as 0x4282 where
the architecture says 0x4102, FAR_EL1 as 0x5300 for 0x4300, the whole
pointer-auth key block was off by a CRn, and so on. Anything reading
one of them got a different register than it asked for.
The field comments were right and only the packed values were wrong,
so most of the file could be rebuilt from its own comments and checked
against llvm-mc. Thirteen needed more: nine had bad comments too
(RGSR/GCR/TFSR/TFSRE0, ID_MMFR4/5, and the three ICC SGI registers --
ICC_SGI*_EL1 are op1=0, not 3), and four carried a "historic collision"
note instead of fields. Those were derived from the ARM ARM and agree
with llvm-mc.
Five of the six encodings that looked like duplicates were simply wrong
values landing on each other; one real pair is left, DBGDTRRX_EL0 and
DBGDTRTX_EL0, which genuinely share an encoding as the read and write
views of one register. The name table prefers the read name, since MRS
is the direction that has to print.
All 230 now verified byte-exact against llvm-mc: 222 through MRS, and
the 8 write-only ones through MSR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
These are what an assembler writes -- and what one prints back -- for
CSINC/CSINV/CSNEG with the condition inverted, and none of the five
existed. A disassembly of `cset w0, eq` came out as
`csinc w0, wzr, wzr, ne`.
Two constraints the table could not state before:
- The condition is stored inverted, so COND_HI_INV packs `cond ~ 1`
and reads it back the same way. The printer needs nothing; the
decoder hands it a plain condition operand.
- cinc/cinv/cneg are only the alias when Rn == Rm, which is a
cross-field equality no mask expresses. One operand fills both
slots on the way in (RN_RM), and decode checks the two fields agree
before accepting the entry -- reached only on a mask match, so it
costs nothing in the scan.
The aliases also require cond != 111x. That one *is* expressible: the
14 legal values are covered exactly by three masked patterns (0xxx,
10xx, 110x), so AL and NV fall through to the underlying instruction
the way llvm-mc does. COND_NOT_AL rejects them on the encode side.
Verified against llvm-mc across all five mnemonics, both widths and all
14 conditions: 140/140 of our printed strings assemble to exactly our
bytes. Disassembly agrees except for cs/hs and cc/lo, which is the
package's existing spelling of those two conditions and shows up on
CSEL and B.cond alike. AL/NV and Rn != Rm both fall through correctly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
`mrs x0, cntvct_el0` came back out of the disassembler as
`mrs x0, #24322`. The 230 sysreg constants were already there and the
encoding was byte-exact against llvm-mc -- the printer simply had no
SYS_REG handling, so the packed 15-bit field printed as an immediate.
Nothing an assembler would take back.
Adds a value -> name table (sorted, binary searched) and prints that
operand by name for MRS, and for the MSR form that takes one. MSR's
other form holds a PSTATE field selector in the same slot, which is a
different namespace; the two are told apart by whether the second
operand is a register or an immediate.
Six encodings carry two names, so a round-trip can come back spelled as
the sibling; the first by source order wins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UmHLRF11EoWwNWCJ7JGaA
Asking for a VFMA with a lane operand emitted a VMLA. NEON's fused multiply-add
has no by-element form at all -- llvm-mc rejects `vfma.f32 d0, d1, d2[0]` -- and
the four rows sitting under VFMA/VFMS held VMLA/VMLS's lane encodings, which
VMLA and VMLS already own. Their data type gave it away too: `.I32` on a
fused multiply-add, which is float-only. Deleted.
They were also the reason `bits & ~mask` looked wrong on those rows: as
authored they were F2A000C0, and clearing the Vn high bit -- correct, since
bit 7 is the top of the register number -- landed them exactly on VMLA's
F2A00040.
Found by asking which (bits, mask, mode) triples more than one mnemonic
claims. That check found 18; this commit takes it to 9.
Three of the eighteen were whole mnemonics duplicating a base:
VRECPE_F, VRSQRTE_F every form already present under VRECPE / VRSQRTE,
which carry both the U32 and F32 variants.
VPADD_F its F32 form duplicated VPADD's; its F16 form was the
only thing it owned, so that moves to VPADD, where the
data type now selects it.
All three were on the list of names no assembler spells, so that count goes
from 14 to 11 -- and nine of the remaining eleven are the *_LANE group, still
waiting on register-list and lane-index modelling. The other two are
`psb csync` and `tsb csync`, which are correct as they are.
Two smoke checks asserted the VFMA/VFMS by-element forms and are gone with
them; a third moved index.
Verified against llvm-mc: vpadd.i8/.f32/.f16, vrecpe.u32/.f32 and vrsqrte.f32
all byte-exact, 1656/1656 decode sweep, every suite at baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Splitting B_COND left three places still describing the old model. The
generated builders were already right -- inst_b_le(label), inst_bc_ne(label),
one per condition, all 32 verified to encode and round-trip -- but the
hand-written scaffolding around them was not.
verify_against_llvm normalised our mnemonic by truncating at the first
underscore, which turned B_COND into "b". LLVM prints b.eq/b.ne/..., so the
tool carried 32 alias rows pairing "b" with each of them to stop the mismatch
being reported. That truncation now collapses all sixteen B_* onto "b" and
makes every condition compare equal to every other -- the check would pass
whatever the table said. Keep the condition instead (B_LE -> "b.le") and the
32 alias rows are unnecessary; they are gone.
specgen's canonicalizer kept B_COND and BC_COND off its rename path by name.
Those names no longer exist, so replace the entry with a rule that matches the
shape (BC?_%u%u), which is what the intent was.
And the note in instructions.odin still pointed at inst_b_cond. It now says
what is actually true: a conditional branch is one builder per condition
because the condition is part of the mnemonic, while the select/compare family
-- CSEL, CSINC, CSINV, CSNEG, CCMP, CCMN, FCSEL -- really does take a
condition operand and keeps one.
specgen still re-derives its 1130 forms from llvm-mc and finds every one
already present, all suites pass, and all 13 packages build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
B_COND was a straight contradiction of the rule the rest of the enum follows.
No assembler has a mnemonic called `b.cond`; it has `b.eq`, `b.le`, `b.ne` and
thirteen more, and x86 in this same library already models exactly this shape
the right way -- JE, JNE, JG, JLE are sixteen separate mnemonics with the
condition in the opcode.
Two things were wrong with folding it into an operand.
The condition is not an operand. It is four bits of the opcode, no different
from x86 putting Jcc's condition in the low nibble of 0x7_. Modelling it as
one has to be papered over everywhere: the printer special-cased B_COND to
rebuild `b.eq` out of operand 0, sbprint had to skip that operand so it did
not print twice, and the public mnemonic_to_string -- which has no instruction
to read the operand from -- returned "b_cond", a string no assembler takes.
All three of those are now gone; the name prints itself.
More importantly it destroyed the flag data. x86 records per condition which
status bits it consults: JE reads {ZF}, JLE reads {ZF, SF, OF}. One B_COND
entry could not say that, so it claimed nzcv_rd = {N, Z, C, V} -- all four
flags, for every condition. Every entry was wrong. Split apart they carry what
they actually read:
b.eq/b.ne Z b.hi/b.ls Z, C
b.cs/b.cc C b.ge/b.lt N, V
b.mi/b.pl N b.gt/b.le N, Z, V
b.vs/b.vc V b.al/b.nv none
which is the data the compiler's asm checker reads for flag liveness, now that
arm64 feeds it.
The mask also covers the condition field for the first time (0xFF000010 ->
0xFF00001F): with the condition in an operand, four opcode bits sat outside
the mask.
BC.cond gets the same treatment. Builders come out per condition, so `b.le` is
`inst_b_le(label)` rather than `inst_b_cond(.LE, label)`. Cond stays exactly as
it is -- CSEL, CSINC, CSINV, CSNEG, CCMP, CCMN and FCSEL take a real condition
operand, and the compiler's OP_COND handling is untouched.
All sixteen verified by disassembling our own output with llvm-mc: b.eq, b.ne,
b.hs, b.lo, b.mi, b.pl, b.vs, b.vc, b.hi, b.ls, b.ge, b.lt, b.gt, b.le, b.al,
b.nv. Rows follow the table's current column formatting. All rexcode suites
pass and the generators stay idempotent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapsing the encoding-shaped mnemonics into their assembler names merged
several form lists that already overlapped, leaving byte-identical Encoding
rows inside one mnemonic. The matcher takes the first form that matches, so
every duplicate was dead weight -- unreachable, and inflating both tables.
VMUL, VMLA and VMLS each gained two from the *_LANE merge; VSHR, VSRA and
VRSHR two each, VQSHRN, VQRSHRN and VSHLL one each.
The branch base had none, so these are mine: the collision check I ran before
each merge compared operand signatures, which is the right test for "can the
matcher tell these apart", but says nothing about two forms being wholly
identical. Comparing the full Encoding would have caught them.
1680 -> 1665 forms. Removing an unreachable exact duplicate cannot change
behaviour, and does not: the load/store, LDM/STM and NEON data-type encode
checks are byte-for-byte what they were, and the decode sweep still round-trips
every form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LDM and STM each held five A32 forms with identical operand shapes -- the four
increment/decrement orders plus a writeback variant -- distinguished only by
their fixed bits. Nothing could tell them apart, so the encoder always took the
first and six of the eight A32 encodings were unreachable: `ldmib`, `ldmda`,
`ldmdb`, `stmib`, `stmda`, `stmdb` could not be produced at all.
They are not variants of one mnemonic in the first place. An assembler spells
them `ldmib` / `ldmda` / `ldmdb`, with plain `ldm` meaning IA, so this follows
the same rule as the rest of the enum: one member per name an assembler
accepts. LDM/STM keep the IA order and the Thumb encodings; the other three
orders become their own mnemonics, and the T32 DB encodings join them.
All eight now encode, byte-exact against llvm-mc:
ldm e8900006 stm e8800006
ldmib e9900006 stmib e9800006
ldmda e8100006 stmda e8000006
ldmdb e9100006 stmdb e9000006
Six test checks referenced these forms by index; they were re-derived by
matching (bits, mask) against the rebuilt table rather than by adjusting
offsets, and every one was found -- so no form was lost in the move.
Writeback (`ldm r0!, {...}`) is still unreachable: it is a property of the base
operand, not a separate mnemonic, and there is nowhere to put it yet. That is
one form per family rather than four.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The encoder builds a word by ORing packed operand fields onto the form's
`bits`. It can only ever set a bit that way, never clear one -- so any bit
`bits` presets that an operand is supposed to drive is stuck at 1 forever.
72 forms did that, and two families show what it cost:
* The U bit (23) on the whole A32 load/store family. U selects add vs
subtract for the displacement, and the encoder derives it from the sign of
mem.disp -- but every form had it preset, so `ldr r0, [r1, #-4]` silently
encoded as `[r1, #4]`. Every negative displacement in the family was wrong.
* The Vn high bit (7) on the NEON lane-indexed forms. That bit is the top of
the register number, so presetting it meant Vn could only ever name
d16..d31; d0..d15 were unreachable.
Which bits are operand-driven was decided by llvm-mc rather than by reading
the manual: for each of the 230 bits a form preset outside its mask, take the
form's canonical word with the bit set and cleared and disassemble both. Same
mnemonic, different operands means the bit belongs to an operand (clear it);
a different mnemonic, or an undecodable word, means the bit is genuinely fixed
for that form. The split was not per-bit -- bit 7 is a register bit for
VMUL/VMLA/VFMA but distinguishes VNEG from VABS and VCMPE from VCMP, and bit
23 is the U bit for LDR but the load/store select for VCX3 -- so every form
was classified individually.
Ten test expectations asserted the old values and were corrected; each had the
bug baked in. Verified byte-exact against llvm-mc across the load/store family
including every negative-displacement form, and the 1680/1680 decode sweep and
all other suites are unchanged.
The other half of `bits & ~mask != 0` -- 130 forms where the bit really is
fixed and the MASK is merely too loose -- is deliberately not in this commit.
Widening those masks alone breaks decode: a bit that distinguishes two
mnemonics has to be added to BOTH forms' masks in the same pass, and doing
only the ones that set it made LSL swallow MOVS, CX3 swallow VADDLVA and VABAV
swallow VRMLSLDAVH. That needs each form's true mask derived empirically
(vary the operands, see which bits move) the way specgen does it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SHIFT_NAMES holds five entries, LSL..RRX, but Shift_Type has ten: the four
register-shifted-register markers (LSL_REG = 6 .. ROR_REG = 9) say the shift
count comes from an Rs register rather than an immediate. Both places that
indexed the table used the raw enum value, and the guard in front of them only
excluded NONE and RRX -- so any operand carrying a register shift indexed a
5-entry array with 6..9 and killed the printer:
printer.odin(473:45) Index 6 is out of range 0..<5
Fold the register-shifted variants back onto the table and give each spelling
its own case: `, lsl #3` for an immediate amount, `, lsl r3` when the count is
in a register (Rs index rides in shift_amt), and a bare `, rrx`, which takes no
amount. All nine now print what an assembler accepts -- verified against
llvm-mc -- where three of them previously crashed and RRX printed nothing.
The memory-operand site indexed the same table the same way and is routed
through the same helper.
Found by printing every entry in the decode table; that sweep now completes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shrinking Instruction to 48 was the wrong call, and measuring it said so.
The premise was that a sub-cache-line struct touches fewer lines. It does --
but `#packed` aligns the struct to 1, so a 48-byte stride straddles a line
boundary 75% of the time, and the heap base is not line-aligned either. The
old 64-byte packed layout was worse still: 100% straddling, getting none of
the benefit its size implied.
Measured on an i7-9750H (L1d 32K/core, L2 256K, L3 12M), best-of-5, median of
3 interleaved rounds, against the 64-byte packed layout this branch started
from:
scan encode decode
64 packed (was) 1.000x 1.000x 1.000x
48 packed 0.909x 1.040x 1.022x
64 align(64) 1.218x 1.034x 0.806x
Decode is ~19% faster aligned, and that holds at every working set including
ones that fit entirely in L1 -- so it is split-store cost at the store ports,
not cache-line fetches. Decode writes whole Instructions, and the aligned
stores are worth more than the 33% extra bytes they move. A fourth variant --
48 bytes with `#packed` removed -- was measured to rule out the obvious
confound, and tracked 48-packed within 0.5% everywhere, so the win is
alignment and not the loss of packing.
Encode is within a few percent throughout (it is compute-bound; the form scan
dominates), and the pure read traversal is slower, but that is a synthetic
loop and its regression is codegen, not cache -- it is present even at
L1-resident sizes where a standalone struct shows no such penalty.
The Operand and Memory work from the previous commit is what makes this
possible: a 45/48-byte payload now sits inside one line with room to spare,
where the original spent all 64 bytes. The 16-19 spare bytes cost nothing over
a straddling 48-byte struct and give new fields somewhere to land.
All 11 rexcode suites match baseline; arm64 is 73/73 byte-exact against
llvm-mc; arm32's 1680/1680 sweep passes and its encode spot-checks are
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
x86 keeps Instruction at 64 bytes -- one cache line -- by packing its memory
operand into a bit_field u64 rather than a struct. arm32 and arm64 both used
a 12-byte Memory struct, and since Memory sits in every Operand that width is
multiplied by four in every Instruction. Adopting x86's trick, plus two
smaller things, takes both ARM ISAs under the cache line.
Instruction ops[4] Operand Memory
x86 64 48 12 8
arm32 88 -> 48 72->40 18->10 12->8
arm64 64 -> 48 56->40 14->10 12->8
Memory -> bit_field u64, both ISAs. Field syntax and composite literals are
unchanged, so callers see nothing. Registers keep their type: an arm64
Register never exceeds 0x0C1F and an arm32 one never exceeds 0x401F, but the
arm64 NONE sentinel is 0xFFFF, so arm64 gives them the full 16 bits and arm32
15. What is left goes to `disp`: 23 bits on arm64 (worst case 65,520, from
LDR Q, [Xn, #imm12*16]) and 19 on arm32 (worst case 4,095, an A32 imm12) --
64x and 32x headroom respectively.
arm32 Operand also carried four tail bytes arm64 does not. `cond` was dead:
nine builders wrote it and nothing in the package ever read it, and
Instruction.cond already exists. shift_type/shift_amt/lane now ride inside
the union alongside the register they describe -- they only ever apply to a
register operand -- via a `using` bit_field, so op.reg, op.shift_type,
op.shift_amt and op.lane still read and write exactly as before.
arm32 Instruction packs cond, operand_count, mode, length and the two flag
bits into one 16-bit word; they need 13 bits between them and were spending
six bytes. `using` again keeps the field names, with the one exception that
inst.flags.sets_flags is now inst.sets_flags (five call sites).
Verified: every rexcode suite matches baseline; both generators stay
idempotent; arm64 is 73/73 byte-exact against llvm-mc on both encode and
decode round-trip; arm32's 1680/1680 sweep still passes and a memory/shift
encode spot-check is byte-identical to what the same code produced before
this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
