rexcode: the x86 test suite could fail and the build said PASS

Found by accident, and worse than the bug that found it. Five deliberately
broken decode cases came back green through `build.lua --test --isa x86`, while
running the same suite directly reported "253/258 PASSED, 5 FAILED".

Two independent holes, both open, so neither backstopped the other:

  - the x86 test binary ended at `print_summary()` and never called `os.exit`,
    so it exited 0 no matter how many cases failed;
  - `do_test` looked for the words "N failed" in the output, and the x86
    harness prints "N FAILED" — a lowercase-only Lua pattern, so the largest
    suite in the tree was exempt from its own gate.

The exit code is the check that should have been load-bearing, so the harness
sets it. The output match stays as a backstop for a suite that forgets to, and
is now case-insensitive.

How long this has been true is not knowable from here — every x86 regression
since the pattern was written would have been reported as PASS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Riok9vMpkLmo78wsVKJHhz
This commit is contained in:
Flāvius
2026-08-16 12:05:52 -04:00
parent 3e3eec090d
commit b54c60b76a

View File

@@ -284,7 +284,11 @@ end
local function do_test(isa)
local ok, out = run(odin_run(pkg(isa, "tests")))
local fails = out:match("([1-9]%d* failed)")
-- Match case-INSENSITIVELY: the x86 harness prints "5 FAILED" where the others print "5 failed",
-- so a lowercase-only pattern silently exempted the largest suite in the tree from its own gate.
-- (The harness now also exits non-zero, which is the check that should have been load-bearing all
-- along -- this one is the backstop for a suite that forgets to.)
local fails = out:lower():match("([1-9]%d* failed)")
if not ok or fails then return false, (fails or "test run failed").."\n"..out:sub(-400) end
local cases = out:match("(%d+ cases? validated)")
if not cases then