IC testing: compare against nim c, and run the real corpus under nim ic

Every invariant `tests/ic` checked was IC-against-IC — clean == incremental, a
no-op edit changes nothing, a body edit moves no interface cookie. An IC that is
*consistently* wrong satisfies all of them, and that is exactly how two silent
miscompilations survived. `koch bootic` has the same blind spot: it proves the
compiler reproduces itself, not that it agrees with the reference backend.

Two mechanisms, at different scales.

**The oracle, in the metamorphic runner.** Every successful step is now also
compiled with `nim c` and run, and the two outputs must agree. Unlike the
hand-written `expect:` strings this needs no foresight from the test author: it
compares everything the program does, not only what someone thought to print,
which is what a silently-skipped destructor evades. `no-oracle` opts out.

The format also grew the expressiveness the recent bug hunt showed was missing —
every one of these described a state the suite could not reach:

* `#!DELETE <file>` removes a module. Deleting a still-imported file moves no
  mtime, so nothing re-fires.
* `#!FLAGS <switches>` changes the compiler switches between steps. A config
  change is not a file either.
* `fails: <substring>` asserts that BOTH compilers reject the program with that
  text. Previously every step had to succeed, so the whole error path — and
  recovery from it — was untested.

Six regression tests cover the eight bugs the last round fixed.
`testament r <file>` now dispatches metamorphic tests like `testament cat ic`.

**`testament --ic` runs the whole corpus through the incremental compiler**, so
IC inherits ~10k programs with expected output instead of 30 bespoke tests.
Two things had to change for that to mean anything:

* `nimcacheDir` now keys on the `matrix:` entry too. Two matrix variants of one
  file are two different compilations; sharing a cache meant each run
  invalidated what the previous left — harmless for a backend that caches only
  object files, useless for an incremental one.
* About half the corpus overrides the command wholesale (`cmd: "nim c --gc:arc
  $file"`), bypassing both `$target` and `$options`. Those are rewritten to `ic`
  and given a private cache.

**Warm cache, hastur-style.** A generated warmup program pulling in `system` and
the most-imported stdlib modules is compiled once per distinct compile
configuration into `nimcache/ic_warmup_<hash>`, and each test's empty cache is
seeded from it with mtimes preserved (nifmake compares output-mtime >
input-mtime, so stamping the copies "now" re-fires the whole graph). Only
program-independent artifacts are copied: the frontend NIFs and cookies plus the
per-module `lower`/`cg` output. The `.c`/`.o` are left behind on purpose — the
merge decision is whole-program, so they are re-rendered for every program
anyway.

`tests/destructor` (97 runs): `nim c` 35s cold / 32s warm; `nim ic` ~3m30 cold /
**9.8s warm**.

Compiler changes this required or uncovered:

* `merge` read the live-module list from a manifest the driver writes instead of
  globbing `*.c.nif` off the nimcache. Globbing absorbed artifacts belonging to
  any other program sharing the directory — which is what made a prefilled cache
  produce undefined symbols at link.
* The build-arg signature no longer includes `--icproject:`/`--icPreparsedConfig:`
  (they name where a build lives, not what it produces, so two caches holding
  identical artifacts got different signatures). The precompiled config still
  counts, by content hash, minus its `(nimcache …)` line.
* `.s.deps` seeding is speculative and runs before the prune, so a sidecar entry
  that has gone stale (an import that a `when` no longer takes) can be dropped
  instead of lingering forever; a pruned module's scan artifacts are deleted so
  an edit-accumulated cache still matches a clean one.
* A failed nifmake run no longer prints an `Error:` of its own. The children have
  already reported; adding a build-system status as the LAST error hid the
  compiler's real message from anything reading the final error — every
  reject-style test under `nim ic` said "nifmake failed with exit code: 1".
* `--mm:hooks` fed the mm mode to an on/off switch and failed outright with
  "'on' or 'off' expected, but 'hooks' found". Pre-existing and unrelated to IC;
  only reachable through the explicit switch, since `--newruntime` sets
  `selectedGC` directly.

Running `tests/destructor` under `--ic` currently leaves 10 failures. They are
genuine IC defects, not harness noise (all 97 pass under `nim c`) — the clearest
is `tglobaldestructor`: `graph.globalDestructors` is accumulated while injecting
destructors into a module's top level, but the main module's `cg` — which emits
the teardown — is a different process, so a module-level `var` with a `=destroy`
is never destroyed. Same shape as the init/datInit metas, and it wants the same
fix: record it in the `.c.nif` head.

Validation: `koch bootic` reaches its byte-identical fixed point; `tests/ic` is
36/36; arc, destructor, macros, template, iter, closure, ccg, codegen, types and
effects pass under `nim c`, with generics showing only its pre-existing failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

testament/categories.nim

@@ -516,7 +516,15 @@ proc testNimblePackages(r: var TResults; cat: Category; packageFilter: string) =
 # accumulated file set is materialised before each `#!STEP`. A `#!STEP`'s
 # attributes are `;`-separated, each either `key: value` or a bare flag:
 #   expect: <stdout>   noop   body-edit   iface-edit   modules: <n>   clean
+#   fails: <substring>   no-oracle
 # The last step always also runs the clean==incremental check.
+#
+# Every successful step is ALSO compiled with `nim c` and run, and the two
+# outputs must agree (`no-oracle` opts out). This is the only check in the suite
+# that is not IC-against-IC; without it a consistently wrong IC passes
+# everything. `#!DELETE <file>` removes a module, `#!FLAGS <switches>` changes
+# the compiler switches from that point on, and `fails: <text>` asserts that
+# BOTH compilers reject the program with that text.
 
 type MetamorphicError = object of CatchableError
   resultKind: TResultEnum
@@ -590,16 +598,34 @@ proc runMetamorphicIcTest(r: var TResults; file: string; cat: Category; options:
   let buildDir = (file.changeFileExt("") & "_mm").absolutePath
   let nc = buildDir / "nc"
   let bin = buildDir / "prog".addFileExt(ExeExt)
+  # The ORACLE: the same sources compiled by the classic backend. Every
+  # invariant this runner checked before was IC-against-IC (clean == incremental,
+  # no-op changes nothing, ...), which a *consistently* wrong IC satisfies
+  # perfectly — that is how a whole class of silent miscompilations (top-level
+  # destructors never injected; `nfFirstWrite`/`nfLastRead` dropped by the
+  # serializer, so every first assignment to a destructor-bearing local became
+  # `=sink` over zeroed memory) stayed invisible. `nim c` is the reference the
+  # suite was missing.
+  let ncRef = buildDir / "ncref"
+  let binRef = buildDir / "progref".addFileExt(ExeExt)
   removeDir(buildDir)
   createDir(buildDir)
 
+  # Extra switches for both compilers, settable per step via `#!FLAGS`.
+  var extraFlags: seq[string] = @[]
+
   template compileIc(): untyped =
-    execCmdEx2(compilerPrefix, ["ic", "--hint:Conf:off", "--warnings:off",
-      "--nimcache:" & nc, "--out:" & bin, "main.nim"],
+    execCmdEx2(compilerPrefix, @["ic", "--hint:Conf:off", "--warnings:off",
+      "--nimcache:" & nc, "--out:" & bin] & extraFlags & @["main.nim"],
+      workingDir = buildDir)
+
+  template compileRef(): untyped =
+    execCmdEx2(compilerPrefix, @["c", "--hint:Conf:off", "--warnings:off",
+      "--nimcache:" & ncRef, "--out:" & binRef] & extraFlags & @["main.nim"],
       workingDir = buildDir)
 
   # Parse the source into a flat op list: ("file", name, content) | ("step", attrs, "").
-  type OpKind = enum opFile, opStep
+  type OpKind = enum opFile, opStep, opDelete, opFlags
   type Op = object
     kind: OpKind
     a, b: string
@@ -615,6 +641,18 @@ proc runMetamorphicIcTest(r: var TResults; file: string; cat: Category; options:
       if s.startsWith("#!FILE"):
         flushFile()
         curName = s["#!FILE".len .. ^1].strip
+      elif s.startsWith("#!DELETE"):
+        # Remove a module from the virtual file system AND from disk. Deleting a
+        # still-imported file moves no mtime, so nothing in an mtime-keyed build
+        # re-fires: `nim ic` used to relink a stale binary where `nim c` reports
+        # `cannot open file`. Untestable until the format could express it.
+        flushFile()
+        ops.add Op(kind: opDelete, a: s["#!DELETE".len .. ^1].strip)
+      elif s.startsWith("#!FLAGS"):
+        # Change the compiler switches for the following steps. Config changes
+        # are not files, so an mtime-keyed build cannot see them either.
+        flushFile()
+        ops.add Op(kind: opFlags, a: s["#!FLAGS".len .. ^1].strip)
       elif s.startsWith("#!STEP"):
         flushFile()
         ops.add Op(kind: opStep, a: s["#!STEP".len .. ^1].strip)
@@ -630,11 +668,21 @@ proc runMetamorphicIcTest(r: var TResults; file: string; cat: Category; options:
   var prevSnap = initTable[string, string]()
   var prevBin = ""
   var stepIdx = 0
+  var deleted: seq[string] = @[]
   try:
     for o in ops:
-      if o.kind == opFile:
+      case o.kind
+      of opFile:
         vfs[o.a] = o.b
         continue
+      of opDelete:
+        vfs.del o.a
+        deleted.add o.a
+        continue
+      of opFlags:
+        extraFlags = o.a.splitWhitespace()
+        continue
+      of opStep: discard
       inc stepIdx
       let where = "step " & $stepIdx
       # Parse step attributes.
@@ -646,8 +694,36 @@ proc runMetamorphicIcTest(r: var TResults; file: string; cat: Category; options:
         if c >= 0: attrs[p[0 ..< c].strip] = p[c+1 .. ^1].strip
         else: attrs[p] = ""
 
+      for fn in deleted:
+        removeFile(buildDir / fn)
+      deleted.setLen 0
       for fn, content in vfs: writeFile(buildDir / fn, content)
       let (_, cout, ccode) = compileIc()
+
+      # `fails: <substring>` — the build MUST fail, with that text in its output.
+      # Without this every step had to succeed, so the whole error path was
+      # untested: a `nim m` that errored still wrote its `.s.bif`, nifmake then
+      # saw the rule satisfied, and the NEXT run reported success for a program
+      # that does not compile.
+      if "fails" in attrs:
+        if ccode == 0:
+          mmRaise(reBuildFailed, "a failed build", where & ": `nim ic` unexpectedly succeeded")
+        let want = attrs["fails"]
+        if want.len > 0 and want notin cout:
+          mmRaise(reOutputsDiffer, want, where & ": error text did not contain it:\n" & cout)
+        # The oracle must reject it too, else the test is asserting an IC-only
+        # error rather than a real one.
+        let (_, refOut, refCode) = compileRef()
+        if refCode == 0:
+          mmRaise(reBuildFailed, "`nim c` to fail too",
+            where & ": `nim ic` failed but `nim c` accepted the program:\n" & cout)
+        if want.len > 0 and want notin refOut:
+          mmRaise(reOutputsDiffer, want,
+            where & ": `nim c` failed differently:\n" & refOut)
+        prevSnap = snapshotDir(nc)
+        prevBin = ""
+        continue
+
       if ccode != 0:
         mmRaise(reBuildFailed, "", where & ": `nim ic` failed:\n" & cout)
       let (_, rout, rcode) = execCmdEx2(bin.absolutePath, [], workingDir = buildDir)
@@ -658,6 +734,22 @@ proc runMetamorphicIcTest(r: var TResults; file: string; cat: Category; options:
         if rout.strip == want.strip: discard
         else: mmRaise(reOutputsDiffer, want, where & " output:\n" & rout.strip)
 
+      # ORACLE: same sources through the classic backend, same observable
+      # behaviour. Unlike `expect:` this needs no foresight from the test author —
+      # it compares everything the program does, not only what someone thought to
+      # print, which is exactly what a silently-skipped destructor evades.
+      block oracle:
+        if "no-oracle" in attrs: break oracle
+        let (_, refCout, refCcode) = compileRef()
+        if refCcode != 0:
+          mmRaise(reBuildFailed, "", where & ": `nim c` (oracle) failed:\n" & refCout)
+        let (_, refRout, refRcode) = execCmdEx2(binRef.absolutePath, [],
+                                                workingDir = buildDir)
+        if refRout.strip != rout.strip or refRcode != rcode:
+          mmRaise(reOutputsDiffer, "`nim c` output:\n" & refRout.strip,
+            where & ": `nim ic` disagrees with `nim c`\n  ic  (exit " & $rcode &
+            "):\n" & rout.strip & "\n  c   (exit " & $refRcode & "):\n" & refRout.strip)
+
       let snap = snapshotDir(nc)
       let binBytes = stableBinary(bin)
       if stepIdx > 1:
@@ -755,6 +847,12 @@ proc processSingleTest(r: var TResults, cat: Category, options, test: string, ta
     let target = if cat.string.normalize == "js": targetJS else: targetC
     targets = {target}
   doAssert fileExists(test), test & " test does not exist"
+  # `testament r <file>` must dispatch metamorphic IC tests the same way
+  # `testament cat ic` does, otherwise a single-test run tries to parse the
+  # header as an ordinary spec and rejects it.
+  if isMetamorphicIcTest(readFile(test)):
+    runMetamorphicIcTest(r, test, cat, options)
+    return
   testSpec r, makeTest(test, options, cat), targets
 
 proc isJoinableSpec(spec: TSpec): bool =

testament/testament.nim

@@ -12,7 +12,7 @@
 import
   std/[strutils, pegs, os, osproc, streams, json,
     parseopt, browsers, terminal, exitprocs,
-    algorithm, times, intsets, macros]
+    algorithm, times, intsets, macros, tables]
 
 import backend, specs, azure, htmlgen
 
@@ -35,6 +35,12 @@ var simulate = false
 var optVerbose = false
 var useMegatest = true
 var valgrindEnabled = true
+var useIc = false
+  ## `--ic`: compile every C-target test with `nim ic` instead of `nim c`, so the
+  ## incremental compiler inherits the whole existing corpus (~10k programs with
+  ## expected output) instead of the handful of tests written for it by hand.
+  ## Every invariant the `tests/ic` suite checks is IC-against-IC; this is the
+  ## part that compares IC against the reference backend at scale.
 
 proc verboseCmd(cmd: string) =
   if optVerbose:
@@ -58,6 +64,7 @@ Arguments:
 Options:
   --print                   print results to the console
   --verbose                 print commands (compiling and running tests)
+  --ic                      compile C-target tests with `nim ic` (incremental)
   --simulate                see what tests would be run but don't run them (for debugging)
   --failing                 only show failing/ignored tests
   --targets:"c cpp js objc" run tests for specified targets (default: c)
@@ -155,23 +162,146 @@ proc execCmdEx2(command: string, args: openArray[string]; workingDir: string = "
       if result.exitCode != -1: break
   close(p)
 
-proc nimcacheDir(filename, options: string, target: TTarget): string =
+proc nimcacheDir(filename, options: string, target: TTarget,
+                 extraOptions = ""): string =
   ## Give each test a private nimcache dir so they don't clobber each other's.
-  let hashInput = options & $target
+  ## `extraOptions` (a `matrix:` entry) is part of the key: two matrix variants
+  ## of one file are two different compilations, and sharing a cache between them
+  ## means each run invalidates what the previous left. Harmless for the classic
+  ## backend, which caches only object files, but it makes an incremental cache
+  ## useless — every variant re-sems the world every time.
+  let hashInput = options & extraOptions & $target
   result = "nimcache" / (filename & '_' & hashInput.getMD5)
 
+const icWarmupSource = """
+# Generated by testament for `--ic`. Compiling this once fills a shared IC cache
+# with `system` and the stdlib modules the test corpus imports most, so each
+# test's own cold build starts from precompiled NIFs instead of re-semming the
+# world. Mirrors nimony's hastur `tools/warmup.nim` + `prefillFromWarmup`.
+import std/[assertions, macros, strutils, tables, os, typetraits, sequtils,
+            sugar, math, options, times, json, sets, algorithm, hashes,
+            strformat, parseutils, streams, unicode]
+
+proc icWarmupAnchor*(): int =
+  # Reference a few generic instantiations the corpus leans on so their
+  # `.c.nif` artifacts are precompiled too, not just the modules' interfaces.
+  var t = initTable[string, int]()
+  t["a"] = 1
+  var s = @[1, 2, 3]
+  s.sort()
+  result = s.len + t.len + "x".repeat(2).len
+"""
+
+var icWarmupCaches: Table[string, string]
+  ## Compile-config key -> shared warm IC cache (or "" when unavailable).
+var buildingIcWarmup = false
+
 proc prepareTestCmd(cmdTemplate, filename, options, nimcache: string,
                      target: TTarget, extraOptions = ""): string =
   var options = target.defaultOptions & ' ' & options
   if nimcache.len > 0: options.add(" --nimCache:$#" % nimcache.quoteShell)
   options.add ' ' & extraOptions
+  # `--ic` swaps the C target's command; every other target is left alone (the
+  # incremental compiler has a C backend only).
+  let targetCmd =
+    if useIc and target == targetC: "ic" else: targetToCmd[target]
   # we avoid using `parseCmdLine` which is buggy, refs bug #14343
-  result = cmdTemplate % ["target", targetToCmd[target],
+  result = cmdTemplate % ["target", targetCmd,
                       "options", options, "file", filename.quoteShell,
                       "filedir", filename.getFileDir(), "nim", compilerPrefix]
+  if useIc and target == targetC:
+    # Roughly half the corpus overrides the command wholesale (`cmd: "nim c
+    # --gc:arc $file"`), which neither goes through `$target` nor picks up
+    # `$options` — so those tests would silently keep using the classic backend
+    # and share one nimcache. Rewrite the compile verb and give them a private
+    # cache, which is what makes them incremental at all.
+    let prefix = compilerPrefix & " c "
+    if result.startsWith(prefix):
+      result = compilerPrefix & " ic " & result[prefix.len .. ^1]
+    if nimcache.len > 0 and "--nimCache:" notin result and "--nimcache:" notin result:
+      # Must land BEFORE the project file: anything after it is swallowed into
+      # `config.arguments`, and a non-empty `arguments` without `--run` is a hard
+      # error ("arguments can only be given if the '--run' option is selected").
+      let fileArg = filename.quoteShell
+      let at = result.find(fileArg)
+      let switch = "--nimCache:" & nimcache.quoteShell & " "
+      if at >= 0: result = result[0 ..< at] & switch & result[at .. ^1]
+      else: result.add " " & switch
+
+proc icWarmupCache(cmdTemplate, filename, options: string, target: TTarget,
+                   extraOptions: string): string =
+  ## The shared warm cache for this test's exact compile configuration, built on
+  ## first use and kept in `nimcache/` across runs. Keyed by the switches AND the
+  ## test's directory, because both decide what the artifacts contain: the
+  ## switches through `-d:`/`--mm:` etc., the directory through the `nim.cfg` /
+  ## `config.nims` it inherits. A cache built under a different configuration
+  ## would just be invalidated wholesale on first use, which is worse than none.
+  if buildingIcWarmup: return ""
+  let dir = filename.getFileDir()
+  let key = options & extraOptions & $target & dir
+  if icWarmupCaches.hasKey(key): return icWarmupCaches[key]
+  result = "nimcache" / ("ic_warmup_" & key.getMD5)
+  icWarmupCaches[key] = result
+  if dirExists(result / "ic.version"): return   # already built by an earlier run
+  if fileExists(result / "ic.version"): return
+  # The warmup must live in the test's own directory so it inherits the same
+  # config files; a stray `.nim` there is not picked up as a test (testament
+  # only collects `t*.nim`). The name must be a valid Nim identifier.
+  let src = dir / "icwarmup_generated.nim"
+  try:
+    writeFile(src, icWarmupSource)
+  except IOError, OSError:
+    icWarmupCaches[key] = ""
+    return ""
+  buildingIcWarmup = true
+  let cmd = prepareTestCmd(cmdTemplate, src, options, result, target, extraOptions)
+  let (outp, code) = execCmdEx(cmd)
+  buildingIcWarmup = false
+  try: removeFile(src)
+  except OSError: discard
+  if code != 0:
+    # Non-fatal: without a warm cache every test just pays its own cold build.
+    if optVerbose: echo "ic warmup failed: ", cmd, "\n", outp
+    icWarmupCaches[key] = ""
+    return ""
+
+proc prefillIcCache(warmup, nimcache: string) =
+  ## Seed a test's empty cache from the shared warm one. Only the artifacts that
+  ## do NOT depend on which program is being built are copied: the frontend NIFs
+  ## and cookies, plus the per-module `lower`/`cg` outputs. The `.c`/`.o` are
+  ## deliberately left out — the merge decision (who owns each emit-everywhere
+  ## definition) is whole-program, so those get re-rendered for every program
+  ## anyway and copying them is pure I/O.
+  ##
+  ## Mtimes are preserved, and that is load-bearing: nifmake decides staleness by
+  ## output-mtime > input-mtime, so stamping every prefilled file with "now"
+  ## would scramble the DAG ordering the warmup established and re-fire the
+  ## whole graph — exactly what the copy is meant to avoid.
+  if warmup.len == 0 or not dirExists(warmup): return
+  if dirExists(nimcache): return          # the test already has its own cache
+  const wanted = [".p.nif", ".p.deps.nif", ".deps.nif", ".s.bif", ".iface.bif",
+                  ".impl.bif", ".edges.bif", ".s.deps.bif", ".t.bif", ".c.nif"]
+  try:
+    createDir(nimcache)
+    for path in walkFiles(warmup / "*"):
+      let name = path.extractFilename
+      var take = name == "ic.version" or name == "ic_build_args.txt"
+      if not take:
+        for ext in wanted:
+          if name.endsWith(ext): take = true; break
+      if not take: continue
+      let dst = nimcache / name
+      copyFile(path, dst)
+      try: setLastModificationTime(dst, getLastModificationTime(path))
+      except OSError, IOError: discard
+  except OSError, IOError:
+    discard  # best effort; a cold build still works
 
 proc callNimCompiler(cmdTemplate, filename, options, nimcache: string,
                      target: TTarget, extraOptions = ""): TSpec =
+  if useIc and target == targetC and nimcache.len > 0 and not buildingIcWarmup:
+    prefillIcCache(icWarmupCache(cmdTemplate, filename, options, target, extraOptions),
+                   nimcache)
   result = TSpec(cmd: prepareTestCmd(cmdTemplate, filename, options, nimcache, target,
                           extraOptions))
   verboseCmd(result.cmd)
@@ -590,7 +720,7 @@ proc targetHelper(r: var TResults, test: TTest, expected: TSpec, extraOptions: s
       inc count
       echo "testSpec count: ", count, " expected: ", expected
     else:
-      let nimcache = nimcacheDir(test.name, test.options, target)
+      let nimcache = nimcacheDir(test.name, test.options, target, extraOptions)
       var testClone = test
       let target = changeTarget(extraOptions, target)
       testSpecHelper(r, testClone, expected, target, extraOptions, nimcache)
@@ -691,6 +821,7 @@ proc main() =
     case p.key.normalize
     of "print": optPrintResults = true
     of "verbose": optVerbose = true
+    of "ic": useIc = true
     of "failing": optFailing = true
     of "pedantic": discard # deadcode refs https://github.com/nim-lang/Nim/issues/16731
     of "targets":