mirror of
https://github.com/nim-lang/Nim.git
synced 2026-09-01 19:33:42 +00:00
467c911dc5
Every invariant `tests/ic` checked was IC-against-IC — clean == incremental, a no-op edit changes nothing, a body edit moves no interface cookie. An IC that is *consistently* wrong satisfies all of them, and that is exactly how two silent miscompilations survived. `koch bootic` has the same blind spot: it proves the compiler reproduces itself, not that it agrees with the reference backend. Two mechanisms, at different scales. **The oracle, in the metamorphic runner.** Every successful step is now also compiled with `nim c` and run, and the two outputs must agree. Unlike the hand-written `expect:` strings this needs no foresight from the test author: it compares everything the program does, not only what someone thought to print, which is what a silently-skipped destructor evades. `no-oracle` opts out. The format also grew the expressiveness the recent bug hunt showed was missing — every one of these described a state the suite could not reach: * `#!DELETE <file>` removes a module. Deleting a still-imported file moves no mtime, so nothing re-fires. * `#!FLAGS <switches>` changes the compiler switches between steps. A config change is not a file either. * `fails: <substring>` asserts that BOTH compilers reject the program with that text. Previously every step had to succeed, so the whole error path — and recovery from it — was untested. Six regression tests cover the eight bugs the last round fixed. `testament r <file>` now dispatches metamorphic tests like `testament cat ic`. **`testament --ic` runs the whole corpus through the incremental compiler**, so IC inherits ~10k programs with expected output instead of 30 bespoke tests. Two things had to change for that to mean anything: * `nimcacheDir` now keys on the `matrix:` entry too. Two matrix variants of one file are two different compilations; sharing a cache meant each run invalidated what the previous left — harmless for a backend that caches only object files, useless for an incremental one. * About half the corpus overrides the command wholesale (`cmd: "nim c --gc:arc $file"`), bypassing both `$target` and `$options`. Those are rewritten to `ic` and given a private cache. **Warm cache, hastur-style.** A generated warmup program pulling in `system` and the most-imported stdlib modules is compiled once per distinct compile configuration into `nimcache/ic_warmup_<hash>`, and each test's empty cache is seeded from it with mtimes preserved (nifmake compares output-mtime > input-mtime, so stamping the copies "now" re-fires the whole graph). Only program-independent artifacts are copied: the frontend NIFs and cookies plus the per-module `lower`/`cg` output. The `.c`/`.o` are left behind on purpose — the merge decision is whole-program, so they are re-rendered for every program anyway. `tests/destructor` (97 runs): `nim c` 35s cold / 32s warm; `nim ic` ~3m30 cold / **9.8s warm**. Compiler changes this required or uncovered: * `merge` read the live-module list from a manifest the driver writes instead of globbing `*.c.nif` off the nimcache. Globbing absorbed artifacts belonging to any other program sharing the directory — which is what made a prefilled cache produce undefined symbols at link. * The build-arg signature no longer includes `--icproject:`/`--icPreparsedConfig:` (they name where a build lives, not what it produces, so two caches holding identical artifacts got different signatures). The precompiled config still counts, by content hash, minus its `(nimcache …)` line. * `.s.deps` seeding is speculative and runs before the prune, so a sidecar entry that has gone stale (an import that a `when` no longer takes) can be dropped instead of lingering forever; a pruned module's scan artifacts are deleted so an edit-accumulated cache still matches a clean one. * A failed nifmake run no longer prints an `Error:` of its own. The children have already reported; adding a build-system status as the LAST error hid the compiler's real message from anything reading the final error — every reject-style test under `nim ic` said "nifmake failed with exit code: 1". * `--mm:hooks` fed the mm mode to an on/off switch and failed outright with "'on' or 'off' expected, but 'hooks' found". Pre-existing and unrelated to IC; only reachable through the explicit switch, since `--newruntime` sets `selectedGC` directly. Running `tests/destructor` under `--ic` currently leaves 10 failures. They are genuine IC defects, not harness noise (all 97 pass under `nim c`) — the clearest is `tglobaldestructor`: `graph.globalDestructors` is accumulated while injecting destructors into a module's top level, but the main module's `cg` — which emits the teardown — is a different process, so a module-level `var` with a `=destroy` is never destroyed. Same shape as the init/datInit metas, and it wants the same fix: record it in the `.c.nif` head. Validation: `koch bootic` reaches its byte-identical fixed point; `tests/ic` is 36/36; arc, destructor, macros, template, iter, closure, ccg, codegen, types and effects pass under `nim c`, with generics showing only its pre-existing failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This directory contains the test cases.
Each test must have a filename of the form: t*.nim
Note: Testament is only aware of tests under a directory (eg tests/foo/) and will ignore
top-level tests like tests/tbar.nim.
Specs
Each test can contain a spec in a discard """ ... """ block.
Check out the parseSpec procedure in the specs module for a full and reliable reference
action
Specifies what action this test should take.
Default: run
Options:
compile- compiles the module and fails the test if compilations fails.run- compiles and runs the module, fails the test if compilation or execution of test code fails.reject- compiles the module and fails the test if compilation succeeds.
There are certain spec keys that imply run, including output and
outputsub.
Categories
Each folder under this directory represents a test category, which can be
tested by running koch tests pcat <category> (or cat to avoid parallel
testing, which is slower).
The folder dll contains simple DLL tests.
The folder realtimeGC contains a test for validating that the realtime GC
can run properly without linking against the nimrtl.dll/so.