55b244efff
`interfHidden` was 1.05s of a cold Atlas build: 1.70M hidden-symbol
stubs against 0.29M exported ones, created by every `nim m` for every
module it imports. The table is reached ONLY through
`modulegraphs.interfSelect` with `optImportHidden`, and that flag is set
in exactly one place — an `import x {.all.}`. Almost nothing reads it.
So it is built when something asks. Every read goes through
`interfSelect`, so guarding its four call sites is complete;
`modulegraphs` already imports `ast2nif`, so the call is direct.
The reason the first attempt at this failed, recorded because it is not
guessable: **a module has two FileIndexes.** `registerNifSuffix` keys
`filenameToIndexTbl` by the NIF SUFFIX and mints a `fikNifModule` entry,
while the graph indexes `g.ifaces` by the module's `fikSource` file, and
`DecodeContext.mods` is keyed by the former. Asking it with the latter
misses every time, silently — `import x {.all.}` then reported
"undeclared identifier" for a symbol that was right there. The lazy
builder therefore takes a SUFFIX. Two more conditions are load-bearing:
clear the pending flag only when the build SUCCEEDS, since an import
whose `.s.bif` the build has not produced yet must be retried rather than
written off for the rest of the process; and build into a LOCAL table
before assigning it back, since loading symbols can grow `g.ifaces` and
leave a `var` alias into it dangling.
Atlas, 204 modules:
frontend 9.98s -> 8.96s loading 4.57s -> 3.55s
InterfTables 1161ms -> 79ms hidden stubs 1.70M -> 0
cold serial 21.59s -> 19.79s cold parallel 11.11s -> 10.47s
Also fixes a scanner gap the test exposed. `import x {.all.}` serialises
as `(pragmax x (pragmas all))`, which `deps.nim.parseImportPath` did not
recognise, so it fell into the unknown-subtree skip and the import was
DROPPED from the static graph — the build only learned about it from the
`.s.deps` sidecar a round later, after a round that failed with
"requires precompiled NIF for import". Correct, but a wasted round and an
alarming error line for an ordinary import.
`tests/ic/timporthidden.nim` covers it: `{.all.}` sees the private
symbols, and the sibling case (a plain `import`) still rejects them under
both `--ic:on` and `--ic:off`. ic 42/42, `koch boot -d:release` equal
executables.
Running tests/ic
./bin/testament --nim:<your compiler> cat ic
The metamorphic tests are expensive, and look hung when they are not
16 of the tests carry #? metamorphic. Each has 3–4 #!STEP directives, and
every step compiles the program twice — once under nim ic, once with
nim c as the reference oracle. That is 100+ full compilations for the
category. Under --ic:on each compilation additionally fans out one backend
process per module per stage, and each of those is a compiler holding its own
module graph (~800MB peak).
A nim ic parent sitting at 0% CPU is normal. It is waiting on its
children. It is not a deadlock, and neither is a metamorphic test that occupies
the runner for many minutes. Before concluding anything is stuck, check that the
test NAME changes over a few minutes — that is the difference between slow and
hung, and it is easy to get wrong.
On a memory-constrained machine the fan-out will swap. The symptoms are exactly
the ones that read as a deadlock: several processes at 0% CPU, no output, a
different test "stuck" on every run, and the same compilation finishing in
seconds when run on its own. Check vm_stat (page-ins per second) and
sysctl vm.swapusage before looking for a bug. This was diagnosed as a
testament/nim ic interaction more than once before anyone measured.
Cap the fan-out to fit the machine — precedence documented at deps.nim's
let parallel:
--parallelBuild:N # standard flag, given meaning under IC
-d:icJobs:N # same cap, legacy define
-d:icNoParallel # serial, and non-interleaved child output
Serial output matters for a second reason: the parallel backend processes share
one stderr, so any per-process diagnostic printing (NIM_IC_BNODE_GRIND,
-d:icCanRaiseLog) interleaves and produces torn lines. Either use
-d:icNoParallel or parse defensively and count what you dropped.
Running a single test
testament r tests/ic/<file>.nim works for the ordinary tests. It does NOT work
for the metamorphic ones — the multi-step files carry several discard """
spec blocks and the single-test path rejects them with "duplicate specStart".
Those only run through cat ic.
Files matching tests/ic/*_temp.nim are ignored by git (see .gitignore) and
are scratch, not tests: several import helper modules that do not exist and fail
for that reason alone.