`for i in k..<n.len: ... n[i] ...` is the dominant shape for walking a `PNode`'s
children in the code generator: 41 such loops across the cgen files, and 777
indexed node accesses in total. It reads worse than iterating, it bounds-checks
every subscript, and it is quadratic the moment the backend reads children off a
NIF `Cursor` rather than a materialised tree (a child is `firstSon` plus one
`skip` per preceding sibling, and `skip` steps over a whole subtree).
31 of the 41 are converted:
* 20 to `sons`/`sonsFrom` — the index only ever subscripted `n`.
* 9 to `isons`, which now takes a `start` index (defaulting to 0, so its eight
existing call sites are unchanged). These genuinely need `i`: a parallel index
into the routine's `PType` (`typ.n[i]`, `typ[i]`), a `needTmp[i-1]` lookup, an
`i == field.position` test, `$i` in a generated struct name, or the index
passed straight to `genOtherArg`.
* 2 to `sonsFrom` with a variable start (`firstParam`, `offset`).
`sonsFrom` is new, next to `sons`/`isons` in astdef.
The remaining 10 are deliberate. Eight are not `PNode` at all — `varargs[Snippet]`,
`seq[PSym]`, `string`, and `PType`, where `sons` is a `proc ...: var TTypeSeq`
rather than an iterator, so a blind rewrite would compile into something quite
different. Two iterate `0..<it.len-1`, excluding the last child, which no
iterator expresses cleanly.
Pure refactor, and verified as one: all 219 generated `.c` files of a 219-module
program and the linked binary are byte-identical to the parent commit. That is
the bar that matters here, because the index arithmetic (`i-1`, `i == position`,
`$i`) is the easy thing to get wrong. It also caught a real slip on the way:
`genFieldCheck` reassigns its loop variable, which a `for` binding cannot do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`--icBackendStage:link` called `loadBackendModules` — a whole-program
deserialization — for exactly two things: each module's `.c` path via
`getCFile`, and its recorded C compile/link directives via
`replayBackendActions`. That was 3.7s of the ~11s serial backend critical path
on a 219-module program, spent recovering a list of paths and a handful of
strings.
Both now come from artifacts the earlier stages already produce:
* `.c` paths from the driver's existing `LiveModulesFile` manifest (which
`merge` already reads); the `.c` sits beside each listed `.c.nif`, so no new
manifest was needed. The merge-decision fallback for conditionally-imported
nodes that own a live symbol is kept.
* C directives from a new `<module>.c.cflags` sidecar. The module's own `cg`
already replays them, so it writes them down too — one tab-separated line per
`compile`/`link`/`passl`/`passc`/`localpassc`/`cppdefine` action. `localpassc`
needs the module's source path, which only the writer can resolve, so it is
baked into the line. Written unconditionally, empty included: it is a declared
nifmake output of the `cg` rule and a missing output re-fires the rule for
ever.
link: 3.70s -> 0.44s. A one-line code edit on the corpus goes 15.2s -> 11.8s;
cold 124s -> 120s; no-op unchanged at 0.27s.
Green: 16/16 metamorphic IC tests, the 17-file `koch ic` suite, 13/13
differential edit checks against `nim c`, and `bootic` iteration 1 (the full
fixed-point check was still running when this was committed).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`registerModuleCode` writes the module's `.c` and registers it for compilation.
Under `--icBackendStage:cg` it must do neither: `cg`'s product is the `.c.nif`,
the `.c` is rendered by `emit` from that artifact plus the GLOBAL merge
decision, and the compile registration belongs to `link`.
Writing it in `cg` puts a second, differently-filtered `.c` — `cg` can only
filter by the liveness its own process sees — at the exact path `emit` declares
as its nifmake output. Two stages claim one output, and the `.c` ends up newer
than `emit`'s own `.c.nif` input, so nifmake considers `emit` up to date and
skips it: `cg`'s unfiltered text goes to the linker and every emit-everywhere
body is defined twice ("multiple definition of eqdup__u727466987__OOZ…").
Nothing surfaces this today because `merge` rewrites the decision file on every
run and every `emit` lists it as an input, so all of them re-fire and overwrite
the stray file. The fire-all is therefore load-bearing, not the "insurance" it
is documented as — and it is what makes a content-stable merge decision
(219 fewer processes and ~13s less backend CPU per edit on a 219-module
program) impossible: that experiment is exactly how this was found.
No behaviour change on its own, since the fire-all still runs. Green: `bootic`
fixed point, 16/16 metamorphic IC tests, the 17-file `koch ic` suite, and 13/13
differential edit checks against `nim c`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`loadTransitiveHooks` loads a module only to register its hooks, macro-cache
replay and generic-instance offers. It is a dep-of-a-dep, not an import, so
none of its symbols are visible to the module being semchecked — yet it went
through the full `loadNifModule`, which builds the interface string tables by
calling `loadSymFromIndexEntry` on every index entry, and `processTopLevel`,
whose `export` branch calls `resolveSym` on every exported symbol. Both write
into an `interf`/`interfHidden` pair that is scratch, shared across all
iterations, and never read. Their only other effect is warming the name-keyed
`c.syms` cache, which `resolveSym` refills lazily from the same index on a miss.
New `LoadFlag.SkipInterfaceTables`, set only by `loadTransitiveHooks`, gates
both. Measured on a 219-module program: a one-line edit rebuild goes 5.58s ->
3.21s (-42%), and the compiler's own cold IC build 202s -> 164s (-19%), since
every `nim m` in a cold build pays this too. Profile of the `nim m` before:
direct imports 17 modules/1.40s, transitive hooks 209 modules/2.99s,
re-exports 39/0.23s.
This is NOT byte-inert, unlike the previous commit: skipping the eager stub
creation shifts the per-module load-order counter, so lazily-created stubs get
different serialized item ids and 44 of 238 `.s.bif` change. The artifacts stay
deterministic (238/238 reproducible across two independent cold builds) and
16/16 metamorphic tests pass, including `clean cache == incremental cache` and
`a no-op edit changes no artifact` — so an existing nimcache re-sems those
modules once and then converges. Also green: the 17-file `koch ic` suite, 13/13
differential edit checks against `nim c`, both `bootic` fixed points (debug and
release), and 482 tests across tests/arc, destructor, closure, iter, gc, async
with a failure set identical to the parent commit's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rule deciding which integer identifies a backend-minted symbol —
content-derived `disamb` for a lifted hook (`setHookDisamb`), `itemId.item`
otherwise — was written out at three sites: `mangleProcNameExt` and
`ccgutils.makeUnique` for the C name, `ast2nif.toNifSymName` for the NIF name,
each carrying its own copy of the ten-line rationale.
They drifted, which is exactly cce17461d: `toNifSymName` lacked the hook
exception, so the loader overwrote a content-derived value and two unrelated
`=destroy` hooks collided on one C name (C accepted the mistyped call, C++
rejected it). `astdef.backendMintedDisamb` is now the single definition and all
three call it. `globalName` still reads `disamb` directly — correct for a loaded
symbol, and the round-trip invariant that makes it agree is now stated in the
shared function instead of left implicit.
Pure de-duplication, verified as such: all 219 generated `.c` files of a
219-module corpus and the linked binary are byte-identical to the previous
commit. Plus 16/16 metamorphic IC tests, the 17-file `koch ic` suite, 13/13
differential edit checks against `nim c`, and the debug and release `bootic`
fixed points.
Note for the record: this started as an attempt to replace the per-process
`_c<itemId.item>` counter with a content hash. Instrumenting both mangling
sites showed that branch is never taken — 0 of 166 backend-minted manglings on
the corpus, 0 of 257 on the compiler, all going through the content-derived
path — so rewriting the scheme would have shifted every backend-minted C name
and forced an `icFormatVersion` bump for no demonstrable benefit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grinding a small figdraw-based program under `nim ic` and diffing its
output against the classic backend surfaced eight bugs, four of which
silently produced a wrong binary rather than an error.
Frontend / build graph (`deps.nim`):
* Dead `when`-guarded imports were compiled anyway. `when someStrdefine
== "x": import y` is `cvUnknown` to the scanner, which conservatively
keeps the edge — right for an edge, but it also gave `y` its own `nim m`
rule, so a build died on a package the user never installed because they
never selected that backend. Track which edges are speculative and drop
a speculative subtree that cannot compile; if the guard was in fact
live, the discovery fixpoint puts the node back with the honest `cannot
open file`.
* Deleting a still-imported module went unnoticed: no mtime moves, so
nothing re-fires and `nim ic` relinked a stale binary while `nim c`
reported `cannot open file`. Report an unresolvable import from a
non-speculatively reached module during the graph scan.
* Macro-generated imports were discovered once and then forgotten.
Discovery only ran after a failure and the graph is re-derived
statically every run, so on a warm build the discovered module had no
rules at all and editing it changed nothing. Seed the graph from the
`.s.deps` sidecars up front.
* Config changes invalidated nothing. nifmake decides staleness from
file mtimes and never looks at a rule's command line, so `-d:foo=bar` /
`--mm:` / `--threads:` regenerated the build file with the new switches
and re-fired zero rules. Reify the configuration as a file and make it
an input of every rule.
* Command-line switches never reached the children: they replay the
project's config files, never the driver's argv, so `nim ic --opt:speed`
produced a byte-identical debug binary (likewise `--panics`,
`--experimental`, `--passC`). Forward the driver's switches, minus the
ones that must differ per child.
Artifacts and codegen:
* A failed `nim m` still wrote its `.s.bif` and cookies, so nifmake saw
the rule as satisfied on the next run: `nim ic` then reported success
for a program that does not compile, and generated code from
error-bearing AST (or hit an internal error in `ccgexprs`). Never
persist an artifact when `errorCounter > 0`.
* Top-level destructors were never injected. `sfInjectDestructors` lives
on the module symbol, which `moduleFromNifFile` rebuilds from scratch,
so `genTopLevelStmt` skipped `injectDestructorCalls` entirely: a
module-level `block: let h = openHandle()` never ran `=destroy`. Persist
the flag as a `(modflags)` record. `injectdestructors` also has to
tolerate the `nkReplayAction` entries the loader prepends to `topLevel`.
* `nfFirstWrite` / `nfLastRead` were dropped by the serializer. A sym
node is written as a bare NIF `SymUse` token, which has nowhere to put
node flags, so the frontend's move analysis never reached the backend:
EVERY first assignment to a destructor-bearing local compiled as
`=sink`, i.e. `=destroy` on still-zeroed memory followed by a copy, and
no read was ever a move. Wrap a sym use in `(nflags ...)` when it
carries persistent node flags.
In the JS backend `toOpenArray` used `slice` (a copy), so writes through
a `var openArray` parameter silently vanished.
This emits `subarray` (a live shared-buffer view) for homogeneous
numeric arrays, otherwise such parameters are passed as a `{base, off,
len}` view that always aliases the caller's storage. Sliced seq/array
args become `{base, off, len}`, whole values `{base, off:0, len}`,
re-slices rebase.
Un-skips the JS guard in tests/openarray/topenarray.nim
Fixes#15952.
A closure's environment type — and the `=destroy`/`=copy`/`=sink` the compiler
lifts for it — is minted by the BACKEND and exists in no module's semmed NIF.
Grinding a corpus of closure programs against `nim c` as the oracle turned up
three ways the per-module backend gets those wrong. `tests/closure` is now green
under `--ic:on`, and `tests/iter` goes from 13 failures to 9.
* **The env hook is emitted by nobody.** `emitsBodyInThisModule` walks up to the
outermost enclosing routine to pick the TU that emits a body. For the env
`=destroy` of a generic closure iterator defined in one module and instantiated
in another, that walk lands on the module of the ORIGINAL generic — a module
that never sees the instance — so the routine was emitted nowhere
(`undefined reference to eqdestroy__c485__…`). A backend-minted routine has no
owning module NIF at all: it is written into the `.t.bif` of every module that
references it, re-homed there with `@bk`. Every referencing TU emits it now and
the merge stage keeps one, like any other content-addressed definition.
* **The `:up` link is stored without an increment.** `env.:up = enclosingEnv` has
to go through `=copy` (with the cyclic incref) or the parent's refcount is one
too low, and at teardown the two envs' mutually recursive `=destroy`s each
believe they hold the last reference and recurse until the stack is gone — a
SIGSEGV after the program's own output has already been printed
(`tests/iter/tnestedclosures`, "Test 3"). Whether it becomes a `=copy` depends
on the up-field type's hooks existing at injection time. Whole-program cgen got
that for free: a LATER lifting pass creates them and it runs before any
routine's injection. The per-module backend injects a routine right after
lifting it (`lower`), long before the module's top level is transformed at all
(`cg`) — so both up-field assignment sites create the ops themselves.
* **Two backend hooks collide on one C name.** A backend-minted symbol is named
`_c<itemId.item>`, a PER-PROCESS counter — fine while "nifc lifts, emits and
compiles them in one run", which is what `mangleProcNameExt` assumed. But
`lower` mints the env hooks of nested routines and `cg` mints those of the
module's top level, and both land in the same translation unit: two unrelated
`=destroy`s became one C function. `mangleProcNameExt` already makes an
exception for hooks whose `disamb` is content-derived (`HookDisambBit`);
`toNifSymName` now mirrors it, so the content-derived value survives the round
trip instead of being overwritten by the loader.
`InstanceDisambBit`/`HookDisambBit` move to `astdef` — `ast2nif` names symbols by
them and cannot import `modulegraphs`.
Two new metamorphic tests, both of which fail on the previous compiler:
`tclosure_hooks` (generic closure iterator instantiated across modules) and
`tclosure_nested_iter` (closure iterator nested in a closure iterator).
`icFormatVersion` 37 -> 38 for the hook naming. `koch bootic` reaches its
byte-identical fixed point; `tests/closure`, `tests/destructor` (3 known
`--newruntime` failures), `tests/cpp` and the classic categories are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`nim ic` was a command of its own, which made it the C backend only and cut it
off from everything the ordinary compile commands accept. `--ic:on` is a switch
on `nim c` / `nim cpp` / `nim objc` instead, so `-r`, `-d:release`,
`--exceptions:`, and a project-wide opt-in from `nim.cfg` / `config.nims` all
work. `nim ic` still resolves to the same driver; `koch bootic`, `koch ic` and
`testament --ic` now go through the switch.
The switch was already parsed into a `conf.ic` nobody read. It is now read in
`passCmd1` as well, because `nim.nim` has to decide whether this run is an IC
DRIVER before config loading (`ensureIcConfig` produces the precompiled config
the driver itself replays); when the switch comes from a config file instead,
`main.nim` produces it late.
**C++.** `tests/cpp` passes under `--ic:on`, matching its classic result. Four
fixes, three of them the shape of every "C++ needs the whole program" problem the
per-module backend has:
* The driver DECLARES each module's translation unit to nifmake without loading a
module, so it cannot ask `cgen.getCFile` — and it hardcoded `.nim.c`, so the
merge stage went looking for `.c.nif` next to the `.cpp.nif` the children had
written. `options.icCFileExt` mirrors the formula at backend granularity.
* C++ has no designated initializers, so the RTTI record is a bare variable that
`DatInit` fills field by field. A bare `TNimTypeV2 x;` is a tentative
definition — C's linker merges those, C++'s does not — so every TU that
demanded the type defined it ("multiple definition of NTIv2__…"). It now gets
the same extern-declaration + owned-`'d'`-definition split the C flavour has.
* `memberProcsPerType` / `initializersPerType` live only in the sem process, so
the backend emitted a struct WITHOUT its in-class member declarations and the
out-of-class definitions did not match ("no declaration matches
'void Doo::memberProc()'"). They are replayed from a new `(repcppmember …)`
log entry; `replayCppMember` re-derives the type from the routine's signature
exactly as `semCppMember` does, so no type key has to survive the round trip.
* Two follow-ons for members: `loc.snippet` is a CALL PATTERN (`#->salute(@)`),
and only `genMemberProcHeader` derives it — whole-program cgen got it for free
by generating the defining module first, but the per-module backend emits that
body in another process, leaving the caller with the mangled Nim name
(`loo->salute_u0__vireouyks1()`). And that pattern is not a linker name: every
`salute` member in every class mints the same one, so the merge stage handed
them all to one artifact and dropped the rest (undefined vtable at link).
Member definitions are keyed by their NIF name there instead.
`--run` is now dropped when re-invoking for the config artifact: the producer has
no output binary and `nim.nim`'s run step asserted on the empty `outFile`.
testament's `--ic` appends the switch rather than rewriting the compile verb, so
a test that overrides `cmd:` wholesale keeps its verb, and the C++ corpus is
covered too (it never was — the old rewrite only matched `nim c `).
`icFormatVersion` 36 -> 37 for the new log entry. `koch bootic` reaches its
byte-identical fixed point through the new entry point; `koch ic` passes;
`tests/ic`, `tests/destructor` (3 known `--newruntime` failures) and the classic
categories are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`testament --ic` on `tests/destructor` went from 10 failures to 3; the three
left are all `--newruntime`, whose owned-ref RTTI destructor is still missing
(the object type's `=destroy` never reaches the `cg` that emits the type info).
* **global destructors are never run.** `graph.globalDestructors` is filled
while a module's top level goes through `injectDestructorCalls`, and
whole-program cgen empties the list into the main module's init proc — which
IS the program body, so the calls land at exit. Under `nim ic` every module's
`cg` is a separate process and main's only ever saw its own entries: a
module-level `var` with a `=destroy` in an imported module was simply never
destroyed. Each module now wraps its own list in a nullary exported proc and
announces the name in its `.c.nif` meta head (`CnifVersion` 4 -> 5); main's
`cg` reads the heads it already reads for init/datInit and calls them in
reverse dependency order. New test `tests/ic/tglobal_dtors.nim` pins the
order against the oracle.
* **`spawn` died with "system module needs: nimArgsPassingDone".** A module
loaded from a NIF is named by its mangled suffix, so `loadCompilerProc`'s
`module.name.s == "threadpool"` could never match. The backend loads the whole
program before codegen, so consult every loaded module's index instead; misses
are final and remembered (`getCompilerProc` doubles as a presence probe).
* **`new(x, finalizer)` died with "environment misses: x".** `semmagic`'s
finalizer-to-destructor wrapper copies the original's AST but only rewrites
the parameter, leaving `ast[namePos]` naming the ORIGINAL proc. ast2nif
re-derives a routine's serialized AST from `ast[namePos].sym.ast`, so the
wrapper serialized the original's body — whose parameter belongs to the
original — and lambda lifting saw it as a captured variable of another proc.
The copy now names itself, the invariant every other routine AST keeps.
* **a top-level `let (a, b) = f()` copied instead of moved.** A module's own
symbol is the owner of every top-level symbol and is written as a real `(sd)`,
so the loader minted a SECOND `skModule` PSym for it — and `sym.owner == owner`
is an identity test in `aliasanalysis.isAnalysableFieldAccess`, which made
every module-level location un-analysable. Hard error for a type with a
disabled `=copy`. Bind the NIF name to the one registered module symbol.
Backend only: doing it under `nim m` costs `times.toDateTimeByWeek` its
inferred `raises`.
* **a doubly linked list leaked its whole contents.** A field USE serializes as
a bare `SymUse` with nowhere to put symbol flags, so `trees.isCursor` said
"not a cursor" for every loaded field and `DoublyLinkedNode.prev` became a
COUNTED reference: every node held its predecessor alive and no refcount ever
hit zero. `{.cursor.}` now rides in the NIF name marker (`` `fc `` next to
`` `f ``), which def and use derive from the same `PSym`.
* **`--expandArc` came out shuffled.** `moduleSymbolStubs` iterated a `Table`,
i.e. hash order, so the `lower` stage transformed a module's routines in an
arbitrary order — not even stable between two compilers. Order by index
offset, which is source order.
Also: `testament`'s `generatedFile` did not include the matrix entry in the
nimcache key its caller uses, so every `ccodeCheck` test with a `matrix:`
reported `reCodeNotFound`.
`koch bootic` still reaches its fixed point; `tests/ic` and the classic
`tests/destructor` are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every invariant `tests/ic` checked was IC-against-IC — clean == incremental, a
no-op edit changes nothing, a body edit moves no interface cookie. An IC that is
*consistently* wrong satisfies all of them, and that is exactly how two silent
miscompilations survived. `koch bootic` has the same blind spot: it proves the
compiler reproduces itself, not that it agrees with the reference backend.
Two mechanisms, at different scales.
**The oracle, in the metamorphic runner.** Every successful step is now also
compiled with `nim c` and run, and the two outputs must agree. Unlike the
hand-written `expect:` strings this needs no foresight from the test author: it
compares everything the program does, not only what someone thought to print,
which is what a silently-skipped destructor evades. `no-oracle` opts out.
The format also grew the expressiveness the recent bug hunt showed was missing —
every one of these described a state the suite could not reach:
* `#!DELETE <file>` removes a module. Deleting a still-imported file moves no
mtime, so nothing re-fires.
* `#!FLAGS <switches>` changes the compiler switches between steps. A config
change is not a file either.
* `fails: <substring>` asserts that BOTH compilers reject the program with that
text. Previously every step had to succeed, so the whole error path — and
recovery from it — was untested.
Six regression tests cover the eight bugs the last round fixed.
`testament r <file>` now dispatches metamorphic tests like `testament cat ic`.
**`testament --ic` runs the whole corpus through the incremental compiler**, so
IC inherits ~10k programs with expected output instead of 30 bespoke tests.
Two things had to change for that to mean anything:
* `nimcacheDir` now keys on the `matrix:` entry too. Two matrix variants of one
file are two different compilations; sharing a cache meant each run
invalidated what the previous left — harmless for a backend that caches only
object files, useless for an incremental one.
* About half the corpus overrides the command wholesale (`cmd: "nim c --gc:arc
$file"`), bypassing both `$target` and `$options`. Those are rewritten to `ic`
and given a private cache.
**Warm cache, hastur-style.** A generated warmup program pulling in `system` and
the most-imported stdlib modules is compiled once per distinct compile
configuration into `nimcache/ic_warmup_<hash>`, and each test's empty cache is
seeded from it with mtimes preserved (nifmake compares output-mtime >
input-mtime, so stamping the copies "now" re-fires the whole graph). Only
program-independent artifacts are copied: the frontend NIFs and cookies plus the
per-module `lower`/`cg` output. The `.c`/`.o` are left behind on purpose — the
merge decision is whole-program, so they are re-rendered for every program
anyway.
`tests/destructor` (97 runs): `nim c` 35s cold / 32s warm; `nim ic` ~3m30 cold /
**9.8s warm**.
Compiler changes this required or uncovered:
* `merge` read the live-module list from a manifest the driver writes instead of
globbing `*.c.nif` off the nimcache. Globbing absorbed artifacts belonging to
any other program sharing the directory — which is what made a prefilled cache
produce undefined symbols at link.
* The build-arg signature no longer includes `--icproject:`/`--icPreparsedConfig:`
(they name where a build lives, not what it produces, so two caches holding
identical artifacts got different signatures). The precompiled config still
counts, by content hash, minus its `(nimcache …)` line.
* `.s.deps` seeding is speculative and runs before the prune, so a sidecar entry
that has gone stale (an import that a `when` no longer takes) can be dropped
instead of lingering forever; a pruned module's scan artifacts are deleted so
an edit-accumulated cache still matches a clean one.
* A failed nifmake run no longer prints an `Error:` of its own. The children have
already reported; adding a build-system status as the LAST error hid the
compiler's real message from anything reading the final error — every
reject-style test under `nim ic` said "nifmake failed with exit code: 1".
* `--mm:hooks` fed the mm mode to an on/off switch and failed outright with
"'on' or 'off' expected, but 'hooks' found". Pre-existing and unrelated to IC;
only reachable through the explicit switch, since `--newruntime` sets
`selectedGC` directly.
Running `tests/destructor` under `--ic` currently leaves 10 failures. They are
genuine IC defects, not harness noise (all 97 pass under `nim c`) — the clearest
is `tglobaldestructor`: `graph.globalDestructors` is accumulated while injecting
destructors into a module's top level, but the main module's `cg` — which emits
the teardown — is a different process, so a module-level `var` with a `=destroy`
is never destroyed. Same shape as the init/datInit metas, and it wants the same
fix: record it in the `.c.nif` head.
Validation: `koch bootic` reaches its byte-identical fixed point; `tests/ic` is
36/36; arc, destructor, macros, template, iter, closure, ccg, codegen, types and
effects pass under `nim c`, with generics showing only its pre-existing failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grinding a small figdraw-based program under `nim ic` and diffing its output
against the classic backend surfaced eight bugs, four of which silently produced
a wrong binary rather than an error.
Frontend / build graph (`deps.nim`):
* Dead `when`-guarded imports were compiled anyway. `when someStrdefine == "x":
import y` is `cvUnknown` to the scanner, which conservatively keeps the edge —
right for an edge, but it also gave `y` its own `nim m` rule, so a build died
on a package the user never installed because they never selected that backend.
Track which edges are speculative and drop a speculative subtree that cannot
compile; if the guard was in fact live, the discovery fixpoint puts the node
back with the honest `cannot open file`.
* Deleting a still-imported module went unnoticed: no mtime moves, so nothing
re-fires and `nim ic` relinked a stale binary while `nim c` reported `cannot
open file`. Report an unresolvable import from a non-speculatively reached
module during the graph scan.
* Macro-generated imports were discovered once and then forgotten. Discovery
only ran after a failure and the graph is re-derived statically every run, so
on a warm build the discovered module had no rules at all and editing it
changed nothing. Seed the graph from the `.s.deps` sidecars up front.
* Config changes invalidated nothing. nifmake decides staleness from file mtimes
and never looks at a rule's command line, so `-d:foo=bar` / `--mm:` / `--threads:`
regenerated the build file with the new switches and re-fired zero rules.
Reify the configuration as a file and make it an input of every rule.
* Command-line switches never reached the children: they replay the project's
config files, never the driver's argv, so `nim ic --opt:speed` produced a
byte-identical debug binary (likewise `--panics`, `--experimental`, `--passC`).
Forward the driver's switches, minus the ones that must differ per child.
Artifacts and codegen:
* A failed `nim m` still wrote its `.s.bif` and cookies, so nifmake saw the rule
as satisfied on the next run: `nim ic` then reported success for a program that
does not compile, and generated code from error-bearing AST (or hit an internal
error in `ccgexprs`). Never persist an artifact when `errorCounter > 0`.
* Top-level destructors were never injected. `sfInjectDestructors` lives on the
module symbol, which `moduleFromNifFile` rebuilds from scratch, so
`genTopLevelStmt` skipped `injectDestructorCalls` entirely: a module-level
`block: let h = openHandle()` never ran `=destroy`. Persist the flag as a
`(modflags)` record. `injectdestructors` also has to tolerate the
`nkReplayAction` entries the loader prepends to `topLevel`.
* `nfFirstWrite` / `nfLastRead` were dropped by the serializer. A sym node is
written as a bare NIF `SymUse` token, which has nowhere to put node flags, so
the frontend's move analysis never reached the backend: EVERY first assignment
to a destructor-bearing local compiled as `=sink`, i.e. `=destroy` on
still-zeroed memory followed by a copy, and no read was ever a move. Wrap a sym
use in `(nflags ...)` when it carries persistent node flags.
`icFormatVersion` 34 -> 35 for the two new NIF records.
Validation: `koch bootic` reaches its byte-identical fixed point; two clean-cache
builds from the same compiler are identical; testament `arc`, `destructor`,
`macros`, `template`, `iter`, `closure`, `ccg`, `codegen`, `types` and `effects`
pass, and `generics`, `ic` and `stdlib` show exactly the pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#23835 tried to do this, but it also switched to `long int` on the GNU
ABI, where it's actually just `int`.
Fixes#26111
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
fixes#26015
Fixes imported global variables with codegenDecl being emitted as
definitions instead of extern declarations.
A variable’s codegenDecl format should customize its definition in the
owning module. Other modules referencing the variable must emit a normal
declaration:
```c
extern NI variable;
```
After the variable-declaration builder refactor, genVarPrototype passed
Extern visibility to addVar. However, the sfCodegenDecl branch returned
before applying that visibility. This caused importing modules to emit
another tentative definition, resulting in duplicate-symbol linker
errors.
The fix restores the previous distinction between the custom definition
and cross-module prototypes. It also adds C and C++ regression coverage
for both direct access and access through an inline procedure.
follows up https://github.com/nim-lang/Nim/pull/24423
Specialize `rawAlloc` for alignment (cherry-picked from the other PR).
This cuts the frame of the normal unaligned path down enough to regain
the performance lost from loading the cold page in #26110
Also cleans up `MemRegion` a bit, the regressions are either gone or
were measurement errors.
fixes#26123
`cast[T](x)` is a transparent path expression for compiler analysis.
Previously, move/alias analysis could fail to see a later use through a
cast and incorrectly mark the source as moved, causing the issue’s
segmentation fault.
for views,
https://nim-lang.org/docs/manual_experimental.html#view-types-path-expressions:
A cast expression cast[T](e) is a path expression.
It also affects skipConvDfa, isAnalysableFieldAccess, and aliases. And I
might narrow it down for the two cases above mentioned if it causes
problems
Issue popped up when using `fromJson` into an array but the JSON passed
is an object
```nim
import std/[jsonutils, json]
let data = parseJson """
{"key": "value"}
"""
var foo: seq[int]
foo.fromJson(data)
echo foo #> @[0]
```
Basically the `setLen` would set the size to be equal to the number of
keys, but `getElems` just returns an empty array if the JSON isn't an
array which lead to it just creating zero'd items in the seq without
letting the user know.
Felt adding the checks was better than just skipping the `setLen` since
it lets the user know that there is a problem with the JSON
fixes#25992
```nim
type
Foo = object
case kind: bool
of true:
a: ref Bar # 8 bytes (pointer)
of false:
b: int # 4 bytes
```
specializeResetT for b emits accessor.b = 0 — writes 4 bytes
But the union is 8 bytes wide (sized by the largest branch)
The remaining 4 bytes where a used to live are untouched
Those stale bytes could contain a heap pointer the GC traces → crash
Add nimZeroMem after specializeResetN for case objects to clear the
entire union including unused branch bytes.
We defer copying the AST during generic instantiation until a cache miss
and fetch from cache based on bindings. On miss, we fall back to the old
logic and populate the cache.
This gains us a roughly 36% reduction in memory usage during bootstrap,
from `799.242MiB` down to `508.672MiB` on my machine.
For another point of reference, nimbus-eth2 goes from `10.5GB` memory
usage to `7.5GB`.
Independent companion to #26090 which together with this one yields a
bit under 20% faster compiles (or at least `--compileOnly` bootstraps)
on ORC.
fixes#26088
in https://github.com/nim-lang/Nim/pull/25772, `beginStores` requires
`newLen` to be passed for setting up the new length. So `streams.nim`
must make the string length exactly newLen.
…haracterization with--warning:ResultUsed:on
fixes#26062
> A return statement with no expression is shorthand for return result.
> ResultUsed: Warn about the usage of the built-in result variable.
> A procedure that does not have any return statement and does not use
the special result variable returns the value of its last expression.
`unapplyBlockContext` reset `inEnforcedGcSafe` and
`inEnforcedNoSideEffects` to false whenever a `{.cast(gcsafe).}` or
`{.cast(noSideEffect).}` block ended. When such a block is nested inside
another block of the same cast, the inner block's exit switched
enforcement back on for the rest of the enclosing block, so statements
lexically inside the outer cast were rejected.
The fix saves both flags in `PragmaBlockContext` when the block context
is created and restores the saved values on exit. That matches how every
other piece of block state there (`locked`, `exc`, `tags`, `forbids`) is
already handled; these two bools were the only ones reset to a constant
instead of restored.
No change for non-nested blocks: entering from the non-enforced state
saves false, so exit still clears the flag. A statement after the outer
block is still rejected as before.
Test covers nested `cast(gcsafe)` and nested `cast(noSideEffect)`.
`tests/effects` passes unchanged (49/49, same as stock).
Boostrap without linking is (conservatively) 17% faster on my machine.
Test setup runs the compiler compiling itself under ORC in release mode
from a clean cache and no C compilation (`--compileOnly`.)
Multiple samples are gathered before judging a potential optimization.
The test suite passes locally before push and bootstrap is tested with
strict views (mostly as a sanity check).
Tests are fair in the sense that they compile the same git worktree and
produce identical C output.
Refc will see less or no benefit.
Changes:
- Save tree traversals in `considerGenSyms` when no mappings exist
- Inline `maybeSkipDistinct` into `typeRel` for a safe cursor
- Only skip to static when there is a static type to reach in
`paramTypesMatchAux`
- Disable overflow checks for hashes
- Disable bounds checks for `nextIdentIter`
- Use cursor annotation when judged safe
- Use lent annotation for ast accessors
Spiritual companion to #26084
## Fix `globalSymbols` support on POSIX
### Summary
Fix `-d:globalSymbols` on POSIX platforms by defining `RTLD_GLOBAL`
alongside `RTLD_NOW` in `system/dyncalls.nim`.
On Linux and macOS, `RTLD_NOW` is defined locally in `dyncalls.nim`, but
`RTLD_GLOBAL` was not. As a result, enabling `-d:globalSymbols` could
fail because `RTLD_GLOBAL` was undeclared.
This change:
* defines `RTLD_GLOBAL` as `0x100` on Linux,
* defines `RTLD_GLOBAL` as `0x8` on macOS,
* imports `RTLD_GLOBAL` from `<dlfcn.h>` on other POSIX platforms.
These values are consistent with the existing POSIX constants already
used elsewhere in the Nim source tree.
### Motivation
`globalSymbols` is intended to load dynamic libraries with `RTLD_GLOBAL`
so that their exported symbols are available to subsequently loaded
shared libraries.
This is needed, for example, when a dynamically loaded library later
loads a plugin or provider that depends on symbols from the first
library.
Without this fix, `-d:globalSymbols` cannot be used reliably for that
purpose on POSIX systems.
### Testing
Tested on Linux with an AArch64 target.
A program using dynamically loaded OpenSSL libraries and a subsequently
loaded OpenSSL provider failed when the OpenSSL libraries were loaded
with the default local symbol visibility.
Using `RTLD_GLOBAL` made the same program work correctly.
After this change, building the original Nim program with:
```text
-d:globalSymbols
```
successfully loads the OpenSSL libraries with global symbol visibility,
and the provider-based TLS 1.2 and TLS 1.3 tests both pass.
The same behavior was also independently reproduced using direct
`dlopen()` / `dlsym()` calls:
```text
RTLD_LOCAL -> TLS 1.2 failed
RTLD_GLOBAL -> TLS 1.2 passed
```
Signed-off-by: Takeyoshi Kikuchi <kikuchi@centurysys.co.jp>
`bundleChecksums` compiles the two NIF host tools through
`nimCompileFold`, which spawns a fresh `nim c` with only the `options`
string. Unlike the other `bundle*` procs it takes no `args`, so the
`--skipUserCfg --skipParentCfg` that `koch boot` was invoked with never
reached these two sub-builds.
For a Nim checked out inside another project (nimbus-eth1/eth2, nimbos
vendor Nim under `vendor/nimbus-build-system/vendor/Nim`) Nim then walks
the parent directories of `dist/nimony/src/nifler` and applies the host
project's `config.nims` to the tool build: an injected
`--import:libbacktrace`, plus `warningAsError[UnusedImport]`,
`hintAsError[DuplicateModuleImport]` and
`hintAsError[ConvFromXtoItselfNotNeeded]` all turn ordinary nimony code
into hard errors and break `koch boot`.
Host tools must not be configurable by whatever directory Nim happens to
sit under, so hardcode the skip flags (and `--noNimblePath`, matching
the other bundlers) rather than threading `args` through all seven call
sites.
`nimDecRefIsLast` always performed an atomic decrement. When the biased
count is already zero the destroying thread holds the only reference, so
there is nothing to adjudicate and the read-modify-write can be skipped.
Soundness: a counted reference can only be derived from the location
being destroyed -- which happens-before this destructor unless the
program races on that location -- or from another counted reference,
whose contribution is already in `rc` and therefore forces the slow
path. Observing zero proves no other thread holds a reference and that
none can appear. This relies on `--mm:atomicArc` having no collector;
ORC and YRC mutate `rc` from a participant that holds no counted
reference at all, so the fast path is deliberately not enabled for them.
The slow path keeps deciding on the value its own RMW returned. That is
what separates this from nim-lang/threading#45, where the "who frees"
role was decided from a separate load and the RMW result was discarded,
so the role could be dropped by every participant at once.
gcbench, -d:danger, median of 21 pinned runs:
--mm:arc (non-atomic RC) 0.1310
--mm:atomicArc 0.1742
--mm:atomicArc + this 0.1330
-23.7%, closing 95% of the gap to non-atomic reference counting. gcbench
builds its trees with `sink` parameters, so it performs almost no
incRefs and the whole atomicArc penalty is decRef traffic. The worst
case -- a decrement that always sees rc > 0, so the load never pays off
-- measures +1.1%.
`-d:nimNoAtomicArcFastPath` restores the previous code path.
`strbasics.add` uses `copymem` when available. CT conditional
scaffolding mirrors the same from system.
Follow-up to #15951
Compile-time test for `strbasics.add` undiscarded and passes, though
pending bug #15952 is still open.
