A closure's environment type — and the `=destroy`/`=copy`/`=sink` the compiler
lifts for it — is minted by the BACKEND and exists in no module's semmed NIF.
Grinding a corpus of closure programs against `nim c` as the oracle turned up
three ways the per-module backend gets those wrong. `tests/closure` is now green
under `--ic:on`, and `tests/iter` goes from 13 failures to 9.
* **The env hook is emitted by nobody.** `emitsBodyInThisModule` walks up to the
outermost enclosing routine to pick the TU that emits a body. For the env
`=destroy` of a generic closure iterator defined in one module and instantiated
in another, that walk lands on the module of the ORIGINAL generic — a module
that never sees the instance — so the routine was emitted nowhere
(`undefined reference to eqdestroy__c485__…`). A backend-minted routine has no
owning module NIF at all: it is written into the `.t.bif` of every module that
references it, re-homed there with `@bk`. Every referencing TU emits it now and
the merge stage keeps one, like any other content-addressed definition.
* **The `:up` link is stored without an increment.** `env.:up = enclosingEnv` has
to go through `=copy` (with the cyclic incref) or the parent's refcount is one
too low, and at teardown the two envs' mutually recursive `=destroy`s each
believe they hold the last reference and recurse until the stack is gone — a
SIGSEGV after the program's own output has already been printed
(`tests/iter/tnestedclosures`, "Test 3"). Whether it becomes a `=copy` depends
on the up-field type's hooks existing at injection time. Whole-program cgen got
that for free: a LATER lifting pass creates them and it runs before any
routine's injection. The per-module backend injects a routine right after
lifting it (`lower`), long before the module's top level is transformed at all
(`cg`) — so both up-field assignment sites create the ops themselves.
* **Two backend hooks collide on one C name.** A backend-minted symbol is named
`_c<itemId.item>`, a PER-PROCESS counter — fine while "nifc lifts, emits and
compiles them in one run", which is what `mangleProcNameExt` assumed. But
`lower` mints the env hooks of nested routines and `cg` mints those of the
module's top level, and both land in the same translation unit: two unrelated
`=destroy`s became one C function. `mangleProcNameExt` already makes an
exception for hooks whose `disamb` is content-derived (`HookDisambBit`);
`toNifSymName` now mirrors it, so the content-derived value survives the round
trip instead of being overwritten by the loader.
`InstanceDisambBit`/`HookDisambBit` move to `astdef` — `ast2nif` names symbols by
them and cannot import `modulegraphs`.
Two new metamorphic tests, both of which fail on the previous compiler:
`tclosure_hooks` (generic closure iterator instantiated across modules) and
`tclosure_nested_iter` (closure iterator nested in a closure iterator).
`icFormatVersion` 37 -> 38 for the hook naming. `koch bootic` reaches its
byte-identical fixed point; `tests/closure`, `tests/destructor` (3 known
`--newruntime` failures), `tests/cpp` and the classic categories are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`testament --ic` on `tests/destructor` went from 10 failures to 3; the three
left are all `--newruntime`, whose owned-ref RTTI destructor is still missing
(the object type's `=destroy` never reaches the `cg` that emits the type info).
* **global destructors are never run.** `graph.globalDestructors` is filled
while a module's top level goes through `injectDestructorCalls`, and
whole-program cgen empties the list into the main module's init proc — which
IS the program body, so the calls land at exit. Under `nim ic` every module's
`cg` is a separate process and main's only ever saw its own entries: a
module-level `var` with a `=destroy` in an imported module was simply never
destroyed. Each module now wraps its own list in a nullary exported proc and
announces the name in its `.c.nif` meta head (`CnifVersion` 4 -> 5); main's
`cg` reads the heads it already reads for init/datInit and calls them in
reverse dependency order. New test `tests/ic/tglobal_dtors.nim` pins the
order against the oracle.
* **`spawn` died with "system module needs: nimArgsPassingDone".** A module
loaded from a NIF is named by its mangled suffix, so `loadCompilerProc`'s
`module.name.s == "threadpool"` could never match. The backend loads the whole
program before codegen, so consult every loaded module's index instead; misses
are final and remembered (`getCompilerProc` doubles as a presence probe).
* **`new(x, finalizer)` died with "environment misses: x".** `semmagic`'s
finalizer-to-destructor wrapper copies the original's AST but only rewrites
the parameter, leaving `ast[namePos]` naming the ORIGINAL proc. ast2nif
re-derives a routine's serialized AST from `ast[namePos].sym.ast`, so the
wrapper serialized the original's body — whose parameter belongs to the
original — and lambda lifting saw it as a captured variable of another proc.
The copy now names itself, the invariant every other routine AST keeps.
* **a top-level `let (a, b) = f()` copied instead of moved.** A module's own
symbol is the owner of every top-level symbol and is written as a real `(sd)`,
so the loader minted a SECOND `skModule` PSym for it — and `sym.owner == owner`
is an identity test in `aliasanalysis.isAnalysableFieldAccess`, which made
every module-level location un-analysable. Hard error for a type with a
disabled `=copy`. Bind the NIF name to the one registered module symbol.
Backend only: doing it under `nim m` costs `times.toDateTimeByWeek` its
inferred `raises`.
* **a doubly linked list leaked its whole contents.** A field USE serializes as
a bare `SymUse` with nowhere to put symbol flags, so `trees.isCursor` said
"not a cursor" for every loaded field and `DoublyLinkedNode.prev` became a
COUNTED reference: every node held its predecessor alive and no refcount ever
hit zero. `{.cursor.}` now rides in the NIF name marker (`` `fc `` next to
`` `f ``), which def and use derive from the same `PSym`.
* **`--expandArc` came out shuffled.** `moduleSymbolStubs` iterated a `Table`,
i.e. hash order, so the `lower` stage transformed a module's routines in an
arbitrary order — not even stable between two compilers. Order by index
offset, which is source order.
Also: `testament`'s `generatedFile` did not include the matrix entry in the
nimcache key its caller uses, so every `ccodeCheck` test with a `matrix:`
reported `reCodeNotFound`.
`koch bootic` still reaches its fixed point; `tests/ic` and the classic
`tests/destructor` are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every invariant `tests/ic` checked was IC-against-IC — clean == incremental, a
no-op edit changes nothing, a body edit moves no interface cookie. An IC that is
*consistently* wrong satisfies all of them, and that is exactly how two silent
miscompilations survived. `koch bootic` has the same blind spot: it proves the
compiler reproduces itself, not that it agrees with the reference backend.
Two mechanisms, at different scales.
**The oracle, in the metamorphic runner.** Every successful step is now also
compiled with `nim c` and run, and the two outputs must agree. Unlike the
hand-written `expect:` strings this needs no foresight from the test author: it
compares everything the program does, not only what someone thought to print,
which is what a silently-skipped destructor evades. `no-oracle` opts out.
The format also grew the expressiveness the recent bug hunt showed was missing —
every one of these described a state the suite could not reach:
* `#!DELETE <file>` removes a module. Deleting a still-imported file moves no
mtime, so nothing re-fires.
* `#!FLAGS <switches>` changes the compiler switches between steps. A config
change is not a file either.
* `fails: <substring>` asserts that BOTH compilers reject the program with that
text. Previously every step had to succeed, so the whole error path — and
recovery from it — was untested.
Six regression tests cover the eight bugs the last round fixed.
`testament r <file>` now dispatches metamorphic tests like `testament cat ic`.
**`testament --ic` runs the whole corpus through the incremental compiler**, so
IC inherits ~10k programs with expected output instead of 30 bespoke tests.
Two things had to change for that to mean anything:
* `nimcacheDir` now keys on the `matrix:` entry too. Two matrix variants of one
file are two different compilations; sharing a cache meant each run
invalidated what the previous left — harmless for a backend that caches only
object files, useless for an incremental one.
* About half the corpus overrides the command wholesale (`cmd: "nim c --gc:arc
$file"`), bypassing both `$target` and `$options`. Those are rewritten to `ic`
and given a private cache.
**Warm cache, hastur-style.** A generated warmup program pulling in `system` and
the most-imported stdlib modules is compiled once per distinct compile
configuration into `nimcache/ic_warmup_<hash>`, and each test's empty cache is
seeded from it with mtimes preserved (nifmake compares output-mtime >
input-mtime, so stamping the copies "now" re-fires the whole graph). Only
program-independent artifacts are copied: the frontend NIFs and cookies plus the
per-module `lower`/`cg` output. The `.c`/`.o` are left behind on purpose — the
merge decision is whole-program, so they are re-rendered for every program
anyway.
`tests/destructor` (97 runs): `nim c` 35s cold / 32s warm; `nim ic` ~3m30 cold /
**9.8s warm**.
Compiler changes this required or uncovered:
* `merge` read the live-module list from a manifest the driver writes instead of
globbing `*.c.nif` off the nimcache. Globbing absorbed artifacts belonging to
any other program sharing the directory — which is what made a prefilled cache
produce undefined symbols at link.
* The build-arg signature no longer includes `--icproject:`/`--icPreparsedConfig:`
(they name where a build lives, not what it produces, so two caches holding
identical artifacts got different signatures). The precompiled config still
counts, by content hash, minus its `(nimcache …)` line.
* `.s.deps` seeding is speculative and runs before the prune, so a sidecar entry
that has gone stale (an import that a `when` no longer takes) can be dropped
instead of lingering forever; a pruned module's scan artifacts are deleted so
an edit-accumulated cache still matches a clean one.
* A failed nifmake run no longer prints an `Error:` of its own. The children have
already reported; adding a build-system status as the LAST error hid the
compiler's real message from anything reading the final error — every
reject-style test under `nim ic` said "nifmake failed with exit code: 1".
* `--mm:hooks` fed the mm mode to an on/off switch and failed outright with
"'on' or 'off' expected, but 'hooks' found". Pre-existing and unrelated to IC;
only reachable through the explicit switch, since `--newruntime` sets
`selectedGC` directly.
Running `tests/destructor` under `--ic` currently leaves 10 failures. They are
genuine IC defects, not harness noise (all 97 pass under `nim c`) — the clearest
is `tglobaldestructor`: `graph.globalDestructors` is accumulated while injecting
destructors into a module's top level, but the main module's `cg` — which emits
the teardown — is a different process, so a module-level `var` with a `=destroy`
is never destroyed. Same shape as the init/datInit metas, and it wants the same
fix: record it in the `.c.nif` head.
Validation: `koch bootic` reaches its byte-identical fixed point; `tests/ic` is
36/36; arc, destructor, macros, template, iter, closure, ccg, codegen, types and
effects pass under `nim c`, with generics showing only its pre-existing failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fixes#25650
This pull request refactors and improves the dependency resolution logic
in the Nim compiler, The most important changes are grouped below:
### Dependency Resolution Refactor
* Replaced the `resolveFile` procedure with two more specialized
procedures: `resolveImport` (which uses the compiler's module lookup
rules for imports) and `resolveInclude` (which resolves includes
relative to the including file or search paths). Updated all usages
accordingly, improving clarity and correctness of dependency handling.
[[1]](diffhunk://#diff-1203947eecb9ef641ce7ee029677f875eb983de050b82c65ca286517feed00e6L82-R94)
[[2]](diffhunk://#diff-1203947eecb9ef641ce7ee029677f875eb983de050b82c65ca286517feed00e6L106-R103)
[[3]](diffhunk://#diff-1203947eecb9ef641ce7ee029677f875eb983de050b82c65ca286517feed00e6L121-R118)
* Removed the unused `strutils` import from `compiler/deps.nim` for
cleaner dependencies.
### Testing Improvements
* Added `import std/strbasics` to `tests/ic/tmiscs.nim` to ensure
required symbols are available for tests.
I tried to improve `resolveFile`, which is harder because either we need
to add `lib/std` to search path and all of other nested directory to
`--path` in `config/nim.cfg`. So I choose toi reuse `findModule` for
imports
fixes#25637
This pull request refactors the way the `sfInjectDestructors` flag is
set on symbols during lambda lifting in the Nim compiler. The main
change is the introduction of a helper procedure to encapsulate the
logic for marking symbols that require destructor injection, improving
code clarity and maintainability.
Refactoring and code quality improvements:
* Introduced the `markInjectDestructors` procedure to encapsulate the
logic for marking a symbol with the `sfInjectDestructors` flag, ensuring
that `backendEnsureMutable` is always called before modifying the
symbol's flags.
* Replaced direct flag manipulation (`owner.incl sfInjectDestructors`
and `prc.incl sfInjectDestructors`) with calls to the new
`markInjectDestructors` procedure in multiple locations, including
`makeClosure`, `createTypeBoundOpsLL`, and `rawClosureCreation`.
[[1]](diffhunk://#diff-19193904ba011a2bcc1e1a9768a7eb57cac57a274cad73d388149776ec2901e6L231-R235)
[[2]](diffhunk://#diff-19193904ba011a2bcc1e1a9768a7eb57cac57a274cad73d388149776ec2901e6L243-R247)
[[3]](diffhunk://#diff-19193904ba011a2bcc1e1a9768a7eb57cac57a274cad73d388149776ec2901e6L639-R643)
```nim
template compute(body: untyped): int =
block:
body
let x = compute:
var sum = 0
for i in 1..10: sum += i
sum
echo x
```
supersedes https://github.com/nim-lang/Nim/pull/25653
which in
02893e2f4c
```nim
of nkSym:
genSingleVar(p, it.sym, newSymNode(it.sym), it.sym.astdef)
```
A new branch for `nkSym` is added, though more changes might be needed
if `nkSym` is handled specifically
This pull request includes a few targeted changes across the codebase,
primarily focusing on improving symbol locality detection in the
compiler, adding a utility function for integer division and modulus,
and simplifying a test case.
- **Compiler Improvements**
* Improved the `isLocalSym` function in `compiler/ast2nif.nim` to more
accurately determine if a symbol is local by checking that the symbol's
owner is not a module.
- **Utility Function Addition**
* Added a new `divmod` procedure in `tests/ic/tmiscs.nim` that returns
both the quotient and remainder of integer division, along with a usage
example.
- **Test Simplification**
* Simplified the `showMeters` test in `tests/ic/tconverter.nim` by
removing a floating-point assertion, leaving only an output statement.
------------------------------------------------------------------------------------------------------------------
```nim
proc divmod(a, b: int): (int, int) =
(a div b, a mod b)
let (q, r) = divmod(17, 5)
echo q
echo r
```
gives `Error: unhandled exception: local symbol 'tmpTuple.0' not found
in localSyms. [AssertionDefect]`
`makeVarTupleSection` uses a temp of which the globalness and localness
is not specified. Turning it a global variable for top level scope broke
some Nim programs. So I think it's better to check the owner of the
symbol
```nim
if useTemp:
# use same symkind for compatibility with original section
let temp = newSym(symkind, getIdent(c.cache, "tmpTuple"), c.idgen, getCurrOwner(c), n.info)
```
fixes#25620
This pull request includes a fix to the type key generation logic in the
compiler and updates to a test file to cover additional language
features. The most important changes are summarized below:
### Compiler logic fix
* In `compiler/typekeys.nim`, the `typeKey` procedure was updated to
iterate over all elements in `t.sonsImpl` starting from index 0 instead
of 1, ensuring that all type sons are considered during type key
generation.
### Test suite improvements
* The test file `tests/ic/tenum.nim` was renamed to
`tests/ic/tmiscs.nim`, and its output expectations were updated to
reflect the new test cases.
* Added new test cases to `tests/ic/tmiscs.nim` to cover sink and move
semantics, including the definition of a `BigObj` type and a `consume`
procedure that demonstrates moving and consuming large objects.
```nim
# Sink and move semantics
type
BigObj = object
data: seq[int]
proc consume(x: sink BigObj) =
echo x.data.len
var b = BigObj(data: @[1, 2, 3, 4, 5])
consume(move b)
```
gives
```
error: passing 'tySequence__qwqHTkRvwhrRyENtudHQ7g' (aka 'struct tySequence__qwqHTkRvwhrRyENtudHQ7g') to parameter of incompatible type 'tySequence__cTyVHeHOWk5jStsToosJ8Q' (aka 'struct tySequence__cTyVHeHOWk5jStsToosJ8Q')
84 | eqdestroy___sysma2dyk_u75((*dest_p0).data);
```
follows up https://github.com/nim-lang/Nim/pull/25614
Attempts to move the generic instantiation to the module that uses it.
This should decrease re-compilation times as the source module where the
generic lives doesnt need to be recompiled
---------
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
* default threads on
* make rst gcsafe
* ignore threads option for nimscript
* threads off
* use createShared for threads
* test without threads
* avr threds off
* avr threads off
* async threads off
* threads off
* fix ci
* restore option
* make CI pleased
* fix ic tests
* Update config.nims
* add changelog
* Update changelog.md
Co-authored-by: konsumlamm <44230978+konsumlamm@users.noreply.github.com>
Co-authored-by: konsumlamm <44230978+konsumlamm@users.noreply.github.com>
* IC exposes typedesc implementation shenanigans; so I change system.default's definition to what it should have been to begin with
* Update lib/system.nim
Co-authored-by: Timothee Cour <timothee.cour2@gmail.com>
* IC: renamed to_packed_ast module to ic module
* IC: don't store the --forceBuild flag, makes it easier to test
* IC: enable hello world test
* Codegen: refactorings for IC; changed the name mangling algorithm
* fixed the HCR regressions
* life is too short for HCR
* tconvexhull is now allowed to use deepCopy
* IC exposed a stdlib bug, required a refactoring
* codegen: code cleanups
* IC: even if a module is outdated, its dependencies might come from disk
* IC: progress
* IC: better name mangling, module IDs are not stable
* IC: another refactoring helping with --ic:on --gc:arc
* disable arraymancer on Windows for the time being
* disable arraymancer altogether
* IC: make basic test work with 'nim cpp'
* IC: progress on --ic:on --gc:arc
* wip; name mangling for type info
* minor improvements
* IC: added the required logic for compilerProcs
* LazySym ftw
* we need this testing logic
* reimplement the old way we use for module package creation
* fixes a regression; don't pick module names if you can avoid it
* IC: C codegen is aware of IC
* manual: minor change to make VSCode's RST plugin render it properly
* IC: minor refactoring
* testament: code refactorings
* rodutils: removed dead code
* IC: always build the compiler with the IC feature
* IC: C codegen improvements
* IC: implement the undocumented -d:nimMustCache option for testing purposes
* IC: added first basic tests
* IC: extensive testing of the deserialization feature
* testament: refactoring; better IC tests
* IC: removes 'nimMustCache' flag; readonly does the same
* testament: minor refactoring
* update Nimble version
* testament: removed dead code and imports; IC: added simple test
* IC: progress
