====================================== Incremental Compilation (IC) ====================================== The ``nim ic`` command provides incremental compilation for Nim projects. It decomposes compilation into per-module steps whose results are cached as NIF files, and uses the external ``nifmake`` build tool to re-run only the steps whose inputs changed. This document describes **how `nim ic` works today**, including the edge cases that shaped the current design, and ends with a **Plan** for the next backend rewrite. Overview ======== The pipeline has two halves driven by one process (`nim ic`, `commandIc` in ``compiler/deps.nim``) that constructs a dependency graph, writes a build file, and hands it to ``nifmake``: 1. **Frontend** — per module: - ``nifler parse --deps`` turns ``.nim`` source into a parsed NIF (``.p.nif``) plus a static dependency list (``.deps.nif``). - ``nim m`` (the *semantic* step, `cmdM`) reads the parsed NIF + the precompiled NIFs of the module's imports, type-checks, and writes the **semmed NIF** (``.nif``) plus invalidation sidecars (see *Cookies*). 2. **Backend** — ``nim nifc`` (`cmdNifC`, ``compiler/nifbackend.nim``) reads the semmed NIFs, generates C, compiles and links. ``nifmake`` orders the steps by their input/output files: every `nim m` runs before the `nim nifc` step that consumes its NIF, and a step re-fires only when one of its inputs is newer than its outputs. The driver invokes ``nifmake run --parallel`` by default, so independent steps at the same DAG depth fan out across cores; pass ``-d:icNoParallel`` to serialize (readable child output when debugging a build). Artifacts (the NIF zoo) ======================= Per module ```` (a content hash of the path; see *NIF symbols* below), under the nimcache directory: | File | Producer | Purpose | | ---- | -------- | ------- | | ``.p.nif`` | nifler | parsed AST (syntactic) | | ``.deps.nif`` | nifler | **static** import list (syntactic `import`s) | | ``.s.deps.nif`` | `nim m` | **real** post-sem imports (incl. macro-generated); see *Discovery* | | ``.nif`` | `nim m` | semmed module (symbols resolved, typed) | | ``.iface.nif`` | `nim m` | **iface cookie**: hash of the importer-visible surface | | ``.impl.nif`` | `nim m` | **impl cookie**: hash of the entire content (bodies included) | | ``.edges.nif`` | `nim m` | **NeedsImpl edges**: modules whose bodies this sem consumed | | ``.c.nif`` | `nim nifc` | the C text as a NIF, with def/ref markers for DCE & dedup | | ``ic_config.cfg.nif`` | driver | precompiled config replayed by every child (`icconfig.nim`) | | ``ic.version`` | driver | format stamp; a mismatch wipes the cache (`icFormatVersion`) | NIF symbols and ownership ========================= (See ``../nifspec/doc/nif-spec.md``.) A global symbol is ``..``. For a **generic instantiation** the `` is not a counter but a *content hash* — `setInstanceDisamb` (``modulegraphs.nim``) MD5s the generic's identity plus the `typeKey` of every concrete type argument, masks it to 30 bits and tags it with `InstanceDisambBit`. So the only part of the name that varies between two modules making the **same** instantiation (`seq[Foo]`) is the ``. Two consequences drive the backend: - **Instance names are content-addressed**: the same instantiation produced in different modules yields the *same* `.`, so a deterministic dedup is possible by the *module-suffix-stripped* name. The cross-TU merge (`ccgtypes.sharedInstanceCName`, keyed via `graph.icSharedSigs`) and the DCE analysis (`dce.computeLiveSymbols`, its `uniq` set) both already key on this stripped form. - **The suffix names a mint-site owner.** Today the `` is the module *that minted the instance* (the instantiation site), so the same instance has a different full name in each module that makes it; *which* TU emits the single definition is then decided at codegen by the reuse/redirect machinery — the root of the *single-writer* hazard below. The driver: graph construction (`commandIc`) ============================================ 1. Stamp/​wipe the cache by ``icFormatVersion``. 2. Seed the graph with the root module and **`system.nim`**. `system`'s entire import closure is folded into one node (one `nim m` invocation) — see *single-writer* below. 3. ``traverseDeps`` runs ``nifler`` per module and reads ``.deps.nif`` to add import edges. 4. **SCC grouping**: strongly-connected import cycles are collapsed (Tarjan). A singleton compiles as ``nim m ``; a cycle compiles as one ``nim m --icGroup:…`` that builds every member *from source* in one process (resolving the recursion in memory) and writes each member's NIF. Only edges *leaving* the component become build-graph inputs. 5. **Discovery fixpoint**: write the build file, run ``nifmake``; if it fails, re-derive the graph from every module's ``.s.deps.nif`` (adding nodes/edges for imports the static scanner missed), and retry. See *Discovery*. 6. The backend step (`nim nifc`) depends on every module's semmed NIF, so ``nifmake`` runs it last. Invalidation: the cookie system ================================ A dependent must re-sem only when a dependency's relevant surface changed. Two hashes per module (``ast2nif.nim``): - **iface cookie** (``.iface.nif``): hashes only the *importer-visible* surface — exported declarations' **signatures** (for *all* routine kinds: plain procs, templates, macros, generics, `inline` procs alike), full content for consts/types, plus import/export/replay/hook records. Routine **bodies are excluded.** It also chains in the iface cookies of its own dependencies, so a surface change anywhere in the import closure propagates. A `nim m` rule for a module depends on its dependencies' iface cookies, so a body-only edit moves no iface cookie and stops the re-sem cascade. - **impl cookie** (``.impl.nif``): hashes the *entire* serialized content (private defs and bodies included), with the module's own iface mixed in. **NeedsImpl edges** (``.edges.nif``): if a module *consumed another module's body* during sem — a macro expansion, a generic instantiation, a `getImpl`, or a compile-time call run in the VM — it records a strong edge. The dependent is then gated on that dependency's **impl** cookie instead of its iface cookie, so e.g. `const x = dep.foo()` re-sems when `foo`'s body changes. Recording sites: `semExprs.semTemplateExpr` (templates), `seminst.generateInstance` (generics), `vmgen.genProc` (VM/macros/CT procs), `vm.opcGetImpl` (`getImpl`). Inline iterators and `inline` procs are *not* tracked — they are inlined at codegen, where the backend's NIF-mtime invalidation re-codegens their users. Discovery of macro-generated imports ==================================== The static scanner only sees syntactic `import`s. A macro can synthesize one (chronicles does `parseStmt("import chronicles/textlines")` driven by the `chronicles_sinks` define). Such an import is invisible until sem runs the macro. Each `nim m` records the imports it *actually* resolved (via the ``semdata.addImportFileDep`` hook → ``graph.importDeps`` → ``ast2nif.writeSemDeps``) into ``.s.deps.nif``; a child that fails on a not-yet-built import flushes it before erroring. The driver re-derives the graph from those sidecars — adding the missing node + the importer→import edge — and reruns to a fixpoint. (This replaced an earlier `icmissing.txt` side channel.) The backend today: `commandNifC` ================================ The current backend is **whole-program and demand-driven**, run as one process (``compiler/nifbackend.nim``): 1. Load `system` then **all** modules' semmed NIFs in dependency order (`loadModuleDependencies`), so all hooks/types are in memory. 2. **DCE**: `computeLiveSymbols` over all NIFs computes a global live set used to filter the top-level routine listing. 3. **`computeModuleReuse`**: decide which modules' cached translation units (``.c.nif``) can be reused — skip codegen for them and use their ``.c``/``.o`` as is. The gate mirrors the m-step's cookie gating; a coarse fallback uses the transitive NIF-mtime closure (`-d:icCoarseReuse`). 4. **Codegen**: for each non-reused module, `generateCodeForModule` runs `cgen`. Codegen is **demand-driven**: emitting one module can *demand* entities (generic instances, type-bound hooks, RTTI) that belong to other modules; a demand whose home TU is reused is **redirected** into the demanding TU (`redirectToLiveModule`), and reused TUs' definitions become prototypes. 5. **`enforceDefRetention`**: an un-reuse cascade. If a regenerated module would stop emitting a definition that a still-reused TU references (the demand chain that placed it no longer arises), the referencing TUs lose their reuse and regenerate so the symbol does not vanish under them. 6. `emitMethodDispatchers`, then `finishModule` for every module (main module **last**, so init-proc registration is complete before `genMainProc`). 7. Emit ``.c``, then `extccomp.callCCompiler` + link. Edge cases (and why the machinery exists) ========================================= - **Single-writer.** Instance type-ids are minted in process-local order, so if two `nim m` processes both write a module's NIF (e.g. a stdlib module pulled into `system`'s from-source closure *and* given its own rule), the second overwrites with different ids and every module checked against the first carries dangling refs ("symbol has no offset"). Fixed by folding `system`'s closure into one SCC and by **forwarding the project's defines** to every child so their `when` bodies (hence import sets and NIF contents) match the scanner's. - **`when … else: import`.** nifler emits `else`-branch imports unguarded, so a dead `else: import` would be scheduled. The compiler's own sources were rewritten to explicit negated `when`s; the vendored nifler later learned to negate prior conditions for the `else`. - **`nil` sons of loaded ASTs.** NIF dot-tokens load as `nil` where from-source ASTs have `nkEmpty`; several passes gained `nil` guards. - **Sealed loaded types.** Loaded types are `Sealed`; sem/transform mutate via `unsealForTransform`/`exactReplica(idgen)` (the latter mints a fresh `uniqueId` so serialized replicas don't collapse). - **Methods/RTTI ownership.** `genTypeInfoV1` already routes a type's RTTI to `t…itemId.module` when that module is open for codegen — an existing ownership notion the rewrite can generalize. - **Config cost.** Each child re-parsing `nim.cfg` + re-running `config.nims` in the VM was ~80 ms; replaced by a precompiled `ic_config.cfg.nif` replayed in `loadConfigs` (`compiler/icconfig.nim`). - **`koch bootic`** bootstraps the compiler through `nim ic` (a 3-iteration fixed-point check). It writes its binary to ``bin/nim_ic`` and never clobbers ``bin/nim``. Known residual hacks (targets for the rewrite) ---------------------------------------------- - `deps.runNifler` uses `setLastModificationTime` to mark its scan up-to-date and deletes a stale parsed file to coordinate with the nifmake nifler rule — the driver duplicating nifmake's freshness logic. - `computeModuleReuse` + `enforceDefRetention` + the redirect/cached-defs machinery is a hand-rolled mini-`nifmake` *inside* the backend process, needed only because one process reuses some TUs while regenerating others. These are legacy artifacts of a code generator that predates IC, not intrinsic requirements. - **Cross-mm reuse / `--force` (`var not init i_`).** Reusing TUs built under one memory-management mode (e.g. an `orc` cache) while rebuilding under another (`refc`), or forcing a full backend re-run, can abort with ``var not init``. The name is a backend-minted closure-capture local whose `loc` is unset; the root cause is that `nifmake` is *mtime-only* and `computeModuleReuse` is not config-aware, so a reuse decision can mix TUs generated under different config signatures. It is **crash-safe** (it aborts, never emits a wrong-mm binary) and does **not** affect normal same-mode warm rebuilds. Deferred to the rewrite, which makes reuse config-aware by construction (each codegen rule's config is a declared input). Status and performance ====================== `nim ic` self-builds the compiler (`koch bootic`'s fixed-point check) under both `orc` and `--mm:refc`, and passes the external-package CI set. The macro sweep is 93/95 (two known-baseline failures: `tmacro7`, `tmacrogetimpl`). Rough numbers on a 32-core box (`-d:release`, building the compiler): | | wall | notes | | - | ---- | ----- | | classic `nim c` full | ~27.5s | ~7 cores busy | | `nim ic` cold, serial | ~81s | one `nim m` at a time | | `nim ic` cold, `--parallel` | ~53s | now the default | | `nim ic` warm no-op | ~0.14s | ~200× vs classic | The cold gap is a *parallelism* deficiency, not extra fundamental work (aggregate work is ≈1.33× classic). The backend's own phases are cheap (load 0.3s, DCE 1.0s, cgen 5.7s, write 0.8s); the C compile+link floor (~21s) is already parallel and shared with the classic backend. The remaining cold cost is the `nim m` sem phase, throttled by `nifmake`'s per-DAG-depth barrier across a narrow, deep tail of heavy modules. A ready-queue scheduler (dependency-ready dispatch instead of a depth barrier) would recover several seconds but cannot beat the import-chain critical path; it folds into the rewrite. **Warm rebuilds — the actual point of IC — already dominate the classic backend by ~200×.** The strategic direction (decided 2026-06-13) is to make this NIF backend (`cmdNifC`) the **default** code generator. The clean translation-unit model it needs — global DCE, content-keyed instance merging, deterministic RTTI — already exists and is always-on inside `cmdNifC`; the remaining work is the per-module backend rewrite below (which also dissolves the cross-mm reuse hazard), then *promotion + deletion*, not new machinery. Plan: a nifmake-driven, per-module backend ========================================== Goal: the backend stops re-implementing `nifmake`. Each module's codegen becomes its own build rule, so "which TUs rebuild" is just "which rules `nifmake` re-fires from input mtimes" — exactly as the frontend already works. The reuse / def-retention / redirect machinery then dissolves. Target build graph (mirrors Nimony's ``src/nimony/deps.nim``): 1. **Frontend split.** Generate a *frontend* build file (nifler + `nim m` rules), run `nifmake`, run the `.s.deps` discovery fixpoint. Then re-derive the graph from `.s.deps` (now complete; dead `when` imports can also be **pruned** here). 2. **Per-module codegen rule.** One ``nifc `` rule per module: inputs are the module's own ``.nif`` plus the ``.iface``/``.impl`` cookies of its dependencies; output is its ``.c.nif``. The process loads `` + its import closure's NIFs (like `nim m`) and emits **only the entities it owns**, referencing everything else `extern`. 3. **Static ownership** replaces runtime redirect. Every emittable entity — generic instances, type-bound hooks, RTTI, lifted procs — gets a deterministic owner module *by symbol suffix*. Because instance names are content-addressed (``ident.disamb.key.owner``), the same instance demanded by several modules has one name and one owner, so there is exactly one writer and no link-time duplicate. (The precise owner rule — minting module vs. root-type's module — is the open design decision; start from `itemId.module` and adjust where it forces a downstream package to own stdlib code.) 4. **DCE as a rule.** A single rule reads all ``.c.nif``, computes the global live set, and (Nimony: ``.live.nif`` / per-module ``.dce.nif``) drives per-module ``.c`` emission filtered to live entities. 5. **Link rule.** Depends on every ``.o`` (each compiled by its own rule) and the DCE output; produces the executable. 6. **Deletions.** `computeModuleReuse`, `enforceDefRetention`, `redirectToLiveModule` and the cached-defs/claim bookkeeping go away. The ``setLastModificationTime`` coordination in `runNifler` goes away with the frontend split (the nifler rule owns parsed+deps; the driver's pre-scan only reads `.deps` to build the graph). Validation bar: `koch bootic` must still reach its byte-identical fixed point, and binary size must not regress (DCE parity), across the external-package CI set. Code, logic & debugging ======================== Core modules: - **`compiler/deps.nim`** — graph construction, SCC grouping, discovery fixpoint, build-file generation; `commandIc`. - **`compiler/ast2nif.nim`** — AST↔NIF, the cookie hashes (`cookieSd`, `writeIfaceCookie`, `writeImplCookie`, `writeEdgesFile`, `writeSemDeps`). - **`compiler/nifbackend.nim`** — the backend (`commandNifC`) and its reuse machinery. - **`compiler/icconfig.nim`** — precompiled config. - **`compiler/pipelines.nim`** / **`modulegraphs.nim`** — pipeline integration and the graph state (`importDeps`, `icImplDeps`, `icReusedModules`, …). Manual workflow: - Frontend a module: ``nim m --nimcache:nifcache path/to/mod.nim`` (writes ``.nif`` + cookies + ``.s.deps``). - Backend: ``nim nifc --nimcache:nifcache main.nim``. - NIF files are text — open/grep them directly; ``diff`` two successive ``.nif`` to see why a module rebuilt. - Force a re-sem: delete the module's ``.nif`` and rerun `nim m`. - A stale-cache crash after editing the serialization layout means bumping ``icFormatVersion`` (`compiler/options.nim`). See also ======== - NIF format spec: [nifspec/doc/nif-spec.md](../nifspec/doc/nif-spec.md) - NIFC (C-like target) spec: dist/nimony/doc/nifc-spec.md