Nim/doc/ic.md

====================================== Incremental Compilation (IC)

The nim ic command provides incremental compilation for Nim projects. It decomposes compilation into per-module steps whose results are cached as NIF files, and uses the external nifmake build tool to re-run only the steps whose inputs changed.

This document describes how nim ic works today, including the edge cases that shaped the current design, and ends with a Plan for the next backend rewrite.

Overview

The pipeline has two halves driven by one process (nim ic, commandIc in compiler/deps.nim) that constructs a dependency graph, writes a build file, and hands it to nifmake:

1. Frontend — per module:
   • nifler parse --deps turns .nim source into a parsed NIF (.p.nif) plus a static dependency list (.deps.nif).
   • nim m (the semantic step, cmdM) reads the parsed NIF + the precompiled NIFs of the module's imports, type-checks, and writes the semmed NIF (.nif) plus invalidation sidecars (see Cookies).
2. Backend — nim nifc (cmdNifC, compiler/nifbackend.nim) reads the semmed NIFs, generates C, compiles and links.

nifmake orders the steps by their input/output files: every nim m runs before the nim nifc step that consumes its NIF, and a step re-fires only when one of its inputs is newer than its outputs. The driver invokes nifmake run --parallel by default, so independent steps at the same DAG depth fan out across cores; pass -d:icNoParallel to serialize (readable child output when debugging a build).

Artifacts (the NIF zoo)

Per module <suffix> (a content hash of the path; see NIF symbols below), under the nimcache directory:

File	Producer	Purpose
<s>.p.nif	nifler	parsed AST (syntactic)
<s>.deps.nif	nifler	static import list (syntactic imports)
<s>.s.deps.nif	nim m	real post-sem imports (incl. macro-generated); see Discovery
<s>.nif	nim m	semmed module (symbols resolved, typed)
<s>.iface.nif	nim m	iface cookie: hash of the importer-visible surface
<s>.impl.nif	nim m	impl cookie: hash of the entire content (bodies included)
<s>.edges.nif	nim m	NeedsImpl edges: modules whose bodies this sem consumed
<s>.c.nif	nim nifc	the C text as a NIF, with def/ref markers for DCE & dedup
ic_config.cfg.nif	driver	precompiled config replayed by every child (icconfig.nim)
ic.version	driver	format stamp; a mismatch wipes the cache (icFormatVersion)

NIF symbols and ownership

(See ../nifspec/doc/nif-spec.md.) A global symbol is <ident>.<disamb>.<moduleSuffix>. For a generic instantiation the <disamb> is not a counter but a content hash — setInstanceDisamb (modulegraphs.nim) MD5s the generic's identity plus the typeKey of every concrete type argument, masks it to 30 bits and tags it with InstanceDisambBit. So the only part of the name that varies between two modules making the same instantiation (seq[Foo]) is the <moduleSuffix>. Two consequences drive the backend:

• Instance names are content-addressed: the same instantiation produced in different modules yields the same <ident>.<disamb>, so a deterministic dedup is possible by the module-suffix-stripped name. The cross-TU merge (ccgtypes.sharedInstanceCName, keyed via graph.icSharedSigs) and the DCE analysis (dce.computeLiveSymbols, its uniq set) both already key on this stripped form.
• The suffix names a mint-site owner. Today the <moduleSuffix> is the module that minted the instance (the instantiation site), so the same instance has a different full name in each module that makes it; which TU emits the single definition is then decided at codegen by the reuse/redirect machinery — the root of the single-writer hazard below.

The driver: graph construction (commandIc)

1. Stamp/​wipe the cache by icFormatVersion.
2. Seed the graph with the root module and system.nim. system's entire import closure is folded into one node (one nim m invocation) — see single-writer below.
3. traverseDeps runs nifler per module and reads .deps.nif to add import edges.
4. SCC grouping: strongly-connected import cycles are collapsed (Tarjan). A singleton compiles as nim m <mod>; a cycle compiles as one nim m <rep> --icGroup:<member>… that builds every member from source in one process (resolving the recursion in memory) and writes each member's NIF. Only edges leaving the component become build-graph inputs.
5. Discovery fixpoint: write the build file, run nifmake; if it fails, re-derive the graph from every module's .s.deps.nif (adding nodes/edges for imports the static scanner missed), and retry. See Discovery.
6. The backend step (nim nifc) depends on every module's semmed NIF, so nifmake runs it last.

Invalidation: the cookie system

A dependent must re-sem only when a dependency's relevant surface changed. Two hashes per module (ast2nif.nim):

• iface cookie (.iface.nif): hashes only the importer-visible surface — exported declarations' signatures (for all routine kinds: plain procs, templates, macros, generics, inline procs alike), full content for consts/types, plus import/export/replay/hook records. Routine bodies are excluded. It also chains in the iface cookies of its own dependencies, so a surface change anywhere in the import closure propagates. A nim m rule for a module depends on its dependencies' iface cookies, so a body-only edit moves no iface cookie and stops the re-sem cascade.
• impl cookie (.impl.nif): hashes the entire serialized content (private defs and bodies included), with the module's own iface mixed in.

NeedsImpl edges (.edges.nif): if a module consumed another module's body during sem — a macro expansion, a generic instantiation, a getImpl, or a compile-time call run in the VM — it records a strong edge. The dependent is then gated on that dependency's impl cookie instead of its iface cookie, so e.g. const x = dep.foo() re-sems when foo's body changes. Recording sites: semExprs.semTemplateExpr (templates), seminst.generateInstance (generics), vmgen.genProc (VM/macros/CT procs), vm.opcGetImpl (getImpl). Inline iterators and inline procs are not tracked — they are inlined at codegen, where the backend's NIF-mtime invalidation re-codegens their users.

Discovery of macro-generated imports

The static scanner only sees syntactic imports. A macro can synthesize one (chronicles does parseStmt("import chronicles/textlines") driven by the chronicles_sinks define). Such an import is invisible until sem runs the macro. Each nim m records the imports it actually resolved (via the semdata.addImportFileDep hook → graph.importDeps → ast2nif.writeSemDeps) into <s>.s.deps.nif; a child that fails on a not-yet-built import flushes it before erroring. The driver re-derives the graph from those sidecars — adding the missing node + the importer→import edge — and reruns to a fixpoint. (This replaced an earlier icmissing.txt side channel.)

The backend today: commandNifC

The current backend is whole-program and demand-driven, run as one process (compiler/nifbackend.nim):

1. Load system then all modules' semmed NIFs in dependency order (loadModuleDependencies), so all hooks/types are in memory.
2. DCE: computeLiveSymbols over all NIFs computes a global live set used to filter the top-level routine listing.
3. computeModuleReuse: decide which modules' cached translation units (.c.nif) can be reused — skip codegen for them and use their .c/.o as is. The gate mirrors the m-step's cookie gating; a coarse fallback uses the transitive NIF-mtime closure (-d:icCoarseReuse).
4. Codegen: for each non-reused module, generateCodeForModule runs cgen. Codegen is demand-driven: emitting one module can demand entities (generic instances, type-bound hooks, RTTI) that belong to other modules; a demand whose home TU is reused is redirected into the demanding TU (redirectToLiveModule), and reused TUs' definitions become prototypes.
5. enforceDefRetention: an un-reuse cascade. If a regenerated module would stop emitting a definition that a still-reused TU references (the demand chain that placed it no longer arises), the referencing TUs lose their reuse and regenerate so the symbol does not vanish under them.
6. emitMethodDispatchers, then finishModule for every module (main module last, so init-proc registration is complete before genMainProc).
7. Emit .c, then extccomp.callCCompiler + link.

Edge cases (and why the machinery exists)

• Single-writer. Instance type-ids are minted in process-local order, so if two nim m processes both write a module's NIF (e.g. a stdlib module pulled into system's from-source closure and given its own rule), the second overwrites with different ids and every module checked against the first carries dangling refs ("symbol has no offset"). Fixed by folding system's closure into one SCC and by forwarding the project's defines to every child so their when bodies (hence import sets and NIF contents) match the scanner's.
• when … else: import. nifler emits else-branch imports unguarded, so a dead else: import would be scheduled. The compiler's own sources were rewritten to explicit negated whens; the vendored nifler later learned to negate prior conditions for the else.
• nil sons of loaded ASTs. NIF dot-tokens load as nil where from-source ASTs have nkEmpty; several passes gained nil guards.
• Sealed loaded types. Loaded types are Sealed; sem/transform mutate via unsealForTransform/exactReplica(idgen) (the latter mints a fresh uniqueId so serialized replicas don't collapse).
• Methods/RTTI ownership. genTypeInfoV1 already routes a type's RTTI to t…itemId.module when that module is open for codegen — an existing ownership notion the rewrite can generalize.
• Config cost. Each child re-parsing nim.cfg + re-running config.nims in the VM was ~80 ms; replaced by a precompiled ic_config.cfg.nif replayed in loadConfigs (compiler/icconfig.nim).
• koch bootic bootstraps the compiler through nim ic (a 3-iteration fixed-point check). It writes its binary to bin/nim_ic and never clobbers bin/nim.

Known residual hacks (targets for the rewrite)

• deps.runNifler uses setLastModificationTime to mark its scan up-to-date and deletes a stale parsed file to coordinate with the nifmake nifler rule — the driver duplicating nifmake's freshness logic.
• computeModuleReuse + enforceDefRetention + the redirect/cached-defs machinery is a hand-rolled mini-nifmake inside the backend process, needed only because one process reuses some TUs while regenerating others.

These are legacy artifacts of a code generator that predates IC, not intrinsic requirements.

• Cross-mm reuse / --force (var not init i_<hash>). Reusing TUs built under one memory-management mode (e.g. an orc cache) while rebuilding under another (refc), or forcing a full backend re-run, can abort with var not init. The name is a backend-minted closure-capture local whose loc is unset; the root cause is that nifmake is mtime-only and computeModuleReuse is not config-aware, so a reuse decision can mix TUs generated under different config signatures. It is crash-safe (it aborts, never emits a wrong-mm binary) and does not affect normal same-mode warm rebuilds. Deferred to the rewrite, which makes reuse config-aware by construction (each codegen rule's config is a declared input).

Status and performance

nim ic self-builds the compiler (koch bootic's fixed-point check) under both orc and --mm:refc, and passes the external-package CI set. The macro sweep is 93/95 (two known-baseline failures: tmacro7, tmacrogetimpl).

Rough numbers on a 32-core box (-d:release, building the compiler):

	wall	notes
classic nim c full	~27.5s	~7 cores busy
nim ic cold, serial	~81s	one nim m at a time
nim ic cold, --parallel	~53s	now the default
nim ic warm no-op	~0.14s	~200× vs classic

The cold gap is a parallelism deficiency, not extra fundamental work (aggregate work is ≈1.33× classic). The backend's own phases are cheap (load 0.3s, DCE 1.0s, cgen 5.7s, write 0.8s); the C compile+link floor (~21s) is already parallel and shared with the classic backend. The remaining cold cost is the nim m sem phase, throttled by nifmake's per-DAG-depth barrier across a narrow, deep tail of heavy modules. A ready-queue scheduler (dependency-ready dispatch instead of a depth barrier) would recover several seconds but cannot beat the import-chain critical path; it folds into the rewrite. Warm rebuilds — the actual point of IC — already dominate the classic backend by ~200×.

The strategic direction (decided 2026-06-13) is to make this NIF backend (cmdNifC) the default code generator. The clean translation-unit model it needs — global DCE, content-keyed instance merging, deterministic RTTI — already exists and is always-on inside cmdNifC; the remaining work is the per-module backend rewrite below (which also dissolves the cross-mm reuse hazard), then promotion + deletion, not new machinery.

Plan: a nifmake-driven, per-module backend

Goal: the backend stops re-implementing nifmake. Each module's codegen becomes its own build rule, so "which TUs rebuild" is just "which rules nifmake re-fires from input mtimes" — exactly as the frontend already works. The reuse / def-retention / redirect machinery then dissolves.

Target build graph (mirrors Nimony's src/nimony/deps.nim):

1. Frontend split. Generate a frontend build file (nifler + nim m rules), run nifmake, run the .s.deps discovery fixpoint. Then re-derive the graph from .s.deps (now complete; dead when imports can also be pruned here).
2. Per-module codegen rule. One nifc <mod> rule per module: inputs are the module's own .nif plus the .iface/.impl cookies of its dependencies; output is its <s>.c.nif. The process loads <mod> + its import closure's NIFs (like nim m) and emits only the entities it owns, referencing everything else extern.
3. Static ownership replaces runtime redirect. Every emittable entity — generic instances, type-bound hooks, RTTI, lifted procs — gets a deterministic owner module by symbol suffix. Because instance names are content-addressed (ident.disamb.key.owner), the same instance demanded by several modules has one name and one owner, so there is exactly one writer and no link-time duplicate. (The precise owner rule — minting module vs. root-type's module — is the open design decision; start from itemId.module and adjust where it forces a downstream package to own stdlib code.)
4. DCE as a rule. A single rule reads all .c.nif, computes the global live set, and (Nimony: .live.nif / per-module .dce.nif) drives per-module .c emission filtered to live entities.
5. Link rule. Depends on every .o (each compiled by its own rule) and the DCE output; produces the executable.
6. Deletions. computeModuleReuse, enforceDefRetention, redirectToLiveModule and the cached-defs/claim bookkeeping go away. The setLastModificationTime coordination in runNifler goes away with the frontend split (the nifler rule owns parsed+deps; the driver's pre-scan only reads .deps to build the graph).

Validation bar: koch bootic must still reach its byte-identical fixed point, and binary size must not regress (DCE parity), across the external-package CI set.

Code, logic & debugging

Core modules:

• compiler/deps.nim — graph construction, SCC grouping, discovery fixpoint, build-file generation; commandIc.
• compiler/ast2nif.nim — AST↔NIF, the cookie hashes (cookieSd, writeIfaceCookie, writeImplCookie, writeEdgesFile, writeSemDeps).
• compiler/nifbackend.nim — the backend (commandNifC) and its reuse machinery.
• compiler/icconfig.nim — precompiled config.
• compiler/pipelines.nim / modulegraphs.nim — pipeline integration and the graph state (importDeps, icImplDeps, icReusedModules, …).

Manual workflow:

• Frontend a module: nim m --nimcache:nifcache path/to/mod.nim (writes .nif + cookies + .s.deps).
• Backend: nim nifc --nimcache:nifcache main.nim.
• NIF files are text — open/grep them directly; diff two successive .nif to see why a module rebuilt.
• Force a re-sem: delete the module's .nif and rerun nim m.
• A stale-cache crash after editing the serialization layout means bumping icFormatVersion (compiler/options.nim).

See also

• NIF format spec: nifspec/doc/nif-spec.md
• NIFC (C-like target) spec: dist/nimony/doc/nifc-spec.md