mirror of
https://github.com/nim-lang/Nim.git
synced 2026-08-02 21:49:02 +00:00
157 lines
44 KiB
HTML
157 lines
44 KiB
HTML
<?xml version="1.0" encoding="utf-8" ?>
|
||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||
<!-- This file is generated by Nim. -->
|
||
<html xmlns="https://www.w3.org/1999/xhtml" xml:lang="en" lang="en" data-theme="auto">
|
||
<head>
|
||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>Incremental Compilation (IC)</title>
|
||
|
||
<!-- Google fonts -->
|
||
<link href='https://fonts.googleapis.com/css?family=Lato:400,600,900' rel='stylesheet' type='text/css'/>
|
||
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro:400,500,600' rel='stylesheet' type='text/css'/>
|
||
|
||
<!-- Favicon -->
|
||
<link rel="shortcut icon" href="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AAAAAAUAAAAF////AP///wD///8A////AP///wD///8A////AP///wD///8A////AAAAAAIAAABbAAAAlQAAAKIAAACbAAAAmwAAAKIAAACVAAAAWwAAAAL///8A////AP///wD///8A////AAAAABQAAADAAAAAYwAAAA3///8A////AP///wD///8AAAAADQAAAGMAAADAAAAAFP///wD///8A////AP///wAAAACdAAAAOv///wD///8A////AP///wD///8A////AP///wD///8AAAAAOgAAAJ3///8A////AP///wAAAAAnAAAAcP///wAAAAAoAAAASv///wD///8A////AP///wAAAABKAAAAKP///wAAAABwAAAAJ////wD///8AAAAAgQAAABwAAACIAAAAkAAAAJMAAACtAAAAFQAAABUAAACtAAAAkwAAAJAAAACIAAAAHAAAAIH///8A////AAAAAKQAAACrAAAAaP///wD///8AAAAARQAAANIAAADSAAAARf///wD///8AAAAAaAAAAKsAAACk////AAAAADMAAACcAAAAnQAAABj///8A////AP///wAAAAAYAAAAGP///wD///8A////AAAAABgAAACdAAAAnAAAADMAAAB1AAAAwwAAAP8AAADpAAAAsQAAAE4AAAAb////AP///wAAAAAbAAAATgAAALEAAADpAAAA/wAAAMMAAAB1AAAAtwAAAOkAAAD/AAAA/wAAAP8AAADvAAAA3gAAAN4AAADeAAAA3gAAAO8AAAD/AAAA/wAAAP8AAADpAAAAtwAAAGUAAAA/AAAA3wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAADfAAAAPwAAAGX///8A////AAAAAEgAAADtAAAAvwAAAL0AAADGAAAA7wAAAO8AAADGAAAAvQAAAL8AAADtAAAASP///wD///8A////AP///wD///8AAAAAO////wD///8A////AAAAAIcAAACH////AP///wD///8AAAAAO////wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A//8AAP//AAD4HwAA7/cAAN/7AAD//wAAoYUAAJ55AACf+QAAh+EAAAAAAADAAwAA4AcAAP5/AAD//wAA//8AAA=="/>
|
||
<link rel="icon" type="image/png" sizes="32x32" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAA3XAAAN1wFCKJt4AAAAB3RJTUUH4QQQEwksSS9ZWwAAAk1JREFUWMPtll2ITVEUx39nn/O7Y5qR8f05wtCUUr6ZIS++8pEnkZInPImneaCQ5METNdOkeFBKUhMPRIkHKfEuUZSUlGlKPN2TrgfncpvmnntnmlEyq1Z7t89/rf9a6+y99oZxGZf/XeIq61EdtgKXgdXA0xrYAvBjOIF1AI9zvjcC74BSpndrJPkBWDScTF8Aa4E3wDlgHbASaANmVqlcCnwHvgDvgVfAJ+AikAAvgfVZwLnSVZHZaOuKoQi3ZOMi4NkYkpe1p4J7A8BpYAD49hfIy/oqG0+hLomiKP2L5L+1ubn5115S+3OAn4EnwBlgMzCjyt6ZAnQCJ4A7wOs88iRJHvw50HoujuPBoCKwHWiosy8MdfZnAdcHk8dxXFJ3VQbQlCTJvRBCGdRbD4M6uc5glpY3eAihpN5S5w12diSEcCCEcKUO4ljdr15T76ur1FDDLIQQ3qv71EdDOe3Kxj3leRXyk+pxdWnFWod6Wt2bY3de3aSuUHcPBVimHs7mK9WrmeOF6lR1o9qnzskh2ar2qm1qizpfXaPeVGdlmGN5pb09qMxz1Xb1kLqgzn1RyH7JUXW52lr5e/Kqi9qpto7V1atuUzfnARrV7jEib1T76gG2qxdGmXyiekkt1GswPTtek0aBfJp6YySGBfWg2tPQ0FAYgf1stUfdmdcjarbYJEniKIq6gY/Aw+zWHAC+p2labGpqiorFYgGYCEzN7oQdQClN07O1/EfDyGgC0ALMBdYAi4FyK+4H3gLPsxfR1zRNi+NP7nH5J+QntnXe5B5mpfQAAAAASUVORK5CYII=">
|
||
|
||
<!-- CSS -->
|
||
<link rel="stylesheet" type="text/css" href="nimdoc.out.css?v=2.3.1">
|
||
|
||
<!-- JS -->
|
||
<script type="text/javascript" src="dochack.js?v=2.3.1"></script>
|
||
</head>
|
||
<body>
|
||
<div class="document" id="documentId">
|
||
<input type="checkbox" id="nav-toggle" hidden>
|
||
<label for="nav-toggle" id="nav-burger">☰</label>
|
||
<label for="nav-toggle" id="nav-overlay"></label>
|
||
<div class="container">
|
||
<h1 class="title">Incremental Compilation (IC)</h1>
|
||
<p>The <tt class="docutils literal"><span class="pre">nim ic</span></tt> command provides incremental compilation for Nim projects. It decomposes compilation into per-module steps whose results are cached as NIF files, and uses the external <tt class="docutils literal"><span class="pre">nifmake</span></tt> build tool to re-run only the steps whose inputs changed.</p>
|
||
<p>This document describes <strong>how `nim ic` works today</strong>, including the edge cases that shaped the current design. The per-module backend rewrite that earlier editions of this document listed as a <em>Plan</em> has <strong>landed</strong>: the whole-program, reuse/redirect/def-retention backend is gone and codegen is now a set of <tt class="docutils literal"><span class="pre">nifmake</span></tt>-driven per-module rules (see <em>The backend</em>).</p>
|
||
|
||
<h1 id="overview">Overview</h1><p>The pipeline has two halves driven by one process (<tt class="docutils literal"><span class="pre">nim ic</span></tt>, <tt class="docutils literal"><span class="pre">commandIc</span></tt> in <tt class="docutils literal"><span class="pre">compiler/deps.nim</span></tt>) that constructs a dependency graph, writes a build file, and hands it to <tt class="docutils literal"><span class="pre">nifmake</span></tt>:</p>
|
||
<ol class="simple"><li><strong>Frontend</strong> — per module:<ul class="simple"><li><tt class="docutils literal"><span class="pre">nifler parse --deps</span></tt> turns <tt class="docutils literal"><span class="pre">.nim</span></tt> source into a parsed NIF (<tt class="docutils literal"><span class="pre">.p.nif</span></tt>) plus a static dependency list (<tt class="docutils literal"><span class="pre">.deps.nif</span></tt>).</li>
|
||
<li><tt class="docutils literal"><span class="pre">nim m</span></tt> (the <em>semantic</em> step, <tt class="docutils literal"><span class="pre">cmdM</span></tt>) reads the parsed NIF + the precompiled NIFs of the module's imports, type-checks, and writes the <strong>semmed NIF</strong> (<tt class="docutils literal"><span class="pre">.nif</span></tt>) plus invalidation sidecars (see <em>Cookies</em>).</li>
|
||
</ul>
|
||
</li>
|
||
<li><strong>Backend</strong> — <tt class="docutils literal"><span class="pre">nim nifc</span></tt> (<tt class="docutils literal"><span class="pre">cmdNifC</span></tt>, <tt class="docutils literal"><span class="pre">compiler/nifbackend.nim</span></tt>) reads the semmed NIFs, generates C, compiles and links.</li>
|
||
</ol>
|
||
<p><tt class="docutils literal"><span class="pre">nifmake</span></tt> orders the steps by their input/output files: every <tt class="docutils literal"><span class="pre">nim m</span></tt> runs before the <tt class="docutils literal"><span class="pre">nim nifc</span></tt> step that consumes its NIF, and a step re-fires only when one of its inputs is newer than its outputs. The driver invokes <tt class="docutils literal"><span class="pre">nifmake run --parallel</span></tt> by default, so independent steps at the same DAG depth fan out across cores; pass <tt class="docutils literal"><span class="pre">-d:icNoParallel</span></tt> to serialize (readable child output when debugging a build).</p>
|
||
|
||
<h1 id="artifacts-the-nif-zoo">Artifacts (the NIF zoo)</h1><p>Per module <tt class="docutils literal"><span class="pre"><suffix></span></tt> (a content hash of the path; see <em>NIF symbols</em> below), under the nimcache directory:</p>
|
||
<table border="1" class="docutils"><tr><th>File</th><th>Producer</th><th>Purpose</th></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.p.nif</span></tt></td><td>nifler</td><td>parsed AST (syntactic)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.deps.nif</span></tt></td><td>nifler</td><td><strong>static</strong> import list (syntactic <tt class="docutils literal"><span class="pre">import</span></tt>s)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.s.deps.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>real</strong> post-sem imports (incl. macro-generated); see <em>Discovery</em></td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td>semmed module (symbols resolved, typed)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.iface.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>iface cookie</strong>: hash of the importer-visible surface</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.impl.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>impl cookie</strong>: hash of the entire content (bodies included)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.edges.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>NeedsImpl edges</strong>: modules whose bodies this sem consumed</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.c.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim nifc</span></tt></td><td>the C text as a NIF, with def/ref markers for DCE & dedup</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre">ic_config.cfg.nif</span></tt></td><td>driver</td><td>precompiled config replayed by every child (<tt class="docutils literal"><span class="pre">icconfig.nim</span></tt>)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre">ic.version</span></tt></td><td>driver</td><td>format stamp; a mismatch wipes the cache (<tt class="docutils literal"><span class="pre">icFormatVersion</span></tt>)</td></tr>
|
||
</table>
|
||
<h1 id="nif-symbols-and-ownership">NIF symbols and ownership</h1><p>(See <tt class="docutils literal"><span class="pre">../nifspec/doc/nif-spec.md</span></tt>.) A global symbol is <tt class="docutils literal"><span class="pre"><ident>.<disamb>.<moduleSuffix></span></tt>. For a <strong>generic instantiation</strong> the <tt class="docutils literal"><span class="pre"><disamb></span></tt> is not a counter but a <em>content hash</em> — <tt class="docutils literal"><span class="pre">setInstanceDisamb</span></tt> (<tt class="docutils literal"><span class="pre">modulegraphs.nim</span></tt>) MD5s the generic's identity plus the <tt class="docutils literal"><span class="pre">typeKey</span></tt> of every concrete type argument, masks it to 30 bits and tags it with <tt class="docutils literal"><span class="pre">InstanceDisambBit</span></tt>. So the only part of the name that varies between two modules making the <strong>same</strong> instantiation (<tt class="docutils literal"><span class="pre">seq[Foo]</span></tt>) is the <tt class="docutils literal"><span class="pre"><moduleSuffix></span></tt>. Two consequences drive the backend:</p>
|
||
<ul class="simple"><li><strong>Instance names are content-addressed</strong>: the same instantiation produced in different modules yields the <em>same</em> <tt class="docutils literal"><span class="pre"><ident>.<disamb></span></tt>, so a deterministic dedup is possible by the <em>module-suffix-stripped</em> name. The cross-TU C name (<tt class="docutils literal"><span class="pre">ccgtypes.sharedInstanceCName</span></tt>) and the <strong>merge</strong> stage's live-set/owner decision (<tt class="docutils literal"><span class="pre">nifbackend.computeMergeDecision</span></tt>) both key on this stripped form.</li>
|
||
<li><strong>The suffix names a mint-site owner.</strong> The <tt class="docutils literal"><span class="pre"><moduleSuffix></span></tt> is the module <em>that minted the instance</em> (the instantiation site), so the same instance has a different full name in each module that makes it. Because every <tt class="docutils literal"><span class="pre">cg</span></tt> process emits the instances it demands (<em>emit-everywhere</em>), the same definition can be produced by several translation units; the <strong>merge</strong> stage then deterministically picks the single artifact allowed to embed each body (smallest claimant), which is the cross-process replacement for the old in-process single-writer machinery.</li>
|
||
</ul>
|
||
|
||
<h1 id="the-drivercolon-graph-construction-commandic">The driver: graph construction (<tt class="docutils literal"><span class="pre">commandIc</span></tt>)</h1><ol class="simple"><li>Stamp/wipe the cache by <tt class="docutils literal"><span class="pre">icFormatVersion</span></tt>.</li>
|
||
<li>Seed the graph with the root module and <strong>`system.nim`</strong>. <tt class="docutils literal"><span class="pre">system</span></tt>'s entire import closure is folded into one node (one <tt class="docutils literal"><span class="pre">nim m</span></tt> invocation) — see <em>single-writer</em> below.</li>
|
||
<li><tt class="docutils literal"><span class="pre">traverseDeps</span></tt> runs <tt class="docutils literal"><span class="pre">nifler</span></tt> per module and reads <tt class="docutils literal"><span class="pre">.deps.nif</span></tt> to add import edges.</li>
|
||
<li><strong>SCC grouping</strong>: strongly-connected import cycles are collapsed (Tarjan). A singleton compiles as <tt class="docutils literal"><span class="pre">nim m <mod></span></tt>; a cycle compiles as one <tt class="docutils literal"><span class="pre">nim m <rep> --icGroup:<member>…</span></tt> that builds every member <em>from source</em> in one process (resolving the recursion in memory) and writes each member's NIF. Only edges <em>leaving</em> the component become build-graph inputs.</li>
|
||
<li><strong>Discovery fixpoint</strong>: write the build file, run <tt class="docutils literal"><span class="pre">nifmake</span></tt>; if it fails, re-derive the graph from every module's <tt class="docutils literal"><span class="pre">.s.deps.nif</span></tt> (adding nodes/edges for imports the static scanner missed), and retry. See <em>Discovery</em>.</li>
|
||
<li>The backend step (<tt class="docutils literal"><span class="pre">nim nifc</span></tt>) depends on every module's semmed NIF, so <tt class="docutils literal"><span class="pre">nifmake</span></tt> runs it last.</li>
|
||
</ol>
|
||
|
||
<h1 id="invalidationcolon-the-cookie-system">Invalidation: the cookie system</h1><p>A dependent must re-sem only when a dependency's relevant surface changed. Two hashes per module (<tt class="docutils literal"><span class="pre">ast2nif.nim</span></tt>):</p>
|
||
<ul class="simple"><li><strong>iface cookie</strong> (<tt class="docutils literal"><span class="pre">.iface.nif</span></tt>): hashes only the <em>importer-visible</em> surface — exported declarations' <strong>signatures</strong> (for <em>all</em> routine kinds: plain procs, templates, macros, generics, <tt class="docutils literal"><span class="pre">inline</span></tt> procs alike), full content for consts/types, plus import/export/replay/hook records. Routine <strong>bodies are excluded.</strong> It also chains in the iface cookies of its own dependencies, so a surface change anywhere in the import closure propagates. A <tt class="docutils literal"><span class="pre">nim m</span></tt> rule for a module depends on its dependencies' iface cookies, so a body-only edit moves no iface cookie and stops the re-sem cascade.</li>
|
||
<li><strong>impl cookie</strong> (<tt class="docutils literal"><span class="pre">.impl.nif</span></tt>): hashes the <em>entire</em> serialized content (private defs and bodies included), with the module's own iface mixed in.</li>
|
||
</ul>
|
||
<p><strong>NeedsImpl edges</strong> (<tt class="docutils literal"><span class="pre">.edges.nif</span></tt>): if a module <em>consumed another module's body</em> during sem — a macro expansion, a generic instantiation, a <tt class="docutils literal"><span class="pre">getImpl</span></tt>, or a compile-time call run in the VM — it records a strong edge. The dependent is then gated on that dependency's <strong>impl</strong> cookie instead of its iface cookie, so e.g. <tt class="docutils literal"><span class="pre">const x = dep.foo()</span></tt> re-sems when <tt class="docutils literal"><span class="pre">foo</span></tt>'s body changes. Recording sites: <tt class="docutils literal"><span class="pre">semExprs.semTemplateExpr</span></tt> (templates), <tt class="docutils literal"><span class="pre">seminst.generateInstance</span></tt> (generics), <tt class="docutils literal"><span class="pre">vmgen.genProc</span></tt> (VM/macros/CT procs), <tt class="docutils literal"><span class="pre">vm.opcGetImpl</span></tt> (<tt class="docutils literal"><span class="pre">getImpl</span></tt>). Inline iterators and <tt class="docutils literal"><span class="pre">inline</span></tt> procs are <em>not</em> tracked — they are inlined at codegen, where the backend's NIF-mtime invalidation re-codegens their users.</p>
|
||
|
||
<h1 id="discovery-of-macrominusgenerated-imports">Discovery of macro-generated imports</h1><p>The static scanner only sees syntactic <tt class="docutils literal"><span class="pre">import</span></tt>s. A macro can synthesize one (chronicles does <tt class="docutils literal"><span class="pre">parseStmt("import chronicles/textlines")</span></tt> driven by the <tt class="docutils literal"><span class="pre">chronicles_sinks</span></tt> define). Such an import is invisible until sem runs the macro. Each <tt class="docutils literal"><span class="pre">nim m</span></tt> records the imports it <em>actually</em> resolved (via the <tt class="docutils literal"><span class="pre">semdata.addImportFileDep</span></tt> hook → <tt class="docutils literal"><span class="pre">graph.importDeps</span></tt> → <tt class="docutils literal"><span class="pre">ast2nif.writeSemDeps</span></tt>) into <tt class="docutils literal"><span class="pre"><s>.s.deps.nif</span></tt>; a child that fails on a not-yet-built import flushes it before erroring. The driver re-derives the graph from those sidecars — adding the missing node + the importer→import edge — and reruns to a fixpoint. (This replaced an earlier <tt class="docutils literal"><span class="pre">icmissing.txt</span></tt> side channel.)</p>
|
||
|
||
<h1 id="the-backendcolon-perminusmodule-nifc-stages">The backend: per-module <tt class="docutils literal"><span class="pre">nifc</span></tt> stages</h1><p>Codegen is no longer one whole-program process. <tt class="docutils literal"><span class="pre">nim nifc</span></tt> (<tt class="docutils literal"><span class="pre">cmdNifC</span></tt>, <tt class="docutils literal"><span class="pre">compiler/nifbackend.nim</span></tt>) is invoked once per <strong>stage</strong> via <tt class="docutils literal"><span class="pre">--icBackendStage:<stage></span></tt>; <tt class="docutils literal"><span class="pre">commandIc</span></tt> emits these as ordinary <tt class="docutils literal"><span class="pre">nifmake</span></tt> rules so "which TUs rebuild" is just "which rules <tt class="docutils literal"><span class="pre">nifmake</span></tt> re-fires from input mtimes" — exactly as the frontend already works. There are four stages:</p>
|
||
<ol class="simple"><li><strong>`cg`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:cg --icBackendModule:<suffix></span></tt>) — generate C for the <em>single</em> named module and write only its <tt class="docutils literal"><span class="pre"><s>.c.nif</span></tt> artifact. A non-main target loads only its own import closure (<tt class="docutils literal"><span class="pre">loadDepClosure</span></tt>), so the whole program is <strong>not</strong> pulled into every parallel <tt class="docutils literal"><span class="pre">cg</span></tt> process. Codegen is still demand-driven and <strong>emit-everywhere</strong>: a <tt class="docutils literal"><span class="pre">cg</span></tt> process emits every entity it demands (generic instances, hooks, RTTI), referencing nothing <tt class="docutils literal"><span class="pre">extern</span></tt>-only. There is no whole-program DCE here — a liveness pass over all ~260 NIFs would cost ~900 MB for a result the merge stage recomputes anyway. The <strong>main</strong> module's <tt class="docutils literal"><span class="pre">cg</span></tt> is special: it loads everything (<tt class="docutils literal"><span class="pre">loadBackendModules</span></tt>), emits the whole-program method dispatchers and <tt class="docutils literal"><span class="pre">NimMain</span></tt>, and registers every other module's init/datInit from the <tt class="docutils literal"><span class="pre">.c.nif</span></tt> meta heads — so it runs <em>last</em>, after every other <tt class="docutils literal"><span class="pre">.c.nif</span></tt> exists. Every <tt class="docutils literal"><span class="pre">cg</span></tt> rule always leaves a <tt class="docutils literal"><span class="pre">.c.nif</span></tt> (empty if the module owns no code) so its nifmake output exists and the rule settles.</li>
|
||
<li><strong>`merge`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:merge</span></tt>) — a pure artifact pass, <em>no module graph loaded</em>. Reads every <tt class="docutils literal"><span class="pre">.c.nif</span></tt>, computes the one program-wide live set and, for each unique definition that several <tt class="docutils literal"><span class="pre">cg</span></tt> processes emitted, the single artifact allowed to embed its body; writes that to a merge-decision file (<tt class="docutils literal"><span class="pre">computeMergeDecision</span></tt> / <tt class="docutils literal"><span class="pre">writeMergeDecision</span></tt>). This is the cross-process replacement for the old in-process first-claimant + DCE coordination.</li>
|
||
<li><strong>`emit`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:emit --icBackendModule:<suffix></span></tt>) — render the target module's final <tt class="docutils literal"><span class="pre">.c</span></tt> from its <tt class="docutils literal"><span class="pre">.c.nif</span></tt> and the merge decision (<tt class="docutils literal"><span class="pre">renderCFromArtifact</span></tt>, dropping globally-dead and non-owned bodies). No codegen runs; the target is loaded only so <tt class="docutils literal"><span class="pre">getCFile</span></tt> yields the path <tt class="docutils literal"><span class="pre">cg</span></tt> wrote.</li>
|
||
<li><strong>`link`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:link</span></tt>) — register every module's emitted <tt class="docutils literal"><span class="pre">.c</span></tt> and run <tt class="docutils literal"><span class="pre">extccomp.callCCompiler</span></tt> once (it parallelizes per-file cc and skips up-to-date objects). Per-module C compile/link directives (<tt class="docutils literal"><span class="pre">{.passL.}</span></tt> etc.) are re-collected here via <tt class="docutils literal"><span class="pre">replayBackendActions</span></tt>, since the <tt class="docutils literal"><span class="pre">cg</span></tt> processes that originally saw them are separate processes (without this, e.g. <tt class="docutils literal"><span class="pre">math</span></tt>'s <tt class="docutils literal"><span class="pre">-lm</span></tt> would be lost → undefined <tt class="docutils literal"><span class="pre">floor</span></tt>/<tt class="docutils literal"><span class="pre">pow</span></tt> at link).</li>
|
||
</ol>
|
||
<p>Because each stage is a <tt class="docutils literal"><span class="pre">nifmake</span></tt> rule keyed on file mtimes, a body-only edit to one module re-fires that module's <tt class="docutils literal"><span class="pre">cg</span></tt>+<tt class="docutils literal"><span class="pre">emit</span></tt> (and the <tt class="docutils literal"><span class="pre">merge</span></tt>/<tt class="docutils literal"><span class="pre">link</span></tt>), not the whole program — and an unchanged module's <tt class="docutils literal"><span class="pre">cg</span></tt> does not run at all.</p>
|
||
|
||
<h1 id="edge-cases-and-why-the-machinery-exists">Edge cases (and why the machinery exists)</h1><ul class="simple"><li><strong>Single-writer.</strong> Instance type-ids are minted in process-local order, so if two <tt class="docutils literal"><span class="pre">nim m</span></tt> processes both write a module's NIF (e.g. a stdlib module pulled into <tt class="docutils literal"><span class="pre">system</span></tt>'s from-source closure <em>and</em> given its own rule), the second overwrites with different ids and every module checked against the first carries dangling refs ("symbol has no offset"). Fixed by folding <tt class="docutils literal"><span class="pre">system</span></tt>'s closure into one SCC and by <strong>forwarding the project's defines</strong> to every child so their <tt class="docutils literal"><span class="pre">when</span></tt> bodies (hence import sets and NIF contents) match the scanner's.</li>
|
||
<li><strong>`when … else: import`.</strong> nifler emits <tt class="docutils literal"><span class="pre">else</span></tt>-branch imports unguarded, so a dead <tt class="docutils literal"><span class="pre">else: import</span></tt> would be scheduled. The compiler's own sources were rewritten to explicit negated <tt class="docutils literal"><span class="pre">when</span></tt>s; the vendored nifler later learned to negate prior conditions for the <tt class="docutils literal"><span class="pre">else</span></tt>.</li>
|
||
<li><strong>`nil` sons of loaded ASTs.</strong> NIF dot-tokens load as <tt class="docutils literal"><span class="pre">nil</span></tt> where from-source ASTs have <tt class="docutils literal"><span class="pre">nkEmpty</span></tt>; several passes gained <tt class="docutils literal"><span class="pre">nil</span></tt> guards.</li>
|
||
<li><strong>Sealed loaded types.</strong> Loaded types are <tt class="docutils literal"><span class="pre">Sealed</span></tt>; sem/transform mutate via <tt class="docutils literal"><span class="pre">unsealForTransform</span></tt>/<tt class="docutils literal"><span class="pre">exactReplica(idgen)</span></tt> (the latter mints a fresh <tt class="docutils literal"><span class="pre">uniqueId</span></tt> so serialized replicas don't collapse).</li>
|
||
<li><strong>Methods/RTTI ownership.</strong> RTTI and type-bound hooks are emit-everywhere at <tt class="docutils literal"><span class="pre">cg</span></tt> and deduplicated by the <tt class="docutils literal"><span class="pre">merge</span></tt> stage, like generic instances; the main module's <tt class="docutils literal"><span class="pre">cg</span></tt> owns the whole-program method dispatchers.</li>
|
||
<li><strong>Config cost.</strong> Each child re-parsing <tt class="docutils literal"><span class="pre">nim.cfg</span></tt> + re-running <tt class="docutils literal"><span class="pre">config.nims</span></tt> in the VM was ~80 ms; replaced by a precompiled <tt class="docutils literal"><span class="pre">ic_config.cfg.nif</span></tt> replayed in <tt class="docutils literal"><span class="pre">loadConfigs</span></tt> (<tt class="docutils literal"><span class="pre">compiler/icconfig.nim</span></tt>).</li>
|
||
<li><strong>`koch bootic`</strong> bootstraps the compiler through <tt class="docutils literal"><span class="pre">nim ic</span></tt> (a 3-iteration fixed-point check). It writes its binary to <tt class="docutils literal"><span class="pre">bin/nim_ic</span></tt> and never clobbers <tt class="docutils literal"><span class="pre">bin/nim</span></tt>.</li>
|
||
</ul>
|
||
|
||
<h2 id="resolved-by-the-rewrite">Resolved by the rewrite</h2><p>The whole-program backend's hand-rolled mini-<tt class="docutils literal"><span class="pre">nifmake</span></tt> — <tt class="docutils literal"><span class="pre">computeModuleReuse</span></tt>, <tt class="docutils literal"><span class="pre">enforceDefRetention</span></tt>, <tt class="docutils literal"><span class="pre">redirectToLiveModule</span></tt>, the cached-defs/claim bookkeeping and the standalone <tt class="docutils literal"><span class="pre">dce.nim</span></tt> — <strong>is gone</strong>. Reuse is now just per-rule <tt class="docutils literal"><span class="pre">nifmake</span></tt> mtime checks, and the single-writer decision is the <tt class="docutils literal"><span class="pre">merge</span></tt> stage. The old <strong>cross-mm / `--force` `var not init`</strong> hazard dissolved with it: every codegen rule's config (including <tt class="docutils literal"><span class="pre">--mm</span></tt>) is a declared <tt class="docutils literal"><span class="pre">nifmake</span></tt> input, so a stale-config TU is simply rebuilt rather than mixed in. <tt class="docutils literal"><span class="pre">koch bootic</span></tt> is green under both <tt class="docutils literal"><span class="pre">orc</span></tt> and <tt class="docutils literal"><span class="pre">--mm:refc</span></tt>.</p>
|
||
|
||
<h2 id="known-residual-hack">Known residual hack</h2><ul class="simple"><li><tt class="docutils literal"><span class="pre">deps.runNifler</span></tt> still uses <tt class="docutils literal"><span class="pre">setLastModificationTime</span></tt> to mark its scan up-to-date and deletes a stale parsed file to coordinate with the nifmake nifler rule — the driver duplicating nifmake's freshness logic. It is explicitly flagged in the source and folds away with a full frontend/nifler split.</li>
|
||
</ul>
|
||
|
||
<h1 id="status-and-performance">Status and performance</h1><p><tt class="docutils literal"><span class="pre">nim ic</span></tt> self-builds the compiler (<tt class="docutils literal"><span class="pre">koch bootic</span></tt>'s byte-identical fixed-point check) under both <tt class="docutils literal"><span class="pre">orc</span></tt> and <tt class="docutils literal"><span class="pre">--mm:refc</span></tt>, and passes the external-package CI set.</p>
|
||
<p>Cold full bootstrap on a 32-core box (<tt class="docutils literal"><span class="pre">-d:release</span></tt>, <strong>no edits</strong> — IC's worst case, since incremental reuse is not exercised):</p>
|
||
<p>| wall | notes |<br/><ul class="simple"><li>| ---- | ----- |</li>
|
||
</ul>
|
||
<br/><tt class="docutils literal"><span class="pre">koch boot</span></tt> (classic) | ~1m00s | reference |<br/><tt class="docutils literal"><span class="pre">koch bootic</span></tt> (<tt class="docutils literal"><span class="pre">nim ic</span></tt>) | ~1m39s | <strong>~1.66×</strong> |<br/></p><p>This is down from ~7.5× in the whole-program-backend era. IC does modestly more aggregate work (more processes, NIF re-parsing of imports per process), but on a many-core box that overhead is absorbed by the parallel <tt class="docutils literal"><span class="pre">nim m</span></tt>/<tt class="docutils literal"><span class="pre">nifc</span></tt> fan-out, and the C compile+link floor is shared with the classic backend. On few-core machines the cold gap is correspondingly wider — IC trades single-build latency for incremental latency.</p>
|
||
<p>The cold number is the <em>least</em> favourable comparison: it pays IC's full per-process overhead while using none of its incremental machinery. <strong>Warm rebuilds — the actual point of IC — recompile only the modules whose inputs changed</strong> (a body-only edit re-fires one module's <tt class="docutils literal"><span class="pre">cg</span></tt>+<tt class="docutils literal"><span class="pre">emit</span></tt>, not the program), so an edit-driven rebuild is a small fraction of either full build.</p>
|
||
<p>The strategic direction (decided 2026-06-13) is to make this NIF backend (<tt class="docutils literal"><span class="pre">cmdNifC</span></tt>) the <strong>default</strong> code generator. The per-module pipeline above is the realization of that direction; remaining work is <em>promotion + deletion</em> of the classic path, not new machinery.</p>
|
||
|
||
<h1 id="design-notes-and-open-decisions">Design notes and open decisions</h1><p>The per-module backend (above) mirrors Nimony's <tt class="docutils literal"><span class="pre">src/nimony/deps.nim</span></tt>: the backend stopped re-implementing <tt class="docutils literal"><span class="pre">nifmake</span></tt>; each stage is a build rule, so reuse is just mtime checks and the merge stage is the only cross-module coordination.</p>
|
||
<p>Settled vs. open:</p>
|
||
<ul class="simple"><li><strong>Ownership.</strong> Emittable entities (generic instances, type-bound hooks, RTTI, lifted procs) are emit-everywhere at <tt class="docutils literal"><span class="pre">cg</span></tt> time and deduplicated at <tt class="docutils literal"><span class="pre">merge</span></tt> time (smallest claimant owns each unique body). The earlier idea of a <em>static</em> per-suffix owner computed before codegen was not needed — content-addressed names make the merge decision deterministic. The precise owner <em>rule</em> (minting module vs. root-type's module) can still be tuned where it would force a downstream package to own stdlib code.</li>
|
||
<li><strong>Remaining cleanup.</strong> The <tt class="docutils literal"><span class="pre">runNifler</span></tt> <tt class="docutils literal"><span class="pre">setLastModificationTime</span></tt> coordination (above) folds away with a full frontend/nifler split; dead <tt class="docutils literal"><span class="pre">when</span></tt> imports could also be pruned during the <tt class="docutils literal"><span class="pre">.s.deps</span></tt> re-derivation.</li>
|
||
</ul>
|
||
<p>Validation bar (held on every change): <tt class="docutils literal"><span class="pre">koch bootic</span></tt> must reach its byte-identical fixed point, and binary size must not regress (DCE parity), across the external-package CI set.</p>
|
||
|
||
<h1 id="further-possible-improvements">Further possible improvements</h1><p>A warm-edit profiling pass (2026-07-02, self-compiling the compiler into a dedicated <tt class="docutils literal"><span class="pre">--nimcache</span></tt>, editing one private proc body — <tt class="docutils literal"><span class="pre">internalErrorImpl</span></tt> — in the hub module <tt class="docutils literal"><span class="pre">compiler/msgs.nim</span></tt>) surfaced where a <strong>hub-module</strong> warm rebuild actually spends its time. The result refines the "a body-only edit re-fires one module" claim above: that holds for the <em>backend</em>, but the <em>frontend</em> can still cascade.</p>
|
||
<p>Measured: no-op <tt class="docutils literal"><span class="pre">0.05s</span></tt>; hub body edit <tt class="docutils literal"><span class="pre">~15s</span></tt>, split <strong>~13s frontend / ~1.6s backend</strong>. Editing a body in a leaf (few importers) is fast; editing a body in a widely-imported module is not, and the cost is almost entirely frontend re-sem.</p>
|
||
<ul class="simple"><li><p><strong>Frontend over-invalidation (the dominant hub-edit cost).</strong> Editing <em>any</em> body in a module — even a private routine that is only ever <em>called</em> — flips that module's whole-module <strong>impl cookie</strong> (<tt class="docutils literal"><span class="pre">writeImplCookie</span></tt> hashes the entire serialized module). Every module carrying a <strong>NeedsImpl</strong> edge on it then re-sems, even though the symbol it actually consumed is unchanged (e.g. a dependent that expanded the <tt class="docutils literal"><span class="pre">internalError</span></tt> <em>template</em> needs the template body, which is untouched; it does <strong>not</strong> need <tt class="docutils literal"><span class="pre">internalErrorImpl</span></tt>'s body). In the msgs edit this re-fires <strong>57</strong> <tt class="docutils literal"><span class="pre">nim m</span></tt> processes. A <tt class="docutils literal"><span class="pre">.s.bif</span></tt> mtime diff <em>hides</em> this — <tt class="docutils literal"><span class="pre">.s.bif</span></tt> is content-stable, so a re-semmed-but-identical module keeps its timestamp; count actual <tt class="docutils literal"><span class="pre">nim m</span></tt> PIDs to see the fan-out.</p>
|
||
<p>The precise fix is <strong>per-symbol NeedsImpl gating</strong>: record which <em>symbols'</em> bodies a dependent consumed (the recording site <tt class="docutils literal"><span class="pre">modulegraphs.recordIcImplDep</span></tt> already receives the <tt class="docutils literal"><span class="pre">PSym</span></tt>; it currently coarsens to <tt class="docutils literal"><span class="pre">module(s.itemId)</span></tt>) and gate the dependent on only those. The obstacle is that <tt class="docutils literal"><span class="pre">nifmake</span></tt> gates on file mtimes, so per-symbol granularity needs either many cookie files or a bucketing scheme, and "which bodies are compile-time-consumable" is entangled with <tt class="docutils literal"><span class="pre">getImpl</span></tt> and the CT call graph (a macro that runs a private helper at CT <em>does</em> consume its body). A conservative narrowing — keep template/generic/macro/<tt class="docutils literal"><span class="pre">sfCompileTime</span></tt> bodies (plus <tt class="docutils literal"><span class="pre">getImpl</span></tt> targets) in the impl cookie but drop ordinary runtime routine bodies — captures the common "edit a private implementation proc" case, at the cost of proving the exclusion is complete.</p>
|
||
</li>
|
||
<li><strong>Serial re-sem chains.</strong> The 57 re-sems above run essentially <strong>one at a time</strong> despite <tt class="docutils literal"><span class="pre">--parallel</span></tt>, because the core modules they belong to form a deep import <em>chain</em> and <tt class="docutils literal"><span class="pre">nifmake</span></tt>'s depth-barriered scheduler runs one depth level at a time (≈1 node per level). This is independent of the invalidation problem: even perfect per-symbol precision leaves a serial tail whenever the re-sem set is a chain. Mitigations live in the scheduler (content-stability already stops the cascade at one level, but does not flatten the chain).</li>
|
||
<li><strong>Emit stage need not load the module graph (done).</strong> <tt class="docutils literal"><span class="pre">generateEmitStage</span></tt> used to <tt class="docutils literal"><span class="pre">loadDepClosure</span></tt>/<tt class="docutils literal"><span class="pre">loadBackendModules</span></tt> — materializing a module's whole transitive import closure as <tt class="docutils literal"><span class="pre">BModule</span></tt>s — solely to reach <tt class="docutils literal"><span class="pre">getCFile(bmod)</span></tt> for the output path. <tt class="docutils literal"><span class="pre">renderCFromArtifact</span></tt> is pure text filtering over the <tt class="docutils literal"><span class="pre">.c.nif</span></tt> plus the merge decision; it needs none of that. Deriving the <tt class="docutils literal"><span class="pre">.c</span></tt> path directly from the suffix (the same pure computation <tt class="docutils literal"><span class="pre">deps.backendCFile</span></tt> uses to <em>declare</em> the stage's output) lets an <tt class="docutils literal"><span class="pre">emit</span></tt> process load nothing. Under the fire-all-every-edit <tt class="docutils literal"><span class="pre">emit</span></tt> barrier (see below) this halved backend CPU (user-time <tt class="docutils literal"><span class="pre">51s → 24s</span></tt> on the msgs edit); wall-clock barely moved because the frontend dominates, but the reduced CPU/RAM contention matters when an editor is running alongside. <tt class="docutils literal"><span class="pre">koch ic</span></tt> stays byte-identical.</li>
|
||
<li><strong>Do NOT make the merge decision content-stable.</strong> A tempting frontend to the above: <tt class="docutils literal"><span class="pre">emit</span></tt> re-fires for <em>every</em> live module whenever <tt class="docutils literal"><span class="pre">merge</span></tt> rewrites the decision file's mtime (deliberate — a decision change must re-render every <tt class="docutils literal"><span class="pre">.c</span></tt> consistently). Writing the decision <tt class="docutils literal"><span class="pre">OnlyIfChanged</span></tt> (with a stamp output so the <tt class="docutils literal"><span class="pre">merge</span></tt> rule is not perpetually stale) makes a warm no-op instant, but a real edit then fires <tt class="docutils literal"><span class="pre">emit</span></tt> only for the modules whose <tt class="docutils literal"><span class="pre">.c.nif</span></tt> changed — and that produces <strong>multiple-definition link errors</strong> even when the decision is byte-identical. Fire-all <tt class="docutils literal"><span class="pre">emit</span></tt> is a correctness invariant, not just insurance (see the comment at <tt class="docutils literal"><span class="pre">generateEmitStage</span></tt>): partial <tt class="docutils literal"><span class="pre">emit</span></tt> leaves inconsistent ownership across the <tt class="docutils literal"><span class="pre">.c</span></tt> set. This path was tried and reverted; do not retry.</li>
|
||
</ul>
|
||
|
||
<h1 id="code-logic-amp-debugging">Code, logic & debugging</h1><p>Core modules:</p>
|
||
<ul class="simple"><li><strong>`compiler/deps.nim`</strong> — graph construction, SCC grouping, discovery fixpoint, build-file generation; <tt class="docutils literal"><span class="pre">commandIc</span></tt>.</li>
|
||
<li><strong>`compiler/ast2nif.nim`</strong> — AST↔NIF, the cookie hashes (<tt class="docutils literal"><span class="pre">cookieSd</span></tt>, <tt class="docutils literal"><span class="pre">writeIfaceCookie</span></tt>, <tt class="docutils literal"><span class="pre">writeImplCookie</span></tt>, <tt class="docutils literal"><span class="pre">writeEdgesFile</span></tt>, <tt class="docutils literal"><span class="pre">writeSemDeps</span></tt>).</li>
|
||
<li><strong>`compiler/nifbackend.nim`</strong> — the per-module backend stages (<tt class="docutils literal"><span class="pre">generateCgStage</span></tt>, <tt class="docutils literal"><span class="pre">generateMergeStage</span></tt>, <tt class="docutils literal"><span class="pre">generateEmitStage</span></tt>, <tt class="docutils literal"><span class="pre">generateLinkStage</span></tt>).</li>
|
||
<li><strong>`compiler/cnif.nim`</strong> — <tt class="docutils literal"><span class="pre">.c.nif</span></tt> artifact read/write, <tt class="docutils literal"><span class="pre">computeMergeDecision</span></tt>, <tt class="docutils literal"><span class="pre">renderCFromArtifact</span></tt>.</li>
|
||
<li><strong>`compiler/icconfig.nim`</strong> — precompiled config.</li>
|
||
<li><strong>`compiler/pipelines.nim`</strong> / <strong>`modulegraphs.nim`</strong> — pipeline integration and the graph state (<tt class="docutils literal"><span class="pre">importDeps</span></tt>, <tt class="docutils literal"><span class="pre">icImplDeps</span></tt>, <tt class="docutils literal"><span class="pre">icCnifFiles</span></tt>, <tt class="docutils literal"><span class="pre">instDisambs</span></tt>, …).</li>
|
||
</ul>
|
||
<p>Manual workflow:</p>
|
||
<ul class="simple"><li>Frontend a module: <tt class="docutils literal"><span class="pre">nim m --nimcache:nifcache path/to/mod.nim</span></tt> (writes <tt class="docutils literal"><span class="pre">.nif</span></tt> + cookies + <tt class="docutils literal"><span class="pre">.s.deps</span></tt>).</li>
|
||
<li>Backend is stage-based (a bare <tt class="docutils literal"><span class="pre">nim nifc main.nim</span></tt> errors — there is no whole-program fallback). The exact per-stage commands <tt class="docutils literal"><span class="pre">nifmake</span></tt> runs are in the <tt class="docutils literal"><span class="pre">*.backend.build.nif</span></tt> build file; rerun one directly against an existing cache, e.g. <tt class="docutils literal"><span class="pre">nim nifc --nimcache:nifcache --icBackendStage:cg --icBackendModule:<suffix> main.nim</span></tt> to regenerate one module's <tt class="docutils literal"><span class="pre">.c.nif</span></tt>, then <tt class="docutils literal"><span class="pre">--icBackendStage:merge</span></tt> / <tt class="docutils literal"><span class="pre">:emit</span></tt> / <tt class="docutils literal"><span class="pre">:link</span></tt>.</li>
|
||
<li>NIF and <tt class="docutils literal"><span class="pre">.c.nif</span></tt> files are text — open/grep them directly; <tt class="docutils literal"><span class="pre">diff</span></tt> two successive <tt class="docutils literal"><span class="pre">.nif</span></tt> to see why a module rebuilt.</li>
|
||
<li>Force a re-sem: delete the module's <tt class="docutils literal"><span class="pre">.nif</span></tt> and rerun <tt class="docutils literal"><span class="pre">nim m</span></tt>.</li>
|
||
<li>A stale-cache crash after editing the serialization layout means bumping <tt class="docutils literal"><span class="pre">icFormatVersion</span></tt> (<tt class="docutils literal"><span class="pre">compiler/options.nim</span></tt>).</li>
|
||
</ul>
|
||
|
||
<h1 id="see-also">See also</h1><ul class="simple"><li>NIF format spec: <a class="reference external" href="../nifspec/doc/nif-spec.md">nifspec/doc/nif-spec.md</a></li>
|
||
<li>NIFC (C-like target) spec: dist/nimony/doc/nifc-spec.md</li>
|
||
</ul>
|
||
|
||
|
||
|
||
<div class="twelve-columns footer">
|
||
<span class="nim-sprite"></span>
|
||
<br>
|
||
<small style="color: var(--hint);">Made with Nim. Generated: 2026-07-16 23:11:19 UTC</small>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<script defer data-domain="nim-lang.org" src="https://plausible.io/js/plausible.js"></script>
|
||
|
||
</body>
|
||
</html>
|