mirror of
https://github.com/nim-lang/Nim.git
synced 2026-07-31 20:49:06 +00:00
159 lines
45 KiB
HTML
159 lines
45 KiB
HTML
<?xml version="1.0" encoding="utf-8" ?>
|
||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||
<!-- This file is generated by Nim. -->
|
||
<html xmlns="https://www.w3.org/1999/xhtml" xml:lang="en" lang="en" data-theme="auto">
|
||
<head>
|
||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>Incremental Compilation (IC)</title>
|
||
|
||
<!-- Google fonts -->
|
||
<link href='https://fonts.googleapis.com/css?family=Lato:400,600,900' rel='stylesheet' type='text/css'/>
|
||
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro:400,500,600' rel='stylesheet' type='text/css'/>
|
||
|
||
<!-- Favicon -->
|
||
<link rel="shortcut icon" href="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AAAAAAUAAAAF////AP///wD///8A////AP///wD///8A////AP///wD///8A////AAAAAAIAAABbAAAAlQAAAKIAAACbAAAAmwAAAKIAAACVAAAAWwAAAAL///8A////AP///wD///8A////AAAAABQAAADAAAAAYwAAAA3///8A////AP///wD///8AAAAADQAAAGMAAADAAAAAFP///wD///8A////AP///wAAAACdAAAAOv///wD///8A////AP///wD///8A////AP///wD///8AAAAAOgAAAJ3///8A////AP///wAAAAAnAAAAcP///wAAAAAoAAAASv///wD///8A////AP///wAAAABKAAAAKP///wAAAABwAAAAJ////wD///8AAAAAgQAAABwAAACIAAAAkAAAAJMAAACtAAAAFQAAABUAAACtAAAAkwAAAJAAAACIAAAAHAAAAIH///8A////AAAAAKQAAACrAAAAaP///wD///8AAAAARQAAANIAAADSAAAARf///wD///8AAAAAaAAAAKsAAACk////AAAAADMAAACcAAAAnQAAABj///8A////AP///wAAAAAYAAAAGP///wD///8A////AAAAABgAAACdAAAAnAAAADMAAAB1AAAAwwAAAP8AAADpAAAAsQAAAE4AAAAb////AP///wAAAAAbAAAATgAAALEAAADpAAAA/wAAAMMAAAB1AAAAtwAAAOkAAAD/AAAA/wAAAP8AAADvAAAA3gAAAN4AAADeAAAA3gAAAO8AAAD/AAAA/wAAAP8AAADpAAAAtwAAAGUAAAA/AAAA3wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAADfAAAAPwAAAGX///8A////AAAAAEgAAADtAAAAvwAAAL0AAADGAAAA7wAAAO8AAADGAAAAvQAAAL8AAADtAAAASP///wD///8A////AP///wD///8AAAAAO////wD///8A////AAAAAIcAAACH////AP///wD///8AAAAAO////wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A//8AAP//AAD4HwAA7/cAAN/7AAD//wAAoYUAAJ55AACf+QAAh+EAAAAAAADAAwAA4AcAAP5/AAD//wAA//8AAA=="/>
|
||
<link rel="icon" type="image/png" sizes="32x32" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAA3XAAAN1wFCKJt4AAAAB3RJTUUH4QQQEwksSS9ZWwAAAk1JREFUWMPtll2ITVEUx39nn/O7Y5qR8f05wtCUUr6ZIS++8pEnkZInPImneaCQ5METNdOkeFBKUhMPRIkHKfEuUZSUlGlKPN2TrgfncpvmnntnmlEyq1Z7t89/rf9a6+y99oZxGZf/XeIq61EdtgKXgdXA0xrYAvBjOIF1AI9zvjcC74BSpndrJPkBWDScTF8Aa4E3wDlgHbASaANmVqlcCnwHvgDvgVfAJ+AikAAvgfVZwLnSVZHZaOuKoQi3ZOMi4NkYkpe1p4J7A8BpYAD49hfIy/oqG0+hLomiKP2L5L+1ubn5115S+3OAn4EnwBlgMzCjyt6ZAnQCJ4A7wOs88iRJHvw50HoujuPBoCKwHWiosy8MdfZnAdcHk8dxXFJ3VQbQlCTJvRBCGdRbD4M6uc5glpY3eAihpN5S5w12diSEcCCEcKUO4ljdr15T76ur1FDDLIQQ3qv71EdDOe3Kxj3leRXyk+pxdWnFWod6Wt2bY3de3aSuUHcPBVimHs7mK9WrmeOF6lR1o9qnzskh2ar2qm1qizpfXaPeVGdlmGN5pb09qMxz1Xb1kLqgzn1RyH7JUXW52lr5e/Kqi9qpto7V1atuUzfnARrV7jEib1T76gG2qxdGmXyiekkt1GswPTtek0aBfJp6YySGBfWg2tPQ0FAYgf1stUfdmdcjarbYJEniKIq6gY/Aw+zWHAC+p2labGpqiorFYgGYCEzN7oQdQClN07O1/EfDyGgC0ALMBdYAi4FyK+4H3gLPsxfR1zRNi+NP7nH5J+QntnXe5B5mpfQAAAAASUVORK5CYII=">
|
||
|
||
<!-- CSS -->
|
||
<link rel="stylesheet" type="text/css" href="nimdoc.out.css?v=2.3.1">
|
||
|
||
<!-- JS -->
|
||
<script type="text/javascript" src="dochack.js?v=2.3.1"></script>
|
||
</head>
|
||
<body>
|
||
<div class="document" id="documentId">
|
||
<input type="checkbox" id="nav-toggle" hidden>
|
||
<label for="nav-toggle" id="nav-burger">☰</label>
|
||
<label for="nav-toggle" id="nav-overlay"></label>
|
||
<div class="container">
|
||
<h1 class="title">Incremental Compilation (IC)</h1>
|
||
<p>The <tt class="docutils literal"><span class="pre">nim ic</span></tt> command provides incremental compilation for Nim projects. It decomposes compilation into per-module steps whose results are cached as NIF files, and uses the external <tt class="docutils literal"><span class="pre">nifmake</span></tt> build tool to re-run only the steps whose inputs changed.</p>
|
||
<p>This document describes <strong>how `nim ic` works today</strong>, including the edge cases that shaped the current design. The per-module backend rewrite that earlier editions of this document listed as a <em>Plan</em> has <strong>landed</strong>: the whole-program, reuse/redirect/def-retention backend is gone and codegen is now a set of <tt class="docutils literal"><span class="pre">nifmake</span></tt>-driven per-module rules (see <em>The backend</em>).</p>
|
||
|
||
<h1 id="overview">Overview</h1><p>The pipeline has two halves driven by one process (<tt class="docutils literal"><span class="pre">nim ic</span></tt>, <tt class="docutils literal"><span class="pre">commandIc</span></tt> in <tt class="docutils literal"><span class="pre">compiler/deps.nim</span></tt>) that constructs a dependency graph, writes a build file, and hands it to <tt class="docutils literal"><span class="pre">nifmake</span></tt>:</p>
|
||
<ol class="simple"><li><strong>Frontend</strong> — per module:<ul class="simple"><li><tt class="docutils literal"><span class="pre">nifler parse --deps</span></tt> turns <tt class="docutils literal"><span class="pre">.nim</span></tt> source into a parsed NIF (<tt class="docutils literal"><span class="pre">.p.nif</span></tt>) plus a static dependency list (<tt class="docutils literal"><span class="pre">.deps.nif</span></tt>).</li>
|
||
<li><tt class="docutils literal"><span class="pre">nim m</span></tt> (the <em>semantic</em> step, <tt class="docutils literal"><span class="pre">cmdM</span></tt>) reads the parsed NIF + the precompiled NIFs of the module's imports, type-checks, and writes the <strong>semmed NIF</strong> (<tt class="docutils literal"><span class="pre">.nif</span></tt>) plus invalidation sidecars (see <em>Cookies</em>).</li>
|
||
</ul>
|
||
</li>
|
||
<li><strong>Backend</strong> — <tt class="docutils literal"><span class="pre">nim nifc</span></tt> (<tt class="docutils literal"><span class="pre">cmdNifC</span></tt>, <tt class="docutils literal"><span class="pre">compiler/nifbackend.nim</span></tt>) reads the semmed NIFs, generates C, compiles and links.</li>
|
||
</ol>
|
||
<p><tt class="docutils literal"><span class="pre">nifmake</span></tt> orders the steps by their input/output files: every <tt class="docutils literal"><span class="pre">nim m</span></tt> runs before the <tt class="docutils literal"><span class="pre">nim nifc</span></tt> step that consumes its NIF, and a step re-fires only when one of its inputs is newer than its outputs. The driver invokes <tt class="docutils literal"><span class="pre">nifmake run --parallel</span></tt> by default, so independent steps at the same DAG depth fan out across cores; pass <tt class="docutils literal"><span class="pre">-d:icNoParallel</span></tt> to serialize (readable child output when debugging a build).</p>
|
||
|
||
<h1 id="artifacts-the-nif-zoo">Artifacts (the NIF zoo)</h1>
|
||
<h2 id="semantic-bif-from-regular-builds">Semantic BIF from regular builds</h2><p><tt class="docutils literal"><span class="pre">--genBif:on</span></tt> makes a regular compiler invocation write each semantically checked module as <tt class="docutils literal"><span class="pre"><suffix>.s.bif</span></tt> under the build's nimcache directory. This reuses the semantic artifact format used by IC without enabling incremental compilation or changing how the program is generated and linked. Tools such as language servers, debuggers, and binding generators can request these artifacts when they need resolved symbols and types from an ordinary build.</p>
|
||
<p>Per module <tt class="docutils literal"><span class="pre"><suffix></span></tt> (a content hash of the path; see <em>NIF symbols</em> below), under the nimcache directory:</p>
|
||
<table border="1" class="docutils"><tr><th>File</th><th>Producer</th><th>Purpose</th></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.p.nif</span></tt></td><td>nifler</td><td>parsed AST (syntactic)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.deps.nif</span></tt></td><td>nifler</td><td><strong>static</strong> import list (syntactic <tt class="docutils literal"><span class="pre">import</span></tt>s)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.s.deps.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>real</strong> post-sem imports (incl. macro-generated); see <em>Discovery</em></td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td>semmed module (symbols resolved, typed)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.iface.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>iface cookie</strong>: hash of the importer-visible surface</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.impl.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>impl cookie</strong>: hash of the entire content (bodies included)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.edges.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim m</span></tt></td><td><strong>NeedsImpl edges</strong>: modules whose bodies this sem consumed</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre"><s>.c.nif</span></tt></td><td><tt class="docutils literal"><span class="pre">nim nifc</span></tt></td><td>the C text as a NIF, with def/ref markers for DCE & dedup</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre">ic_config.cfg.nif</span></tt></td><td>driver</td><td>precompiled config replayed by every child (<tt class="docutils literal"><span class="pre">icconfig.nim</span></tt>)</td></tr>
|
||
<tr><td><tt class="docutils literal"><span class="pre">ic.version</span></tt></td><td>driver</td><td>format stamp; a mismatch wipes the cache (<tt class="docutils literal"><span class="pre">icFormatVersion</span></tt>)</td></tr>
|
||
</table>
|
||
<h1 id="nif-symbols-and-ownership">NIF symbols and ownership</h1><p>(See <tt class="docutils literal"><span class="pre">../nifspec/doc/nif-spec.md</span></tt>.) A global symbol is <tt class="docutils literal"><span class="pre"><ident>.<disamb>.<moduleSuffix></span></tt>. For a <strong>generic instantiation</strong> the <tt class="docutils literal"><span class="pre"><disamb></span></tt> is not a counter but a <em>content hash</em> — <tt class="docutils literal"><span class="pre">setInstanceDisamb</span></tt> (<tt class="docutils literal"><span class="pre">modulegraphs.nim</span></tt>) MD5s the generic's identity plus the <tt class="docutils literal"><span class="pre">typeKey</span></tt> of every concrete type argument, masks it to 30 bits and tags it with <tt class="docutils literal"><span class="pre">InstanceDisambBit</span></tt>. So the only part of the name that varies between two modules making the <strong>same</strong> instantiation (<tt class="docutils literal"><span class="pre">seq[Foo]</span></tt>) is the <tt class="docutils literal"><span class="pre"><moduleSuffix></span></tt>. Two consequences drive the backend:</p>
|
||
<ul class="simple"><li><strong>Instance names are content-addressed</strong>: the same instantiation produced in different modules yields the <em>same</em> <tt class="docutils literal"><span class="pre"><ident>.<disamb></span></tt>, so a deterministic dedup is possible by the <em>module-suffix-stripped</em> name. The cross-TU C name (<tt class="docutils literal"><span class="pre">ccgtypes.sharedInstanceCName</span></tt>) and the <strong>merge</strong> stage's live-set/owner decision (<tt class="docutils literal"><span class="pre">nifbackend.computeMergeDecision</span></tt>) both key on this stripped form.</li>
|
||
<li><strong>The suffix names a mint-site owner.</strong> The <tt class="docutils literal"><span class="pre"><moduleSuffix></span></tt> is the module <em>that minted the instance</em> (the instantiation site), so the same instance has a different full name in each module that makes it. Because every <tt class="docutils literal"><span class="pre">cg</span></tt> process emits the instances it demands (<em>emit-everywhere</em>), the same definition can be produced by several translation units; the <strong>merge</strong> stage then deterministically picks the single artifact allowed to embed each body (smallest claimant), which is the cross-process replacement for the old in-process single-writer machinery.</li>
|
||
</ul>
|
||
|
||
<h1 id="the-drivercolon-graph-construction-commandic">The driver: graph construction (<tt class="docutils literal"><span class="pre">commandIc</span></tt>)</h1><ol class="simple"><li>Stamp/wipe the cache by <tt class="docutils literal"><span class="pre">icFormatVersion</span></tt>.</li>
|
||
<li>Seed the graph with the root module and <strong>`system.nim`</strong>. <tt class="docutils literal"><span class="pre">system</span></tt>'s entire import closure is folded into one node (one <tt class="docutils literal"><span class="pre">nim m</span></tt> invocation) — see <em>single-writer</em> below.</li>
|
||
<li><tt class="docutils literal"><span class="pre">traverseDeps</span></tt> runs <tt class="docutils literal"><span class="pre">nifler</span></tt> per module and reads <tt class="docutils literal"><span class="pre">.deps.nif</span></tt> to add import edges.</li>
|
||
<li><strong>SCC grouping</strong>: strongly-connected import cycles are collapsed (Tarjan). A singleton compiles as <tt class="docutils literal"><span class="pre">nim m <mod></span></tt>; a cycle compiles as one <tt class="docutils literal"><span class="pre">nim m <rep> --icGroup:<member>…</span></tt> that builds every member <em>from source</em> in one process (resolving the recursion in memory) and writes each member's NIF. Only edges <em>leaving</em> the component become build-graph inputs.</li>
|
||
<li><strong>Discovery fixpoint</strong>: write the build file, run <tt class="docutils literal"><span class="pre">nifmake</span></tt>; if it fails, re-derive the graph from every module's <tt class="docutils literal"><span class="pre">.s.deps.nif</span></tt> (adding nodes/edges for imports the static scanner missed), and retry. See <em>Discovery</em>.</li>
|
||
<li>The backend step (<tt class="docutils literal"><span class="pre">nim nifc</span></tt>) depends on every module's semmed NIF, so <tt class="docutils literal"><span class="pre">nifmake</span></tt> runs it last.</li>
|
||
</ol>
|
||
|
||
<h1 id="invalidationcolon-the-cookie-system">Invalidation: the cookie system</h1><p>A dependent must re-sem only when a dependency's relevant surface changed. Two hashes per module (<tt class="docutils literal"><span class="pre">ast2nif.nim</span></tt>):</p>
|
||
<ul class="simple"><li><strong>iface cookie</strong> (<tt class="docutils literal"><span class="pre">.iface.nif</span></tt>): hashes only the <em>importer-visible</em> surface — exported declarations' <strong>signatures</strong> (for <em>all</em> routine kinds: plain procs, templates, macros, generics, <tt class="docutils literal"><span class="pre">inline</span></tt> procs alike), full content for consts/types, plus import/export/replay/hook records. Routine <strong>bodies are excluded.</strong> It also chains in the iface cookies of its own dependencies, so a surface change anywhere in the import closure propagates. A <tt class="docutils literal"><span class="pre">nim m</span></tt> rule for a module depends on its dependencies' iface cookies, so a body-only edit moves no iface cookie and stops the re-sem cascade.</li>
|
||
<li><strong>impl cookie</strong> (<tt class="docutils literal"><span class="pre">.impl.nif</span></tt>): hashes the <em>entire</em> serialized content (private defs and bodies included), with the module's own iface mixed in.</li>
|
||
</ul>
|
||
<p><strong>NeedsImpl edges</strong> (<tt class="docutils literal"><span class="pre">.edges.nif</span></tt>): if a module <em>consumed another module's body</em> during sem — a macro expansion, a generic instantiation, a <tt class="docutils literal"><span class="pre">getImpl</span></tt>, or a compile-time call run in the VM — it records a strong edge. The dependent is then gated on that dependency's <strong>impl</strong> cookie instead of its iface cookie, so e.g. <tt class="docutils literal"><span class="pre">const x = dep.foo()</span></tt> re-sems when <tt class="docutils literal"><span class="pre">foo</span></tt>'s body changes. Recording sites: <tt class="docutils literal"><span class="pre">semExprs.semTemplateExpr</span></tt> (templates), <tt class="docutils literal"><span class="pre">seminst.generateInstance</span></tt> (generics), <tt class="docutils literal"><span class="pre">vmgen.genProc</span></tt> (VM/macros/CT procs), <tt class="docutils literal"><span class="pre">vm.opcGetImpl</span></tt> (<tt class="docutils literal"><span class="pre">getImpl</span></tt>). Inline iterators and <tt class="docutils literal"><span class="pre">inline</span></tt> procs are <em>not</em> tracked — they are inlined at codegen, where the backend's NIF-mtime invalidation re-codegens their users.</p>
|
||
|
||
<h1 id="discovery-of-macrominusgenerated-imports">Discovery of macro-generated imports</h1><p>The static scanner only sees syntactic <tt class="docutils literal"><span class="pre">import</span></tt>s. A macro can synthesize one (chronicles does <tt class="docutils literal"><span class="pre">parseStmt("import chronicles/textlines")</span></tt> driven by the <tt class="docutils literal"><span class="pre">chronicles_sinks</span></tt> define). Such an import is invisible until sem runs the macro. Each <tt class="docutils literal"><span class="pre">nim m</span></tt> records the imports it <em>actually</em> resolved (via the <tt class="docutils literal"><span class="pre">semdata.addImportFileDep</span></tt> hook → <tt class="docutils literal"><span class="pre">graph.importDeps</span></tt> → <tt class="docutils literal"><span class="pre">ast2nif.writeSemDeps</span></tt>) into <tt class="docutils literal"><span class="pre"><s>.s.deps.nif</span></tt>; a child that fails on a not-yet-built import flushes it before erroring. The driver re-derives the graph from those sidecars — adding the missing node + the importer→import edge — and reruns to a fixpoint. (This replaced an earlier <tt class="docutils literal"><span class="pre">icmissing.txt</span></tt> side channel.)</p>
|
||
|
||
<h1 id="the-backendcolon-perminusmodule-nifc-stages">The backend: per-module <tt class="docutils literal"><span class="pre">nifc</span></tt> stages</h1><p>Codegen is no longer one whole-program process. <tt class="docutils literal"><span class="pre">nim nifc</span></tt> (<tt class="docutils literal"><span class="pre">cmdNifC</span></tt>, <tt class="docutils literal"><span class="pre">compiler/nifbackend.nim</span></tt>) is invoked once per <strong>stage</strong> via <tt class="docutils literal"><span class="pre">--icBackendStage:<stage></span></tt>; <tt class="docutils literal"><span class="pre">commandIc</span></tt> emits these as ordinary <tt class="docutils literal"><span class="pre">nifmake</span></tt> rules so "which TUs rebuild" is just "which rules <tt class="docutils literal"><span class="pre">nifmake</span></tt> re-fires from input mtimes" — exactly as the frontend already works. There are four stages:</p>
|
||
<ol class="simple"><li><strong>`cg`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:cg --icBackendModule:<suffix></span></tt>) — generate C for the <em>single</em> named module and write only its <tt class="docutils literal"><span class="pre"><s>.c.nif</span></tt> artifact. A non-main target loads only its own import closure (<tt class="docutils literal"><span class="pre">loadDepClosure</span></tt>), so the whole program is <strong>not</strong> pulled into every parallel <tt class="docutils literal"><span class="pre">cg</span></tt> process. Codegen is still demand-driven and <strong>emit-everywhere</strong>: a <tt class="docutils literal"><span class="pre">cg</span></tt> process emits every entity it demands (generic instances, hooks, RTTI), referencing nothing <tt class="docutils literal"><span class="pre">extern</span></tt>-only. There is no whole-program DCE here — a liveness pass over all ~260 NIFs would cost ~900 MB for a result the merge stage recomputes anyway. The <strong>main</strong> module's <tt class="docutils literal"><span class="pre">cg</span></tt> is special: it loads everything (<tt class="docutils literal"><span class="pre">loadBackendModules</span></tt>), emits the whole-program method dispatchers and <tt class="docutils literal"><span class="pre">NimMain</span></tt>, and registers every other module's init/datInit from the <tt class="docutils literal"><span class="pre">.c.nif</span></tt> meta heads — so it runs <em>last</em>, after every other <tt class="docutils literal"><span class="pre">.c.nif</span></tt> exists. Every <tt class="docutils literal"><span class="pre">cg</span></tt> rule always leaves a <tt class="docutils literal"><span class="pre">.c.nif</span></tt> (empty if the module owns no code) so its nifmake output exists and the rule settles.</li>
|
||
<li><strong>`merge`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:merge</span></tt>) — a pure artifact pass, <em>no module graph loaded</em>. Reads every <tt class="docutils literal"><span class="pre">.c.nif</span></tt>, computes the one program-wide live set and, for each unique definition that several <tt class="docutils literal"><span class="pre">cg</span></tt> processes emitted, the single artifact allowed to embed its body; writes that to a merge-decision file (<tt class="docutils literal"><span class="pre">computeMergeDecision</span></tt> / <tt class="docutils literal"><span class="pre">writeMergeDecision</span></tt>). This is the cross-process replacement for the old in-process first-claimant + DCE coordination.</li>
|
||
<li><strong>`emit`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:emit --icBackendModule:<suffix></span></tt>) — render the target module's final <tt class="docutils literal"><span class="pre">.c</span></tt> from its <tt class="docutils literal"><span class="pre">.c.nif</span></tt> and the merge decision (<tt class="docutils literal"><span class="pre">renderCFromArtifact</span></tt>, dropping globally-dead and non-owned bodies). No codegen runs; the target is loaded only so <tt class="docutils literal"><span class="pre">getCFile</span></tt> yields the path <tt class="docutils literal"><span class="pre">cg</span></tt> wrote.</li>
|
||
<li><strong>`link`</strong> (<tt class="docutils literal"><span class="pre">--icBackendStage:link</span></tt>) — register every module's emitted <tt class="docutils literal"><span class="pre">.c</span></tt> and run <tt class="docutils literal"><span class="pre">extccomp.callCCompiler</span></tt> once (it parallelizes per-file cc and skips up-to-date objects). Per-module C compile/link directives (<tt class="docutils literal"><span class="pre">{.passL.}</span></tt> etc.) are re-collected here via <tt class="docutils literal"><span class="pre">replayBackendActions</span></tt>, since the <tt class="docutils literal"><span class="pre">cg</span></tt> processes that originally saw them are separate processes (without this, e.g. <tt class="docutils literal"><span class="pre">math</span></tt>'s <tt class="docutils literal"><span class="pre">-lm</span></tt> would be lost → undefined <tt class="docutils literal"><span class="pre">floor</span></tt>/<tt class="docutils literal"><span class="pre">pow</span></tt> at link).</li>
|
||
</ol>
|
||
<p>Because each stage is a <tt class="docutils literal"><span class="pre">nifmake</span></tt> rule keyed on file mtimes, a body-only edit to one module re-fires that module's <tt class="docutils literal"><span class="pre">cg</span></tt>+<tt class="docutils literal"><span class="pre">emit</span></tt> (and the <tt class="docutils literal"><span class="pre">merge</span></tt>/<tt class="docutils literal"><span class="pre">link</span></tt>), not the whole program — and an unchanged module's <tt class="docutils literal"><span class="pre">cg</span></tt> does not run at all.</p>
|
||
|
||
<h1 id="edge-cases-and-why-the-machinery-exists">Edge cases (and why the machinery exists)</h1><ul class="simple"><li><strong>Single-writer.</strong> Instance type-ids are minted in process-local order, so if two <tt class="docutils literal"><span class="pre">nim m</span></tt> processes both write a module's NIF (e.g. a stdlib module pulled into <tt class="docutils literal"><span class="pre">system</span></tt>'s from-source closure <em>and</em> given its own rule), the second overwrites with different ids and every module checked against the first carries dangling refs ("symbol has no offset"). Fixed by folding <tt class="docutils literal"><span class="pre">system</span></tt>'s closure into one SCC and by <strong>forwarding the project's defines</strong> to every child so their <tt class="docutils literal"><span class="pre">when</span></tt> bodies (hence import sets and NIF contents) match the scanner's.</li>
|
||
<li><strong>`when … else: import`.</strong> nifler emits <tt class="docutils literal"><span class="pre">else</span></tt>-branch imports unguarded, so a dead <tt class="docutils literal"><span class="pre">else: import</span></tt> would be scheduled. The compiler's own sources were rewritten to explicit negated <tt class="docutils literal"><span class="pre">when</span></tt>s; the vendored nifler later learned to negate prior conditions for the <tt class="docutils literal"><span class="pre">else</span></tt>.</li>
|
||
<li><strong>`nil` sons of loaded ASTs.</strong> NIF dot-tokens load as <tt class="docutils literal"><span class="pre">nil</span></tt> where from-source ASTs have <tt class="docutils literal"><span class="pre">nkEmpty</span></tt>; several passes gained <tt class="docutils literal"><span class="pre">nil</span></tt> guards.</li>
|
||
<li><strong>Sealed loaded types.</strong> Loaded types are <tt class="docutils literal"><span class="pre">Sealed</span></tt>; sem/transform mutate via <tt class="docutils literal"><span class="pre">unsealForTransform</span></tt>/<tt class="docutils literal"><span class="pre">exactReplica(idgen)</span></tt> (the latter mints a fresh <tt class="docutils literal"><span class="pre">uniqueId</span></tt> so serialized replicas don't collapse).</li>
|
||
<li><strong>Methods/RTTI ownership.</strong> RTTI and type-bound hooks are emit-everywhere at <tt class="docutils literal"><span class="pre">cg</span></tt> and deduplicated by the <tt class="docutils literal"><span class="pre">merge</span></tt> stage, like generic instances; the main module's <tt class="docutils literal"><span class="pre">cg</span></tt> owns the whole-program method dispatchers.</li>
|
||
<li><strong>Config cost.</strong> Each child re-parsing <tt class="docutils literal"><span class="pre">nim.cfg</span></tt> + re-running <tt class="docutils literal"><span class="pre">config.nims</span></tt> in the VM was ~80 ms; replaced by a precompiled <tt class="docutils literal"><span class="pre">ic_config.cfg.nif</span></tt> replayed in <tt class="docutils literal"><span class="pre">loadConfigs</span></tt> (<tt class="docutils literal"><span class="pre">compiler/icconfig.nim</span></tt>).</li>
|
||
<li><strong>`koch bootic`</strong> bootstraps the compiler through <tt class="docutils literal"><span class="pre">nim ic</span></tt> (a 3-iteration fixed-point check). It writes its binary to <tt class="docutils literal"><span class="pre">bin/nim_ic</span></tt> and never clobbers <tt class="docutils literal"><span class="pre">bin/nim</span></tt>.</li>
|
||
</ul>
|
||
|
||
<h2 id="resolved-by-the-rewrite">Resolved by the rewrite</h2><p>The whole-program backend's hand-rolled mini-<tt class="docutils literal"><span class="pre">nifmake</span></tt> — <tt class="docutils literal"><span class="pre">computeModuleReuse</span></tt>, <tt class="docutils literal"><span class="pre">enforceDefRetention</span></tt>, <tt class="docutils literal"><span class="pre">redirectToLiveModule</span></tt>, the cached-defs/claim bookkeeping and the standalone <tt class="docutils literal"><span class="pre">dce.nim</span></tt> — <strong>is gone</strong>. Reuse is now just per-rule <tt class="docutils literal"><span class="pre">nifmake</span></tt> mtime checks, and the single-writer decision is the <tt class="docutils literal"><span class="pre">merge</span></tt> stage. The old <strong>cross-mm / `--force` `var not init`</strong> hazard dissolved with it: every codegen rule's config (including <tt class="docutils literal"><span class="pre">--mm</span></tt>) is a declared <tt class="docutils literal"><span class="pre">nifmake</span></tt> input, so a stale-config TU is simply rebuilt rather than mixed in. <tt class="docutils literal"><span class="pre">koch bootic</span></tt> is green under both <tt class="docutils literal"><span class="pre">orc</span></tt> and <tt class="docutils literal"><span class="pre">--mm:refc</span></tt>.</p>
|
||
|
||
<h2 id="known-residual-hack">Known residual hack</h2><ul class="simple"><li><tt class="docutils literal"><span class="pre">deps.runNifler</span></tt> still uses <tt class="docutils literal"><span class="pre">setLastModificationTime</span></tt> to mark its scan up-to-date and deletes a stale parsed file to coordinate with the nifmake nifler rule — the driver duplicating nifmake's freshness logic. It is explicitly flagged in the source and folds away with a full frontend/nifler split.</li>
|
||
</ul>
|
||
|
||
<h1 id="status-and-performance">Status and performance</h1><p><tt class="docutils literal"><span class="pre">nim ic</span></tt> self-builds the compiler (<tt class="docutils literal"><span class="pre">koch bootic</span></tt>'s byte-identical fixed-point check) under both <tt class="docutils literal"><span class="pre">orc</span></tt> and <tt class="docutils literal"><span class="pre">--mm:refc</span></tt>, and passes the external-package CI set.</p>
|
||
<p>Cold full bootstrap on a 32-core box (<tt class="docutils literal"><span class="pre">-d:release</span></tt>, <strong>no edits</strong> — IC's worst case, since incremental reuse is not exercised):</p>
|
||
<p>| wall | notes |<br/><ul class="simple"><li>| ---- | ----- |</li>
|
||
</ul>
|
||
<br/><tt class="docutils literal"><span class="pre">koch boot</span></tt> (classic) | ~1m00s | reference |<br/><tt class="docutils literal"><span class="pre">koch bootic</span></tt> (<tt class="docutils literal"><span class="pre">nim ic</span></tt>) | ~1m39s | <strong>~1.66×</strong> |<br/></p><p>This is down from ~7.5× in the whole-program-backend era. IC does modestly more aggregate work (more processes, NIF re-parsing of imports per process), but on a many-core box that overhead is absorbed by the parallel <tt class="docutils literal"><span class="pre">nim m</span></tt>/<tt class="docutils literal"><span class="pre">nifc</span></tt> fan-out, and the C compile+link floor is shared with the classic backend. On few-core machines the cold gap is correspondingly wider — IC trades single-build latency for incremental latency.</p>
|
||
<p>The cold number is the <em>least</em> favourable comparison: it pays IC's full per-process overhead while using none of its incremental machinery. <strong>Warm rebuilds — the actual point of IC — recompile only the modules whose inputs changed</strong> (a body-only edit re-fires one module's <tt class="docutils literal"><span class="pre">cg</span></tt>+<tt class="docutils literal"><span class="pre">emit</span></tt>, not the program), so an edit-driven rebuild is a small fraction of either full build.</p>
|
||
<p>The strategic direction (decided 2026-06-13) is to make this NIF backend (<tt class="docutils literal"><span class="pre">cmdNifC</span></tt>) the <strong>default</strong> code generator. The per-module pipeline above is the realization of that direction; remaining work is <em>promotion + deletion</em> of the classic path, not new machinery.</p>
|
||
|
||
<h1 id="design-notes-and-open-decisions">Design notes and open decisions</h1><p>The per-module backend (above) mirrors Nimony's <tt class="docutils literal"><span class="pre">src/nimony/deps.nim</span></tt>: the backend stopped re-implementing <tt class="docutils literal"><span class="pre">nifmake</span></tt>; each stage is a build rule, so reuse is just mtime checks and the merge stage is the only cross-module coordination.</p>
|
||
<p>Settled vs. open:</p>
|
||
<ul class="simple"><li><strong>Ownership.</strong> Emittable entities (generic instances, type-bound hooks, RTTI, lifted procs) are emit-everywhere at <tt class="docutils literal"><span class="pre">cg</span></tt> time and deduplicated at <tt class="docutils literal"><span class="pre">merge</span></tt> time (smallest claimant owns each unique body). The earlier idea of a <em>static</em> per-suffix owner computed before codegen was not needed — content-addressed names make the merge decision deterministic. The precise owner <em>rule</em> (minting module vs. root-type's module) can still be tuned where it would force a downstream package to own stdlib code.</li>
|
||
<li><strong>Remaining cleanup.</strong> The <tt class="docutils literal"><span class="pre">runNifler</span></tt> <tt class="docutils literal"><span class="pre">setLastModificationTime</span></tt> coordination (above) folds away with a full frontend/nifler split; dead <tt class="docutils literal"><span class="pre">when</span></tt> imports could also be pruned during the <tt class="docutils literal"><span class="pre">.s.deps</span></tt> re-derivation.</li>
|
||
</ul>
|
||
<p>Validation bar (held on every change): <tt class="docutils literal"><span class="pre">koch bootic</span></tt> must reach its byte-identical fixed point, and binary size must not regress (DCE parity), across the external-package CI set.</p>
|
||
|
||
<h1 id="further-possible-improvements">Further possible improvements</h1><p>A warm-edit profiling pass (2026-07-02, self-compiling the compiler into a dedicated <tt class="docutils literal"><span class="pre">--nimcache</span></tt>, editing one private proc body — <tt class="docutils literal"><span class="pre">internalErrorImpl</span></tt> — in the hub module <tt class="docutils literal"><span class="pre">compiler/msgs.nim</span></tt>) surfaced where a <strong>hub-module</strong> warm rebuild actually spends its time. The result refines the "a body-only edit re-fires one module" claim above: that holds for the <em>backend</em>, but the <em>frontend</em> can still cascade.</p>
|
||
<p>Measured: no-op <tt class="docutils literal"><span class="pre">0.05s</span></tt>; hub body edit <tt class="docutils literal"><span class="pre">~15s</span></tt>, split <strong>~13s frontend / ~1.6s backend</strong>. Editing a body in a leaf (few importers) is fast; editing a body in a widely-imported module is not, and the cost is almost entirely frontend re-sem.</p>
|
||
<ul class="simple"><li><p><strong>Frontend over-invalidation (the dominant hub-edit cost).</strong> Editing <em>any</em> body in a module — even a private routine that is only ever <em>called</em> — flips that module's whole-module <strong>impl cookie</strong> (<tt class="docutils literal"><span class="pre">writeImplCookie</span></tt> hashes the entire serialized module). Every module carrying a <strong>NeedsImpl</strong> edge on it then re-sems, even though the symbol it actually consumed is unchanged (e.g. a dependent that expanded the <tt class="docutils literal"><span class="pre">internalError</span></tt> <em>template</em> needs the template body, which is untouched; it does <strong>not</strong> need <tt class="docutils literal"><span class="pre">internalErrorImpl</span></tt>'s body). In the msgs edit this re-fires <strong>57</strong> <tt class="docutils literal"><span class="pre">nim m</span></tt> processes. A <tt class="docutils literal"><span class="pre">.s.bif</span></tt> mtime diff <em>hides</em> this — <tt class="docutils literal"><span class="pre">.s.bif</span></tt> is content-stable, so a re-semmed-but-identical module keeps its timestamp; count actual <tt class="docutils literal"><span class="pre">nim m</span></tt> PIDs to see the fan-out.</p>
|
||
<p>The precise fix is <strong>per-symbol NeedsImpl gating</strong>: record which <em>symbols'</em> bodies a dependent consumed (the recording site <tt class="docutils literal"><span class="pre">modulegraphs.recordIcImplDep</span></tt> already receives the <tt class="docutils literal"><span class="pre">PSym</span></tt>; it currently coarsens to <tt class="docutils literal"><span class="pre">module(s.itemId)</span></tt>) and gate the dependent on only those. The obstacle is that <tt class="docutils literal"><span class="pre">nifmake</span></tt> gates on file mtimes, so per-symbol granularity needs either many cookie files or a bucketing scheme, and "which bodies are compile-time-consumable" is entangled with <tt class="docutils literal"><span class="pre">getImpl</span></tt> and the CT call graph (a macro that runs a private helper at CT <em>does</em> consume its body). A conservative narrowing — keep template/generic/macro/<tt class="docutils literal"><span class="pre">sfCompileTime</span></tt> bodies (plus <tt class="docutils literal"><span class="pre">getImpl</span></tt> targets) in the impl cookie but drop ordinary runtime routine bodies — captures the common "edit a private implementation proc" case, at the cost of proving the exclusion is complete.</p>
|
||
</li>
|
||
<li><strong>Serial re-sem chains.</strong> The 57 re-sems above run essentially <strong>one at a time</strong> despite <tt class="docutils literal"><span class="pre">--parallel</span></tt>, because the core modules they belong to form a deep import <em>chain</em> and <tt class="docutils literal"><span class="pre">nifmake</span></tt>'s depth-barriered scheduler runs one depth level at a time (≈1 node per level). This is independent of the invalidation problem: even perfect per-symbol precision leaves a serial tail whenever the re-sem set is a chain. Mitigations live in the scheduler (content-stability already stops the cascade at one level, but does not flatten the chain).</li>
|
||
<li><strong>Emit stage need not load the module graph (done).</strong> <tt class="docutils literal"><span class="pre">generateEmitStage</span></tt> used to <tt class="docutils literal"><span class="pre">loadDepClosure</span></tt>/<tt class="docutils literal"><span class="pre">loadBackendModules</span></tt> — materializing a module's whole transitive import closure as <tt class="docutils literal"><span class="pre">BModule</span></tt>s — solely to reach <tt class="docutils literal"><span class="pre">getCFile(bmod)</span></tt> for the output path. <tt class="docutils literal"><span class="pre">renderCFromArtifact</span></tt> is pure text filtering over the <tt class="docutils literal"><span class="pre">.c.nif</span></tt> plus the merge decision; it needs none of that. Deriving the <tt class="docutils literal"><span class="pre">.c</span></tt> path directly from the suffix (the same pure computation <tt class="docutils literal"><span class="pre">deps.backendCFile</span></tt> uses to <em>declare</em> the stage's output) lets an <tt class="docutils literal"><span class="pre">emit</span></tt> process load nothing. Under the fire-all-every-edit <tt class="docutils literal"><span class="pre">emit</span></tt> barrier (see below) this halved backend CPU (user-time <tt class="docutils literal"><span class="pre">51s → 24s</span></tt> on the msgs edit); wall-clock barely moved because the frontend dominates, but the reduced CPU/RAM contention matters when an editor is running alongside. <tt class="docutils literal"><span class="pre">koch ic</span></tt> stays byte-identical.</li>
|
||
<li><strong>Do NOT make the merge decision content-stable.</strong> A tempting frontend to the above: <tt class="docutils literal"><span class="pre">emit</span></tt> re-fires for <em>every</em> live module whenever <tt class="docutils literal"><span class="pre">merge</span></tt> rewrites the decision file's mtime (deliberate — a decision change must re-render every <tt class="docutils literal"><span class="pre">.c</span></tt> consistently). Writing the decision <tt class="docutils literal"><span class="pre">OnlyIfChanged</span></tt> (with a stamp output so the <tt class="docutils literal"><span class="pre">merge</span></tt> rule is not perpetually stale) makes a warm no-op instant, but a real edit then fires <tt class="docutils literal"><span class="pre">emit</span></tt> only for the modules whose <tt class="docutils literal"><span class="pre">.c.nif</span></tt> changed — and that produces <strong>multiple-definition link errors</strong> even when the decision is byte-identical. Fire-all <tt class="docutils literal"><span class="pre">emit</span></tt> is a correctness invariant, not just insurance (see the comment at <tt class="docutils literal"><span class="pre">generateEmitStage</span></tt>): partial <tt class="docutils literal"><span class="pre">emit</span></tt> leaves inconsistent ownership across the <tt class="docutils literal"><span class="pre">.c</span></tt> set. This path was tried and reverted; do not retry.</li>
|
||
</ul>
|
||
|
||
<h1 id="code-logic-amp-debugging">Code, logic & debugging</h1><p>Core modules:</p>
|
||
<ul class="simple"><li><strong>`compiler/deps.nim`</strong> — graph construction, SCC grouping, discovery fixpoint, build-file generation; <tt class="docutils literal"><span class="pre">commandIc</span></tt>.</li>
|
||
<li><strong>`compiler/ast2nif.nim`</strong> — AST↔NIF, the cookie hashes (<tt class="docutils literal"><span class="pre">cookieSd</span></tt>, <tt class="docutils literal"><span class="pre">writeIfaceCookie</span></tt>, <tt class="docutils literal"><span class="pre">writeImplCookie</span></tt>, <tt class="docutils literal"><span class="pre">writeEdgesFile</span></tt>, <tt class="docutils literal"><span class="pre">writeSemDeps</span></tt>).</li>
|
||
<li><strong>`compiler/nifbackend.nim`</strong> — the per-module backend stages (<tt class="docutils literal"><span class="pre">generateCgStage</span></tt>, <tt class="docutils literal"><span class="pre">generateMergeStage</span></tt>, <tt class="docutils literal"><span class="pre">generateEmitStage</span></tt>, <tt class="docutils literal"><span class="pre">generateLinkStage</span></tt>).</li>
|
||
<li><strong>`compiler/cnif.nim`</strong> — <tt class="docutils literal"><span class="pre">.c.nif</span></tt> artifact read/write, <tt class="docutils literal"><span class="pre">computeMergeDecision</span></tt>, <tt class="docutils literal"><span class="pre">renderCFromArtifact</span></tt>.</li>
|
||
<li><strong>`compiler/icconfig.nim`</strong> — precompiled config.</li>
|
||
<li><strong>`compiler/pipelines.nim`</strong> / <strong>`modulegraphs.nim`</strong> — pipeline integration and the graph state (<tt class="docutils literal"><span class="pre">importDeps</span></tt>, <tt class="docutils literal"><span class="pre">icImplDeps</span></tt>, <tt class="docutils literal"><span class="pre">icCnifFiles</span></tt>, <tt class="docutils literal"><span class="pre">instDisambs</span></tt>, …).</li>
|
||
</ul>
|
||
<p>Manual workflow:</p>
|
||
<ul class="simple"><li>Frontend a module: <tt class="docutils literal"><span class="pre">nim m --nimcache:nifcache path/to/mod.nim</span></tt> (writes <tt class="docutils literal"><span class="pre">.nif</span></tt> + cookies + <tt class="docutils literal"><span class="pre">.s.deps</span></tt>).</li>
|
||
<li>Backend is stage-based (a bare <tt class="docutils literal"><span class="pre">nim nifc main.nim</span></tt> errors — there is no whole-program fallback). The exact per-stage commands <tt class="docutils literal"><span class="pre">nifmake</span></tt> runs are in the <tt class="docutils literal"><span class="pre">*.backend.build.nif</span></tt> build file; rerun one directly against an existing cache, e.g. <tt class="docutils literal"><span class="pre">nim nifc --nimcache:nifcache --icBackendStage:cg --icBackendModule:<suffix> main.nim</span></tt> to regenerate one module's <tt class="docutils literal"><span class="pre">.c.nif</span></tt>, then <tt class="docutils literal"><span class="pre">--icBackendStage:merge</span></tt> / <tt class="docutils literal"><span class="pre">:emit</span></tt> / <tt class="docutils literal"><span class="pre">:link</span></tt>.</li>
|
||
<li>NIF and <tt class="docutils literal"><span class="pre">.c.nif</span></tt> files are text — open/grep them directly; <tt class="docutils literal"><span class="pre">diff</span></tt> two successive <tt class="docutils literal"><span class="pre">.nif</span></tt> to see why a module rebuilt.</li>
|
||
<li>Force a re-sem: delete the module's <tt class="docutils literal"><span class="pre">.nif</span></tt> and rerun <tt class="docutils literal"><span class="pre">nim m</span></tt>.</li>
|
||
<li>A stale-cache crash after editing the serialization layout means bumping <tt class="docutils literal"><span class="pre">icFormatVersion</span></tt> (<tt class="docutils literal"><span class="pre">compiler/options.nim</span></tt>).</li>
|
||
</ul>
|
||
|
||
<h1 id="see-also">See also</h1><ul class="simple"><li>NIF format spec: <a class="reference external" href="../nifspec/doc/nif-spec.md">nifspec/doc/nif-spec.md</a></li>
|
||
<li>NIFC (C-like target) spec: dist/nimony/doc/nifc-spec.md</li>
|
||
</ul>
|
||
|
||
|
||
|
||
<div class="twelve-columns footer">
|
||
<span class="nim-sprite"></span>
|
||
<br>
|
||
<small style="color: var(--hint);">Made with Nim. Generated: 2026-07-31 03:43:00 UTC</small>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<script defer data-domain="nim-lang.org" src="https://plausible.io/js/plausible.js"></script>
|
||
|
||
</body>
|
||
</html>
|