diff --git a/doc/ic.md b/doc/ic.md new file mode 100644 index 0000000000..81dadf4f89 --- /dev/null +++ b/doc/ic.md @@ -0,0 +1,181 @@ +====================================== + Incremental Compilation (IC) +====================================== + +The ``nim ic`` command provides incremental compilation support for Nim projects, +allowing faster rebuilds by reusing previously compiled intermediate representations +of modules that haven't changed. + +Overview +======== + +Incremental compilation works by decomposing the compilation process into several stages: + +1. **Parsing** - Source files are parsed into an abstract syntax tree (AST) +2. **Semantic Analysis** - Symbols are resolved and type checking is performed +3. **Code Generation** - Platform-specific code is generated from the analyzed AST +4. **Linking** - The generated code is linked into an executable + +The IC mechanism caches the results of earlier stages in ``.nif`` files +(Nim frontend intermediate format). When recompiling, only modules that have +changed need to be reprocessed through the semantic analysis and code generation +stages, significantly reducing compilation time for large projects. + +NIF File Format +=============== + +NIF (Nim Frontend Intermediate Format) files are text-based files that use a Lisp-like +syntax. They employ a hybrid format where byte offsets into the text are used for +efficient access, making them simultaneously human-readable and machine-efficient. +The text representation is particularly valuable for debugging and introspection. + +Each ``.nim`` module produces its own ``.nif`` file during compilation. +The NIF format contains: + +- **Header** - Version information (e.g., `(.nif24)`) +- **Dependencies** - List of source file checksums and their dependencies +- **Interface** - Exported symbols and their indices +- **Body** - The intermediate representation of the module's code in Lisp-like syntax + +The NIF format is designed specifically for Nim and allows efficient serialization +and deserialization of the compiler's intermediate representation while remaining +readable and debuggable by tools and developers. + +The ``nim ic`` Switch +===================== + +The ``nim ic`` command initiates incremental compilation for a project. +It automatically manages the build process by: + +1. Parsing all source files into ``.nif`` format (using the ``nifler`` tool) +2. Performing semantic analysis on modified modules +3. Generating code only for modules with changes or dependencies on changed modules +4. Generating a build file (in NIFMake format) that orchestrates the compilation +5. Executing the build file through ``nifmake`` + +Prerequisites +------------- + +- **nifler** - Tool for parsing Nim source files into NIF format +- **nifmake** - Build orchestration tool that follows dependencies + +If these tools are not available, ``nim ic`` will display instructions on how to +obtain them. + +Key Modules for IC Logic +========================= + +The primary modules in the compiler that handle incremental compilation logic are: + +- **deps.nim** - Dependency analysis and build file generation. Contains the + ``commandIc`` procedure which is the main entry point for the ``nim ic`` command. + This module orchestrates the incremental compilation process, handling NIF generation + and build file creation. + +- **ic.nim** - Core incremental compilation module handling the main IC logic, + module caching, and NIF serialization/deserialization. + +Additionally, various utility modules in the ``compiler/ic/`` directory support +the IC infrastructure for handling NIF data structures, line information mapping, +and state replay for pragmas and VM-specific compilations. + +Caching and Consistency +======================= + +The IC system ensures correctness through several mechanisms: + +- **Dependency Tracking** - Every module's dependencies are recorded and their + checksums stored in the NIF file + +- **Configuration Hashing** - The compiler configuration (options, GC mode, backend) + is hashed and stored, invalidating caches when configuration changes + +- **Atomic Operations** - By construction, either a `.nif` file is completely + read or completely written, preventing partial/inconsistent updates + +- **No Global State** - Each module's IC cache is independent, avoiding + complex global state synchronization issues + +**Code, Logic & Debugging** +=========================== + +This section focuses on the compiler-side code paths, the logic you will +inspect while debugging IC, and a pragmatic manual workflow for bug hunting +using local invocations such as ``nim m --nimcache:nifcache``. + +Core places to inspect +- **`compiler/deps.nim`**: generates the NIF-based build file and implements + ``commandIc`` (entry point for ``nim ic``). Look for how build rules are + emitted (calls to the NIF builder) and how inputs/outputs are wired. +- **`compiler/modulegraphs.nim`** and **`compiler/pipelines.nim`**: + dependency graph and compilation pipeline integration — useful when a module + is rebuilt unexpectedly. + +Understanding the NIF text +- NIF files are human-readable; open the per-module ``.nif`` files in + ``nifcache/`` to inspect parsed ASTs, dependency lists and interface tables. +- Because NIF uses textual nodes and byte offsets, tools can quickly seek to + positions in the file — but for debugging you usually only need to read the + file top-to-bottom. + +Manual bug-hunting workflow +- Prepare a clean nimcache directory (relative to your project): + + ```bash + mkdir -p nifcache + ``` + +- Parse/semantic-check a single module and write NIF/sem artifacts: + + ```bash + nim m --nimcache:nifcache path/to/module.nim + ``` + + - ``nim m`` runs the compiler up to the semantic checking stage for the + specified module and emits intermediate cache files into ``nifcache/``. + - Use this to reproduce and isolate failures in the semantic stage. + +- Inspect the generated files for that module under ``nifcache/`` (look for + ``.nif``, sem/parsed artifacts). Because NIF is text-based you can open and + grep it directly: + + ```bash + sed -n '1,200p' nifcache/ModuleName.nif + grep -n "someSymbol" -n nifcache/ModuleName.nif + ``` + +- To reproduce a full incremental compilation of the project, generate the + build file and run it (``nim ic`` automates this). To debug an individual + build step, run the command that the build file would execute manually + (for example, the semantic step uses ``nim m``; code generation uses ``nim nifc``). + +- Force a cache invalidation for a single module by removing its NIF/sem + artifact and re-running the semantic step: + + ```bash + rm nifcache/ModuleName.nif + nim m --nimcache:nifcache path/to/ModuleName.nim + ``` + +- When investigating incorrect replayed state (pragmas, `{.compile: ...}`): + inspect the replay actions in ``compiler/ic/replayer.nim`` and open the + module's NIF to find the ``toReplay``/action entries that will be executed + during reload. + +Tips for efficient debugging +- Use ``--path:...`` flags when invoking ``nim m`` to emulate the exact + search paths used in your project, e.g. ``--path:lib --path:vendor``. +- Compare two successive ``.nif`` files with ``diff`` to see what changed and + why a module was rebuilt. + +Where to change behavior +- Cache invalidation decisions and build-rule emission are implemented in + ``compiler/deps.nim``. When investigating surprising + rebuilds, instrument those modules to log the footprint/hash/comparison + outcome. + +See also +======== + +- `nif-spec` - NIF format specification (text format and node grammar): + [nifspec/doc/nif-spec.md](../nifspec/doc/nif-spec.md)