From ca4b971bc81b2e751e0388d80896fde7079b1679 Mon Sep 17 00:00:00 2001 From: zah Date: Tue, 26 Feb 2019 16:48:55 +0200 Subject: [PATCH] Initial version of the hot-code reloading support for native targets (#10729) * squashed work by Zahary * squashing a ton of useful history... otherwise rebasing on top of upstream Nim after commit 82c009a2cbc5d07ab9a847f1c58228a20efaf219 would be impossible. * Code review changes; Working test suite (without code reloading enabled) * - documentation - implemented the HCR test - almost works... - fix the issue on Unix where for executable targets the source file for the main module of a project in nimcache was being overwritten with the binary itself (and thus the actual source code was lost) - fixing embedded paths to shared objects on unix (the "lib" prefix was being prepended to the entire path instead of just the filename) - other fixes - removing unnecessary includes since that file is already included in chcks.nim which is in turn included in system.nim (and previously was getting imported in chcks.nim but then system.nim improts something... and that breaks HCR (perhaps it could be fixed but it would be nice not to import anything in system)) * fix for clang & C++ - explicitly casting a function pointer to void* more stable mangling of parameter names when HCR is on the length of the static arrays in the DatInit functions is now part of the name of the variables, so when they get resized they get also recreated more stable mangling for inline functions - no longer depends on the module which first used them work on the new complicated HCR test - turned surprisingly complex - WIP test now successfully passes even when re-running `koch test` (previously when the nimcache wasn't cold that lead to errors) better documentation calling setStackBottomWith for PreMain passes over the HcrInit/DatInit/Init calls of all modules are now in the proper order (first all of one type, then all of the next). Also typeinfo globals are registered (created) in a single pass before the DatInit pass (because of the way generic instantiations are handled) Fix the test suite execution on macOs fix for being able to query the program arguments when using HCR on posix! other fixes * Bugfix: Fix a compilation error in C++ mode when a function pointer is converted to a raw pointer * basic documentation for the new hot code reloading semantics * Add change log entry * Don't re-execute the top-level statements while reloading JS code * fix a number of tests broken in a recent bugfix * Review changes * Added {.executeOnReload.} pragma that indicates top-level statements that should be executed on each reload. To make this work, I've modified the way the `if (hcr_init_) {...}` guards are produced in the init code. This still needs more work as the new guards seem to be inserted within the previously generated guards. This change also removes the need for `lastRegistedGlobal` in nimhcr. * Implemented the `signatureHash` magic and the `hasModuleChanged` API depending on it (the actual logic is not imlemented yet). * Add the "hcr" prefix to all HCR-related symbols in the system module. Added a new `hotcodereloading` module exporting the high-level API to the user. Besides being more hygienic, this was also required in order to make it possible to use macros in the high-level API. Without the split, `system` would have to import `macros`, which was going to produce the well-known init problems. * Attempted to solve the "GC markers problem". Crashes were expected with the previous code, because the GC markers were compiled as normal procs are registered in the GC. When their module is unloaded, dangling pointers will remain in the GC tables. To solve this issue, I don't register any GC markers when HCR is on, but I add them to the HCR globals metadata and I use a single marker registed in nimhcr during the initialization of the system module that will be responsible for marking all globals. * fix a compilation error * - implemented the hasModuleChanged functionality - tuples can be returned and broken into different vars in global scope - added comments for the closnig scopes of the if statements in the init proc - the new executeOnReload pragma works now! - other fixes * finally! fixing this hack in a proper way - declaring the destructor out of line (out of the class body) - we no longer need to forward-declare popCurrentExceptionEx * Force full module parsing This is a temporary hack that breaks some tests. I'll investigate later how these can be fixed. * tuples are now properly handled when global! * these comments mess up the codegen in debug mode when $n is not actually a new line (or something like that) - these labels are intended only for GOTO labels anyway... * "solved" the issue with the .pdb locks on windows when a binary is being debugged and hot code reloading is used at the same time * fixes after rebasing... * small fixes for the test * better handling of globals! no more compiler crashes for locals with the global pragma, also simplified code around loops in global scope which have local vars (actually globals) * we can now use the global pragma even for ... globals! * the right output * lets try those boehm GC tests * after the test is ran it will be at its starting state - no git modifications * clarification in the docs * removed unnecessary line directives for forward declarations of functions - they were causing trouble with hot code reloading when no semantic change propagates to the main module but a line directive got changed and thus the main module had to be recompiled since the .c code had changed * fixed bug! was inserting duplicate keys into the table and later was removing only 1 copy of all the duplicates (after a few reloads) * no longer breaking into DatInit code when not supposed to * fixes after rebasing * yet more fixes after rebasing * Update jssys.nim * Rework the HCR path-handling logic After reviewing the code more carefully, I've noticed that the old logic will be broken when the user overrides the '--out:f' compiler option. Besides fixing this issues, I took the opportunity to implement the missing '--outdir:d' option. Other changes: * ./koch test won't overwrite any HCR and RTL builds located in nim/lib * HCR and RTL are compiled with --threads:on by default * Clean up the globals registration logic * Handle non-flattened top-level stmtlists in JS as well * The HCR is not supported with the Boehm GC yet Also fixes some typos and the expected output of the HCR integration test * The GC marker procs are now properly used as trampolines * Fix the HCR integration test in release builds * Fix ./koch tools * this forward declaration doesn't seem to be necessary, and in fact breaks HCR because a 2nd function pointer is emitted for this externed/rtl func * the forward declaration I removed in the last commit was actually necessary * Attempt to make all tests green * Fix tgenscript * BAT file for running the HCR integration test on Windows [skip ci] * Fix the docgen tests * A final fix for Travis (hopefully) --- .gitignore | 4 +- changelog.md | 10 +- compiler/ast.nim | 12 +- compiler/ccgcalls.nim | 6 +- compiler/ccgexprs.nim | 22 + compiler/ccgmerge.nim | 2 +- compiler/ccgstmts.nim | 84 +++- compiler/ccgtrav.nim | 21 +- compiler/ccgtypes.nim | 151 ++++-- compiler/cgen.nim | 407 +++++++++++---- compiler/cgendata.nim | 9 +- compiler/commands.nim | 20 +- compiler/condsyms.nim | 2 + compiler/docgen.nim | 14 +- compiler/docgen2.nim | 1 - compiler/extccomp.nim | 162 ++++-- compiler/jsgen.nim | 101 ++-- compiler/layouter.nim | 7 +- compiler/main.nim | 42 +- compiler/modulegraphs.nim | 2 + compiler/modules.nim | 5 +- compiler/nim.nim | 19 +- compiler/nodejs.nim | 1 + compiler/options.nim | 26 +- compiler/passes.nim | 4 +- compiler/pathutils.nim | 4 + compiler/pragmas.nim | 6 +- compiler/semexprs.nim | 5 +- compiler/semfold.nim | 2 - compiler/sighashes.nim | 16 +- compiler/transf.nim | 18 +- compiler/vm.nim | 11 +- compiler/vmdef.nim | 1 + compiler/vmgen.nim | 1 + compiler/wordrecg.nim | 2 + doc/advopt.txt | 1 + doc/nimc.rst | 98 +++- lib/core/hotcodereloading.nim | 27 + lib/core/macros.nim | 7 + lib/core/strs.nim | 2 +- lib/core/typeinfo.nim | 2 - lib/nimbase.h | 5 + lib/nimhcr.nim | 652 +++++++++++++++++++++++++ lib/nimhcr.nim.cfg | 5 + lib/nimrtl.nim.cfg | 1 + lib/pure/collections/sharedstrings.nim | 2 - lib/pure/os.nim | 2 +- lib/pure/reservedmem.nim | 241 +++++++++ lib/pure/strformat.nim | 6 +- lib/pure/strtabs.nim | 87 ++-- lib/system.nim | 20 +- lib/system/cgprocs.nim | 8 +- lib/system/chcks.nim | 2 +- lib/system/dyncalls.nim | 2 +- lib/system/excpt.nim | 4 +- lib/system/gc_common.nim | 8 +- lib/system/memory.nim | 6 +- lib/system/sysstr.nim | 2 +- lib/windows/winlean.nim | 4 + nimpretty/nimpretty.nim | 2 +- testament/categories.nim | 44 +- testament/tester.nim | 59 ++- tests/dll/nimhcr_0.nim | 4 + tests/dll/nimhcr_0_1.nim | 14 + tests/dll/nimhcr_0_2.nim | 18 + tests/dll/nimhcr_0_3.nim | 18 + tests/dll/nimhcr_0_4.nim | 19 + tests/dll/nimhcr_0_5.nim | 2 + tests/dll/nimhcr_0_6.nim | 4 + tests/dll/nimhcr_1.nim | 0 tests/dll/nimhcr_1_1.nim | 51 ++ tests/dll/nimhcr_1_2.nim | 4 + tests/dll/nimhcr_1_3.nim | 0 tests/dll/nimhcr_2.nim | 0 tests/dll/nimhcr_2_1.nim | 15 + tests/dll/nimhcr_2_2.nim | 7 + tests/dll/nimhcr_2_3.nim | 0 tests/dll/nimhcr_integration.nim | 152 ++++++ tests/dll/nimhcr_unit.nim | 147 ++++++ tests/dll/nimhcr_unit.nim.cfg | 2 + tests/dll/test_nimhcr_integration.bat | 10 + tests/dll/test_nimhcr_integration.sh | 17 + tests/types/tissues_types.nim | 7 + tools/kochdocs.nim | 2 +- 84 files changed, 2506 insertions(+), 486 deletions(-) create mode 100644 lib/core/hotcodereloading.nim create mode 100644 lib/nimhcr.nim create mode 100644 lib/nimhcr.nim.cfg create mode 100644 lib/pure/reservedmem.nim create mode 100644 tests/dll/nimhcr_0.nim create mode 100644 tests/dll/nimhcr_0_1.nim create mode 100644 tests/dll/nimhcr_0_2.nim create mode 100644 tests/dll/nimhcr_0_3.nim create mode 100644 tests/dll/nimhcr_0_4.nim create mode 100644 tests/dll/nimhcr_0_5.nim create mode 100644 tests/dll/nimhcr_0_6.nim create mode 100644 tests/dll/nimhcr_1.nim create mode 100644 tests/dll/nimhcr_1_1.nim create mode 100644 tests/dll/nimhcr_1_2.nim create mode 100644 tests/dll/nimhcr_1_3.nim create mode 100644 tests/dll/nimhcr_2.nim create mode 100644 tests/dll/nimhcr_2_1.nim create mode 100644 tests/dll/nimhcr_2_2.nim create mode 100644 tests/dll/nimhcr_2_3.nim create mode 100644 tests/dll/nimhcr_integration.nim create mode 100644 tests/dll/nimhcr_unit.nim create mode 100644 tests/dll/nimhcr_unit.nim.cfg create mode 100644 tests/dll/test_nimhcr_integration.bat create mode 100755 tests/dll/test_nimhcr_integration.sh diff --git a/.gitignore b/.gitignore index 38b7976120..9de4569bfd 100644 --- a/.gitignore +++ b/.gitignore @@ -11,7 +11,9 @@ dnimcache/ !/icons/*.o *.obj *.ilk +*.exp *.pdb +*.lib *.dll *.exe *.so @@ -19,7 +21,6 @@ dnimcache/ *.zip *.iss *.log -*.ilk *.pdb mapping.txt @@ -42,6 +43,7 @@ bin/* *.perspectivev3 *.swp .DS_Store +.tags project.xcworkspace/ xcuserdata/ diff --git a/changelog.md b/changelog.md index 7f6ad1ecb8..297a6c87bc 100644 --- a/changelog.md +++ b/changelog.md @@ -136,6 +136,9 @@ proc enumToString*(enums: openArray[enum]): string = slightly. The `dumpLisp` macro in this module now outputs an indented proper Lisp, devoid of commas. +- Added `macros.signatureHash` that returns a stable identifier + derived from the signature of a symbol. + - In `strutils` empty strings now no longer matched as substrings anymore. @@ -192,6 +195,11 @@ proc enumToString*(enums: openArray[enum]): string = ### Compiler changes - The deprecated `fmod` proc is now unavailable on the VM'. - +- A new `--outdir` option was added. +- The compiled JavaScript file for the project produced by executing `nim js` + will no longer be placed in the nimcache directory. +- The `--hotCodeReloading` has been implemented for the native targets. + The compiler also provides a new more flexible API for handling the + hot code reloading events in the code. ### Bugfixes diff --git a/compiler/ast.nim b/compiler/ast.nim index e691cc175a..607b497fb8 100644 --- a/compiler/ast.nim +++ b/compiler/ast.nim @@ -227,7 +227,7 @@ type TNodeKinds* = set[TNodeKind] type - TSymFlag* = enum # already 33 flags! + TSymFlag* = enum # already 34 flags! sfUsed, # read access of sym (for warnings) or simply used sfExported, # symbol is exported from module sfFromGeneric, # symbol is instantiation of a generic; this is needed @@ -276,6 +276,8 @@ type # the calling side of the macro, not from the # implementation. sfGenSym # symbol is 'gensym'ed; do not add to symbol table + sfNonReloadable # symbol will be left as-is when hot code reloading is on - + # meaning that it won't be renamed and/or changed in any way TSymFlags* = set[TSymFlag] @@ -468,6 +470,7 @@ type nfDefaultParam # an automatically inserter default parameter nfDefaultRefsParam # a default param value references another parameter # the flag is applied to proc default values and to calls + nfExecuteOnReload # A top-level statement that will be executed during reloads TNodeFlags* = set[TNodeFlag] TTypeFlag* = enum # keep below 32 for efficiency reasons (now: beyond that) @@ -654,7 +657,7 @@ type mNIntVal, mNFloatVal, mNSymbol, mNIdent, mNGetType, mNStrVal, mNSetIntVal, mNSetFloatVal, mNSetSymbol, mNSetIdent, mNSetType, mNSetStrVal, mNLineInfo, - mNNewNimNode, mNCopyNimNode, mNCopyNimTree, mStrToIdent, + mNNewNimNode, mNCopyNimNode, mNCopyNimTree, mStrToIdent, mNSigHash, mNBindSym, mLocals, mNCallSite, mEqIdent, mEqNimrodNode, mSameNodeType, mGetImpl, mNGenSym, mNHint, mNWarning, mNError, @@ -854,7 +857,7 @@ type offset*: int # offset of record field loc*: TLoc annex*: PLib # additional fields (seldom used, so we use a - # reference to another object to safe space) + # reference to another object to save space) constraint*: PNode # additional constraints like 'lit|result'; also # misused for the codegenDecl pragma in the hope # it won't cause problems @@ -978,7 +981,8 @@ const PersistentNodeFlags*: TNodeFlags = {nfBase2, nfBase8, nfBase16, nfDotSetter, nfDotField, nfIsRef, nfPreventCg, nfLL, - nfFromTemplate, nfDefaultRefsParam} + nfFromTemplate, nfDefaultRefsParam, + nfExecuteOnReload} namePos* = 0 patternPos* = 1 # empty except for term rewriting macros genericParamsPos* = 2 diff --git a/compiler/ccgcalls.nim b/compiler/ccgcalls.nim index d177e1f881..4c8fa7147c 100644 --- a/compiler/ccgcalls.nim +++ b/compiler/ccgcalls.nim @@ -185,7 +185,7 @@ proc genPrefixCall(p: BProc, le, ri: PNode, d: var TLoc) = var length = sonsLen(ri) for i in countup(1, length - 1): genParamLoop(params) - fixupCall(p, le, ri, d, op.r, params) + fixupCall(p, le, ri, d, rdLoc(op), params) proc genClosureCall(p: BProc, le, ri: PNode, d: var TLoc) = @@ -209,7 +209,7 @@ proc genClosureCall(p: BProc, le, ri: PNode, d: var TLoc) = genParamLoop(pl) template genCallPattern {.dirty.} = - lineF(p, cpsStmts, callPattern & ";$n", [op.r, pl, pl.addComma, rawProc]) + lineF(p, cpsStmts, callPattern & ";$n", [rdLoc(op), pl, pl.addComma, rawProc]) let rawProc = getRawProcType(p, typ) let callPattern = if tfIterator in typ.flags: PatIter else: PatProc @@ -237,7 +237,7 @@ proc genClosureCall(p: BProc, le, ri: PNode, d: var TLoc) = assert(d.t != nil) # generate an assignment to d: var list: TLoc initLoc(list, locCall, d.lode, OnUnknown) - list.r = callPattern % [op.r, pl, pl.addComma, rawProc] + list.r = callPattern % [rdLoc(op), pl, pl.addComma, rawProc] genAssignment(p, d, list, {}) # no need for deep copying else: genCallPattern() diff --git a/compiler/ccgexprs.nim b/compiler/ccgexprs.nim index f92f2e4de8..a40c60e6d0 100644 --- a/compiler/ccgexprs.nim +++ b/compiler/ccgexprs.nim @@ -1534,6 +1534,7 @@ proc genDollar(p: BProc, n: PNode, d: var TLoc, frmt: string) = var a: TLoc initLocExpr(p, n.sons[1], a) a.r = ropecg(p.module, frmt, [rdLoc(a)]) + a.flags = a.flags - {lfIndirect} # this flag should not be propagated here (not just for HCR) if d.k == locNone: getTemp(p, n.typ, d) genAssignment(p, d, a, {}) gcUsage(p.config, n) @@ -2034,8 +2035,28 @@ proc genMagicExpr(p: BProc, e: PNode, d: var TLoc, op: TMagic) = genCall(p, e, d) of mNewString, mNewStringOfCap, mExit, mParseBiggestFloat: var opr = e.sons[0].sym + # Why would anyone want to set nodecl to one of these hardcoded magics? + # - not sure, and it wouldn't work if the symbol behind the magic isn't + # somehow forward-declared from some other usage, but it is *possible* if lfNoDecl notin opr.loc.flags: + let prc = magicsys.getCompilerProc(p.module.g.graph, $opr.loc.r) + # HACK: + # Explicitly add this proc as declared here so the cgsym call doesn't + # add a forward declaration - without this we could end up with the same + # 2 forward declarations. That happens because the magic symbol and the original + # one that shall be used have different ids (even though a call to one is + # actually a call to the other) so checking into m.declaredProtos with the 2 different ids doesn't work. + # Why would 2 identical forward declarations be a problem? + # - in the case of hot code-reloading we generate function pointers instead + # of forward declarations and in C++ it is an error to redefine a global + let wasDeclared = containsOrIncl(p.module.declaredProtos, prc.id) + # Make the function behind the magic get actually generated - this will + # not lead to a forward declaration! The genCall will lead to one. discard cgsym(p.module, $opr.loc.r) + # make sure we have pointer-initialising code for hot code reloading + if not wasDeclared and p.hcrOn: + addf(p.module.s[cfsDynLibInit], "\t$1 = ($2) hcrGetProc($3, \"$1\");$n", + [mangleDynLibProc(prc), getTypeDesc(p.module, prc.loc.t), getModuleDllPath(p.module, prc)]) genCall(p, e, d) of mReset: genReset(p, e) of mEcho: genEcho(p, e[1].skipConv) @@ -2292,6 +2313,7 @@ proc exprComplexConst(p: BProc, n: PNode, d: var TLoc) = proc expr(p: BProc, n: PNode, d: var TLoc) = p.currLineInfo = n.info + case n.kind of nkSym: var sym = n.sym diff --git a/compiler/ccgmerge.nim b/compiler/ccgmerge.nim index 2d68a198e3..ccb5a7635d 100644 --- a/compiler/ccgmerge.nim +++ b/compiler/ccgmerge.nim @@ -31,7 +31,7 @@ const cfsData: "NIM_merge_DATA", cfsProcs: "NIM_merge_PROCS", cfsInitProc: "NIM_merge_INIT_PROC", - cfsDatInitProc: "NIM_merge_DATINIT_PROC", + cfsDatInitProc: "NIM_merge_DATINIT_PROC", cfsTypeInit1: "NIM_merge_TYPE_INIT1", cfsTypeInit2: "NIM_merge_TYPE_INIT2", cfsTypeInit3: "NIM_merge_TYPE_INIT3", diff --git a/compiler/ccgstmts.nim b/compiler/ccgstmts.nim index bc87353977..6dc10db3d6 100644 --- a/compiler/ccgstmts.nim +++ b/compiler/ccgstmts.nim @@ -15,18 +15,20 @@ const stringCaseThreshold = 8 # above X strings a hash-switch for strings is generated -proc registerGcRoot(p: BProc, v: PSym) = +proc getTraverseProc(p: BProc, v: Psym): Rope = if p.config.selectedGC in {gcMarkAndSweep, gcDestructors, gcV2, gcRefc} and containsGarbageCollectedRef(v.loc.t): # we register a specialized marked proc here; this has the advantage # that it works out of the box for thread local storage then :-) - let prc = genTraverseProcForGlobal(p.module, v, v.info) - if sfThread in v.flags: - appcg(p.module, p.module.initProc.procSec(cpsInit), - "#nimRegisterThreadLocalMarker($1);$n", [prc]) - else: - appcg(p.module, p.module.initProc.procSec(cpsInit), - "#nimRegisterGlobalMarker($1);$n", [prc]) + result = genTraverseProcForGlobal(p.module, v, v.info) + +proc registerTraverseProc(p: BProc, v: PSym, traverseProc: Rope) = + if sfThread in v.flags: + appcg(p.module, p.module.initProc.procSec(cpsInit), + "$n\t#nimRegisterThreadLocalMarker($1);$n$n", [traverseProc]) + else: + appcg(p.module, p.module.initProc.procSec(cpsInit), + "$n\t#nimRegisterGlobalMarker($1);$n$n", [traverseProc]) proc isAssignedImmediately(conf: ConfigRef; n: PNode): bool {.inline.} = if n.kind == nkEmpty: return false @@ -41,6 +43,10 @@ proc inExceptBlockLen(p: BProc): int = for x in p.nestedTryStmts: if x.inExcept: result.inc +proc startBlock(p: BProc, start: FormatStr = "{$n", + args: varargs[Rope]): int {.discardable.} +proc endBlock(p: BProc) + proc genVarTuple(p: BProc, n: PNode) = var tup, field: TLoc if n.kind != nkVarTuple: internalError(p.config, n.info, "genVarTuple") @@ -52,6 +58,32 @@ proc genVarTuple(p: BProc, n: PNode) = genStmts(p, lowerTupleUnpacking(p.module.g.graph, n, p.prc)) return + # check only the first son + var forHcr = treatGlobalDifferentlyForHCR(p.module, n.sons[0].sym) + let hcrCond = if forHcr: getTempName(p.module) else: nil + var hcrGlobals: seq[tuple[loc: TLoc, tp: Rope]] + # determine if the tuple is constructed at top-level scope or inside of a block (if/while/block) + let isGlobalInBlock = forHcr and p.blocks.len > 2 + # do not close and reopen blocks if this is a 'global' but inside of a block (if/while/block) + forHcr = forHcr and not isGlobalInBlock + + if forHcr: + # check with the boolean if the initializing code for the tuple should be ran + lineCg(p, cpsStmts, "if ($1)$n", hcrCond) + startBlock(p) + defer: + if forHcr: + # end the block where the tuple gets initialized + endBlock(p) + if forHcr or isGlobalInBlock: + # insert the registration of the globals for the different parts of the tuple at the + # start of the current scope (after they have been iterated) and init a boolean to + # check if any of them is newly introduced and the initializing code has to be ran + lineCg(p, cpsLocals, "NIM_BOOL $1 = NIM_FALSE;$n", hcrCond) + for curr in hcrGlobals: + lineCg(p, cpsLocals, "$1 |= hcrRegisterGlobal($4, \"$2\", sizeof($3), $5, (void**)&$2);$N", + hcrCond, curr.loc.r, rdLoc(curr.loc), getModuleDllPath(p.module, n.sons[0].sym), curr.tp) + genLineDir(p, n) initLocExpr(p, n.sons[L-1], tup) var t = tup.t.skipTypes(abstractInst) @@ -59,10 +91,13 @@ proc genVarTuple(p: BProc, n: PNode) = let vn = n.sons[i] let v = vn.sym if sfCompileTime in v.flags: continue + var traverseProc: Rope if sfGlobal in v.flags: assignGlobalVar(p, vn) genObjectInit(p, cpsInit, v.typ, v.loc, true) - registerGcRoot(p, v) + traverseProc = getTraverseProc(p, v) + if traverseProc != nil and not p.hcrOn: + registerTraverseProc(p, v, traverseProc) else: assignLocalVar(p, vn) initLocalVar(p, v, immediateAsgn=isAssignedImmediately(p.config, n[L-1])) @@ -73,6 +108,8 @@ proc genVarTuple(p: BProc, n: PNode) = if t.n.sons[i].kind != nkSym: internalError(p.config, n.info, "genVarTuple") field.r = "$1.$2" % [rdLoc(tup), mangleRecFieldName(p.module, t.n.sons[i].sym)] putLocIntoDest(p, v.loc, field) + if forHcr or isGlobalInBlock: + hcrGlobals.add((loc: v.loc, tp: if traverseProc == nil: ~"NULL" else: traverseProc)) proc genDeref(p: BProc, e: PNode, d: var TLoc; enforceDeref=false) @@ -242,6 +279,7 @@ proc genSingleVar(p: BProc, a: PNode) = genGotoVar(p, a.sons[2]) return var targetProc = p + var traverseProc: Rope if sfGlobal in v.flags: if v.flags * {sfImportc, sfExportc} == {sfImportc} and a.sons[2].kind == nkEmpty and @@ -270,7 +308,9 @@ proc genSingleVar(p: BProc, a: PNode) = # if sfImportc notin v.flags: constructLoc(p.module.preInitProc, v.loc) if sfExportc in v.flags and p.module.g.generatedHeader != nil: genVarPrototype(p.module.g.generatedHeader, vn) - registerGcRoot(p, v) + traverseProc = getTraverseProc(p, v) + if traverseProc != nil and not p.hcrOn: + registerTraverseProc(p, v, traverseProc) else: let value = a.sons[2] let imm = isAssignedImmediately(p.config, value) @@ -302,6 +342,30 @@ proc genSingleVar(p: BProc, a: PNode) = assignLocalVar(p, vn) initLocalVar(p, v, imm) + if traverseProc == nil: traverseProc = ~"NULL" + # If the var is in a block (control flow like if/while or a block) in global scope just + # register the so called "global" so it can be used later on. There is no need to close + # and reopen of if (nim_hcr_do_init_) blocks because we are in one already anyway. + var forHcr = treatGlobalDifferentlyForHCR(p.module, v) + if forHcr and targetProc.blocks.len > 3 and v.owner.kind == skModule: + # put it in the locals section - mainly because of loops which + # use the var in a call to resetLoc() in the statements section + lineCg(targetProc, cpsLocals, "hcrRegisterGlobal($3, \"$1\", sizeof($2), $4, (void**)&$1);$n", + v.loc.r, rdLoc(v.loc), getModuleDllPath(p.module, v), traverseProc) + # nothing special left to do later on - let's avoid closing and reopening blocks + forHcr = false + + # we close and reopen the global if (nim_hcr_do_init_) blocks in the main Init function + # for the module so we can have globals and top-level code be interleaved and still + # be able to re-run it but without the top level code - just the init of globals + if forHcr: + lineCg(targetProc, cpsStmts, "if (hcrRegisterGlobal($3, \"$1\", sizeof($2), $4, (void**)&$1))$N", + v.loc.r, rdLoc(v.loc), getModuleDllPath(p.module, v), traverseProc) + startBlock(targetProc) + defer: + if forHcr: + endBlock(targetProc) + if a.sons[2].kind != nkEmpty: genLineDir(targetProc, a) loadInto(targetProc, a.sons[0], a.sons[2], v.loc) diff --git a/compiler/ccgtrav.nim b/compiler/ccgtrav.nim index 0a2bbf93bd..87e7c9d489 100644 --- a/compiler/ccgtrav.nim +++ b/compiler/ccgtrav.nim @@ -134,11 +134,13 @@ proc genTraverseProc(m: BModule, origTyp: PType; sig: SigHash): Rope = var c: TTraversalClosure var p = newProc(nil, m) result = "Marker_" & getTypeName(m, origTyp, sig) - var typ = origTyp.skipTypes(abstractInstOwned) + let + hcrOn = m.hcrOn + typ = origTyp.skipTypes(abstractInstOwned) + markerName = if hcrOn: result & "_actual" else: result + header = "static N_NIMCALL(void, $1)(void* p, NI op)" % [markerName] + t = getTypeDesc(m, typ) - let header = "static N_NIMCALL(void, $1)(void* p, NI op)" % [result] - - let t = getTypeDesc(m, typ) lineF(p, cpsLocals, "$1 a;$n", [t]) lineF(p, cpsInit, "a = ($1)p;$n", [t]) @@ -155,18 +157,23 @@ proc genTraverseProc(m: BModule, origTyp: PType; sig: SigHash): Rope = else: genTraverseProc(c, "(*a)".rope, typ.sons[0]) - let generatedProc = "$1 {$n$2$3$4}$n" % + let generatedProc = "$1 {$n$2$3$4}\n" % [header, p.s(cpsLocals), p.s(cpsInit), p.s(cpsStmts)] - m.s[cfsProcHeaders].addf("$1;$n", [header]) + m.s[cfsProcHeaders].addf("$1;\n", [header]) m.s[cfsProcs].add(generatedProc) + if hcrOn: + addf(m.s[cfsProcHeaders], "N_NIMCALL_PTR(void, $1)(void*, NI);\n", [result]) + addf(m.s[cfsDynLibInit], "\t$1 = (N_NIMCALL_PTR(void, )(void*, NI)) hcrRegisterProc($3, \"$1\", (void*)$2);\n", + [result, markerName, getModuleDllPath(m)]) + proc genTraverseProcForGlobal(m: BModule, s: PSym; info: TLineInfo): Rope = discard genTypeInfo(m, s.loc.t, info) var c: TTraversalClosure var p = newProc(nil, m) - var sLoc = s.loc.r + var sLoc = rdLoc(s.loc) result = getTempName(m) if sfThread in s.flags and emulatedThreadVars(m.config): diff --git a/compiler/ccgtypes.nim b/compiler/ccgtypes.nim index afe90544d1..063c02df9a 100644 --- a/compiler/ccgtypes.nim +++ b/compiler/ccgtypes.nim @@ -14,7 +14,7 @@ import sighashes from lowerings import createObj -proc genProcHeader(m: BModule, prc: PSym): Rope +proc genProcHeader(m: BModule, prc: PSym, asPtr: bool = false): Rope proc isKeyword(w: PIdent): bool = # Nim and C++ share some keywords @@ -59,7 +59,23 @@ proc mangleParamName(m: BModule; s: PSym): Rope = result = s.loc.r if result == nil: var res = s.name.s.mangle - if isKeyword(s.name) or m.g.config.cppDefines.contains(res): + # Take into account if HCR is on because of the following scenario: + # if a module gets imported and it has some more importc symbols in it, + # some param names might recieve the "_0" suffix to distinguish from what + # is newly available. That might lead to changes in the C code in nimcache + # that contain only a parameter name change, but that is enough to mandate + # recompilation of that source file and thus a new shared object will be + # relinked. That may lead to a module getting reloaded which wasn't intended + # and that may be fatal when parts of the current active callstack when + # performCodeReload() was called are from the module being reloaded + # unintentionally - example (3 modules which import one another): + # main => proxy => reloadable + # we call performCodeReload() in proxy to reload only changes in reloadable + # but there is a new import which introduces an importc symbol `socket` + # and a function called in main or proxy uses `socket` as a parameter name. + # That would lead to either needing to reload `proxy` or to overwrite the + # executable file for the main module, which is running (or both!) -> error. + if m.hcrOn or isKeyword(s.name) or m.g.config.cppDefines.contains(res): res.add "_0" result = res.rope s.loc.r = result @@ -507,6 +523,8 @@ proc fillObjectFields*(m: BModule; typ: PType) = var check = initIntSet() discard getRecordFields(m, typ, check) +proc mangleDynLibProc(sym: PSym): Rope + proc getRecordDesc(m: BModule, typ: PType, name: Rope, check: var IntSet): Rope = # declare the record: @@ -535,22 +553,13 @@ proc getRecordDesc(m: BModule, typ: PType, name: Rope, appcg(m, result, " : public $1 {$n", [getTypeDescAux(m, typ.sons[0].skipTypes(skipPtrs), check)]) if typ.isException: - appcg(m, result, "virtual void raise() {throw *this;}$n") # required for polymorphic exceptions + appcg(m, result, "virtual void raise() { throw *this; }$n") # required for polymorphic exceptions if typ.sym.magic == mException: # Add cleanup destructor to Exception base class - appcg(m, result, "~$1() {if(this->raiseId) #popCurrentExceptionEx(this->raiseId);}$n", [name]) - # hack: forward declare popCurrentExceptionEx() on top of type description, - # proper request to generate popCurrentExceptionEx not possible for 2 reasons: - # generated function will be below declared Exception type and circular dependency - # between Exception and popCurrentExceptionEx function - - let popExSym = magicsys.getCompilerProc(m.g.graph, "popCurrentExceptionEx") - if lfDynamicLib in popExSym.loc.flags and sfImportc in popExSym.flags: - # echo popExSym.flags, " ma flags ", popExSym.loc.flags - result = "extern " & getTypeDescAux(m, popExSym.typ, check) & " " & - mangleName(m, popExSym) & ";\L" & result - else: - result = genProcHeader(m, popExSym) & ";\L" & result + appcg(m, result, "~$1();$n", [name]) + # define it out of the class body and into the procs section so we don't have to + # artificially forward-declare popCurrentExceptionEx (very VERY troublesome for HCR) + appcg(m, cfsProcs, "inline $1::~$1() {if(this->raiseId) #popCurrentExceptionEx(this->raiseId);}$n", [name]) hasField = true else: appcg(m, result, " {$n $1 Sup;$n", @@ -888,31 +897,43 @@ proc finishTypeDescriptions(m: BModule) = template cgDeclFrmt*(s: PSym): string = s.constraint.strVal -proc genProcHeader(m: BModule, prc: PSym): Rope = +proc isReloadable(m: BModule, prc: PSym): bool = + return m.hcrOn and sfNonReloadable notin prc.flags + +proc isNonReloadable(m: BModule, prc: PSym): bool = + return m.hcrOn and sfNonReloadable in prc.flags + +proc genProcHeader(m: BModule, prc: PSym, asPtr: bool = false): Rope = var rettype, params: Rope - genCLineDir(result, prc.info, m.config) # using static is needed for inline procs if lfExportLib in prc.loc.flags: if isHeaderFile in m.flags: result.add "N_LIB_IMPORT " else: result.add "N_LIB_EXPORT " - elif prc.typ.callConv == ccInline: + elif prc.typ.callConv == ccInline or asPtr or isNonReloadable(m, prc): result.add "static " elif {sfImportc, sfExportc} * prc.flags == {}: result.add "N_LIB_PRIVATE " var check = initIntSet() fillLoc(prc.loc, locProc, prc.ast[namePos], mangleName(m, prc), OnUnknown) genProcParams(m, prc.typ, rettype, params, check) + # handle the 2 options for hotcodereloading codegen - function pointer + # (instead of forward declaration) or header for function budy with "_actual" postfix + let asPtrStr = rope(if asPtr: "_PTR" else: "") + var name = prc.loc.r + if isReloadable(m, prc) and not asPtr: + add(name, "_actual") # careful here! don't access ``prc.ast`` as that could reload large parts of # the object graph! if prc.constraint.isNil: - addf(result, "$1($2, $3)$4", - [rope(CallingConvToStr[prc.typ.callConv]), rettype, prc.loc.r, + addf(result, "$1$2($3, $4)$5", + [rope(CallingConvToStr[prc.typ.callConv]), asPtrStr, rettype, name, params]) else: - result = prc.cgDeclFrmt % [rettype, prc.loc.r, params] + let asPtrStr = if asPtr: (rope("(*") & name & ")") else: name + result = prc.cgDeclFrmt % [rettype, asPtrStr, params] # ------------------ type info generation ------------------------------------- @@ -921,6 +942,9 @@ proc getNimNode(m: BModule): Rope = result = "$1[$2]" % [m.typeNodesName, rope(m.typeNodes)] inc(m.typeNodes) +proc TINameForHcr(m: BModule, name: Rope): Rope = + return if m.hcrOn: "(*".rope & name & ")" else: name + proc genTypeInfoAuxBase(m: BModule; typ, origType: PType; name, base: Rope; info: TLineInfo) = var nimtypeKind: int @@ -930,19 +954,21 @@ proc genTypeInfoAuxBase(m: BModule; typ, origType: PType; else: nimtypeKind = ord(typ.kind) + let nameHcr = TINameForHcr(m, name) + var size: Rope if tfIncompleteStruct in typ.flags: size = rope"void*" else: size = getTypeDesc(m, origType) addf(m.s[cfsTypeInit3], "$1.size = sizeof($2);$n" & "$1.kind = $3;$n" & "$1.base = $4;$n", - [name, size, rope(nimtypeKind), base]) + [nameHcr, size, rope(nimtypeKind), base]) # compute type flags for GC optimization var flags = 0 if not containsGarbageCollectedRef(typ): flags = flags or 1 if not canFormAcycle(typ): flags = flags or 2 #else MessageOut("can contain a cycle: " & typeToString(typ)) if flags != 0: - addf(m.s[cfsTypeInit3], "$1.flags = $2;$n", [name, rope(flags)]) + addf(m.s[cfsTypeInit3], "$1.flags = $2;$n", [nameHcr, rope(flags)]) discard cgsym(m, "TNimType") if isDefined(m.config, "nimTypeNames"): var typename = typeToString(if origType.typeInst != nil: origType.typeInst @@ -950,11 +976,17 @@ proc genTypeInfoAuxBase(m: BModule; typ, origType: PType; if typename == "ref object" and origType.skipTypes(skipPtrs).sym != nil: typename = "anon ref object from " & m.config$origType.skipTypes(skipPtrs).sym.info addf(m.s[cfsTypeInit3], "$1.name = $2;$n", - [name, makeCstring typename]) + [nameHcr, makeCstring typename]) discard cgsym(m, "nimTypeRoot") addf(m.s[cfsTypeInit3], "$1.nextType = nimTypeRoot; nimTypeRoot=&$1;$n", - [name]) - addf(m.s[cfsVars], "TNimType $1;$n", [name]) + [nameHcr]) + + if m.hcrOn: + addf(m.s[cfsVars], "static TNimType* $1;$n", [name]) + addf(m.hcrCreateTypeInfosProc, "\thcrRegisterGlobal($2, \"$1\", sizeof(TNimType), NULL, (void**)&$1);$n", + [name, getModuleDllPath(m, m.module)]) + else: + addf(m.s[cfsVars], "TNimType $1;$n", [name]) proc genTypeInfoAux(m: BModule, typ, origType: PType, name: Rope; info: TLineInfo) = @@ -984,6 +1016,14 @@ proc discriminatorTableDecl(m: BModule, objtype: PType, d: PSym): Rope = var tmp = discriminatorTableName(m, objtype, d) result = "TNimNode* $1[$2];$n" % [tmp, rope(lengthOrd(m.config, d.typ)+1)] +proc genTNimNodeArray(m: BModule, name: Rope, size: Rope) = + if m.hcrOn: + addf(m.s[cfsVars], "static TNimNode** $1;$n", [name]) + addf(m.hcrCreateTypeInfosProc, "\thcrRegisterGlobal($3, \"$1\", sizeof(TNimNode*) * $2, NULL, (void**)&$1);$n", + [name, size, getModuleDllPath(m, m.module)]) + else: + addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n", [name, size]) + proc genObjectFields(m: BModule, typ, origType: PType, n: PNode, expr: Rope; info: TLineInfo) = case n.kind @@ -992,8 +1032,8 @@ proc genObjectFields(m: BModule, typ, origType: PType, n: PNode, expr: Rope; if L == 1: genObjectFields(m, typ, origType, n.sons[0], expr, info) elif L > 0: - var tmp = getTempName(m) - addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n", [tmp, rope(L)]) + var tmp = getTempName(m) & "_" & $L + genTNimNodeArray(m, tmp, rope(L)) for i in countup(0, L-1): var tmp2 = getNimNode(m) addf(m.s[cfsTypeInit3], "$1[$2] = &$3;$n", [tmp, rope(i), tmp2]) @@ -1066,7 +1106,7 @@ proc genObjectInfo(m: BModule, typ, origType: PType, name: Rope; info: TLineInfo var tmp = getNimNode(m) if not isImportedType(typ): genObjectFields(m, typ, origType, typ.n, tmp, info) - addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [name, tmp]) + addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [TINameForHcr(m, name), tmp]) var t = typ.sons[0] while t != nil: t = t.skipTypes(skipPtrs) @@ -1078,8 +1118,8 @@ proc genTupleInfo(m: BModule, typ, origType: PType, name: Rope; info: TLineInfo) var expr = getNimNode(m) var length = sonsLen(typ) if length > 0: - var tmp = getTempName(m) - addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n", [tmp, rope(length)]) + var tmp = getTempName(m) & "_" & $length + genTNimNodeArray(m, tmp, rope(length)) for i in countup(0, length - 1): var a = typ.sons[i] var tmp2 = getNimNode(m) @@ -1094,7 +1134,7 @@ proc genTupleInfo(m: BModule, typ, origType: PType, name: Rope; info: TLineInfo) else: addf(m.s[cfsTypeInit3], "$1.len = $2; $1.kind = 2;$n", [expr, rope(length)]) - addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [name, expr]) + addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [TINameForHcr(m, name), expr]) proc genEnumInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) = # Type information for enumerations is quite heavy, so we do some @@ -1102,10 +1142,9 @@ proc genEnumInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) = # anyway. We generate a cstring array and a loop over it. Exceptional # positions will be reset after the loop. genTypeInfoAux(m, typ, typ, name, info) - var nodePtrs = getTempName(m) var length = sonsLen(typ.n) - addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n", - [nodePtrs, rope(length)]) + var nodePtrs = getTempName(m) & "_" & $length + genTNimNodeArray(m, nodePtrs, rope(length)) var enumNames, specialCases: Rope var firstNimNode = m.typeNodes var hasHoles = false @@ -1134,17 +1173,17 @@ proc genEnumInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) = add(m.s[cfsTypeInit3], specialCases) addf(m.s[cfsTypeInit3], "$1.len = $2; $1.kind = 2; $1.sons = &$3[0];$n$4.node = &$1;$n", - [getNimNode(m), rope(length), nodePtrs, name]) + [getNimNode(m), rope(length), nodePtrs, TINameForHcr(m, name)]) if hasHoles: # 1 << 2 is {ntfEnumHole} - addf(m.s[cfsTypeInit3], "$1.flags = 1<<2;$n", [name]) + addf(m.s[cfsTypeInit3], "$1.flags = 1<<2;$n", [TINameForHcr(m, name)]) proc genSetInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) = assert(typ.sons[0] != nil) genTypeInfoAux(m, typ, typ, name, info) var tmp = getNimNode(m) addf(m.s[cfsTypeInit3], "$1.len = $2; $1.kind = 0;$n" & "$3.node = &$1;$n", - [tmp, rope(firstOrd(m.config, typ)), name]) + [tmp, rope(firstOrd(m.config, typ)), TINameForHcr(m, name)]) proc genArrayInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) = genTypeInfoAuxBase(m, typ, typ, name, genTypeInfo(m, typ.sons[1], info), info) @@ -1169,19 +1208,29 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope = let origType = t var t = skipTypes(origType, irrelevantForBackend + tyUserTypeClasses) + let prefixTI = if m.hcrOn: "(" else: "(&" + let sig = hashType(origType) result = m.typeInfoMarker.getOrDefault(sig) if result != nil: - return "(&".rope & result & ")".rope + return prefixTI.rope & result & ")".rope - result = m.g.typeInfoMarker.getOrDefault(sig) - if result != nil: + proc declareNimType(m: BModule, str: Rope, ownerModule: PSym) = + if m.hcrOn: + addf(m.s[cfsVars], "static TNimType* $1;$n", [str]) + addf(m.s[cfsTypeInit1], "\t$1 = (TNimType*)hcrGetGlobal($2, \"$1\");$n", + [str, getModuleDllPath(m, ownerModule)]) + else: + addf(m.s[cfsVars], "extern TNimType $1;$n", [str]) + + let marker = m.g.typeInfoMarker.getOrDefault(sig) + if marker.str != nil: discard cgsym(m, "TNimType") discard cgsym(m, "TNimNode") - addf(m.s[cfsVars], "extern TNimType $1;$n", [result]) + declareNimType(m, marker.str, marker.owner) # also store in local type section: - m.typeInfoMarker[sig] = result - return "(&".rope & result & ")".rope + m.typeInfoMarker[sig] = marker.str + return prefixTI.rope & marker.str & ")".rope result = "NTI$1_" % [rope($sig)] m.typeInfoMarker[sig] = result @@ -1194,10 +1243,10 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope = # reference the type info as extern here discard cgsym(m, "TNimType") discard cgsym(m, "TNimNode") - addf(m.s[cfsVars], "extern TNimType $1;$n", [result]) - return "(&".rope & result & ")".rope + declareNimType(m, result, owner) + return prefixTI.rope & result & ")".rope - m.g.typeInfoMarker[sig] = result + m.g.typeInfoMarker[sig] = (str: result, owner: owner) case t.kind of tyEmpty, tyVoid: result = rope"0" of tyPointer, tyBool, tyChar, tyCString, tyString, tyInt..tyUInt64, tyVar, tyLent: @@ -1219,12 +1268,12 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope = if m.config.selectedGC != gcDestructors: if m.config.selectedGC >= gcMarkAndSweep: let markerProc = genTraverseProc(m, origType, sig) - addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [result, markerProc]) + addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [TINameForHcr(m, result), markerProc]) of tyRef: genTypeInfoAux(m, t, t, result, info) if m.config.selectedGC >= gcMarkAndSweep: let markerProc = genTraverseProc(m, origType, sig) - addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [result, markerProc]) + addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [TINameForHcr(m, result), markerProc]) of tyPtr, tyRange, tyUncheckedArray: genTypeInfoAux(m, t, t, result, info) of tyArray: genArrayInfo(m, t, result, info) of tySet: genSetInfo(m, t, result, info) @@ -1241,7 +1290,7 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope = genDeepCopyProc(m, t.deepCopy, result) elif origType.deepCopy != nil: genDeepCopyProc(m, origType.deepCopy, result) - result = "(&".rope & result & ")".rope + result = prefixTI.rope & result & ")".rope proc genTypeSection(m: BModule, n: PNode) = discard diff --git a/compiler/cgen.nim b/compiler/cgen.nim index cb186de101..d8f426f05d 100644 --- a/compiler/cgen.nim +++ b/compiler/cgen.nim @@ -16,8 +16,6 @@ import condsyms, rodutils, renderer, idgen, cgendata, ccgmerge, semfold, aliases, lowerings, tables, sets, ndi, lineinfos, pathutils, transf -import system/indexerrors - when not defined(leanCompiler): import semparallel @@ -46,6 +44,9 @@ when options.hasTinyCBackend: # implementation +proc hcrOn(m: BModule): bool = m.config.hcrOn +proc hcrOn(p: BProc): bool = p.module.config.hcrOn + proc addForwardedProc(m: BModule, prc: PSym) = m.g.forwardedProcs.add(prc) @@ -92,6 +93,17 @@ proc useHeader(m: BModule, sym: PSym) = proc cgsym(m: BModule, name: string): Rope +proc getCFile(m: BModule): AbsoluteFile + +proc getModuleDllPath(m: BModule): Rope = + let (dir, name, ext) = splitFile(getCFile(m)) + let filename = strutils.`%`(platform.OS[m.g.config.target.targetOS].dllFrmt, [name & ext]) + return makeCString(dir.string & "/" & filename) + +proc getModuleDllPath(m: BModule, s: PSym): Rope = + return getModuleDllPath(findPendingModule(m, s)) + +# TODO: please document proc ropecg(m: BModule, frmt: FormatStr, args: varargs[Rope]): Rope = assert m != nil var i = 0 @@ -440,12 +452,18 @@ proc assignLocalVar(p: BProc, n: PNode) = include ccgthreadvars proc varInDynamicLib(m: BModule, sym: PSym) -proc mangleDynLibProc(sym: PSym): Rope + +proc treatGlobalDifferentlyForHCR(m: BModule, s: PSym): bool = + return m.hcrOn and {sfThread, sfGlobal} * s.flags == {sfGlobal} and + ({lfNoDecl, lfHeader} * s.loc.flags == {}) + # and s.owner.kind == skModule # owner isn't always a module (global pragma on local var) + # and s.loc.k == locGlobalVar # loc isn't always initialized when this proc is used proc assignGlobalVar(p: BProc, n: PNode) = let s = n.sym if s.loc.k == locNone: fillLoc(s.loc, locGlobalVar, n, mangleName(p.module, s), OnHeap) + if treatGlobalDifferentlyForHCR(p.module, s): incl(s.loc.flags, lfIndirect) if lfDynamicLib in s.loc.flags: var q = findPendingModule(p.module, s) @@ -463,8 +481,10 @@ proc assignGlobalVar(p: BProc, n: PNode) = var decl: Rope = nil var td = getTypeDesc(p.module, s.loc.t) if s.constraint.isNil: - if sfImportc in s.flags: add(decl, "extern ") + if p.hcrOn: add(decl, "static ") + elif sfImportc in s.flags: add(decl, "extern ") add(decl, td) + if p.hcrOn: add(decl, "*") if sfRegister in s.flags: add(decl, " register") if sfVolatile in s.flags: add(decl, " volatile") addf(decl, " $1;$n", [s.loc.r]) @@ -853,6 +873,14 @@ proc allPathsAsgnResult(n: PNode): InitResultEnum = for i in 0.. 0: m.s[cfsProcs].add openNamespaceNim(m.config.cppCustomNamespace) -proc getSomeInitName(m: PSym, suffix: string): Rope = - assert m.kind == skModule - assert m.owner.kind == skPackage - if {sfSystemModule, sfMainModule} * m.flags == {}: - result = m.owner.name.s.mangle.rope - result.add "_" - result.add m.name.s.mangle - result.add suffix - -proc getInitName(m: PSym): Rope = - if sfMainModule in m.flags: - # generate constant name for main module, for "easy" debugging. - result = rope"NimMainModule" - else: - result = getSomeInitName(m, "Init000") - -proc getDatInitName(m: PSym): Rope = getSomeInitName(m, "DatInit000") - - proc registerModuleToMain(g: BModuleList; m: BModule) = + let + init = m.getInitName + datInit = m.getDatInitName + + if m.hcrOn: + var hcr_module_meta = "$nN_LIB_PRIVATE const char* hcr_module_list[] = {$n" % [] + let systemModulePath = getModuleDllPath(m, g.modules[g.graph.config.m.systemFileIdx.int].module) + let mainModulePath = getModuleDllPath(m, m.module) + if sfMainModule in m.module.flags: + addf(hcr_module_meta, "\t$1,$n", [systemModulePath]) + g.graph.importDeps.withValue(FileIndex(m.module.position), deps): + for curr in deps[]: + addf(hcr_module_meta, "\t$1,$n", [getModuleDllPath(m, g.modules[curr.int].module)]) + addf(hcr_module_meta, "\t\"\"};$n", []) + addf(hcr_module_meta, "$nN_LIB_EXPORT N_NIMCALL(void**, HcrGetImportedModules)() { return (void**)hcr_module_list; }$n", []) + addf(hcr_module_meta, "$nN_LIB_EXPORT N_NIMCALL(char*, HcrGetSigHash)() { return \"$1\"; }$n$n", + [($sigHash(m.module)).rope]) + if sfMainModule in m.module.flags: + add(g.mainModProcs, hcr_module_meta) + addf(g.mainModProcs, "static void* hcr_handle;$N", []) + addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, $1)(void);$N", [init]) + addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, $1)(void);$N", [datInit]) + addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, $1)(void*, N_NIMCALL_PTR(void*, getProcAddr)(void*, char*));$N", [m.getHcrInitName]) + addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, HcrCreateTypeInfos)(void);$N", []) + addf(g.mainModInit, "\t$1();$N", [init]) + addf(g.otherModsInit, "\thcrInit((void**)hcr_module_list, $1, $2, $3, hcr_handle, nimGetProcAddr);$n", + [mainModulePath, systemModulePath, datInit]) + addf(g.mainDatInit, "\t$1(hcr_handle, nimGetProcAddr);$N", [m.getHcrInitName]) + addf(g.mainDatInit, "\thcrAddModule($1);\n", [mainModulePath]) + addf(g.mainDatInit, "\tHcrCreateTypeInfos();$N", []) + # nasty nasty hack to get the command line functionality working with HCR + # register the 2 variables on behalf of the os module which might not even + # be loaded (in which case it will get collected but that is not a problem) + let osModulePath = ($systemModulePath).replace("stdlib_system", "stdlib_os").rope + addf(g.mainDatInit, "\thcrAddModule($1);\n", [osModulePath]) + add(g.mainDatInit, "\tint* cmd_count;\n") + add(g.mainDatInit, "\tchar*** cmd_line;\n") + addf(g.mainDatInit, "\thcrRegisterGlobal($1, \"cmdCount\", sizeof(cmd_count), NULL, (void**)&cmd_count);$N", [osModulePath]) + addf(g.mainDatInit, "\thcrRegisterGlobal($1, \"cmdLine\", sizeof(cmd_line), NULL, (void**)&cmd_line);$N", [osModulePath]) + add(g.mainDatInit, "\t*cmd_count = cmdCount;\n") + add(g.mainDatInit, "\t*cmd_line = cmdLine;\n") + else: + add(m.s[cfsInitProc], hcr_module_meta) + return + if m.s[cfsDatInitProc].len > 0: - let datInit = m.module.getDatInitName addf(g.mainModProcs, "N_LIB_PRIVATE N_NIMCALL(void, $1)(void);$N", [datInit]) addf(g.mainDatInit, "\t$1();$N", [datInit]) @@ -1287,7 +1419,6 @@ proc registerModuleToMain(g: BModuleList; m: BModule) = add(g.mainDatInit, ropecg(m, "\t#initStackBottomWith((void *)&inner);$N")) if m.s[cfsInitProc].len > 0: - let init = m.module.getInitName addf(g.mainModProcs, "N_LIB_PRIVATE N_NIMCALL(void, $1)(void);$N", [init]) let initCall = "\t$1();$N" % [init] if sfMainModule in m.module.flags: @@ -1302,10 +1433,14 @@ proc genDatInitCode(m: BModule) = ## it means raising dependency on the symbols is too late as it will not propogate ## into other modules, only simple rope manipulations are allowed - var moduleDatInitRequired = false + var moduleDatInitRequired = m.hcrOn - var prc = "N_LIB_PRIVATE N_NIMCALL(void, $1)(void) {$N" % - [getDatInitName(m.module)] + var prc = "$1 N_NIMCALL(void, $2)(void) {$N" % + [rope(if m.hcrOn: "N_LIB_EXPORT" else: "N_LIB_PRIVATE"), getDatInitName(m)] + + # we don't want to break into such init code - could happen if a line + # directive from a function written by the user spills after itself + genCLineDir(prc, "generated_not_to_break_here", 999999, m.config) for i in cfsTypeInit1..cfsDynLibInit: if m.s[i].len != 0: @@ -1319,45 +1454,73 @@ proc genDatInitCode(m: BModule) = if moduleDatInitRequired: add(m.s[cfsDatInitProc], prc) +# Very similar to the contents of symInDynamicLib - basically only the +# things needed for the hot code reloading runtime procs to be loaded +proc hcrGetProcLoadCode(m: BModule, sym, prefix, handle, getProcFunc: string): Rope = + let prc = magicsys.getCompilerProc(m.g.graph, sym) + assert prc != nil + fillProcLoc(m, prc.ast[namePos]) + + var extname = prefix & sym + var tmp = mangleDynLibProc(prc) + prc.loc.r = tmp + prc.typ.sym = nil + + if not containsOrIncl(m.declaredThings, prc.id): + addf(m.s[cfsVars], "static $2 $1;$n", [prc.loc.r, getTypeDesc(m, prc.loc.t)]) + + result = "\t$1 = ($2) $3($4, $5);$n" % + [tmp, getTypeDesc(m, prc.typ), getProcFunc.rope, handle.rope, makeCString(prefix & sym)] + proc genInitCode(m: BModule) = ## this function is called in cgenWriteModules after all modules are closed, ## it means raising dependency on the symbols is too late as it will not propogate ## into other modules, only simple rope manipulations are allowed appcg(m, m.s[cfsForwardTypes], frameDefines, [rope("#")]) - var moduleInitRequired = false - let initname = getInitName(m.module) - var prc = "N_LIB_PRIVATE N_NIMCALL(void, $1)(void) {$N" % [initname] + var moduleInitRequired = m.hcrOn + let initname = getInitName(m) + var prc = "$1 N_NIMCALL(void, $2)(void) {$N" % + [rope(if m.hcrOn: "N_LIB_EXPORT" else: "N_LIB_PRIVATE"), initname] + # we don't want to break into such init code - could happen if a line + # directive from a function written by the user spills after itself + genCLineDir(prc, "generated_not_to_break_here", 999999, m.config) if m.typeNodes > 0: - appcg(m, m.s[cfsTypeInit1], "static #TNimNode $1[$2];$n", - [m.typeNodesName, rope(m.typeNodes)]) + if m.hcrOn: + appcg(m, m.s[cfsTypeInit1], "\t#TNimNode* $1;$N", [m.typeNodesName]) + appcg(m, m.s[cfsTypeInit1], "\thcrRegisterGlobal($3, \"$1_$2\", sizeof(TNimNode) * $2, NULL, (void**)&$1);$N", + [m.typeNodesName, rope(m.typeNodes), getModuleDllPath(m, m.module)]) + else: + appcg(m, m.s[cfsTypeInit1], "static #TNimNode $1[$2];$n", + [m.typeNodesName, rope(m.typeNodes)]) if m.nimTypes > 0: appcg(m, m.s[cfsTypeInit1], "static #TNimType $1[$2];$n", [m.nimTypesName, rope(m.nimTypes)]) + if m.hcrOn: + addf(prc, "\tint* nim_hcr_dummy_ = 0;$n" & + "\tNIM_BOOL nim_hcr_do_init_ = " & + "hcrRegisterGlobal($1, \"module_initialized_\", 1, NULL, (void**)&nim_hcr_dummy_);$n", + [getModuleDllPath(m, m.module)]) + + template writeSection(thing: untyped, section: TCProcSection, addHcrGuards = false) = + if m.thing.s(section).len > 0: + moduleInitRequired = true + if addHcrGuards: add(prc, "\tif (nim_hcr_do_init_) {\n\n") + add(prc, genSectionStart(section, m.config)) + add(prc, m.thing.s(section)) + add(prc, genSectionEnd(section, m.config)) + if addHcrGuards: add(prc, "\n\t} // nim_hcr_do_init_\n") + if m.preInitProc.s(cpsInit).len > 0 or m.preInitProc.s(cpsStmts).len > 0: # Give this small function its own scope addf(prc, "{$N", []) # Keep a bogus frame in case the code needs one add(prc, ~"\tTFrame FR_; FR_.len = 0;$N") - if m.preInitProc.s(cpsLocals).len > 0: - moduleInitRequired = true - add(prc, genSectionStart(cpsLocals, m.config)) - add(prc, m.preInitProc.s(cpsLocals)) - add(prc, genSectionEnd(cpsLocals, m.config)) - - if m.preInitProc.s(cpsInit).len > 0: - moduleInitRequired = true - add(prc, genSectionStart(cpsInit, m.config)) - add(prc, m.preInitProc.s(cpsInit)) - add(prc, genSectionEnd(cpsInit, m.config)) - - if m.preInitProc.s(cpsStmts).len > 0: - moduleInitRequired = true - add(prc, genSectionStart(cpsStmts, m.config)) - add(prc, m.preInitProc.s(cpsStmts)) - add(prc, genSectionEnd(cpsStmts, m.config)) + writeSection(preInitProc, cpsLocals) + writeSection(preInitProc, cpsInit, m.hcrOn) + writeSection(preInitProc, cpsStmts) addf(prc, "}$N", []) # add new scope for following code, because old vcc compiler need variable @@ -1367,11 +1530,7 @@ proc genInitCode(m: BModule) = moduleInitRequired = true add(prc, initGCFrame(m.initProc)) - if m.initProc.s(cpsLocals).len > 0: - moduleInitRequired = true - add(prc, genSectionStart(cpsLocals, m.config)) - add(prc, m.initProc.s(cpsLocals)) - add(prc, genSectionEnd(cpsLocals, m.config)) + writeSection(initProc, cpsLocals) if m.initProc.s(cpsInit).len > 0 or m.initProc.s(cpsStmts).len > 0: moduleInitRequired = true @@ -1385,13 +1544,8 @@ proc genInitCode(m: BModule) = else: add(prc, ~"\tTFrame FR_; FR_.len = 0;$N") - add(prc, genSectionStart(cpsInit, m.config)) - add(prc, m.initProc.s(cpsInit)) - add(prc, genSectionEnd(cpsInit, m.config)) - - add(prc, genSectionStart(cpsStmts, m.config)) - add(prc, m.initProc.s(cpsStmts)) - add(prc, genSectionEnd(cpsStmts, m.config)) + writeSection(initProc, cpsInit, m.hcrOn) + writeSection(initProc, cpsStmts) if optStackTrace in m.initProc.options and preventStackTrace notin m.flags: add(prc, deinitFrame(m.initProc)) @@ -1407,6 +1561,19 @@ proc genInitCode(m: BModule) = # that would lead to a *nesting* of merge sections which the merger does # not support. So we add it to another special section: ``cfsInitProc`` + if m.hcrOn: + var procsToLoad = @["hcrRegisterProc", "hcrGetProc", "hcrRegisterGlobal", "hcrGetGlobal"] + + addf(m.s[cfsInitProc], "N_LIB_EXPORT N_NIMCALL(void, $1)(void* handle, N_NIMCALL_PTR(void*, getProcAddr)(void*, char*)) {$N", [getHcrInitName(m)]) + if sfMainModule in m.module.flags: + # additional procs to load + procsToLoad.add("hcrInit") + procsToLoad.add("hcrAddModule") + # load procs + for curr in procsToLoad: + add(m.s[cfsInitProc], hcrGetProcLoadCode(m, curr, "", "handle", "getProcAddr")) + addf(m.s[cfsInitProc], "}$N$N", []) + for i, el in pairs(m.extensionLoaders): if el != nil: let ex = "NIM_EXTERNC N_NIMCALL(void, nimLoadProcs$1)(void) {$2}$N$N" % @@ -1418,6 +1585,12 @@ proc genInitCode(m: BModule) = add(m.s[cfsInitProc], prc) genDatInitCode(m) + + if m.hcrOn: + addf(m.s[cfsInitProc], "N_LIB_EXPORT N_NIMCALL(void, HcrCreateTypeInfos)(void) {$N", []) + add(m.s[cfsInitProc], m.hcrCreateTypeInfosProc) + addf(m.s[cfsInitProc], "}$N$N", []) + registerModuleToMain(m.g, m) proc genModule(m: BModule, cfile: Cfile): Rope = @@ -1445,7 +1618,7 @@ proc genModule(m: BModule, cfile: Cfile): Rope = if m.s[cfsInitProc].len > 0: moduleIsEmpty = false add(result, m.s[cfsInitProc]) - if m.s[cfsDatInitProc].len > 0: + if m.s[cfsDatInitProc].len > 0 or m.hcrOn: moduleIsEmpty = false add(result, m.s[cfsDatInitProc]) @@ -1565,6 +1738,25 @@ when false: readMergeInfo(getCFile(m), m) result = m +proc addHcrInitGuards(p: BProc, n: PNode, inInitGuard: var bool) = + if n.kind == nkStmtList: + for child in n: + addHcrInitGuards(p, child, inInitGuard) + else: + let stmtShouldExecute = n.kind in {nkVarSection, nkLetSection} or + nfExecuteOnReload in n.flags + if inInitGuard: + if stmtShouldExecute: + endBlock(p) + inInitGuard = false + else: + if not stmtShouldExecute: + line(p, cpsStmts, "if (nim_hcr_do_init_)\n") + startBlock(p) + inInitGuard = true + + genStmts(p, n) + proc myProcess(b: PPassContext, n: PNode): PNode = result = n if b == nil: return @@ -1573,8 +1765,11 @@ proc myProcess(b: PPassContext, n: PNode): PNode = m.initProc.options = initProcOptions(m) #softRnl = if optLineDir in m.config.options: noRnl else: rnl # XXX replicate this logic! - let tranformed_n = transformStmt(m.g.graph, m.module, n) - genStmts(m.initProc, tranformed_n) + let transformed_n = transformStmt(m.g.graph, m.module, n) + if m.hcrOn: + addHcrInitGuards(m.initProc, transformed_n, m.inHcrInitGuard) + else: + genStmts(m.initProc, transformed_n) proc shouldRecompile(m: BModule; code: Rope, cfile: Cfile): bool = result = true @@ -1674,7 +1869,24 @@ proc myClose(graph: ModuleGraph; b: PPassContext, n: PNode): PNode = m.initProc.options = initProcOptions(m) genStmts(m.initProc, n) + if m.hcrOn: + # make sure this is pulled in (meaning hcrGetGlobal() is called for it during init) + discard cgsym(m, "programResult") + if m.inHcrInitGuard: + endBlock(m.initProc) + if sfMainModule in m.module.flags: + if m.hcrOn: + # pull ("define" since they are inline when HCR is on) these functions in the main file + # so it can load the HCR runtime and later pass the library handle to the HCR runtime which + # will in turn pass it to the other modules it initializes so they can initialize the + # register/get procs so they don't have to have the definitions of these functions as well + discard cgsym(m, "nimLoadLibrary") + discard cgsym(m, "nimLoadLibraryError") + discard cgsym(m, "nimGetProcAddr") + discard cgsym(m, "procAddrError") + discard cgsym(m, "rawWrite") + # raise dependencies on behalf of genMainProc if m.config.target.targetOS != osStandalone and m.config.selectedGC != gcNone: discard cgsym(m, "initStackBottomWith") @@ -1709,14 +1921,11 @@ proc genForwardedProcs(g: BModuleList) = proc cgenWriteModules*(backend: RootRef, config: ConfigRef) = let g = BModuleList(backend) + g.config = config + # we need to process the transitive closure because recursive module # deps are allowed (and the system module is processed in the wrong # order anyway) - g.config = config - let (outDir, _, _) = splitFile(config.outfile) - if not outDir.isEmpty: - createDir(outDir) - genForwardedProcs(g) for m in cgenModules(g): diff --git a/compiler/cgendata.nim b/compiler/cgendata.nim index 50b484e31c..7b1e7c1233 100644 --- a/compiler/cgendata.nim +++ b/compiler/cgendata.nim @@ -98,6 +98,7 @@ type TTypeSeq* = seq[PType] TypeCache* = Table[SigHash, Rope] + TypeCacheWithOwner* = Table[SigHash, tuple[str: Rope, owner: PSym]] Codegenflag* = enum preventStackTrace, # true if stack traces need to be prevented @@ -118,7 +119,7 @@ type generatedHeader*: BModule breakPointId*: int breakpoints*: Rope # later the breakpoints are inserted into the main proc - typeInfoMarker*: TypeCache + typeInfoMarker*: TypeCacheWithOwner config*: ConfigRef graph*: ModuleGraph strVersion*, seqVersion*: int # version of the string/seq implementation to use @@ -150,6 +151,8 @@ type typeInfoMarker*: TypeCache # needed for generating type information initProc*: BProc # code for init procedure preInitProc*: BProc # code executed before the init proc + hcrCreateTypeInfosProc*: Rope # type info globals are in here when HCR=on + inHcrInitGuard*: bool # We are currently withing a HCR reloading guard. typeStack*: TTypeSeq # used for type generation dataCache*: TNodeTable typeNodes*, nimTypes*: int # used for type info generation @@ -189,8 +192,8 @@ proc newProc*(prc: PSym, module: BModule): BProc = result.sigConflicts = initCountTable[string]() proc newModuleList*(g: ModuleGraph): BModuleList = - BModuleList(typeInfoMarker: initTable[SigHash, Rope](), config: g.config, - graph: g, nimtvDeclared: initIntSet()) + BModuleList(typeInfoMarker: initTable[SigHash, tuple[str: Rope, owner: PSym]](), + config: g.config, graph: g, nimtvDeclared: initIntSet()) iterator cgenModules*(g: BModuleList): BModule = for m in g.modules_closed: diff --git a/compiler/commands.nim b/compiler/commands.nim index 56fe8f2057..1f46abca45 100644 --- a/compiler/commands.nim +++ b/compiler/commands.nim @@ -387,7 +387,12 @@ proc processSwitch*(switch, arg: string, pass: TCmdLinePass, info: TLineInfo; conf.nimcacheDir = processPath(conf, arg, info, true) of "out", "o": expectArg(conf, switch, arg, pass, info) - conf.outFile = AbsoluteFile arg + let f = splitFile(arg.expandTilde) + conf.outFile = RelativeFile f.name + conf.outDir = toAbsoluteDir f.dir + of "outdir": + expectArg(conf, switch, arg, pass, info) + conf.outDir = toAbsoluteDir arg.expandTilde of "docseesrcurl": expectArg(conf, switch, arg, pass, info) conf.docSeeSrcUrl = arg @@ -487,9 +492,13 @@ proc processSwitch*(switch, arg: string, pass: TCmdLinePass, info: TLineInfo; if optMemTracker in conf.options: defineSymbol(conf.symbols, "memtracker") else: undefSymbol(conf.symbols, "memtracker") of "hotcodereloading": - processOnOffSwitch(conf, {optHotCodeReloading}, arg, pass, info) - if optHotCodeReloading in conf.options: defineSymbol(conf.symbols, "hotcodereloading") - else: undefSymbol(conf.symbols, "hotcodereloading") + processOnOffSwitchG(conf, {optHotCodeReloading}, arg, pass, info) + if conf.hcrOn: + defineSymbol(conf.symbols, "hotcodereloading") + defineSymbol(conf.symbols, "useNimRtl") + else: + undefSymbol(conf.symbols, "hotcodereloading") + undefSymbol(conf.symbols, "useNimRtl") of "oldnewlines": case arg.normalize of "","on": @@ -787,7 +796,8 @@ proc processArgument*(pass: TCmdLinePass; p: OptParser; if pass == passCmd1: config.commandArgs.add p.key if argsCount == 1: # support UNIX style filenames everywhere for portable build scripts: - config.projectName = unixToNativePath(p.key) + if config.projectName.len == 0: + config.projectName = unixToNativePath(p.key) config.arguments = cmdLineRest(p) result = true inc argsCount diff --git a/compiler/condsyms.nim b/compiler/condsyms.nim index 5e7ce3a081..57dd551324 100644 --- a/compiler/condsyms.nim +++ b/compiler/condsyms.nim @@ -82,7 +82,9 @@ proc initDefines*(symbols: StringTableRef) = defineSymbol("nimHasTypeof") defineSymbol("nimErrorProcCanHaveBody") defineSymbol("nimHasInstantiationOfInMacro") + defineSymbol("nimHasHotCodeReloading") defineSymbol("nimHasNilSeqs") + defineSymbol("nimHasSignatureHashInMacro") for f in low(Feature)..high(Feature): defineSymbol("nimHas" & $f) diff --git a/compiler/docgen.nim b/compiler/docgen.nim index 5af4c464e1..28d53533f7 100644 --- a/compiler/docgen.nim +++ b/compiler/docgen.nim @@ -100,14 +100,12 @@ proc parseRst(text, filename: string, proc getOutFile2(conf: ConfigRef; filename: RelativeFile, ext: string, dir: RelativeDir; guessTarget: bool): AbsoluteFile = if optWholeProject in conf.globalOptions: - # This is correct, for 'nim doc --project' we interpret the '--out' option as an - # absolute directory, not as a filename! - let d = if conf.outFile.isEmpty: conf.projectPath / dir else: AbsoluteDir(conf.outFile) + let d = if conf.outDir.isEmpty: conf.projectPath / dir else: conf.outDir createDir(d) result = d / changeFileExt(filename, ext) elif guessTarget: - let d = if not conf.outFile.isEmpty: splitFile(conf.outFile).dir - else: conf.projectPath + let d = if not conf.outDir.isEmpty: conf.outDir + else: conf.projectPath createDir(d) result = d / changeFileExt(filename, ext) else: @@ -952,9 +950,8 @@ proc genOutFile(d: PDoc): Rope = proc generateIndex*(d: PDoc) = if optGenIndex in d.conf.globalOptions: - let dir = if d.conf.outFile.isEmpty: d.conf.projectPath / RelativeDir"htmldocs" - elif optWholeProject in d.conf.globalOptions: AbsoluteDir(d.conf.outFile) - else: AbsoluteDir(d.conf.outFile.string.splitFile.dir) + let dir = if not d.conf.outDir.isEmpty: d.conf.outDir + else: d.conf.projectPath / RelativeDir"htmldocs" createDir(dir) let dest = dir / changeFileExt(relativeTo(AbsoluteFile d.filename, d.conf.projectPath), IndexExt) @@ -995,6 +992,7 @@ proc writeOutputJson*(d: PDoc, useWarning = false) = "\" for writing") proc commandDoc*(cache: IdentCache, conf: ConfigRef) = + conf.outDir = AbsoluteDir(conf.outDir / conf.outFile) var ast = parseFile(conf.projectMainIdx, cache, conf) if ast == nil: return var d = newDocumentor(conf.projectFull, cache, conf) diff --git a/compiler/docgen2.nim b/compiler/docgen2.nim index f2ac6c1a9d..2232601f4f 100644 --- a/compiler/docgen2.nim +++ b/compiler/docgen2.nim @@ -29,7 +29,6 @@ template shouldProcess(g): bool = template closeImpl(body: untyped) {.dirty.} = var g = PGen(p) let useWarning = sfMainModule notin g.module.flags - #echo g.module.name.s, " ", g.module.owner.id, " ", gMainPackageId if shouldProcess(g): body try: diff --git a/compiler/extccomp.nim b/compiler/extccomp.nim index 2c5af6433b..18a81ca967 100644 --- a/compiler/extccomp.nim +++ b/compiler/extccomp.nim @@ -14,7 +14,7 @@ import ropes, os, strutils, osproc, platform, condsyms, options, msgs, - lineinfos, std / sha1, streams, pathutils + lineinfos, std / sha1, streams, pathutils, sequtils, times type TInfoCCProp* = enum # properties of the C compiler: @@ -468,8 +468,8 @@ proc execExternalProgram*(conf: ConfigRef; cmd: string, msg = hintExecuting) = rawMessage(conf, errGenerated, "execution of an external program failed: '$1'" % cmd) -proc generateScript(conf: ConfigRef; projectFile: AbsoluteFile, script: Rope) = - let (_, name, _) = splitFile(projectFile) +proc generateScript(conf: ConfigRef; script: Rope) = + let (_, name, _) = splitFile(conf.outFile.string) let filename = getNimcacheDir(conf) / RelativeFile(addFileExt("compile_" & name, platform.OS[conf.target.targetOS].scriptExt)) if writeRope(script, filename): @@ -551,14 +551,14 @@ proc getLinkerExe(conf: ConfigRef; compiler: TSystemCC): string = elif optMixedMode in conf.globalOptions and conf.cmd != cmdCompileToCpp: CC[compiler].cppCompiler else: getCompilerExe(conf, compiler, AbsoluteFile"") -proc getCompileCFileCmd*(conf: ConfigRef; cfile: Cfile): string = +proc getCompileCFileCmd*(conf: ConfigRef; cfile: Cfile, isMainFile = false): string = var c = conf.cCompiler var options = cFileSpecificOptions(conf, cfile.cname) var exe = getConfigVar(conf, c, ".exe") if exe.len == 0: exe = getCompilerExe(conf, c, cfile.cname) if needsExeExt(conf): exe = addFileExt(exe, "exe") - if optGenDynLib in conf.globalOptions and + if (optGenDynLib in conf.globalOptions or (conf.hcrOn and not isMainFile)) and ospNeedsPIC in platform.OS[conf.target.targetOS].props: add(options, ' ' & CC[c].pic) @@ -648,12 +648,14 @@ proc addExternalFileToCompile*(conf: ConfigRef; filename: AbsoluteFile) = flags: {CfileFlag.External}) addExternalFileToCompile(conf, c) -proc compileCFile(conf: ConfigRef; list: CFileList, script: var Rope, cmds: var TStringSeq, +proc compileCFiles(conf: ConfigRef; list: CFileList, script: var Rope, cmds: var TStringSeq, prettyCmds: var TStringSeq) = + var currIdx = 0 for it in list: # call the C compiler for the .c file: if it.flags.contains(CfileFlag.Cached): continue - var compileCmd = getCompileCFileCmd(conf, it) + var compileCmd = getCompileCFileCmd(conf, it, currIdx == list.len - 1) + inc currIdx if optCompileOnly notin conf.globalOptions: add(cmds, compileCmd) let (_, name, _) = splitFile(it.cname) @@ -662,7 +664,8 @@ proc compileCFile(conf: ConfigRef; list: CFileList, script: var Rope, cmds: var add(script, compileCmd) add(script, "\n") -proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): string = +proc getLinkCmd(conf: ConfigRef; output: AbsoluteFile, + objfiles: string, isDllBuild: bool): string = if optGenStaticLib in conf.globalOptions: var libname: string if not conf.outFile.isEmpty: @@ -672,7 +675,7 @@ proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): s else: libname = (libNameTmpl(conf) % splitFile(conf.projectName).name) result = CC[conf.cCompiler].buildLib % ["libfile", quoteShell(libname), - "objfiles", objfiles] + "objfiles", objfiles] else: var linkerExe = getConfigVar(conf, conf.cCompiler, ".linkerexe") if len(linkerExe) == 0: linkerExe = getLinkerExe(conf, conf.cCompiler) @@ -684,28 +687,16 @@ proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): s CC[conf.cCompiler].buildGui else: "" - var exefile, builddll: string - if optGenDynLib in conf.globalOptions: - exefile = platform.OS[conf.target.targetOS].dllFrmt % splitFile(projectfile).name - builddll = CC[conf.cCompiler].buildDll - else: - exefile = splitFile(projectfile).name & platform.OS[conf.target.targetOS].exeExt - builddll = "" - if not conf.outFile.isEmpty: - exefile = conf.outFile.string.expandTilde - if not exefile.isAbsolute(): - exefile = getCurrentDir() / exefile - if not noAbsolutePaths(conf): - if not exefile.isAbsolute(): - exefile = string(splitFile(projectfile).dir / RelativeFile(exefile)) + let builddll = if isDllBuild: CC[conf.cCompiler].buildDll else: "" + let exefile = quoteShell(output) + when false: if optCDebug in conf.globalOptions: writeDebugInfo(exefile.changeFileExt("ndb")) - exefile = quoteShell(exefile) # Map files are required by Nintendo Switch compilation. They are a list # of all function calls in the library and where they come from. - let mapfile = quoteShell(getNimcacheDir(conf) / RelativeFile(splitFile(projectFile).name & ".map")) + let mapfile = quoteShell(getNimcacheDir(conf) / RelativeFile(splitFile(output).name & ".map")) let linkOptions = getLinkOptions(conf) & " " & getConfigVar(conf, conf.cCompiler, ".options.linker") @@ -723,6 +714,46 @@ proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): s "objfiles", objfiles, "exefile", exefile, "nim", quoteShell(getPrefixDir(conf)), "lib", quoteShell(conf.libpath)]) + # On windows the debug information for binaries is emitted in a separate .pdb + # file and the binaries (.dll and .exe) contain a full path to that .pdb file. + # This is a problem for hot code reloading because even when we copy the .dll + # and load the copy so the build process may overwrite the original .dll on + # the disk (windows locks the files of running binaries) the copy still points + # to the original .pdb (and a simple copy of the .pdb won't help). This is a + # problem when a debugger is attached to the program we are hot-reloading. + # This problem is nonexistent on Unix since there by default debug symbols + # are embedded in the binaries so loading a copy of a .so will be fine. There + # is the '/Z7' flag for the MSVC compiler to embed the debug info of source + # files into their respective .obj files but the linker still produces a .pdb + # when a final .dll or .exe is linked so the debug info isn't embedded. + # There is also the issue that even when a .dll is unloaded the debugger + # still keeps the .pdb for that .dll locked. This is a major problem and + # because of this we cannot just alternate between 2 names for a .pdb file + # when rebuilding a .dll - instead we need to accumulate differently named + # .pdb files in the nimcache folder - this is the easiest and most reliable + # way of being able to debug and rebuild the program at the same time. This + # is accomplished using the /PDB: flag (there also exists the + # /PDBALTPATH: flag). The only downside is that the .pdb files are + # atleast 5-10mb big and will quickly accumulate. There is a hacky solution: + # we could try to delete all .pdb files with a pattern and swallow exceptions. + # + # links about .pdb files and hot code reloading: + # https://ourmachinery.com/post/dll-hot-reloading-in-theory-and-practice/ + # https://ourmachinery.com/post/little-machines-working-together-part-2/ + # https://github.com/fungos/cr + # https://fungos.github.io/blog/2017/11/20/cr.h-a-simple-c-hot-reload-header-only-library/ + # on forcing the debugger to unlock a locked .pdb of an unloaded library: + # https://blog.molecular-matters.com/2017/05/09/deleting-pdb-files-locked-by-visual-studio/ + # and a bit about the .pdb format in case that is ever needed: + # https://github.com/crosire/blink + # http://www.debuginfo.com/articles/debuginfomatch.html#pdbfiles + if conf.hcrOn and conf.cCompiler == ccVcc: + let t = now() + let pdb = output.string & "." & format(t, "MMMM-yyyy-HH-mm-") & $t.nanosecond & ".pdb" + result.add " /link /PDB:" & pdb + +template getLinkCmd(conf: ConfigRef; output: AbsoluteFile, objfiles: string): string = + getLinkCmd(conf, output, objfiles, optGenDynLib in conf.globalOptions) template tryExceptOSErrorMessage(conf: ConfigRef; errorPrefix: string = "", body: untyped): typed = try: @@ -799,7 +830,17 @@ proc linkViaResponseFile(conf: ConfigRef; cmd: string) = finally: removeFile(linkerArgs) -proc callCCompiler*(conf: ConfigRef; projectfile: AbsoluteFile) = +proc getObjFilePath(conf: ConfigRef, f: CFile): string = + if noAbsolutePaths(conf): f.obj.extractFilename + else: f.obj.string + +proc hcrLinkTargetName(conf: ConfigRef, objFile: string, isMain = false): AbsoluteFile = + let basename = splitFile(objFile).name + let targetName = if isMain: basename & ".exe" + else: platform.OS[conf.target.targetOS].dllFrmt % basename + result = conf.nimcacheDir / RelativeFile(targetName) + +proc callCCompiler*(conf: ConfigRef) = var linkCmd: string if conf.globalOptions * {optCompileOnly, optGenScript} == {optCompileOnly}: @@ -813,7 +854,7 @@ proc callCCompiler*(conf: ConfigRef; projectfile: AbsoluteFile) = when declared(echo): let cmd = prettyCmds[idx] if cmd != "": echo cmd - compileCFile(conf, conf.toCompile, script, cmds, prettyCmds) + compileCFiles(conf, conf.toCompile, script, cmds, prettyCmds) if optCompileOnly notin conf.globalOptions: execCmdsInParallel(conf, cmds, prettyCb) if optNoLinking notin conf.globalOptions: @@ -824,31 +865,64 @@ proc callCCompiler*(conf: ConfigRef; projectfile: AbsoluteFile) = add(objfiles, ' ') add(objfiles, quoteShell( addFileExt(objFile, CC[conf.cCompiler].objExt))) - for x in conf.toCompile: - let objFile = if noAbsolutePaths(conf): x.obj.extractFilename else: x.obj.string - add(objfiles, ' ') - add(objfiles, quoteShell(objFile)) - linkCmd = getLinkCmd(conf, projectfile, objfiles) - if optCompileOnly notin conf.globalOptions: - if defined(windows) and linkCmd.len > 8_000: - # Windows's command line limit is about 8K (don't laugh...) so C compilers on - # Windows support a feature where the command line can be passed via ``@linkcmd`` - # to them. - linkViaResponseFile(conf, linkCmd) - else: - execLinkCmd(conf, linkCmd) + if conf.hcrOn: # lets assume that optCompileOnly isn't on + cmds = @[] + let mainFileIdx = conf.toCompile.len - 1 + for idx, x in conf.toCompile: + # don't relink each of the many binaries (one for each source file) if the nim code is + # cached because that would take too much time for small changes - the only downside to + # this is that if an external-to-link file changes the final target wouldn't be relinked + if x.flags.contains(CfileFlag.Cached): continue + # we pass each object file as if it is the project file - a .dll will be created for each such + # object file in the nimcache directory, and only in the case of the main project file will + # there be probably an executable (if the project is such) which will be copied out of the nimcache + let objFile = conf.getObjFilePath(x) + let buildDll = idx != mainFileIdx + let linkTarget = conf.hcrLinkTargetName(objFile, not buildDll) + add(cmds, getLinkCmd(conf, linkTarget, objfiles & " " & quoteShell(objFile), buildDll)) + # try to remove all .pdb files for the current binary so they don't accumulate endlessly in the nimcache + # for more info check the comment inside of getLinkCmd() where the /PDB: MSVC flag is used + if conf.cCompiler == ccVcc: + for pdb in walkFiles(objFile & ".*.pdb"): + discard tryRemoveFile(pdb) + # execute link commands in parallel - output will be a bit different + # if it fails than that from execLinkCmd() but that doesn't matter + prettyCmds = map(prettyCmds, proc (curr: string): string = return curr.replace("CC", "Link")) + execCmdsInParallel(conf, cmds, prettyCb) + # only if not cached - copy the resulting main file from the nimcache folder to its originally intended destination + if not conf.toCompile[mainFileIdx].flags.contains(CfileFlag.Cached): + let mainObjFile = getObjFilePath(conf, conf.toCompile[mainFileIdx]) + var src = conf.hcrLinkTargetName(mainObjFile, true) + var dst = conf.prepareToWriteOutput + copyFileWithPermissions(src.string, dst.string) + else: + for x in conf.toCompile: + let objFile = if noAbsolutePaths(conf): x.obj.extractFilename else: x.obj.string + add(objfiles, ' ') + add(objfiles, quoteShell(objFile)) + let mainOutput = if optGenScript notin conf.globalOptions: conf.prepareToWriteOutput + else: AbsoluteFile(conf.projectName) + linkCmd = getLinkCmd(conf, mainOutput, objfiles) + if optCompileOnly notin conf.globalOptions: + if defined(windows) and linkCmd.len > 8_000: + # Windows's command line limit is about 8K (don't laugh...) so C compilers on + # Windows support a feature where the command line can be passed via ``@linkcmd`` + # to them. + linkViaResponseFile(conf, linkCmd) + else: + execLinkCmd(conf, linkCmd) else: linkCmd = "" if optGenScript in conf.globalOptions: add(script, linkCmd) add(script, "\n") - generateScript(conf, projectfile, script) + generateScript(conf, script) #from json import escapeJson import json -proc writeJsonBuildInstructions*(conf: ConfigRef; projectfile: AbsoluteFile) = +proc writeJsonBuildInstructions*(conf: ConfigRef) = template lit(x: untyped) = f.write x template str(x: untyped) = when compiles(escapeJson(x, buf)): @@ -895,7 +969,7 @@ proc writeJsonBuildInstructions*(conf: ConfigRef; projectfile: AbsoluteFile) = var buf = newStringOfCap(50) - let jsonFile = toGeneratedFile(conf, projectfile, "json") + let jsonFile = conf.nimcacheDir / RelativeFile(conf.projectName & ".json") var f: File if open(f, jsonFile.string, fmWrite): @@ -907,7 +981,7 @@ proc writeJsonBuildInstructions*(conf: ConfigRef; projectfile: AbsoluteFile) = linkfiles(conf, f, buf, objfiles, conf.toCompile, conf.externalToLink) lit "],\L\"linkcmd\": " - str getLinkCmd(conf, projectfile, objfiles) + str getLinkCmd(conf, conf.absOutFile, objfiles) lit "\L}\L" close(f) diff --git a/compiler/jsgen.nim b/compiler/jsgen.nim index cad8fc9909..e3ca6830c1 100644 --- a/compiler/jsgen.nim +++ b/compiler/jsgen.nim @@ -83,7 +83,6 @@ type forwarded: seq[PSym] generatedSyms: IntSet typeInfoGenerated: IntSet - classes: seq[(PType, Rope)] unique: int # for temp identifier generation PProc = ref TProc @@ -133,7 +132,6 @@ proc newGlobals(): PGlobals = result.forwarded = @[] result.generatedSyms = initIntSet() result.typeInfoGenerated = initIntSet() - result.classes = @[] proc initCompRes(r: var TCompRes) = r.address = nil @@ -250,7 +248,7 @@ proc mangleName(m: BModule, s: PSym): Rope = result = rope(x) # From ES5 on reserved words can be used as object field names if s.kind != skField: - if optHotCodeReloading in m.config.options: + if m.config.hcrOn: # When hot reloading is enabled, we must ensure that the names # of functions and types will be preserved across rebuilds: add(result, idOrSig(s, m.module.name.s, m.sigConflicts)) @@ -1206,19 +1204,8 @@ proc genAddr(p: PProc, n: PNode, r: var TCompRes) = gen(p, n.sons[0].sons[0], r) else: internalError(p.config, n.sons[0].info, "genAddr: " & $n.sons[0].kind) -proc thisParam(p: PProc; typ: PType): PType = - discard - proc attachProc(p: PProc; content: Rope; s: PSym) = - let otyp = thisParam(p, s.typ) - if otyp != nil: - for i, cls in p.g.classes: - if sameType(cls[0], otyp): - add(p.g.classes[i][1], content) - return - p.g.classes.add((otyp, content)) - else: - add(p.g.code, content) + add(p.g.code, content) proc attachProc(p: PProc; s: PSym) = let newp = genProc(p, s) @@ -1460,9 +1447,6 @@ proc genInfixCall(p: PProc, n: PNode, r: var TCompRes) = genArgs(p, n, r, 2) proc genCall(p: PProc, n: PNode, r: var TCompRes) = - if n.sons[0].kind == nkSym and thisParam(p, n.sons[0].typ) != nil: - genInfixCall(p, n, r) - return gen(p, n.sons[0], r) genArgs(p, n, r) if n.typ != nil: @@ -1606,13 +1590,14 @@ proc genVarInit(p: PProc, v: PSym, n: PNode) = s: Rope varCode: string varName = mangleName(p.module, v) - useReloadingGuard = sfGlobal in v.flags and optHotCodeReloading in p.config.options + useReloadingGuard = sfGlobal in v.flags and p.config.hcrOn if v.constraint.isNil: if useReloadingGuard: lineF(p, "var $1;$n", varName) lineF(p, "if ($1 === undefined) {$n", varName) varCode = $varName + inc p.extraIndent else: varCode = "var $2" else: @@ -1664,6 +1649,7 @@ proc genVarInit(p: PProc, v: PSym, n: PNode) = lineF(p, varCode & " = $3;$n", [returnType, v.loc.r, s]) if useReloadingGuard: + dec p.extraIndent lineF(p, "}$n") proc genVarStmt(p: PProc, n: PNode) = @@ -2204,7 +2190,7 @@ proc genProc(oldProc: PProc, prc: PSym): Rope = else: result = ~"\L" - if optHotCodeReloading in p.config.options: + if p.config.hcrOn: # Here, we introduce thunks that create the equivalent of a jump table # for all global functions, because references to them may be stored # in JavaScript variables. The added indirection ensures that such @@ -2440,13 +2426,52 @@ proc genHeader(): Rope = "if (typeof Float64Array === 'undefined') Float64Array = Array;$n") % [rope(VersionAsString)] +proc addHcrInitGuards(p: PProc, n: PNode, + moduleLoadedVar: Rope, inInitGuard: var bool) = + if n.kind == nkStmtList: + for child in n: + addHcrInitGuards(p, child, moduleLoadedVar, inInitGuard) + else: + let stmtShouldExecute = n.kind in { + nkProcDef, nkFuncDef, nkMethodDef,nkConverterDef, + nkVarSection, nkLetSection} or nfExecuteOnReload in n.flags + + if inInitGuard: + if stmtShouldExecute: + dec p.extraIndent + line(p, "}\L") + inInitGuard = false + else: + if not stmtShouldExecute: + lineF(p, "if ($1 == undefined) {$n", [moduleLoadedVar]) + inc p.extraIndent + inInitGuard = true + + genStmt(p, n) + proc genModule(p: PProc, n: PNode) = if optStackTrace in p.options: add(p.body, frameCreate(p, makeJSString("module " & p.module.module.name.s), makeJSString(toFilename(p.config, p.module.module.info)))) let n_transformed = transformStmt(p.module.graph, p.module.module, n) - genStmt(p, n_transformed) + if p.config.hcrOn and n.kind == nkStmtList: + let moduleSym = p.module.module + var moduleLoadedVar = rope(moduleSym.name.s) & "_loaded" & + idOrSig(moduleSym, moduleSym.name.s, p.module.sigConflicts) + lineF(p, "var $1;$n", [moduleLoadedVar]) + var inGuardedBlock = false + + addHcrInitGuards(p, n_transformed, moduleLoadedVar, inGuardedBlock) + + if inGuardedBlock: + dec p.extraIndent + line(p, "}\L") + + lineF(p, "$1 = true;$n", [moduleLoadedVar]) + else: + genStmt(p, n_transformed) + if optStackTrace in p.options: add(p.body, frameDestroy(p)) @@ -2487,44 +2512,14 @@ proc getClassName(t: PType): Rope = if s.loc.r != nil: result = s.loc.r else: result = rope(s.name.s) -proc genClass(conf: ConfigRef; obj: PType; content: Rope; ext: string) = - let cls = getClassName(obj) - let t = skipTypes(obj, abstractPtrs) - let extends = if t.kind == tyObject and t.sons[0] != nil: - " extends " & getClassName(t.sons[0]) - else: nil - let result = ("` or `BrowserSync ` -to implement the actual reloading behavior in your project. +Once your code is compiled for hot reloading, the ``nim-livereload`` NPM +package provides a convenient solution for implementing the actual reloading +in the browser using a framework such as [LiveReload](http://livereload.com/) +or [BrowserSync](https://browsersync.io/). DynlibOverride diff --git a/lib/core/hotcodereloading.nim b/lib/core/hotcodereloading.nim new file mode 100644 index 0000000000..8b48b3d691 --- /dev/null +++ b/lib/core/hotcodereloading.nim @@ -0,0 +1,27 @@ +when defined(hotcodereloading): + import + macros + + template beforeCodeReload*(body: untyped) = + hcrAddEventHandler(true, proc = body) {.executeOnReload.} + + template afterCodeReload*(body: untyped) = + hcrAddEventHandler(false, proc = body) {.executeOnReload.} + + macro hasModuleChanged*(module: typed): untyped = + if module.kind != nnkSym or module.symKind != nskModule: + error "hasModuleChanged expects a module symbol", module + return newCall(bindSym"hcrHasModuleChanged", newLit(module.signatureHash)) + + proc hasAnyModuleChanged*(): bool = hcrReloadNeeded() + + when not defined(JS): + template performCodeReload* = hcrPerformCodeReload() + else: + template performCodeReload* = discard +else: + template beforeCodeReload*(body: untyped) = discard + template afterCodeReload*(body: untyped) = discard + template hasModuleChanged*(module: typed): bool = false + proc hasAnyModuleChanged*(): bool = false + template performCodeReload*() = discard diff --git a/lib/core/macros.nim b/lib/core/macros.nim index 461afb963a..8e6b93a116 100644 --- a/lib/core/macros.nim +++ b/lib/core/macros.nim @@ -347,6 +347,13 @@ object doAssert(dumpTypeImpl(b) == t) doAssert(dumpTypeImpl(c) == t) +when defined(nimHasSignatureHashInMacro): + proc signatureHash*(n: NimNode): string {.magic: "NSigHash", noSideEffect.} + ## Returns a stable identifier derived from the signature of a symbol. + ## The signature combines many factors such as the type of the symbol, + ## the owning module of the symbol and others. The same identifier is + ## used in the back-end to produce the mangled symbol name. + proc getTypeImpl*(n: typedesc): NimNode {.magic: "NGetType", noSideEffect.} ## Version of ``getTypeImpl`` which takes a ``typedesc``. diff --git a/lib/core/strs.nim b/lib/core/strs.nim index ccbde76fe1..e55c884935 100644 --- a/lib/core/strs.nim +++ b/lib/core/strs.nim @@ -125,7 +125,7 @@ proc cstrToNimstr(str: cstring): NimStringV2 {.compilerRtl.} = if str == nil: toNimStr(str, 0) else: toNimStr(str, str.len) -proc nimToCStringConv(s: NimStringV2): cstring {.compilerProc, inline.} = +proc nimToCStringConv(s: NimStringV2): cstring {.compilerProc, nonReloadable, inline.} = if s.len == 0: result = cstring"" else: result = cstring(unsafeAddr s.p.data) diff --git a/lib/core/typeinfo.nim b/lib/core/typeinfo.nim index d6dd16b54c..fe958c7f53 100644 --- a/lib/core/typeinfo.nim +++ b/lib/core/typeinfo.nim @@ -27,8 +27,6 @@ include "system/inclrtl.nim" include "system/hti.nim" -import system/indexerrors - {.pop.} type diff --git a/lib/nimbase.h b/lib/nimbase.h index ba42737260..9fd475c85a 100644 --- a/lib/nimbase.h +++ b/lib/nimbase.h @@ -354,7 +354,12 @@ typedef NU8 NU; # endif #endif +// for now there isn't an easy way for C code to reach the program result +// when hot code reloading is ON - users will have to: +// load the nimhcr.dll, get the hcrGetGlobal proc from there and use it +#ifndef NIM_HOT_CODE_RELOADING extern NI nim_program_result; +#endif typedef float NF32; typedef double NF64; diff --git a/lib/nimhcr.nim b/lib/nimhcr.nim new file mode 100644 index 0000000000..f3afac3470 --- /dev/null +++ b/lib/nimhcr.nim @@ -0,0 +1,652 @@ +# +# +# Nim's Runtime Library +# (c) Copyright 2018 Nim Contributors +# +# See the file "copying.txt", included in this +# distribution, for details about the copyright. +# + +## This is the Nim hot code reloading run-time for the native targets. +## +## This minimal dynamic library is not subject to reloading when the +## `hotCodeReloading` build mode is enabled. It's responsible for providing +## a permanent memory location for all globals and procs within a program +## and orchestrating the reloading. For globals, this is easily achieved +## by storing them on the heap. For procs, we produce on the fly simple +## trampolines that can be dynamically overwritten to jump to a different +## target. In the host program, all globals and procs are first registered +## here with ``hcrRegisterGlobal`` and ``hcrRegisterProc`` and then the +## returned permanent locations are used in every reference to these symbols +## onwards. +## +## Detailed description: +## +## When code is compiled with the hotCodeReloading option for native targets +## a couple of things happen for all modules in a project: +## - the useNimRtl option is forced (including when building the HCR runtime too) +## - all modules of a target get built into separate shared libraries +## - the smallest granularity of reloads is modules +## - for each .c (or .cpp) in the corresponding nimcache folder of the project +## a shared object is built with the name of the source file + DLL extension +## - only the main module produces whatever the original project type intends +## (again in nimcache) and is then copied to its original destination +## - linking is done in parallel - just like compilation +## - function calls to functions from the same project go through function pointers: +## - with a few exceptions - see the nonReloadable pragma +## - the forward declarations of the original functions become function +## pointers as static globals with the same names +## - the original function definitions get suffixed with _actual +## - the function pointers get initialized with the address of the corresponding +## function in the DatInit of their module through a call to either hcrRegisterProc +## or hcrGetProc. When being registered, the _actual address is passed to +## hcrRegisterProc and a permanent location is returned and assigned to the pointer. +## This way the implementation (_actual) can change but the address for it +## will be the same - this works by just updating a jump instruction (trampoline). +## For functions from other modules hcrGetProc is used (after they are registered). +## - globals are initialized only once and their state is preserved +## - including locals with the {.global.} pragma +## - their definitions are changed into pointer definitions which are initialized +## in the DatInit() of their module with calls to hcrRegisterGlobal (supplying the +## size of the type that this HCR runtime should allocate) and a bool is returned +## which when true triggers the initialization code for the global (only once). +## Globals from other modules: a global pointer coupled with a hcrGetGlobal call. +## - globals which have already been initialized cannot have their values changed +## by changing their initialization - use a handler or some other mechanism +## - new globals can be introduced when reloading +## - top-level code (global scope) is executed only once - at the first module load +## - the runtime knows every symbol's module owner (globals and procs) +## - both the RTL and HCR shared libraries need to be near the program for execution +## - same folder, in the PATH or LD_LIBRARY_PATH env var, etc (depending on OS) +## - the main module is responsible for initializing the HCR runtime +## - the main module loads the RTL and HCR shared objects +## - after that a call to hcrInit() is done in the main module which triggers +## the loading of all modules the main one imports, and doing that for the +## dependencies of each module recursively. Basically a DFS traversal. +## - then initialization takes place with several passes over all modules: +## - HcrInit - initializes the pointers for HCR procs such as hcrRegisterProc +## - HcrCreateTypeInfos - creates globals which will be referenced in the next pass +## - DatInit - usual dat init + register/get procs and get globals +## - Init - it does the following multiplexed operations: +## - register globals (if already registered - then just retrieve pointer) +## - execute top level scope (only if loaded for the first time) +## - when modules are loaded the originally built shared libraries get copied in +## the same folder and the copies are loaded instead of the original files +## - a module import tree is built in the runtime (and maintained when reloading) +## - hcrPerformCodeReload +## - named `performCodeReload`, requires the hotcodereloading module +## - explicitly called by the user - the current active callstack shouldn't contain +## any functions which are defined in modules that will be reloaded (or crash!). +## The reason is that old dynalic libraries get unloaded. +## Example: +## if A is the main module and it imports B, then only B is reloadable and only +## if when calling hcrPerformCodeReload there is no function defined in B in the +## current active callstack at the point of the call (it has to be done from A) +## - for reloading to take place the user has to have rebuilt parts of the application +## without changes affecting the main module in any way - it shouldn't be rebuilt. +## - to determine what needs to be reloaded the runtime starts traversing the import +## tree from the root and checks the timestamps of the loaded shared objects +## - modules that are no longer referenced are unloaded and cleaned up properly +## - symbols (procs/globals) that have been removed in the code are also cleaned up +## - so changing the init of a global does nothing, but removing it, reloading, +## and then re-introducing it with a new initializer works +## - new modules can be imported, and imports can also be reodereded/removed +## - hcrReloadNeeded() can be used to determine if any module needs reloading +## - named `hasAnyModuleChanged`, requires the hotcodereloading module +## - code in the beforeCodeReload/afterCodeReload handlers is executed on each reload +## - require the hotcodereloading module +## - such handlers can be added and removed +## - before each reload all "beforeCodeReload" handlers are executed and after +## that all handlers (including "after") from the particular module are deleted +## - the order of execution is the same as the order of top-level code execution. +## Example: if A imports B which imports C, then all handlers in C will be executed +## first (from top to bottom) followed by all from B and lastly all from A +## - after the reload all "after" handlers are executed the same way as "before" +## - the handlers for a reloaded module are always removed when reloading and then +## registered when the top-level scope is executed (thanks to `executeOnReload`) +## +## TODO - after first merge in upstream Nim: +## +## - profile +## - build speed with and without hot code reloading - difference should be small +## - runtime degradation of HCR-enabled code - important!!! +## - ARM support for the trampolines +## - investigate: +## - rethink the closure iterators +## - ability to keep old versions of dynamic libraries alive +## - because of async server code +## - perhaps with refcounting of .dlls for unfinished closures +## - linking with static libs +## - all shared objects for each module will (probably) have to link to them +## - state in static libs gets duplicated +## - linking is slow and therefore iteration time suffers +## - have just a single .dll for all .nim files and bulk reload? +## - think about the compile/link/passC/passL/emit/injectStmt pragmas +## - if a passC pragma is introduced (either written or dragged in by a new +## import) the whole command line for compilation changes - for example: +## winlean.nim: {.passC: "-DWIN32_LEAN_AND_MEAN".} +## - play with plugins/dlls/lfIndirect/lfDynamicLib/lfExportLib - shouldn't add an extra '*' +## - everything thread-local related +## - tests +## - add a new travis build matrix entry which builds everything with HCR enabled +## - currently building with useNimRtl is problematic - lots of problems... +## - how to supply the nimrtl/nimhcr shared objects to all test binaries...? +## - think about building to C++ instead of only to C - added type safety +## - run tests through valgrind and the sanitizers! of HUGE importance! +## +## TODO - nice to have cool stuff: +## +## - separate handling of global state for much faster reloading and manipulation +## - imagine sliders in an IDE for tweaking variables +## - perhaps using shared memory +## - multi-dll projects - how everything can be reloaded..? +## - a single HCR instance shared across multiple .dlls +## - instead of having to call hcrPerformCodeReload from a function in each dll +## - which currently renders the main module of each dll not reloadable +## - ability to check with the current callstack if a reload is "legal" +## - if it is in any function which is in a module about to be reloaded ==> error +## - pragma annotations for files - to be excluded from dll shenanigans +## - for such file-global pragmas look at codeReordering or injectStmt +## - how would the initialization order be kept? messy... +## - per function exclude pragmas would be TOO messy and hard... +## - C code calling stable exportc interface of nim code (for bindings) +## - generate proxy functions with the stable names +## - in a non-reloadable part (the main binary) that call the function pointers +## - parameter passing/forwarding - how? use the same trampoline jumping? +## - extracting the dependencies for these stubs/proxies will be hard... +## - changing memory layout of types - detecting this..? +## - implement with registerType() call to HCR runtime...? +## - and checking if a previously registered type matches +## - issue an error +## - or let the user handle this by transferring the state properly +## - perhaps in the before/afterCodeReload handlers +## - optimization: calls to procs within a module (+inlined) to use the _actual versions +## - implement executeOnReload for global vars too - not just statements (and document!) +## - cleanup at shutdown - freeing all globals +## +## TODO - unimportant: +## +## - have a "bad call" trampoline that all no-longer-present functions are routed to call there +## - so the user gets some error msg if he calls a dangling pointer instead of a crash +## - before/afterCodeReload and hasModuleChanged should be accessible only where appropriate +## - nim_program_result is inaccessible in HCR mode from external C code (see nimbase.h) +## - proper .json build file - but the format is different... multiple link commands... +## - avoid registering globals on each loop when using an iterator in global scope +## +## TODO - REPL: +## - proper way (as proposed by Zahary): +## - parse the input code and put everything in global scope except for +## statements with side effects only - those go in afterCodeReload blocks +## - my very hacky idea: just append to a closure iterator the new statements +## followed by a yield statement. So far I can think of 2 problems: +## - import and some other code cannot be written inside of a proc - +## has to be parsed and extracted in the outer scope +## - when new variables are created they are actually locals to the closure +## so the struct for the closure state grows in memory, but it has already +## been allocated when the closure was created with the previous smaller size. +## That would lead to working with memory outside of the initially allocated +## block. Perhaps something can be done about this - some way of re-allocating +## the state and transferring the old... + +when not defined(JS) and (defined(hotcodereloading) or + defined(createNimHcr) or + defined(testNimHcr)): + const + dllExt = when defined(windows): "dll" + elif defined(macosx): "dylib" + else: "so" + type + HcrProcGetter* = proc (libHandle: pointer, procName: cstring): pointer {.nimcall.} + HcrGcMarkerProc = proc () {.nimcall.} + HcrModuleInitializer* = proc () {.nimcall.} + +when defined(createNimHcr): + when system.appType != "lib": + {.error: "This file has to be compiled as a library!".} + + import os, tables, sets, times, strutils, reservedmem, dynlib + + template trace(args: varargs[untyped]) = + when defined(testNimHcr) or defined(traceHcr): + echo args + + proc sanitize(arg: Time): string = + when defined(testNimHcr): return "