mirrors/Nim
1
0
Fork 0
mirror of https://github.com/nim-lang/Nim.git synced 2025-12-28 08:54:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

Initial version of the hot-code reloading support for native targets (#10729)

Browse Source 
* squashed work by Zahary

* squashing a ton of useful history... otherwise rebasing on top of upstream Nim after commit 82c009a2cb would be impossible.

* Code review changes; Working test suite (without code reloading enabled)

* - documentation
- implemented the HCR test - almost works...
- fix the issue on Unix where for executable targets the source file for the main module of a project in nimcache was being overwritten with the binary itself (and thus the actual source code was lost)
- fixing embedded paths to shared objects on unix (the "lib" prefix was being prepended to the entire path instead of just the filename)
- other fixes
- removing unnecessary includes since that file is already included in chcks.nim which is in turn included in system.nim (and previously was getting imported in chcks.nim but then system.nim improts something... and that breaks HCR (perhaps it could be fixed but it would be nice not to import anything in system))

* fix for clang & C++ - explicitly casting a function pointer to void*
more stable mangling of parameter names when HCR is on
the length of the static arrays in the DatInit functions is now part of the name of the variables, so when they get resized they get also recreated
more stable mangling for inline functions - no longer depends on the module which first used them
work on the new complicated HCR test - turned surprisingly complex - WIP
test now successfully passes even when re-running `koch test` (previously when the nimcache wasn't cold that lead to errors)
better documentation
calling setStackBottomWith for PreMain
passes over the HcrInit/DatInit/Init calls of all modules are now in the proper order (first all of one type, then all of the next). Also typeinfo globals are registered (created) in a single pass before the DatInit pass (because of the way generic instantiations are handled)
Fix the test suite execution on macOs
fix for being able to query the program arguments when using HCR on posix!
other fixes

* Bugfix: Fix a compilation error in C++ mode when a function pointer
is converted to a raw pointer

* basic documentation for the new hot code reloading semantics

* Add change log entry

* Don't re-execute the top-level statements while reloading JS code

* fix a number of tests broken in a recent bugfix

* Review changes

* Added {.executeOnReload.} pragma that indicates top-level statements
  that should be executed on each reload. To make this work, I've modified
  the way the `if (hcr_init_) {...}` guards are produced in the init code.
  This still needs more work as the new guards seem to be inserted within
  the previously generated guards.

  This change also removes the need for `lastRegistedGlobal` in nimhcr.

* Implemented the `signatureHash` magic and the `hasModuleChanged` API
  depending on it (the actual logic is not imlemented yet).

* Add the "hcr" prefix to all HCR-related symbols in the system module.
  Added a new `hotcodereloading` module exporting the high-level API to
  the user.

  Besides being more hygienic, this was also required in order to make
  it possible to use macros in the high-level API. Without the split,
  `system` would have to import `macros`, which was going to produce
  the well-known init problems.

* Attempted to solve the "GC markers problem".

  Crashes were expected with the previous code, because the GC markers
  were compiled as normal procs are registered in the GC. When their
  module is unloaded, dangling pointers will remain in the GC tables.
  To solve this issue, I don't register any GC markers when HCR is on,
  but I add them to the HCR globals metadata and I use a single marker
  registed in nimhcr during the initialization of the system module that
  will be responsible for marking all globals.

* fix a compilation error

* - implemented the hasModuleChanged functionality
- tuples can be returned and broken into different vars in global scope
- added comments for the closnig scopes of the if statements in the init proc
- the new executeOnReload pragma works now!
- other fixes

* finally! fixing this hack in a proper way - declaring the destructor out of line (out of the class body) - we no longer need to forward-declare popCurrentExceptionEx

* Force full module parsing

This is a temporary hack that breaks some tests. I'll investigate
later how these can be fixed.

* tuples are now properly handled when global!

* these comments mess up the codegen in debug mode when $n is not actually a new line (or something like that) - these labels are intended only for GOTO labels anyway...

* "solved" the issue with the .pdb locks on windows when a binary is being debugged and hot code reloading is used at the same time

* fixes after rebasing...

* small fixes for the test

* better handling of globals! no more compiler crashes for locals with the global pragma, also simplified code around loops in global scope which have local vars (actually globals)

* we can now use the global pragma even for ... globals!

* the right output

* lets try those boehm GC tests

* after the test is ran it will be at its starting state - no git modifications

* clarification in the docs

* removed unnecessary line directives for forward declarations of functions - they were causing trouble with hot code reloading when no semantic change propagates to the main module but a line directive got changed and thus the main module had to be recompiled since the .c code had changed

* fixed bug! was inserting duplicate keys into the table and later was removing only 1 copy of all the duplicates (after a few reloads)

* no longer breaking into DatInit code when not supposed to

* fixes after rebasing

* yet more fixes after rebasing

* Update jssys.nim

* Rework the HCR path-handling logic

After reviewing the code more carefully, I've noticed that the old logic
will be broken when the user overrides the '--out:f' compiler option.

Besides fixing this issues, I took the opportunity to implement the
missing '--outdir:d' option.

Other changes:

* ./koch test won't overwrite any HCR and RTL builds located in nim/lib
* HCR and RTL are compiled with --threads:on by default

* Clean up the globals registration logic

* Handle non-flattened top-level stmtlists in JS as well

* The HCR is not supported with the Boehm GC yet

Also fixes some typos and the expected output of the HCR integration test

* The GC marker procs are now properly used as trampolines

* Fix the HCR integration test in release builds

* Fix ./koch tools

* this forward declaration doesn't seem to be necessary, and in fact breaks HCR because a 2nd function pointer is emitted for this externed/rtl func

* the forward declaration I removed in the last commit was actually necessary

* Attempt to make all tests green

* Fix tgenscript

* BAT file for running the HCR integration test on Windows [skip ci]

* Fix the docgen tests

* A final fix for Travis (hopefully)
This commit is contained in:
zah
2019-02-26 16:48:55 +02:00
committed by Andreas Rumpf
parent ba38c05eb6
commit ca4b971bc8
84 changed files with 2506 additions and 486 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.gitignore vendored
View File

@@ -11,7 +11,9 @@ dnimcache/
!/icons/*.o
*.obj
*.ilk
*.exp
*.pdb
*.lib
*.dll
*.exe
*.so
@@ -19,7 +21,6 @@ dnimcache/
*.zip
*.iss
*.log
*.ilk
*.pdb
mapping.txt
@@ -42,6 +43,7 @@ bin/*
*.perspectivev3
*.swp
.DS_Store
.tags
project.xcworkspace/
xcuserdata/

10
changelog.md
View File

@@ -136,6 +136,9 @@ proc enumToString*(enums: openArray[enum]): string =
slightly. The `dumpLisp` macro in this module now outputs an
indented proper Lisp, devoid of commas.
- Added `macros.signatureHash` that returns a stable identifier
derived from the signature of a symbol.
- In `strutils` empty strings now no longer matched as substrings
anymore.
@@ -192,6 +195,11 @@ proc enumToString*(enums: openArray[enum]): string =
### Compiler changes
- The deprecated `fmod` proc is now unavailable on the VM'.
- A new `--outdir` option was added.
- The compiled JavaScript file for the project produced by executing `nim js`
will no longer be placed in the nimcache directory.
- The `--hotCodeReloading` has been implemented for the native targets.
The compiler also provides a new more flexible API for handling the
hot code reloading events in the code.
### Bugfixes

12
compiler/ast.nim
View File

@@ -227,7 +227,7 @@ type
TNodeKinds* = set[TNodeKind]
type
TSymFlag* = enum # already 33 flags!
TSymFlag* = enum # already 34 flags!
sfUsed, # read access of sym (for warnings) or simply used
sfExported, # symbol is exported from module
sfFromGeneric, # symbol is instantiation of a generic; this is needed
@@ -276,6 +276,8 @@ type
# the calling side of the macro, not from the
# implementation.
sfGenSym # symbol is 'gensym'ed; do not add to symbol table
sfNonReloadable # symbol will be left as-is when hot code reloading is on -
# meaning that it won't be renamed and/or changed in any way
TSymFlags* = set[TSymFlag]
@@ -468,6 +470,7 @@ type
nfDefaultParam # an automatically inserter default parameter
nfDefaultRefsParam # a default param value references another parameter
# the flag is applied to proc default values and to calls
nfExecuteOnReload # A top-level statement that will be executed during reloads
TNodeFlags* = set[TNodeFlag]
TTypeFlag* = enum # keep below 32 for efficiency reasons (now: beyond that)
@@ -654,7 +657,7 @@ type
mNIntVal, mNFloatVal, mNSymbol, mNIdent, mNGetType, mNStrVal, mNSetIntVal,
mNSetFloatVal, mNSetSymbol, mNSetIdent, mNSetType, mNSetStrVal, mNLineInfo,
mNNewNimNode, mNCopyNimNode, mNCopyNimTree, mStrToIdent,
mNNewNimNode, mNCopyNimNode, mNCopyNimTree, mStrToIdent, mNSigHash,
mNBindSym, mLocals, mNCallSite,
mEqIdent, mEqNimrodNode, mSameNodeType, mGetImpl, mNGenSym,
mNHint, mNWarning, mNError,
@@ -854,7 +857,7 @@ type
offset*: int # offset of record field
loc*: TLoc
annex*: PLib # additional fields (seldom used, so we use a
# reference to another object to safe space)
# reference to another object to save space)
constraint*: PNode # additional constraints like 'lit|result'; also
# misused for the codegenDecl pragma in the hope
# it won't cause problems
@@ -978,7 +981,8 @@ const
PersistentNodeFlags*: TNodeFlags = {nfBase2, nfBase8, nfBase16,
nfDotSetter, nfDotField,
nfIsRef, nfPreventCg, nfLL,
nfFromTemplate, nfDefaultRefsParam}
nfFromTemplate, nfDefaultRefsParam,
nfExecuteOnReload}
namePos* = 0
patternPos* = 1 # empty except for term rewriting macros
genericParamsPos* = 2

6
compiler/ccgcalls.nim
View File

@@ -185,7 +185,7 @@ proc genPrefixCall(p: BProc, le, ri: PNode, d: var TLoc) =
var length = sonsLen(ri)
for i in countup(1, length - 1):
genParamLoop(params)
fixupCall(p, le, ri, d, op.r, params)
fixupCall(p, le, ri, d, rdLoc(op), params)
proc genClosureCall(p: BProc, le, ri: PNode, d: var TLoc) =
@@ -209,7 +209,7 @@ proc genClosureCall(p: BProc, le, ri: PNode, d: var TLoc) =
genParamLoop(pl)
template genCallPattern {.dirty.} =
lineF(p, cpsStmts, callPattern & ";$n", [op.r, pl, pl.addComma, rawProc])
lineF(p, cpsStmts, callPattern & ";$n", [rdLoc(op), pl, pl.addComma, rawProc])
let rawProc = getRawProcType(p, typ)
let callPattern = if tfIterator in typ.flags: PatIter else: PatProc
@@ -237,7 +237,7 @@ proc genClosureCall(p: BProc, le, ri: PNode, d: var TLoc) =
assert(d.t != nil) # generate an assignment to d:
var list: TLoc
initLoc(list, locCall, d.lode, OnUnknown)
list.r = callPattern % [op.r, pl, pl.addComma, rawProc]
list.r = callPattern % [rdLoc(op), pl, pl.addComma, rawProc]
genAssignment(p, d, list, {}) # no need for deep copying
else:
genCallPattern()

22
compiler/ccgexprs.nim
View File

@@ -1534,6 +1534,7 @@ proc genDollar(p: BProc, n: PNode, d: var TLoc, frmt: string) =
var a: TLoc
initLocExpr(p, n.sons[1], a)
a.r = ropecg(p.module, frmt, [rdLoc(a)])
a.flags = a.flags - {lfIndirect} # this flag should not be propagated here (not just for HCR)
if d.k == locNone: getTemp(p, n.typ, d)
genAssignment(p, d, a, {})
gcUsage(p.config, n)
@@ -2034,8 +2035,28 @@ proc genMagicExpr(p: BProc, e: PNode, d: var TLoc, op: TMagic) =
genCall(p, e, d)
of mNewString, mNewStringOfCap, mExit, mParseBiggestFloat:
var opr = e.sons[0].sym
# Why would anyone want to set nodecl to one of these hardcoded magics?
# - not sure, and it wouldn't work if the symbol behind the magic isn't
# somehow forward-declared from some other usage, but it is *possible*
if lfNoDecl notin opr.loc.flags:
let prc = magicsys.getCompilerProc(p.module.g.graph, $opr.loc.r)
# HACK:
# Explicitly add this proc as declared here so the cgsym call doesn't
# add a forward declaration - without this we could end up with the same
# 2 forward declarations. That happens because the magic symbol and the original
# one that shall be used have different ids (even though a call to one is
# actually a call to the other) so checking into m.declaredProtos with the 2 different ids doesn't work.
# Why would 2 identical forward declarations be a problem?
# - in the case of hot code-reloading we generate function pointers instead
# of forward declarations and in C++ it is an error to redefine a global
let wasDeclared = containsOrIncl(p.module.declaredProtos, prc.id)
# Make the function behind the magic get actually generated - this will
# not lead to a forward declaration! The genCall will lead to one.
discard cgsym(p.module, $opr.loc.r)
# make sure we have pointer-initialising code for hot code reloading
if not wasDeclared and p.hcrOn:
addf(p.module.s[cfsDynLibInit], "\t$1 = ($2) hcrGetProc($3, \"$1\");$n",
[mangleDynLibProc(prc), getTypeDesc(p.module, prc.loc.t), getModuleDllPath(p.module, prc)])
genCall(p, e, d)
of mReset: genReset(p, e)
of mEcho: genEcho(p, e[1].skipConv)
@@ -2292,6 +2313,7 @@ proc exprComplexConst(p: BProc, n: PNode, d: var TLoc) =
proc expr(p: BProc, n: PNode, d: var TLoc) =
p.currLineInfo = n.info
case n.kind
of nkSym:
var sym = n.sym

2
compiler/ccgmerge.nim
View File

@@ -31,7 +31,7 @@ const
cfsData: "NIM_merge_DATA",
cfsProcs: "NIM_merge_PROCS",
cfsInitProc: "NIM_merge_INIT_PROC",
cfsDatInitProc: "NIM_merge_DATINIT_PROC",
cfsDatInitProc: "NIM_merge_DATINIT_PROC",
cfsTypeInit1: "NIM_merge_TYPE_INIT1",
cfsTypeInit2: "NIM_merge_TYPE_INIT2",
cfsTypeInit3: "NIM_merge_TYPE_INIT3",

84
compiler/ccgstmts.nim
View File

@@ -15,18 +15,20 @@ const
stringCaseThreshold = 8
# above X strings a hash-switch for strings is generated
proc registerGcRoot(p: BProc, v: PSym) =
proc getTraverseProc(p: BProc, v: Psym): Rope =
if p.config.selectedGC in {gcMarkAndSweep, gcDestructors, gcV2, gcRefc} and
containsGarbageCollectedRef(v.loc.t):
# we register a specialized marked proc here; this has the advantage
# that it works out of the box for thread local storage then :-)
let prc = genTraverseProcForGlobal(p.module, v, v.info)
if sfThread in v.flags:
appcg(p.module, p.module.initProc.procSec(cpsInit),
"#nimRegisterThreadLocalMarker($1);$n", [prc])
else:
appcg(p.module, p.module.initProc.procSec(cpsInit),
"#nimRegisterGlobalMarker($1);$n", [prc])
result = genTraverseProcForGlobal(p.module, v, v.info)
proc registerTraverseProc(p: BProc, v: PSym, traverseProc: Rope) =
if sfThread in v.flags:
appcg(p.module, p.module.initProc.procSec(cpsInit),
"$n\t#nimRegisterThreadLocalMarker($1);$n$n", [traverseProc])
else:
appcg(p.module, p.module.initProc.procSec(cpsInit),
"$n\t#nimRegisterGlobalMarker($1);$n$n", [traverseProc])
proc isAssignedImmediately(conf: ConfigRef; n: PNode): bool {.inline.} =
if n.kind == nkEmpty: return false
@@ -41,6 +43,10 @@ proc inExceptBlockLen(p: BProc): int =
for x in p.nestedTryStmts:
if x.inExcept: result.inc
proc startBlock(p: BProc, start: FormatStr = "{$n",
args: varargs[Rope]): int {.discardable.}
proc endBlock(p: BProc)
proc genVarTuple(p: BProc, n: PNode) =
var tup, field: TLoc
if n.kind != nkVarTuple: internalError(p.config, n.info, "genVarTuple")
@@ -52,6 +58,32 @@ proc genVarTuple(p: BProc, n: PNode) =
genStmts(p, lowerTupleUnpacking(p.module.g.graph, n, p.prc))
return
# check only the first son
var forHcr = treatGlobalDifferentlyForHCR(p.module, n.sons[0].sym)
let hcrCond = if forHcr: getTempName(p.module) else: nil
var hcrGlobals: seq[tuple[loc: TLoc, tp: Rope]]
# determine if the tuple is constructed at top-level scope or inside of a block (if/while/block)
let isGlobalInBlock = forHcr and p.blocks.len > 2
# do not close and reopen blocks if this is a 'global' but inside of a block (if/while/block)
forHcr = forHcr and not isGlobalInBlock
if forHcr:
# check with the boolean if the initializing code for the tuple should be ran
lineCg(p, cpsStmts, "if ($1)$n", hcrCond)
startBlock(p)
defer:
if forHcr:
# end the block where the tuple gets initialized
endBlock(p)
if forHcr or isGlobalInBlock:
# insert the registration of the globals for the different parts of the tuple at the
# start of the current scope (after they have been iterated) and init a boolean to
# check if any of them is newly introduced and the initializing code has to be ran
lineCg(p, cpsLocals, "NIM_BOOL $1 = NIM_FALSE;$n", hcrCond)
for curr in hcrGlobals:
lineCg(p, cpsLocals, "$1 |= hcrRegisterGlobal($4, \"$2\", sizeof($3), $5, (void**)&$2);$N",
hcrCond, curr.loc.r, rdLoc(curr.loc), getModuleDllPath(p.module, n.sons[0].sym), curr.tp)
genLineDir(p, n)
initLocExpr(p, n.sons[L-1], tup)
var t = tup.t.skipTypes(abstractInst)
@@ -59,10 +91,13 @@ proc genVarTuple(p: BProc, n: PNode) =
let vn = n.sons[i]
let v = vn.sym
if sfCompileTime in v.flags: continue
var traverseProc: Rope
if sfGlobal in v.flags:
assignGlobalVar(p, vn)
genObjectInit(p, cpsInit, v.typ, v.loc, true)
registerGcRoot(p, v)
traverseProc = getTraverseProc(p, v)
if traverseProc != nil and not p.hcrOn:
registerTraverseProc(p, v, traverseProc)
else:
assignLocalVar(p, vn)
initLocalVar(p, v, immediateAsgn=isAssignedImmediately(p.config, n[L-1]))
@@ -73,6 +108,8 @@ proc genVarTuple(p: BProc, n: PNode) =
if t.n.sons[i].kind != nkSym: internalError(p.config, n.info, "genVarTuple")
field.r = "$1.$2" % [rdLoc(tup), mangleRecFieldName(p.module, t.n.sons[i].sym)]
putLocIntoDest(p, v.loc, field)
if forHcr or isGlobalInBlock:
hcrGlobals.add((loc: v.loc, tp: if traverseProc == nil: ~"NULL" else: traverseProc))
proc genDeref(p: BProc, e: PNode, d: var TLoc; enforceDeref=false)
@@ -242,6 +279,7 @@ proc genSingleVar(p: BProc, a: PNode) =
genGotoVar(p, a.sons[2])
return
var targetProc = p
var traverseProc: Rope
if sfGlobal in v.flags:
if v.flags * {sfImportc, sfExportc} == {sfImportc} and
a.sons[2].kind == nkEmpty and
@@ -270,7 +308,9 @@ proc genSingleVar(p: BProc, a: PNode) =
# if sfImportc notin v.flags: constructLoc(p.module.preInitProc, v.loc)
if sfExportc in v.flags and p.module.g.generatedHeader != nil:
genVarPrototype(p.module.g.generatedHeader, vn)
registerGcRoot(p, v)
traverseProc = getTraverseProc(p, v)
if traverseProc != nil and not p.hcrOn:
registerTraverseProc(p, v, traverseProc)
else:
let value = a.sons[2]
let imm = isAssignedImmediately(p.config, value)
@@ -302,6 +342,30 @@ proc genSingleVar(p: BProc, a: PNode) =
assignLocalVar(p, vn)
initLocalVar(p, v, imm)
if traverseProc == nil: traverseProc = ~"NULL"
# If the var is in a block (control flow like if/while or a block) in global scope just
# register the so called "global" so it can be used later on. There is no need to close
# and reopen of if (nim_hcr_do_init_) blocks because we are in one already anyway.
var forHcr = treatGlobalDifferentlyForHCR(p.module, v)
if forHcr and targetProc.blocks.len > 3 and v.owner.kind == skModule:
# put it in the locals section - mainly because of loops which
# use the var in a call to resetLoc() in the statements section
lineCg(targetProc, cpsLocals, "hcrRegisterGlobal($3, \"$1\", sizeof($2), $4, (void**)&$1);$n",
v.loc.r, rdLoc(v.loc), getModuleDllPath(p.module, v), traverseProc)
# nothing special left to do later on - let's avoid closing and reopening blocks
forHcr = false
# we close and reopen the global if (nim_hcr_do_init_) blocks in the main Init function
# for the module so we can have globals and top-level code be interleaved and still
# be able to re-run it but without the top level code - just the init of globals
if forHcr:
lineCg(targetProc, cpsStmts, "if (hcrRegisterGlobal($3, \"$1\", sizeof($2), $4, (void**)&$1))$N",
v.loc.r, rdLoc(v.loc), getModuleDllPath(p.module, v), traverseProc)
startBlock(targetProc)
defer:
if forHcr:
endBlock(targetProc)
if a.sons[2].kind != nkEmpty:
genLineDir(targetProc, a)
loadInto(targetProc, a.sons[0], a.sons[2], v.loc)

21
compiler/ccgtrav.nim
View File

@@ -134,11 +134,13 @@ proc genTraverseProc(m: BModule, origTyp: PType; sig: SigHash): Rope =
var c: TTraversalClosure
var p = newProc(nil, m)
result = "Marker_" & getTypeName(m, origTyp, sig)
var typ = origTyp.skipTypes(abstractInstOwned)
let
hcrOn = m.hcrOn
typ = origTyp.skipTypes(abstractInstOwned)
markerName = if hcrOn: result & "_actual" else: result
header = "static N_NIMCALL(void, $1)(void* p, NI op)" % [markerName]
t = getTypeDesc(m, typ)
let header = "static N_NIMCALL(void, $1)(void* p, NI op)" % [result]
let t = getTypeDesc(m, typ)
lineF(p, cpsLocals, "$1 a;$n", [t])
lineF(p, cpsInit, "a = ($1)p;$n", [t])
@@ -155,18 +157,23 @@ proc genTraverseProc(m: BModule, origTyp: PType; sig: SigHash): Rope =
else:
genTraverseProc(c, "(*a)".rope, typ.sons[0])
let generatedProc = "$1 {$n$2$3$4}$n" %
let generatedProc = "$1 {$n$2$3$4}\n" %
[header, p.s(cpsLocals), p.s(cpsInit), p.s(cpsStmts)]
m.s[cfsProcHeaders].addf("$1;$n", [header])
m.s[cfsProcHeaders].addf("$1;\n", [header])
m.s[cfsProcs].add(generatedProc)
if hcrOn:
addf(m.s[cfsProcHeaders], "N_NIMCALL_PTR(void, $1)(void*, NI);\n", [result])
addf(m.s[cfsDynLibInit], "\t$1 = (N_NIMCALL_PTR(void, )(void*, NI)) hcrRegisterProc($3, \"$1\", (void*)$2);\n",
[result, markerName, getModuleDllPath(m)])
proc genTraverseProcForGlobal(m: BModule, s: PSym; info: TLineInfo): Rope =
discard genTypeInfo(m, s.loc.t, info)
var c: TTraversalClosure
var p = newProc(nil, m)
var sLoc = s.loc.r
var sLoc = rdLoc(s.loc)
result = getTempName(m)
if sfThread in s.flags and emulatedThreadVars(m.config):

151
compiler/ccgtypes.nim
View File

@@ -14,7 +14,7 @@
import sighashes
from lowerings import createObj
proc genProcHeader(m: BModule, prc: PSym): Rope
proc genProcHeader(m: BModule, prc: PSym, asPtr: bool = false): Rope
proc isKeyword(w: PIdent): bool =
# Nim and C++ share some keywords
@@ -59,7 +59,23 @@ proc mangleParamName(m: BModule; s: PSym): Rope =
result = s.loc.r
if result == nil:
var res = s.name.s.mangle
if isKeyword(s.name) or m.g.config.cppDefines.contains(res):
# Take into account if HCR is on because of the following scenario:
# if a module gets imported and it has some more importc symbols in it,
# some param names might recieve the "_0" suffix to distinguish from what
# is newly available. That might lead to changes in the C code in nimcache
# that contain only a parameter name change, but that is enough to mandate
# recompilation of that source file and thus a new shared object will be
# relinked. That may lead to a module getting reloaded which wasn't intended
# and that may be fatal when parts of the current active callstack when
# performCodeReload() was called are from the module being reloaded
# unintentionally - example (3 modules which import one another):
# main => proxy => reloadable
# we call performCodeReload() in proxy to reload only changes in reloadable
# but there is a new import which introduces an importc symbol `socket`
# and a function called in main or proxy uses `socket` as a parameter name.
# That would lead to either needing to reload `proxy` or to overwrite the
# executable file for the main module, which is running (or both!) -> error.
if m.hcrOn or isKeyword(s.name) or m.g.config.cppDefines.contains(res):
res.add "_0"
result = res.rope
s.loc.r = result
@@ -507,6 +523,8 @@ proc fillObjectFields*(m: BModule; typ: PType) =
var check = initIntSet()
discard getRecordFields(m, typ, check)
proc mangleDynLibProc(sym: PSym): Rope
proc getRecordDesc(m: BModule, typ: PType, name: Rope,
check: var IntSet): Rope =
# declare the record:
@@ -535,22 +553,13 @@ proc getRecordDesc(m: BModule, typ: PType, name: Rope,
appcg(m, result, " : public $1 {$n",
[getTypeDescAux(m, typ.sons[0].skipTypes(skipPtrs), check)])
if typ.isException:
appcg(m, result, "virtual void raise() {throw *this;}$n") # required for polymorphic exceptions
appcg(m, result, "virtual void raise() { throw *this; }$n") # required for polymorphic exceptions
if typ.sym.magic == mException:
# Add cleanup destructor to Exception base class
appcg(m, result, "~$1() {if(this->raiseId) #popCurrentExceptionEx(this->raiseId);}$n", [name])
# hack: forward declare popCurrentExceptionEx() on top of type description,
# proper request to generate popCurrentExceptionEx not possible for 2 reasons:
# generated function will be below declared Exception type and circular dependency
# between Exception and popCurrentExceptionEx function
let popExSym = magicsys.getCompilerProc(m.g.graph, "popCurrentExceptionEx")
if lfDynamicLib in popExSym.loc.flags and sfImportc in popExSym.flags:
# echo popExSym.flags, " ma flags ", popExSym.loc.flags
result = "extern " & getTypeDescAux(m, popExSym.typ, check) & " " &
mangleName(m, popExSym) & ";\L" & result
else:
result = genProcHeader(m, popExSym) & ";\L" & result
appcg(m, result, "~$1();$n", [name])
# define it out of the class body and into the procs section so we don't have to
# artificially forward-declare popCurrentExceptionEx (very VERY troublesome for HCR)
appcg(m, cfsProcs, "inline $1::~$1() {if(this->raiseId) #popCurrentExceptionEx(this->raiseId);}$n", [name])
hasField = true
else:
appcg(m, result, " {$n $1 Sup;$n",
@@ -888,31 +897,43 @@ proc finishTypeDescriptions(m: BModule) =
template cgDeclFrmt*(s: PSym): string = s.constraint.strVal
proc genProcHeader(m: BModule, prc: PSym): Rope =
proc isReloadable(m: BModule, prc: PSym): bool =
return m.hcrOn and sfNonReloadable notin prc.flags
proc isNonReloadable(m: BModule, prc: PSym): bool =
return m.hcrOn and sfNonReloadable in prc.flags
proc genProcHeader(m: BModule, prc: PSym, asPtr: bool = false): Rope =
var
rettype, params: Rope
genCLineDir(result, prc.info, m.config)
# using static is needed for inline procs
if lfExportLib in prc.loc.flags:
if isHeaderFile in m.flags:
result.add "N_LIB_IMPORT "
else:
result.add "N_LIB_EXPORT "
elif prc.typ.callConv == ccInline:
elif prc.typ.callConv == ccInline or asPtr or isNonReloadable(m, prc):
result.add "static "
elif {sfImportc, sfExportc} * prc.flags == {}:
result.add "N_LIB_PRIVATE "
var check = initIntSet()
fillLoc(prc.loc, locProc, prc.ast[namePos], mangleName(m, prc), OnUnknown)
genProcParams(m, prc.typ, rettype, params, check)
# handle the 2 options for hotcodereloading codegen - function pointer
# (instead of forward declaration) or header for function budy with "_actual" postfix
let asPtrStr = rope(if asPtr: "_PTR" else: "")
var name = prc.loc.r
if isReloadable(m, prc) and not asPtr:
add(name, "_actual")
# careful here! don't access ``prc.ast`` as that could reload large parts of
# the object graph!
if prc.constraint.isNil:
addf(result, "$1($2, $3)$4",
[rope(CallingConvToStr[prc.typ.callConv]), rettype, prc.loc.r,
addf(result, "$1$2($3, $4)$5",
[rope(CallingConvToStr[prc.typ.callConv]), asPtrStr, rettype, name,
params])
else:
result = prc.cgDeclFrmt % [rettype, prc.loc.r, params]
let asPtrStr = if asPtr: (rope("(*") & name & ")") else: name
result = prc.cgDeclFrmt % [rettype, asPtrStr, params]
# ------------------ type info generation -------------------------------------
@@ -921,6 +942,9 @@ proc getNimNode(m: BModule): Rope =
result = "$1[$2]" % [m.typeNodesName, rope(m.typeNodes)]
inc(m.typeNodes)
proc TINameForHcr(m: BModule, name: Rope): Rope =
return if m.hcrOn: "(*".rope & name & ")" else: name
proc genTypeInfoAuxBase(m: BModule; typ, origType: PType;
name, base: Rope; info: TLineInfo) =
var nimtypeKind: int
@@ -930,19 +954,21 @@ proc genTypeInfoAuxBase(m: BModule; typ, origType: PType;
else:
nimtypeKind = ord(typ.kind)
let nameHcr = TINameForHcr(m, name)
var size: Rope
if tfIncompleteStruct in typ.flags: size = rope"void*"
else: size = getTypeDesc(m, origType)
addf(m.s[cfsTypeInit3],
"$1.size = sizeof($2);$n" & "$1.kind = $3;$n" & "$1.base = $4;$n",
[name, size, rope(nimtypeKind), base])
[nameHcr, size, rope(nimtypeKind), base])
# compute type flags for GC optimization
var flags = 0
if not containsGarbageCollectedRef(typ): flags = flags or 1
if not canFormAcycle(typ): flags = flags or 2
#else MessageOut("can contain a cycle: " & typeToString(typ))
if flags != 0:
addf(m.s[cfsTypeInit3], "$1.flags = $2;$n", [name, rope(flags)])
addf(m.s[cfsTypeInit3], "$1.flags = $2;$n", [nameHcr, rope(flags)])
discard cgsym(m, "TNimType")
if isDefined(m.config, "nimTypeNames"):
var typename = typeToString(if origType.typeInst != nil: origType.typeInst
@@ -950,11 +976,17 @@ proc genTypeInfoAuxBase(m: BModule; typ, origType: PType;
if typename == "ref object" and origType.skipTypes(skipPtrs).sym != nil:
typename = "anon ref object from " & m.config$origType.skipTypes(skipPtrs).sym.info
addf(m.s[cfsTypeInit3], "$1.name = $2;$n",
[name, makeCstring typename])
[nameHcr, makeCstring typename])
discard cgsym(m, "nimTypeRoot")
addf(m.s[cfsTypeInit3], "$1.nextType = nimTypeRoot; nimTypeRoot=&$1;$n",
[name])
addf(m.s[cfsVars], "TNimType $1;$n", [name])
[nameHcr])
if m.hcrOn:
addf(m.s[cfsVars], "static TNimType* $1;$n", [name])
addf(m.hcrCreateTypeInfosProc, "\thcrRegisterGlobal($2, \"$1\", sizeof(TNimType), NULL, (void**)&$1);$n",
[name, getModuleDllPath(m, m.module)])
else:
addf(m.s[cfsVars], "TNimType $1;$n", [name])
proc genTypeInfoAux(m: BModule, typ, origType: PType, name: Rope;
info: TLineInfo) =
@@ -984,6 +1016,14 @@ proc discriminatorTableDecl(m: BModule, objtype: PType, d: PSym): Rope =
var tmp = discriminatorTableName(m, objtype, d)
result = "TNimNode* $1[$2];$n" % [tmp, rope(lengthOrd(m.config, d.typ)+1)]
proc genTNimNodeArray(m: BModule, name: Rope, size: Rope) =
if m.hcrOn:
addf(m.s[cfsVars], "static TNimNode** $1;$n", [name])
addf(m.hcrCreateTypeInfosProc, "\thcrRegisterGlobal($3, \"$1\", sizeof(TNimNode*) * $2, NULL, (void**)&$1);$n",
[name, size, getModuleDllPath(m, m.module)])
else:
addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n", [name, size])
proc genObjectFields(m: BModule, typ, origType: PType, n: PNode, expr: Rope;
info: TLineInfo) =
case n.kind
@@ -992,8 +1032,8 @@ proc genObjectFields(m: BModule, typ, origType: PType, n: PNode, expr: Rope;
if L == 1:
genObjectFields(m, typ, origType, n.sons[0], expr, info)
elif L > 0:
var tmp = getTempName(m)
addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n", [tmp, rope(L)])
var tmp = getTempName(m) & "_" & $L
genTNimNodeArray(m, tmp, rope(L))
for i in countup(0, L-1):
var tmp2 = getNimNode(m)
addf(m.s[cfsTypeInit3], "$1[$2] = &$3;$n", [tmp, rope(i), tmp2])
@@ -1066,7 +1106,7 @@ proc genObjectInfo(m: BModule, typ, origType: PType, name: Rope; info: TLineInfo
var tmp = getNimNode(m)
if not isImportedType(typ):
genObjectFields(m, typ, origType, typ.n, tmp, info)
addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [name, tmp])
addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [TINameForHcr(m, name), tmp])
var t = typ.sons[0]
while t != nil:
t = t.skipTypes(skipPtrs)
@@ -1078,8 +1118,8 @@ proc genTupleInfo(m: BModule, typ, origType: PType, name: Rope; info: TLineInfo)
var expr = getNimNode(m)
var length = sonsLen(typ)
if length > 0:
var tmp = getTempName(m)
addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n", [tmp, rope(length)])
var tmp = getTempName(m) & "_" & $length
genTNimNodeArray(m, tmp, rope(length))
for i in countup(0, length - 1):
var a = typ.sons[i]
var tmp2 = getNimNode(m)
@@ -1094,7 +1134,7 @@ proc genTupleInfo(m: BModule, typ, origType: PType, name: Rope; info: TLineInfo)
else:
addf(m.s[cfsTypeInit3], "$1.len = $2; $1.kind = 2;$n",
[expr, rope(length)])
addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [name, expr])
addf(m.s[cfsTypeInit3], "$1.node = &$2;$n", [TINameForHcr(m, name), expr])
proc genEnumInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) =
# Type information for enumerations is quite heavy, so we do some
@@ -1102,10 +1142,9 @@ proc genEnumInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) =
# anyway. We generate a cstring array and a loop over it. Exceptional
# positions will be reset after the loop.
genTypeInfoAux(m, typ, typ, name, info)
var nodePtrs = getTempName(m)
var length = sonsLen(typ.n)
addf(m.s[cfsTypeInit1], "static TNimNode* $1[$2];$n",
[nodePtrs, rope(length)])
var nodePtrs = getTempName(m) & "_" & $length
genTNimNodeArray(m, nodePtrs, rope(length))
var enumNames, specialCases: Rope
var firstNimNode = m.typeNodes
var hasHoles = false
@@ -1134,17 +1173,17 @@ proc genEnumInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) =
add(m.s[cfsTypeInit3], specialCases)
addf(m.s[cfsTypeInit3],
"$1.len = $2; $1.kind = 2; $1.sons = &$3[0];$n$4.node = &$1;$n",
[getNimNode(m), rope(length), nodePtrs, name])
[getNimNode(m), rope(length), nodePtrs, TINameForHcr(m, name)])
if hasHoles:
# 1 << 2 is {ntfEnumHole}
addf(m.s[cfsTypeInit3], "$1.flags = 1<<2;$n", [name])
addf(m.s[cfsTypeInit3], "$1.flags = 1<<2;$n", [TINameForHcr(m, name)])
proc genSetInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) =
assert(typ.sons[0] != nil)
genTypeInfoAux(m, typ, typ, name, info)
var tmp = getNimNode(m)
addf(m.s[cfsTypeInit3], "$1.len = $2; $1.kind = 0;$n" & "$3.node = &$1;$n",
[tmp, rope(firstOrd(m.config, typ)), name])
[tmp, rope(firstOrd(m.config, typ)), TINameForHcr(m, name)])
proc genArrayInfo(m: BModule, typ: PType, name: Rope; info: TLineInfo) =
genTypeInfoAuxBase(m, typ, typ, name, genTypeInfo(m, typ.sons[1], info), info)
@@ -1169,19 +1208,29 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope =
let origType = t
var t = skipTypes(origType, irrelevantForBackend + tyUserTypeClasses)
let prefixTI = if m.hcrOn: "(" else: "(&"
let sig = hashType(origType)
result = m.typeInfoMarker.getOrDefault(sig)
if result != nil:
return "(&".rope & result & ")".rope
return prefixTI.rope & result & ")".rope
result = m.g.typeInfoMarker.getOrDefault(sig)
if result != nil:
proc declareNimType(m: BModule, str: Rope, ownerModule: PSym) =
if m.hcrOn:
addf(m.s[cfsVars], "static TNimType* $1;$n", [str])
addf(m.s[cfsTypeInit1], "\t$1 = (TNimType*)hcrGetGlobal($2, \"$1\");$n",
[str, getModuleDllPath(m, ownerModule)])
else:
addf(m.s[cfsVars], "extern TNimType $1;$n", [str])
let marker = m.g.typeInfoMarker.getOrDefault(sig)
if marker.str != nil:
discard cgsym(m, "TNimType")
discard cgsym(m, "TNimNode")
addf(m.s[cfsVars], "extern TNimType $1;$n", [result])
declareNimType(m, marker.str, marker.owner)
# also store in local type section:
m.typeInfoMarker[sig] = result
return "(&".rope & result & ")".rope
m.typeInfoMarker[sig] = marker.str
return prefixTI.rope & marker.str & ")".rope
result = "NTI$1_" % [rope($sig)]
m.typeInfoMarker[sig] = result
@@ -1194,10 +1243,10 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope =
# reference the type info as extern here
discard cgsym(m, "TNimType")
discard cgsym(m, "TNimNode")
addf(m.s[cfsVars], "extern TNimType $1;$n", [result])
return "(&".rope & result & ")".rope
declareNimType(m, result, owner)
return prefixTI.rope & result & ")".rope
m.g.typeInfoMarker[sig] = result
m.g.typeInfoMarker[sig] = (str: result, owner: owner)
case t.kind
of tyEmpty, tyVoid: result = rope"0"
of tyPointer, tyBool, tyChar, tyCString, tyString, tyInt..tyUInt64, tyVar, tyLent:
@@ -1219,12 +1268,12 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope =
if m.config.selectedGC != gcDestructors:
if m.config.selectedGC >= gcMarkAndSweep:
let markerProc = genTraverseProc(m, origType, sig)
addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [result, markerProc])
addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [TINameForHcr(m, result), markerProc])
of tyRef:
genTypeInfoAux(m, t, t, result, info)
if m.config.selectedGC >= gcMarkAndSweep:
let markerProc = genTraverseProc(m, origType, sig)
addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [result, markerProc])
addf(m.s[cfsTypeInit3], "$1.marker = $2;$n", [TINameForHcr(m, result), markerProc])
of tyPtr, tyRange, tyUncheckedArray: genTypeInfoAux(m, t, t, result, info)
of tyArray: genArrayInfo(m, t, result, info)
of tySet: genSetInfo(m, t, result, info)
@@ -1241,7 +1290,7 @@ proc genTypeInfo(m: BModule, t: PType; info: TLineInfo): Rope =
genDeepCopyProc(m, t.deepCopy, result)
elif origType.deepCopy != nil:
genDeepCopyProc(m, origType.deepCopy, result)
result = "(&".rope & result & ")".rope
result = prefixTI.rope & result & ")".rope
proc genTypeSection(m: BModule, n: PNode) =
discard

407
compiler/cgen.nim
View File

@@ -16,8 +16,6 @@ import
condsyms, rodutils, renderer, idgen, cgendata, ccgmerge, semfold, aliases,
lowerings, tables, sets, ndi, lineinfos, pathutils, transf
import system/indexerrors
when not defined(leanCompiler):
import semparallel
@@ -46,6 +44,9 @@ when options.hasTinyCBackend:
# implementation
proc hcrOn(m: BModule): bool = m.config.hcrOn
proc hcrOn(p: BProc): bool = p.module.config.hcrOn
proc addForwardedProc(m: BModule, prc: PSym) =
m.g.forwardedProcs.add(prc)
@@ -92,6 +93,17 @@ proc useHeader(m: BModule, sym: PSym) =
proc cgsym(m: BModule, name: string): Rope
proc getCFile(m: BModule): AbsoluteFile
proc getModuleDllPath(m: BModule): Rope =
let (dir, name, ext) = splitFile(getCFile(m))
let filename = strutils.`%`(platform.OS[m.g.config.target.targetOS].dllFrmt, [name & ext])
return makeCString(dir.string & "/" & filename)
proc getModuleDllPath(m: BModule, s: PSym): Rope =
return getModuleDllPath(findPendingModule(m, s))
# TODO: please document
proc ropecg(m: BModule, frmt: FormatStr, args: varargs[Rope]): Rope =
assert m != nil
var i = 0
@@ -440,12 +452,18 @@ proc assignLocalVar(p: BProc, n: PNode) =
include ccgthreadvars
proc varInDynamicLib(m: BModule, sym: PSym)
proc mangleDynLibProc(sym: PSym): Rope
proc treatGlobalDifferentlyForHCR(m: BModule, s: PSym): bool =
return m.hcrOn and {sfThread, sfGlobal} * s.flags == {sfGlobal} and
({lfNoDecl, lfHeader} * s.loc.flags == {})
# and s.owner.kind == skModule # owner isn't always a module (global pragma on local var)
# and s.loc.k == locGlobalVar # loc isn't always initialized when this proc is used
proc assignGlobalVar(p: BProc, n: PNode) =
let s = n.sym
if s.loc.k == locNone:
fillLoc(s.loc, locGlobalVar, n, mangleName(p.module, s), OnHeap)
if treatGlobalDifferentlyForHCR(p.module, s): incl(s.loc.flags, lfIndirect)
if lfDynamicLib in s.loc.flags:
var q = findPendingModule(p.module, s)
@@ -463,8 +481,10 @@ proc assignGlobalVar(p: BProc, n: PNode) =
var decl: Rope = nil
var td = getTypeDesc(p.module, s.loc.t)
if s.constraint.isNil:
if sfImportc in s.flags: add(decl, "extern ")
if p.hcrOn: add(decl, "static ")
elif sfImportc in s.flags: add(decl, "extern ")
add(decl, td)
if p.hcrOn: add(decl, "*")
if sfRegister in s.flags: add(decl, " register")
if sfVolatile in s.flags: add(decl, " volatile")
addf(decl, " $1;$n", [s.loc.r])
@@ -853,6 +873,14 @@ proc allPathsAsgnResult(n: PNode): InitResultEnum =
for i in 0..<safeLen(n):
allPathsInBranch(n[i])
proc getProcTypeCast(m: BModule, prc: PSym): Rope =
result = getTypeDesc(m, prc.loc.t)
if prc.typ.callConv == ccClosure:
var rettype, params: Rope
var check = initIntSet()
genProcParams(m, prc.typ, rettype, params, check)
result = "$1(*)$2" % [rettype, params]
proc genProcAux(m: BModule, prc: PSym) =
var p = newProc(prc, m)
var header = genProcHeader(m, prc)
@@ -931,6 +959,9 @@ proc genProcAux(m: BModule, prc: PSym) =
add(generatedProc, returnStmt)
add(generatedProc, ~"}$N")
add(m.s[cfsProcs], generatedProc)
if isReloadable(m, prc):
addf(m.s[cfsDynLibInit], "\t$1 = ($3) hcrRegisterProc($4, \"$1\", (void*)$2);$n",
[prc.loc.r, prc.loc.r & "_actual", getProcTypeCast(m, prc), getModuleDllPath(m, prc)])
proc requiresExternC(m: BModule; sym: PSym): bool {.inline.} =
result = (sfCompileToCpp in m.module.flags and
@@ -946,20 +977,27 @@ proc genProcPrototype(m: BModule, sym: PSym) =
if lfDynamicLib in sym.loc.flags:
if getModule(sym).id != m.module.id and
not containsOrIncl(m.declaredThings, sym.id):
add(m.s[cfsVars], ropecg(m, "extern $1 $2;$n",
add(m.s[cfsVars], ropecg(m, "$1 $2 $3;$n",
rope(if isReloadable(m, sym): "static" else: "extern"),
getTypeDesc(m, sym.loc.t), mangleDynLibProc(sym)))
if isReloadable(m, sym):
addf(m.s[cfsDynLibInit], "\t$1 = ($2) hcrGetProc($3, \"$1\");$n",
[mangleDynLibProc(sym), getTypeDesc(m, sym.loc.t), getModuleDllPath(m, sym)])
elif not containsOrIncl(m.declaredProtos, sym.id):
var header = genProcHeader(m, sym)
if sfNoReturn in sym.flags and hasDeclspec in extccomp.CC[m.config.cCompiler].props:
header = "__declspec(noreturn) " & header
if sym.typ.callConv != ccInline and requiresExternC(m, sym):
header = "extern \"C\" " & header
if sfPure in sym.flags and hasAttribute in CC[m.config.cCompiler].props:
header.add(" __attribute__((naked))")
if sfNoReturn in sym.flags and hasAttribute in CC[m.config.cCompiler].props:
header.add(" __attribute__((noreturn))")
add(m.s[cfsProcHeaders], ropecg(m, "$1;$n", header))
let asPtr = isReloadable(m, sym)
var header = genProcHeader(m, sym, asPtr)
if not asPtr:
if sfNoReturn in sym.flags and hasDeclspec in extccomp.CC[m.config.cCompiler].props:
header = "__declspec(noreturn) " & header
if sym.typ.callConv != ccInline and requiresExternC(m, sym):
header = "extern \"C\" " & header
if sfPure in sym.flags and hasAttribute in CC[m.config.cCompiler].props:
header.add(" __attribute__((naked))")
if sfNoReturn in sym.flags and hasAttribute in CC[m.config.cCompiler].props:
header.add(" __attribute__((noreturn))")
add(m.s[cfsProcHeaders], ropecg(m, "$1;$N", header))
# TODO: figure out how to rename this - it DOES generate a forward declaration
proc genProcNoForward(m: BModule, prc: PSym) =
if lfImportCompilerProc in prc.loc.flags:
fillProcLoc(m, prc.ast[namePos])
@@ -969,7 +1007,6 @@ proc genProcNoForward(m: BModule, prc: PSym) =
return
if lfNoDecl in prc.loc.flags:
fillProcLoc(m, prc.ast[namePos])
useHeader(m, prc)
genProcPrototype(m, prc)
elif prc.typ.callConv == ccInline:
# We add inline procs to the calling module to enable C based inlining.
@@ -977,30 +1014,45 @@ proc genProcNoForward(m: BModule, prc: PSym) =
# a check for ``m.declaredThings``.
if not containsOrIncl(m.declaredThings, prc.id):
#if prc.loc.k == locNone:
fillProcLoc(m, prc.ast[namePos])
# mangle the inline proc based on the module where it is defined - not on the first module that uses it
fillProcLoc(findPendingModule(m, prc), prc.ast[namePos])
#elif {sfExportc, sfImportc} * prc.flags == {}:
# # reset name to restore consistency in case of hashing collisions:
# echo "resetting ", prc.id, " by ", m.module.name.s
# prc.loc.r = nil
# prc.loc.r = mangleName(m, prc)
useHeader(m, prc)
genProcPrototype(m, prc)
genProcAux(m, prc)
elif lfDynamicLib in prc.loc.flags:
var q = findPendingModule(m, prc)
fillProcLoc(q, prc.ast[namePos])
useHeader(m, prc)
genProcPrototype(m, prc)
if q != nil and not containsOrIncl(q.declaredThings, prc.id):
symInDynamicLib(q, prc)
# register the procedure even though it is in a different dynamic library and will not be
# reloadable (and has no _actual suffix) - other modules will need to be able to get it through
# the hcr dynlib (also put it in the DynLibInit section - right after it gets loaded)
if isReloadable(q, prc):
addf(q.s[cfsDynLibInit], "\t$1 = ($2) hcrRegisterProc($3, \"$1\", (void*)$1);$n",
[prc.loc.r, getTypeDesc(q, prc.loc.t), getModuleDllPath(m, q.module)])
else:
symInDynamicLibPartial(m, prc)
elif sfImportc notin prc.flags:
var q = findPendingModule(m, prc)
fillProcLoc(q, prc.ast[namePos])
useHeader(m, prc)
# generate a getProc call to initialize the pointer for this
# externally-to-the-current-module defined proc, also important
# to do the declaredProtos check before the call to genProcPrototype
if isReloadable(m, prc) and prc.id notin m.declaredProtos and
q != nil and q.module.id != m.module.id:
addf(m.s[cfsDynLibInit], "\t$1 = ($2) hcrGetProc($3, \"$1\");$n",
[prc.loc.r, getProcTypeCast(m, prc), getModuleDllPath(m, prc)])
genProcPrototype(m, prc)
if q != nil and not containsOrIncl(q.declaredThings, prc.id):
# make sure there is a "prototype" in the external module
# which will actually become a function pointer
if isReloadable(m, prc):
genProcPrototype(q, prc)
genProcAux(q, prc)
else:
fillProcLoc(m, prc.ast[namePos])
@@ -1049,6 +1101,8 @@ proc genVarPrototype(m: BModule, n: PNode) =
let sym = n.sym
useHeader(m, sym)
fillLoc(sym.loc, locGlobalVar, n, mangleName(m, sym), OnHeap)
if treatGlobalDifferentlyForHCR(m, sym): incl(sym.loc.flags, lfIndirect)
if (lfNoDecl in sym.loc.flags) or contains(m.declaredThings, sym.id):
return
if sym.owner.id != m.module.id:
@@ -1057,12 +1111,17 @@ proc genVarPrototype(m: BModule, n: PNode) =
if sfThread in sym.flags:
declareThreadVar(m, sym, true)
else:
add(m.s[cfsVars], "extern ")
incl(m.declaredThings, sym.id)
add(m.s[cfsVars], if m.hcrOn: "static " else: "extern ")
add(m.s[cfsVars], getTypeDesc(m, sym.loc.t))
if m.hcrOn: add(m.s[cfsVars], "*")
if lfDynamicLib in sym.loc.flags: add(m.s[cfsVars], "*")
if sfRegister in sym.flags: add(m.s[cfsVars], " register")
if sfVolatile in sym.flags: add(m.s[cfsVars], " volatile")
addf(m.s[cfsVars], " $1;$n", [sym.loc.r])
if m.hcrOn: addf(m.initProc.procSec(cpsLocals),
"\t$1 = ($2*)hcrGetGlobal($3, \"$1\");$n", [sym.loc.r,
getTypeDesc(m, sym.loc.t), getModuleDllPath(m, sym)])
const
frameDefines = """
@@ -1106,22 +1165,76 @@ proc getCopyright(conf: ConfigRef; cfile: Cfile): Rope =
proc getFileHeader(conf: ConfigRef; cfile: Cfile): Rope =
result = getCopyright(conf, cfile)
if conf.hcrOn: add(result, "#define NIM_HOT_CODE_RELOADING\L")
addIntTypes(result, conf)
proc getSomeNameForModule(m: PSym): Rope =
assert m.kind == skModule
assert m.owner.kind == skPackage
if {sfSystemModule, sfMainModule} * m.flags == {}:
result = m.owner.name.s.mangle.rope
result.add "_"
result.add m.name.s.mangle
proc getSomeInitName(m: BModule, suffix: string): Rope =
if not m.hcrOn:
result = getSomeNameForModule(m.module)
result.add suffix
proc getInitName(m: BModule): Rope =
if sfMainModule in m.module.flags:
# generate constant name for main module, for "easy" debugging.
result = rope"NimMainModule"
else:
result = getSomeInitName(m, "Init000")
proc getDatInitName(m: BModule): Rope = getSomeInitName(m, "DatInit000")
proc getHcrInitName(m: BModule): Rope = getSomeInitName(m, "HcrInit000")
proc hcrGetProcLoadCode(m: BModule, sym, prefix, handle, getProcFunc: string): Rope
proc genMainProc(m: BModule) =
## this function is called in cgenWriteModules after all modules are closed,
## it means raising dependency on the symbols is too late as it will not propogate
## into other modules, only simple rope manipulations are allowed
const
var preMainCode: Rope
if m.hcrOn:
proc loadLib(handle: string, name: string): Rope =
let prc = magicsys.getCompilerProc(m.g.graph, name)
assert prc != nil
let n = newStrNode(nkStrLit, prc.annex.path.strVal)
n.info = prc.annex.path.info
appcg(m, result, "\tif (!($1 = #nimLoadLibrary($2)))$N" &
"\t\t#nimLoadLibraryError($2);$N",
[handle.rope, genStringLiteral(m, n)])
add(preMainCode, loadLib("hcr_handle", "hcrGetProc"))
add(preMainCode, "\tvoid* rtl_handle;$N")
add(preMainCode, loadLib("rtl_handle", "nimGC_setStackBottom"))
add(preMainCode, hcrGetProcLoadCode(m, "nimGC_setStackBottom", "nimrtl_", "rtl_handle", "nimGetProcAddr"))
add(preMainCode, "\tinner = PreMain;$N")
add(preMainCode, "\tinitStackBottomWith_actual((void *)&inner);$N")
add(preMainCode, "\t(*inner)();$N")
else:
add(preMainCode, "\tPreMain();$N")
let
# not a big deal if we always compile these 3 global vars... makes the HCR code easier
PosixCmdLine =
"int cmdCount;$N" &
"char** cmdLine;$N" &
"char** gEnv;$N"
# The use of a volatile function pointer to call Pre/NimMainInner
# prevents inlining of the NimMainInner function and dependent
# functions, which might otherwise merge their stack frames.
PreMainBody =
PreMainBody = "$N" &
"void PreMainInner(void) {$N" &
"$2" &
"$3" &
"}$N$N" &
PosixCmdLine &
"void PreMain(void) {$N" &
"\tvoid (*volatile inner)(void);$N" &
"\tinner = PreMainInner;$N" &
@@ -1133,7 +1246,7 @@ proc genMainProc(m: BModule) =
"\tNimMain();$N"
MainProcsWithResult =
MainProcs & "\treturn nim_program_result;$N"
MainProcs & ("\treturn " & (if m.hcrOn: "*" else: "") & "nim_program_result;$N")
NimMainInner = "N_CDECL(void, NimMainInner)(void) {$N" &
"$1" &
@@ -1142,7 +1255,7 @@ proc genMainProc(m: BModule) =
NimMainProc =
"N_CDECL(void, NimMain)(void) {$N" &
"\tvoid (*volatile inner)(void);$N" &
"\tPreMain();$N" &
$preMainCode &
"\tinner = NimMainInner;$N" &
"$2" &
"\t(*inner)();$N" &
@@ -1150,12 +1263,6 @@ proc genMainProc(m: BModule) =
NimMainBody = NimMainInner & NimMainProc
PosixNimMain =
"int cmdCount;$N" &
"char** cmdLine;$N" &
"char** gEnv;$N" &
NimMainBody
PosixCMain =
"int main(int argc, char** args, char** env) {$N" &
"\tcmdLine = args;$N" &
@@ -1227,10 +1334,10 @@ proc genMainProc(m: BModule) =
nimMain = PosixNimDllMain
otherMain = PosixCDllMain
elif m.config.target.targetOS == osStandalone:
nimMain = PosixNimMain
nimMain = NimMainBody
otherMain = StandaloneCMain
else:
nimMain = PosixNimMain
nimMain = NimMainBody
otherMain = PosixCMain
if optEndb in m.config.options:
for i in 0..<m.config.m.fileInfos.len:
@@ -1253,28 +1360,53 @@ proc genMainProc(m: BModule) =
appcg(m, m.s[cfsProcs], otherMain, [])
if m.config.cppCustomNamespace.len > 0: m.s[cfsProcs].add openNamespaceNim(m.config.cppCustomNamespace)
proc getSomeInitName(m: PSym, suffix: string): Rope =
assert m.kind == skModule
assert m.owner.kind == skPackage
if {sfSystemModule, sfMainModule} * m.flags == {}:
result = m.owner.name.s.mangle.rope
result.add "_"
result.add m.name.s.mangle
result.add suffix
proc getInitName(m: PSym): Rope =
if sfMainModule in m.flags:
# generate constant name for main module, for "easy" debugging.
result = rope"NimMainModule"
else:
result = getSomeInitName(m, "Init000")
proc getDatInitName(m: PSym): Rope = getSomeInitName(m, "DatInit000")
proc registerModuleToMain(g: BModuleList; m: BModule) =
let
init = m.getInitName
datInit = m.getDatInitName
if m.hcrOn:
var hcr_module_meta = "$nN_LIB_PRIVATE const char* hcr_module_list[] = {$n" % []
let systemModulePath = getModuleDllPath(m, g.modules[g.graph.config.m.systemFileIdx.int].module)
let mainModulePath = getModuleDllPath(m, m.module)
if sfMainModule in m.module.flags:
addf(hcr_module_meta, "\t$1,$n", [systemModulePath])
g.graph.importDeps.withValue(FileIndex(m.module.position), deps):
for curr in deps[]:
addf(hcr_module_meta, "\t$1,$n", [getModuleDllPath(m, g.modules[curr.int].module)])
addf(hcr_module_meta, "\t\"\"};$n", [])
addf(hcr_module_meta, "$nN_LIB_EXPORT N_NIMCALL(void**, HcrGetImportedModules)() { return (void**)hcr_module_list; }$n", [])
addf(hcr_module_meta, "$nN_LIB_EXPORT N_NIMCALL(char*, HcrGetSigHash)() { return \"$1\"; }$n$n",
[($sigHash(m.module)).rope])
if sfMainModule in m.module.flags:
add(g.mainModProcs, hcr_module_meta)
addf(g.mainModProcs, "static void* hcr_handle;$N", [])
addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, $1)(void);$N", [init])
addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, $1)(void);$N", [datInit])
addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, $1)(void*, N_NIMCALL_PTR(void*, getProcAddr)(void*, char*));$N", [m.getHcrInitName])
addf(g.mainModProcs, "N_LIB_EXPORT N_NIMCALL(void, HcrCreateTypeInfos)(void);$N", [])
addf(g.mainModInit, "\t$1();$N", [init])
addf(g.otherModsInit, "\thcrInit((void**)hcr_module_list, $1, $2, $3, hcr_handle, nimGetProcAddr);$n",
[mainModulePath, systemModulePath, datInit])
addf(g.mainDatInit, "\t$1(hcr_handle, nimGetProcAddr);$N", [m.getHcrInitName])
addf(g.mainDatInit, "\thcrAddModule($1);\n", [mainModulePath])
addf(g.mainDatInit, "\tHcrCreateTypeInfos();$N", [])
# nasty nasty hack to get the command line functionality working with HCR
# register the 2 variables on behalf of the os module which might not even
# be loaded (in which case it will get collected but that is not a problem)
let osModulePath = ($systemModulePath).replace("stdlib_system", "stdlib_os").rope
addf(g.mainDatInit, "\thcrAddModule($1);\n", [osModulePath])
add(g.mainDatInit, "\tint* cmd_count;\n")
add(g.mainDatInit, "\tchar*** cmd_line;\n")
addf(g.mainDatInit, "\thcrRegisterGlobal($1, \"cmdCount\", sizeof(cmd_count), NULL, (void**)&cmd_count);$N", [osModulePath])
addf(g.mainDatInit, "\thcrRegisterGlobal($1, \"cmdLine\", sizeof(cmd_line), NULL, (void**)&cmd_line);$N", [osModulePath])
add(g.mainDatInit, "\t*cmd_count = cmdCount;\n")
add(g.mainDatInit, "\t*cmd_line = cmdLine;\n")
else:
add(m.s[cfsInitProc], hcr_module_meta)
return
if m.s[cfsDatInitProc].len > 0:
let datInit = m.module.getDatInitName
addf(g.mainModProcs, "N_LIB_PRIVATE N_NIMCALL(void, $1)(void);$N", [datInit])
addf(g.mainDatInit, "\t$1();$N", [datInit])
@@ -1287,7 +1419,6 @@ proc registerModuleToMain(g: BModuleList; m: BModule) =
add(g.mainDatInit, ropecg(m, "\t#initStackBottomWith((void *)&inner);$N"))
if m.s[cfsInitProc].len > 0:
let init = m.module.getInitName
addf(g.mainModProcs, "N_LIB_PRIVATE N_NIMCALL(void, $1)(void);$N", [init])
let initCall = "\t$1();$N" % [init]
if sfMainModule in m.module.flags:
@@ -1302,10 +1433,14 @@ proc genDatInitCode(m: BModule) =
## it means raising dependency on the symbols is too late as it will not propogate
## into other modules, only simple rope manipulations are allowed
var moduleDatInitRequired = false
var moduleDatInitRequired = m.hcrOn
var prc = "N_LIB_PRIVATE N_NIMCALL(void, $1)(void) {$N" %
[getDatInitName(m.module)]
var prc = "$1 N_NIMCALL(void, $2)(void) {$N" %
[rope(if m.hcrOn: "N_LIB_EXPORT" else: "N_LIB_PRIVATE"), getDatInitName(m)]
# we don't want to break into such init code - could happen if a line
# directive from a function written by the user spills after itself
genCLineDir(prc, "generated_not_to_break_here", 999999, m.config)
for i in cfsTypeInit1..cfsDynLibInit:
if m.s[i].len != 0:
@@ -1319,45 +1454,73 @@ proc genDatInitCode(m: BModule) =
if moduleDatInitRequired:
add(m.s[cfsDatInitProc], prc)
# Very similar to the contents of symInDynamicLib - basically only the
# things needed for the hot code reloading runtime procs to be loaded
proc hcrGetProcLoadCode(m: BModule, sym, prefix, handle, getProcFunc: string): Rope =
let prc = magicsys.getCompilerProc(m.g.graph, sym)
assert prc != nil
fillProcLoc(m, prc.ast[namePos])
var extname = prefix & sym
var tmp = mangleDynLibProc(prc)
prc.loc.r = tmp
prc.typ.sym = nil
if not containsOrIncl(m.declaredThings, prc.id):
addf(m.s[cfsVars], "static $2 $1;$n", [prc.loc.r, getTypeDesc(m, prc.loc.t)])
result = "\t$1 = ($2) $3($4, $5);$n" %
[tmp, getTypeDesc(m, prc.typ), getProcFunc.rope, handle.rope, makeCString(prefix & sym)]
proc genInitCode(m: BModule) =
## this function is called in cgenWriteModules after all modules are closed,
## it means raising dependency on the symbols is too late as it will not propogate
## into other modules, only simple rope manipulations are allowed
appcg(m, m.s[cfsForwardTypes], frameDefines, [rope("#")])
var moduleInitRequired = false
let initname = getInitName(m.module)
var prc = "N_LIB_PRIVATE N_NIMCALL(void, $1)(void) {$N" % [initname]
var moduleInitRequired = m.hcrOn
let initname = getInitName(m)
var prc = "$1 N_NIMCALL(void, $2)(void) {$N" %
[rope(if m.hcrOn: "N_LIB_EXPORT" else: "N_LIB_PRIVATE"), initname]
# we don't want to break into such init code - could happen if a line
# directive from a function written by the user spills after itself
genCLineDir(prc, "generated_not_to_break_here", 999999, m.config)
if m.typeNodes > 0:
appcg(m, m.s[cfsTypeInit1], "static #TNimNode $1[$2];$n",
[m.typeNodesName, rope(m.typeNodes)])
if m.hcrOn:
appcg(m, m.s[cfsTypeInit1], "\t#TNimNode* $1;$N", [m.typeNodesName])
appcg(m, m.s[cfsTypeInit1], "\thcrRegisterGlobal($3, \"$1_$2\", sizeof(TNimNode) * $2, NULL, (void**)&$1);$N",
[m.typeNodesName, rope(m.typeNodes), getModuleDllPath(m, m.module)])
else:
appcg(m, m.s[cfsTypeInit1], "static #TNimNode $1[$2];$n",
[m.typeNodesName, rope(m.typeNodes)])
if m.nimTypes > 0:
appcg(m, m.s[cfsTypeInit1], "static #TNimType $1[$2];$n",
[m.nimTypesName, rope(m.nimTypes)])
if m.hcrOn:
addf(prc, "\tint* nim_hcr_dummy_ = 0;$n" &
"\tNIM_BOOL nim_hcr_do_init_ = " &
"hcrRegisterGlobal($1, \"module_initialized_\", 1, NULL, (void**)&nim_hcr_dummy_);$n",
[getModuleDllPath(m, m.module)])
template writeSection(thing: untyped, section: TCProcSection, addHcrGuards = false) =
if m.thing.s(section).len > 0:
moduleInitRequired = true
if addHcrGuards: add(prc, "\tif (nim_hcr_do_init_) {\n\n")
add(prc, genSectionStart(section, m.config))
add(prc, m.thing.s(section))
add(prc, genSectionEnd(section, m.config))
if addHcrGuards: add(prc, "\n\t} // nim_hcr_do_init_\n")
if m.preInitProc.s(cpsInit).len > 0 or m.preInitProc.s(cpsStmts).len > 0:
# Give this small function its own scope
addf(prc, "{$N", [])
# Keep a bogus frame in case the code needs one
add(prc, ~"\tTFrame FR_; FR_.len = 0;$N")
if m.preInitProc.s(cpsLocals).len > 0:
moduleInitRequired = true
add(prc, genSectionStart(cpsLocals, m.config))
add(prc, m.preInitProc.s(cpsLocals))
add(prc, genSectionEnd(cpsLocals, m.config))
if m.preInitProc.s(cpsInit).len > 0:
moduleInitRequired = true
add(prc, genSectionStart(cpsInit, m.config))
add(prc, m.preInitProc.s(cpsInit))
add(prc, genSectionEnd(cpsInit, m.config))
if m.preInitProc.s(cpsStmts).len > 0:
moduleInitRequired = true
add(prc, genSectionStart(cpsStmts, m.config))
add(prc, m.preInitProc.s(cpsStmts))
add(prc, genSectionEnd(cpsStmts, m.config))
writeSection(preInitProc, cpsLocals)
writeSection(preInitProc, cpsInit, m.hcrOn)
writeSection(preInitProc, cpsStmts)
addf(prc, "}$N", [])
# add new scope for following code, because old vcc compiler need variable
@@ -1367,11 +1530,7 @@ proc genInitCode(m: BModule) =
moduleInitRequired = true
add(prc, initGCFrame(m.initProc))
if m.initProc.s(cpsLocals).len > 0:
moduleInitRequired = true
add(prc, genSectionStart(cpsLocals, m.config))
add(prc, m.initProc.s(cpsLocals))
add(prc, genSectionEnd(cpsLocals, m.config))
writeSection(initProc, cpsLocals)
if m.initProc.s(cpsInit).len > 0 or m.initProc.s(cpsStmts).len > 0:
moduleInitRequired = true
@@ -1385,13 +1544,8 @@ proc genInitCode(m: BModule) =
else:
add(prc, ~"\tTFrame FR_; FR_.len = 0;$N")
add(prc, genSectionStart(cpsInit, m.config))
add(prc, m.initProc.s(cpsInit))
add(prc, genSectionEnd(cpsInit, m.config))
add(prc, genSectionStart(cpsStmts, m.config))
add(prc, m.initProc.s(cpsStmts))
add(prc, genSectionEnd(cpsStmts, m.config))
writeSection(initProc, cpsInit, m.hcrOn)
writeSection(initProc, cpsStmts)
if optStackTrace in m.initProc.options and preventStackTrace notin m.flags:
add(prc, deinitFrame(m.initProc))
@@ -1407,6 +1561,19 @@ proc genInitCode(m: BModule) =
# that would lead to a *nesting* of merge sections which the merger does
# not support. So we add it to another special section: ``cfsInitProc``
if m.hcrOn:
var procsToLoad = @["hcrRegisterProc", "hcrGetProc", "hcrRegisterGlobal", "hcrGetGlobal"]
addf(m.s[cfsInitProc], "N_LIB_EXPORT N_NIMCALL(void, $1)(void* handle, N_NIMCALL_PTR(void*, getProcAddr)(void*, char*)) {$N", [getHcrInitName(m)])
if sfMainModule in m.module.flags:
# additional procs to load
procsToLoad.add("hcrInit")
procsToLoad.add("hcrAddModule")
# load procs
for curr in procsToLoad:
add(m.s[cfsInitProc], hcrGetProcLoadCode(m, curr, "", "handle", "getProcAddr"))
addf(m.s[cfsInitProc], "}$N$N", [])
for i, el in pairs(m.extensionLoaders):
if el != nil:
let ex = "NIM_EXTERNC N_NIMCALL(void, nimLoadProcs$1)(void) {$2}$N$N" %
@@ -1418,6 +1585,12 @@ proc genInitCode(m: BModule) =
add(m.s[cfsInitProc], prc)
genDatInitCode(m)
if m.hcrOn:
addf(m.s[cfsInitProc], "N_LIB_EXPORT N_NIMCALL(void, HcrCreateTypeInfos)(void) {$N", [])
add(m.s[cfsInitProc], m.hcrCreateTypeInfosProc)
addf(m.s[cfsInitProc], "}$N$N", [])
registerModuleToMain(m.g, m)
proc genModule(m: BModule, cfile: Cfile): Rope =
@@ -1445,7 +1618,7 @@ proc genModule(m: BModule, cfile: Cfile): Rope =
if m.s[cfsInitProc].len > 0:
moduleIsEmpty = false
add(result, m.s[cfsInitProc])
if m.s[cfsDatInitProc].len > 0:
if m.s[cfsDatInitProc].len > 0 or m.hcrOn:
moduleIsEmpty = false
add(result, m.s[cfsDatInitProc])
@@ -1565,6 +1738,25 @@ when false:
readMergeInfo(getCFile(m), m)
result = m
proc addHcrInitGuards(p: BProc, n: PNode, inInitGuard: var bool) =
if n.kind == nkStmtList:
for child in n:
addHcrInitGuards(p, child, inInitGuard)
else:
let stmtShouldExecute = n.kind in {nkVarSection, nkLetSection} or
nfExecuteOnReload in n.flags
if inInitGuard:
if stmtShouldExecute:
endBlock(p)
inInitGuard = false
else:
if not stmtShouldExecute:
line(p, cpsStmts, "if (nim_hcr_do_init_)\n")
startBlock(p)
inInitGuard = true
genStmts(p, n)
proc myProcess(b: PPassContext, n: PNode): PNode =
result = n
if b == nil: return
@@ -1573,8 +1765,11 @@ proc myProcess(b: PPassContext, n: PNode): PNode =
m.initProc.options = initProcOptions(m)
#softRnl = if optLineDir in m.config.options: noRnl else: rnl
# XXX replicate this logic!
let tranformed_n = transformStmt(m.g.graph, m.module, n)
genStmts(m.initProc, tranformed_n)
let transformed_n = transformStmt(m.g.graph, m.module, n)
if m.hcrOn:
addHcrInitGuards(m.initProc, transformed_n, m.inHcrInitGuard)
else:
genStmts(m.initProc, transformed_n)
proc shouldRecompile(m: BModule; code: Rope, cfile: Cfile): bool =
result = true
@@ -1674,7 +1869,24 @@ proc myClose(graph: ModuleGraph; b: PPassContext, n: PNode): PNode =
m.initProc.options = initProcOptions(m)
genStmts(m.initProc, n)
if m.hcrOn:
# make sure this is pulled in (meaning hcrGetGlobal() is called for it during init)
discard cgsym(m, "programResult")
if m.inHcrInitGuard:
endBlock(m.initProc)
if sfMainModule in m.module.flags:
if m.hcrOn:
# pull ("define" since they are inline when HCR is on) these functions in the main file
# so it can load the HCR runtime and later pass the library handle to the HCR runtime which
# will in turn pass it to the other modules it initializes so they can initialize the
# register/get procs so they don't have to have the definitions of these functions as well
discard cgsym(m, "nimLoadLibrary")
discard cgsym(m, "nimLoadLibraryError")
discard cgsym(m, "nimGetProcAddr")
discard cgsym(m, "procAddrError")
discard cgsym(m, "rawWrite")
# raise dependencies on behalf of genMainProc
if m.config.target.targetOS != osStandalone and m.config.selectedGC != gcNone:
discard cgsym(m, "initStackBottomWith")
@@ -1709,14 +1921,11 @@ proc genForwardedProcs(g: BModuleList) =
proc cgenWriteModules*(backend: RootRef, config: ConfigRef) =
let g = BModuleList(backend)
g.config = config
# we need to process the transitive closure because recursive module
# deps are allowed (and the system module is processed in the wrong
# order anyway)
g.config = config
let (outDir, _, _) = splitFile(config.outfile)
if not outDir.isEmpty:
createDir(outDir)
genForwardedProcs(g)
for m in cgenModules(g):

9
compiler/cgendata.nim
View File

@@ -98,6 +98,7 @@ type
TTypeSeq* = seq[PType]
TypeCache* = Table[SigHash, Rope]
TypeCacheWithOwner* = Table[SigHash, tuple[str: Rope, owner: PSym]]
Codegenflag* = enum
preventStackTrace, # true if stack traces need to be prevented
@@ -118,7 +119,7 @@ type
generatedHeader*: BModule
breakPointId*: int
breakpoints*: Rope # later the breakpoints are inserted into the main proc
typeInfoMarker*: TypeCache
typeInfoMarker*: TypeCacheWithOwner
config*: ConfigRef
graph*: ModuleGraph
strVersion*, seqVersion*: int # version of the string/seq implementation to use
@@ -150,6 +151,8 @@ type
typeInfoMarker*: TypeCache # needed for generating type information
initProc*: BProc # code for init procedure
preInitProc*: BProc # code executed before the init proc
hcrCreateTypeInfosProc*: Rope # type info globals are in here when HCR=on
inHcrInitGuard*: bool # We are currently withing a HCR reloading guard.
typeStack*: TTypeSeq # used for type generation
dataCache*: TNodeTable
typeNodes*, nimTypes*: int # used for type info generation
@@ -189,8 +192,8 @@ proc newProc*(prc: PSym, module: BModule): BProc =
result.sigConflicts = initCountTable[string]()
proc newModuleList*(g: ModuleGraph): BModuleList =
BModuleList(typeInfoMarker: initTable[SigHash, Rope](), config: g.config,
graph: g, nimtvDeclared: initIntSet())
BModuleList(typeInfoMarker: initTable[SigHash, tuple[str: Rope, owner: PSym]](),
config: g.config, graph: g, nimtvDeclared: initIntSet())
iterator cgenModules*(g: BModuleList): BModule =
for m in g.modules_closed:

20
compiler/commands.nim
View File

@@ -387,7 +387,12 @@ proc processSwitch*(switch, arg: string, pass: TCmdLinePass, info: TLineInfo;
conf.nimcacheDir = processPath(conf, arg, info, true)
of "out", "o":
expectArg(conf, switch, arg, pass, info)
conf.outFile = AbsoluteFile arg
let f = splitFile(arg.expandTilde)
conf.outFile = RelativeFile f.name
conf.outDir = toAbsoluteDir f.dir
of "outdir":
expectArg(conf, switch, arg, pass, info)
conf.outDir = toAbsoluteDir arg.expandTilde
of "docseesrcurl":
expectArg(conf, switch, arg, pass, info)
conf.docSeeSrcUrl = arg
@@ -487,9 +492,13 @@ proc processSwitch*(switch, arg: string, pass: TCmdLinePass, info: TLineInfo;
if optMemTracker in conf.options: defineSymbol(conf.symbols, "memtracker")
else: undefSymbol(conf.symbols, "memtracker")
of "hotcodereloading":
processOnOffSwitch(conf, {optHotCodeReloading}, arg, pass, info)
if optHotCodeReloading in conf.options: defineSymbol(conf.symbols, "hotcodereloading")
else: undefSymbol(conf.symbols, "hotcodereloading")
processOnOffSwitchG(conf, {optHotCodeReloading}, arg, pass, info)
if conf.hcrOn:
defineSymbol(conf.symbols, "hotcodereloading")
defineSymbol(conf.symbols, "useNimRtl")
else:
undefSymbol(conf.symbols, "hotcodereloading")
undefSymbol(conf.symbols, "useNimRtl")
of "oldnewlines":
case arg.normalize
of "","on":
@@ -787,7 +796,8 @@ proc processArgument*(pass: TCmdLinePass; p: OptParser;
if pass == passCmd1: config.commandArgs.add p.key
if argsCount == 1:
# support UNIX style filenames everywhere for portable build scripts:
config.projectName = unixToNativePath(p.key)
if config.projectName.len == 0:
config.projectName = unixToNativePath(p.key)
config.arguments = cmdLineRest(p)
result = true
inc argsCount

2
compiler/condsyms.nim
View File

@@ -82,7 +82,9 @@ proc initDefines*(symbols: StringTableRef) =
defineSymbol("nimHasTypeof")
defineSymbol("nimErrorProcCanHaveBody")
defineSymbol("nimHasInstantiationOfInMacro")
defineSymbol("nimHasHotCodeReloading")
defineSymbol("nimHasNilSeqs")
defineSymbol("nimHasSignatureHashInMacro")
for f in low(Feature)..high(Feature):
defineSymbol("nimHas" & $f)

14
compiler/docgen.nim
View File

@@ -100,14 +100,12 @@ proc parseRst(text, filename: string,
proc getOutFile2(conf: ConfigRef; filename: RelativeFile,
ext: string, dir: RelativeDir; guessTarget: bool): AbsoluteFile =
if optWholeProject in conf.globalOptions:
# This is correct, for 'nim doc --project' we interpret the '--out' option as an
# absolute directory, not as a filename!
let d = if conf.outFile.isEmpty: conf.projectPath / dir else: AbsoluteDir(conf.outFile)
let d = if conf.outDir.isEmpty: conf.projectPath / dir else: conf.outDir
createDir(d)
result = d / changeFileExt(filename, ext)
elif guessTarget:
let d = if not conf.outFile.isEmpty: splitFile(conf.outFile).dir
else: conf.projectPath
let d = if not conf.outDir.isEmpty: conf.outDir
else: conf.projectPath
createDir(d)
result = d / changeFileExt(filename, ext)
else:
@@ -952,9 +950,8 @@ proc genOutFile(d: PDoc): Rope =
proc generateIndex*(d: PDoc) =
if optGenIndex in d.conf.globalOptions:
let dir = if d.conf.outFile.isEmpty: d.conf.projectPath / RelativeDir"htmldocs"
elif optWholeProject in d.conf.globalOptions: AbsoluteDir(d.conf.outFile)
else: AbsoluteDir(d.conf.outFile.string.splitFile.dir)
let dir = if not d.conf.outDir.isEmpty: d.conf.outDir
else: d.conf.projectPath / RelativeDir"htmldocs"
createDir(dir)
let dest = dir / changeFileExt(relativeTo(AbsoluteFile d.filename,
d.conf.projectPath), IndexExt)
@@ -995,6 +992,7 @@ proc writeOutputJson*(d: PDoc, useWarning = false) =
"\" for writing")
proc commandDoc*(cache: IdentCache, conf: ConfigRef) =
conf.outDir = AbsoluteDir(conf.outDir / conf.outFile)
var ast = parseFile(conf.projectMainIdx, cache, conf)
if ast == nil: return
var d = newDocumentor(conf.projectFull, cache, conf)

1
compiler/docgen2.nim
View File

@@ -29,7 +29,6 @@ template shouldProcess(g): bool =
template closeImpl(body: untyped) {.dirty.} =
var g = PGen(p)
let useWarning = sfMainModule notin g.module.flags
#echo g.module.name.s, " ", g.module.owner.id, " ", gMainPackageId
if shouldProcess(g):
body
try:

162
compiler/extccomp.nim
View File

@@ -14,7 +14,7 @@
import
ropes, os, strutils, osproc, platform, condsyms, options, msgs,
lineinfos, std / sha1, streams, pathutils
lineinfos, std / sha1, streams, pathutils, sequtils, times
type
TInfoCCProp* = enum # properties of the C compiler:
@@ -468,8 +468,8 @@ proc execExternalProgram*(conf: ConfigRef; cmd: string, msg = hintExecuting) =
rawMessage(conf, errGenerated, "execution of an external program failed: '$1'" %
cmd)
proc generateScript(conf: ConfigRef; projectFile: AbsoluteFile, script: Rope) =
let (_, name, _) = splitFile(projectFile)
proc generateScript(conf: ConfigRef; script: Rope) =
let (_, name, _) = splitFile(conf.outFile.string)
let filename = getNimcacheDir(conf) / RelativeFile(addFileExt("compile_" & name,
platform.OS[conf.target.targetOS].scriptExt))
if writeRope(script, filename):
@@ -551,14 +551,14 @@ proc getLinkerExe(conf: ConfigRef; compiler: TSystemCC): string =
elif optMixedMode in conf.globalOptions and conf.cmd != cmdCompileToCpp: CC[compiler].cppCompiler
else: getCompilerExe(conf, compiler, AbsoluteFile"")
proc getCompileCFileCmd*(conf: ConfigRef; cfile: Cfile): string =
proc getCompileCFileCmd*(conf: ConfigRef; cfile: Cfile, isMainFile = false): string =
var c = conf.cCompiler
var options = cFileSpecificOptions(conf, cfile.cname)
var exe = getConfigVar(conf, c, ".exe")
if exe.len == 0: exe = getCompilerExe(conf, c, cfile.cname)
if needsExeExt(conf): exe = addFileExt(exe, "exe")
if optGenDynLib in conf.globalOptions and
if (optGenDynLib in conf.globalOptions or (conf.hcrOn and not isMainFile)) and
ospNeedsPIC in platform.OS[conf.target.targetOS].props:
add(options, ' ' & CC[c].pic)
@@ -648,12 +648,14 @@ proc addExternalFileToCompile*(conf: ConfigRef; filename: AbsoluteFile) =
flags: {CfileFlag.External})
addExternalFileToCompile(conf, c)
proc compileCFile(conf: ConfigRef; list: CFileList, script: var Rope, cmds: var TStringSeq,
proc compileCFiles(conf: ConfigRef; list: CFileList, script: var Rope, cmds: var TStringSeq,
prettyCmds: var TStringSeq) =
var currIdx = 0
for it in list:
# call the C compiler for the .c file:
if it.flags.contains(CfileFlag.Cached): continue
var compileCmd = getCompileCFileCmd(conf, it)
var compileCmd = getCompileCFileCmd(conf, it, currIdx == list.len - 1)
inc currIdx
if optCompileOnly notin conf.globalOptions:
add(cmds, compileCmd)
let (_, name, _) = splitFile(it.cname)
@@ -662,7 +664,8 @@ proc compileCFile(conf: ConfigRef; list: CFileList, script: var Rope, cmds: var
add(script, compileCmd)
add(script, "\n")
proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): string =
proc getLinkCmd(conf: ConfigRef; output: AbsoluteFile,
objfiles: string, isDllBuild: bool): string =
if optGenStaticLib in conf.globalOptions:
var libname: string
if not conf.outFile.isEmpty:
@@ -672,7 +675,7 @@ proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): s
else:
libname = (libNameTmpl(conf) % splitFile(conf.projectName).name)
result = CC[conf.cCompiler].buildLib % ["libfile", quoteShell(libname),
"objfiles", objfiles]
"objfiles", objfiles]
else:
var linkerExe = getConfigVar(conf, conf.cCompiler, ".linkerexe")
if len(linkerExe) == 0: linkerExe = getLinkerExe(conf, conf.cCompiler)
@@ -684,28 +687,16 @@ proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): s
CC[conf.cCompiler].buildGui
else:
""
var exefile, builddll: string
if optGenDynLib in conf.globalOptions:
exefile = platform.OS[conf.target.targetOS].dllFrmt % splitFile(projectfile).name
builddll = CC[conf.cCompiler].buildDll
else:
exefile = splitFile(projectfile).name & platform.OS[conf.target.targetOS].exeExt
builddll = ""
if not conf.outFile.isEmpty:
exefile = conf.outFile.string.expandTilde
if not exefile.isAbsolute():
exefile = getCurrentDir() / exefile
if not noAbsolutePaths(conf):
if not exefile.isAbsolute():
exefile = string(splitFile(projectfile).dir / RelativeFile(exefile))
let builddll = if isDllBuild: CC[conf.cCompiler].buildDll else: ""
let exefile = quoteShell(output)
when false:
if optCDebug in conf.globalOptions:
writeDebugInfo(exefile.changeFileExt("ndb"))
exefile = quoteShell(exefile)
# Map files are required by Nintendo Switch compilation. They are a list
# of all function calls in the library and where they come from.
let mapfile = quoteShell(getNimcacheDir(conf) / RelativeFile(splitFile(projectFile).name & ".map"))
let mapfile = quoteShell(getNimcacheDir(conf) / RelativeFile(splitFile(output).name & ".map"))
let linkOptions = getLinkOptions(conf) & " " &
getConfigVar(conf, conf.cCompiler, ".options.linker")
@@ -723,6 +714,46 @@ proc getLinkCmd(conf: ConfigRef; projectfile: AbsoluteFile, objfiles: string): s
"objfiles", objfiles, "exefile", exefile,
"nim", quoteShell(getPrefixDir(conf)),
"lib", quoteShell(conf.libpath)])
# On windows the debug information for binaries is emitted in a separate .pdb
# file and the binaries (.dll and .exe) contain a full path to that .pdb file.
# This is a problem for hot code reloading because even when we copy the .dll
# and load the copy so the build process may overwrite the original .dll on
# the disk (windows locks the files of running binaries) the copy still points
# to the original .pdb (and a simple copy of the .pdb won't help). This is a
# problem when a debugger is attached to the program we are hot-reloading.
# This problem is nonexistent on Unix since there by default debug symbols
# are embedded in the binaries so loading a copy of a .so will be fine. There
# is the '/Z7' flag for the MSVC compiler to embed the debug info of source
# files into their respective .obj files but the linker still produces a .pdb
# when a final .dll or .exe is linked so the debug info isn't embedded.
# There is also the issue that even when a .dll is unloaded the debugger
# still keeps the .pdb for that .dll locked. This is a major problem and
# because of this we cannot just alternate between 2 names for a .pdb file
# when rebuilding a .dll - instead we need to accumulate differently named
# .pdb files in the nimcache folder - this is the easiest and most reliable
# way of being able to debug and rebuild the program at the same time. This
# is accomplished using the /PDB:<filename> flag (there also exists the
# /PDBALTPATH:<filename> flag). The only downside is that the .pdb files are
# atleast 5-10mb big and will quickly accumulate. There is a hacky solution:
# we could try to delete all .pdb files with a pattern and swallow exceptions.
#
# links about .pdb files and hot code reloading:
# https://ourmachinery.com/post/dll-hot-reloading-in-theory-and-practice/
# https://ourmachinery.com/post/little-machines-working-together-part-2/
# https://github.com/fungos/cr
# https://fungos.github.io/blog/2017/11/20/cr.h-a-simple-c-hot-reload-header-only-library/
# on forcing the debugger to unlock a locked .pdb of an unloaded library:
# https://blog.molecular-matters.com/2017/05/09/deleting-pdb-files-locked-by-visual-studio/
# and a bit about the .pdb format in case that is ever needed:
# https://github.com/crosire/blink
# http://www.debuginfo.com/articles/debuginfomatch.html#pdbfiles
if conf.hcrOn and conf.cCompiler == ccVcc:
let t = now()
let pdb = output.string & "." & format(t, "MMMM-yyyy-HH-mm-") & $t.nanosecond & ".pdb"
result.add " /link /PDB:" & pdb
template getLinkCmd(conf: ConfigRef; output: AbsoluteFile, objfiles: string): string =
getLinkCmd(conf, output, objfiles, optGenDynLib in conf.globalOptions)
template tryExceptOSErrorMessage(conf: ConfigRef; errorPrefix: string = "", body: untyped): typed =
try:
@@ -799,7 +830,17 @@ proc linkViaResponseFile(conf: ConfigRef; cmd: string) =
finally:
removeFile(linkerArgs)
proc callCCompiler*(conf: ConfigRef; projectfile: AbsoluteFile) =
proc getObjFilePath(conf: ConfigRef, f: CFile): string =
if noAbsolutePaths(conf): f.obj.extractFilename
else: f.obj.string
proc hcrLinkTargetName(conf: ConfigRef, objFile: string, isMain = false): AbsoluteFile =
let basename = splitFile(objFile).name
let targetName = if isMain: basename & ".exe"
else: platform.OS[conf.target.targetOS].dllFrmt % basename
result = conf.nimcacheDir / RelativeFile(targetName)
proc callCCompiler*(conf: ConfigRef) =
var
linkCmd: string
if conf.globalOptions * {optCompileOnly, optGenScript} == {optCompileOnly}:
@@ -813,7 +854,7 @@ proc callCCompiler*(conf: ConfigRef; projectfile: AbsoluteFile) =
when declared(echo):
let cmd = prettyCmds[idx]
if cmd != "": echo cmd
compileCFile(conf, conf.toCompile, script, cmds, prettyCmds)
compileCFiles(conf, conf.toCompile, script, cmds, prettyCmds)
if optCompileOnly notin conf.globalOptions:
execCmdsInParallel(conf, cmds, prettyCb)
if optNoLinking notin conf.globalOptions:
@@ -824,31 +865,64 @@ proc callCCompiler*(conf: ConfigRef; projectfile: AbsoluteFile) =
add(objfiles, ' ')
add(objfiles, quoteShell(
addFileExt(objFile, CC[conf.cCompiler].objExt)))
for x in conf.toCompile:
let objFile = if noAbsolutePaths(conf): x.obj.extractFilename else: x.obj.string
add(objfiles, ' ')
add(objfiles, quoteShell(objFile))
linkCmd = getLinkCmd(conf, projectfile, objfiles)
if optCompileOnly notin conf.globalOptions:
if defined(windows) and linkCmd.len > 8_000:
# Windows's command line limit is about 8K (don't laugh...) so C compilers on
# Windows support a feature where the command line can be passed via ``@linkcmd``
# to them.
linkViaResponseFile(conf, linkCmd)
else:
execLinkCmd(conf, linkCmd)
if conf.hcrOn: # lets assume that optCompileOnly isn't on
cmds = @[]
let mainFileIdx = conf.toCompile.len - 1
for idx, x in conf.toCompile:
# don't relink each of the many binaries (one for each source file) if the nim code is
# cached because that would take too much time for small changes - the only downside to
# this is that if an external-to-link file changes the final target wouldn't be relinked
if x.flags.contains(CfileFlag.Cached): continue
# we pass each object file as if it is the project file - a .dll will be created for each such
# object file in the nimcache directory, and only in the case of the main project file will
# there be probably an executable (if the project is such) which will be copied out of the nimcache
let objFile = conf.getObjFilePath(x)
let buildDll = idx != mainFileIdx
let linkTarget = conf.hcrLinkTargetName(objFile, not buildDll)
add(cmds, getLinkCmd(conf, linkTarget, objfiles & " " & quoteShell(objFile), buildDll))
# try to remove all .pdb files for the current binary so they don't accumulate endlessly in the nimcache
# for more info check the comment inside of getLinkCmd() where the /PDB:<filename> MSVC flag is used
if conf.cCompiler == ccVcc:
for pdb in walkFiles(objFile & ".*.pdb"):
discard tryRemoveFile(pdb)
# execute link commands in parallel - output will be a bit different
# if it fails than that from execLinkCmd() but that doesn't matter
prettyCmds = map(prettyCmds, proc (curr: string): string = return curr.replace("CC", "Link"))
execCmdsInParallel(conf, cmds, prettyCb)
# only if not cached - copy the resulting main file from the nimcache folder to its originally intended destination
if not conf.toCompile[mainFileIdx].flags.contains(CfileFlag.Cached):
let mainObjFile = getObjFilePath(conf, conf.toCompile[mainFileIdx])
var src = conf.hcrLinkTargetName(mainObjFile, true)
var dst = conf.prepareToWriteOutput
copyFileWithPermissions(src.string, dst.string)
else:
for x in conf.toCompile:
let objFile = if noAbsolutePaths(conf): x.obj.extractFilename else: x.obj.string
add(objfiles, ' ')
add(objfiles, quoteShell(objFile))
let mainOutput = if optGenScript notin conf.globalOptions: conf.prepareToWriteOutput
else: AbsoluteFile(conf.projectName)
linkCmd = getLinkCmd(conf, mainOutput, objfiles)
if optCompileOnly notin conf.globalOptions:
if defined(windows) and linkCmd.len > 8_000:
# Windows's command line limit is about 8K (don't laugh...) so C compilers on
# Windows support a feature where the command line can be passed via ``@linkcmd``
# to them.
linkViaResponseFile(conf, linkCmd)
else:
execLinkCmd(conf, linkCmd)
else:
linkCmd = ""
if optGenScript in conf.globalOptions:
add(script, linkCmd)
add(script, "\n")
generateScript(conf, projectfile, script)
generateScript(conf, script)
#from json import escapeJson
import json
proc writeJsonBuildInstructions*(conf: ConfigRef; projectfile: AbsoluteFile) =
proc writeJsonBuildInstructions*(conf: ConfigRef) =
template lit(x: untyped) = f.write x
template str(x: untyped) =
when compiles(escapeJson(x, buf)):
@@ -895,7 +969,7 @@ proc writeJsonBuildInstructions*(conf: ConfigRef; projectfile: AbsoluteFile) =
var buf = newStringOfCap(50)
let jsonFile = toGeneratedFile(conf, projectfile, "json")
let jsonFile = conf.nimcacheDir / RelativeFile(conf.projectName & ".json")
var f: File
if open(f, jsonFile.string, fmWrite):
@@ -907,7 +981,7 @@ proc writeJsonBuildInstructions*(conf: ConfigRef; projectfile: AbsoluteFile) =
linkfiles(conf, f, buf, objfiles, conf.toCompile, conf.externalToLink)
lit "],\L\"linkcmd\": "
str getLinkCmd(conf, projectfile, objfiles)
str getLinkCmd(conf, conf.absOutFile, objfiles)
lit "\L}\L"
close(f)

101
compiler/jsgen.nim
View File

@@ -83,7 +83,6 @@ type
forwarded: seq[PSym]
generatedSyms: IntSet
typeInfoGenerated: IntSet
classes: seq[(PType, Rope)]
unique: int # for temp identifier generation
PProc = ref TProc
@@ -133,7 +132,6 @@ proc newGlobals(): PGlobals =
result.forwarded = @[]
result.generatedSyms = initIntSet()
result.typeInfoGenerated = initIntSet()
result.classes = @[]
proc initCompRes(r: var TCompRes) =
r.address = nil
@@ -250,7 +248,7 @@ proc mangleName(m: BModule, s: PSym): Rope =
result = rope(x)
# From ES5 on reserved words can be used as object field names
if s.kind != skField:
if optHotCodeReloading in m.config.options:
if m.config.hcrOn:
# When hot reloading is enabled, we must ensure that the names
# of functions and types will be preserved across rebuilds:
add(result, idOrSig(s, m.module.name.s, m.sigConflicts))
@@ -1206,19 +1204,8 @@ proc genAddr(p: PProc, n: PNode, r: var TCompRes) =
gen(p, n.sons[0].sons[0], r)
else: internalError(p.config, n.sons[0].info, "genAddr: " & $n.sons[0].kind)
proc thisParam(p: PProc; typ: PType): PType =
discard
proc attachProc(p: PProc; content: Rope; s: PSym) =
let otyp = thisParam(p, s.typ)
if otyp != nil:
for i, cls in p.g.classes:
if sameType(cls[0], otyp):
add(p.g.classes[i][1], content)
return
p.g.classes.add((otyp, content))
else:
add(p.g.code, content)
add(p.g.code, content)
proc attachProc(p: PProc; s: PSym) =
let newp = genProc(p, s)
@@ -1460,9 +1447,6 @@ proc genInfixCall(p: PProc, n: PNode, r: var TCompRes) =
genArgs(p, n, r, 2)
proc genCall(p: PProc, n: PNode, r: var TCompRes) =
if n.sons[0].kind == nkSym and thisParam(p, n.sons[0].typ) != nil:
genInfixCall(p, n, r)
return
gen(p, n.sons[0], r)
genArgs(p, n, r)
if n.typ != nil:
@@ -1606,13 +1590,14 @@ proc genVarInit(p: PProc, v: PSym, n: PNode) =
s: Rope
varCode: string
varName = mangleName(p.module, v)
useReloadingGuard = sfGlobal in v.flags and optHotCodeReloading in p.config.options
useReloadingGuard = sfGlobal in v.flags and p.config.hcrOn
if v.constraint.isNil:
if useReloadingGuard:
lineF(p, "var $1;$n", varName)
lineF(p, "if ($1 === undefined) {$n", varName)
varCode = $varName
inc p.extraIndent
else:
varCode = "var $2"
else:
@@ -1664,6 +1649,7 @@ proc genVarInit(p: PProc, v: PSym, n: PNode) =
lineF(p, varCode & " = $3;$n", [returnType, v.loc.r, s])
if useReloadingGuard:
dec p.extraIndent
lineF(p, "}$n")
proc genVarStmt(p: PProc, n: PNode) =
@@ -2204,7 +2190,7 @@ proc genProc(oldProc: PProc, prc: PSym): Rope =
else:
result = ~"\L"
if optHotCodeReloading in p.config.options:
if p.config.hcrOn:
# Here, we introduce thunks that create the equivalent of a jump table
# for all global functions, because references to them may be stored
# in JavaScript variables. The added indirection ensures that such
@@ -2440,13 +2426,52 @@ proc genHeader(): Rope =
"if (typeof Float64Array === 'undefined') Float64Array = Array;$n") %
[rope(VersionAsString)]
proc addHcrInitGuards(p: PProc, n: PNode,
moduleLoadedVar: Rope, inInitGuard: var bool) =
if n.kind == nkStmtList:
for child in n:
addHcrInitGuards(p, child, moduleLoadedVar, inInitGuard)
else:
let stmtShouldExecute = n.kind in {
nkProcDef, nkFuncDef, nkMethodDef,nkConverterDef,
nkVarSection, nkLetSection} or nfExecuteOnReload in n.flags
if inInitGuard:
if stmtShouldExecute:
dec p.extraIndent
line(p, "}\L")
inInitGuard = false
else:
if not stmtShouldExecute:
lineF(p, "if ($1 == undefined) {$n", [moduleLoadedVar])
inc p.extraIndent
inInitGuard = true
genStmt(p, n)
proc genModule(p: PProc, n: PNode) =
if optStackTrace in p.options:
add(p.body, frameCreate(p,
makeJSString("module " & p.module.module.name.s),
makeJSString(toFilename(p.config, p.module.module.info))))
let n_transformed = transformStmt(p.module.graph, p.module.module, n)
genStmt(p, n_transformed)
if p.config.hcrOn and n.kind == nkStmtList:
let moduleSym = p.module.module
var moduleLoadedVar = rope(moduleSym.name.s) & "_loaded" &
idOrSig(moduleSym, moduleSym.name.s, p.module.sigConflicts)
lineF(p, "var $1;$n", [moduleLoadedVar])
var inGuardedBlock = false
addHcrInitGuards(p, n_transformed, moduleLoadedVar, inGuardedBlock)
if inGuardedBlock:
dec p.extraIndent
line(p, "}\L")
lineF(p, "$1 = true;$n", [moduleLoadedVar])
else:
genStmt(p, n_transformed)
if optStackTrace in p.options:
add(p.body, frameDestroy(p))
@@ -2487,44 +2512,14 @@ proc getClassName(t: PType): Rope =
if s.loc.r != nil: result = s.loc.r
else: result = rope(s.name.s)
proc genClass(conf: ConfigRef; obj: PType; content: Rope; ext: string) =
let cls = getClassName(obj)
let t = skipTypes(obj, abstractPtrs)
let extends = if t.kind == tyObject and t.sons[0] != nil:
" extends " & getClassName(t.sons[0])
else: nil
let result = ("<?php$n" &
"/* Generated by the Nim Compiler v$# */$n" &
"/* (c) " & copyrightYear & " Andreas Rumpf */$n$n" &
"require_once \"nimsystem.php\";$n" &
"class $#$# {$n$#$n}$n") %
[rope(VersionAsString), cls, extends, content]
let outfile = changeFileExt(completeCFilePath(conf, AbsoluteFile($cls)), ext)
discard writeRopeIfNotEqual(result, outfile)
proc myClose(graph: ModuleGraph; b: PPassContext, n: PNode): PNode =
result = myProcess(b, n)
var m = BModule(b)
if passes.skipCodegen(m.config, n): return n
if sfMainModule in m.module.flags:
let globals = PGlobals(graph.backend)
let ext = "js"
let f = if globals.classes.len == 0: toFilename(m.config, FileIndex m.module.position)
else: "nimsystem"
let code = wholeCode(graph, m)
let outfile =
if not m.config.outFile.isEmpty:
if m.config.outFile.string.isAbsolute: m.config.outFile
else: AbsoluteFile(getCurrentDir() / m.config.outFile.string)
else:
changeFileExt(completeCFilePath(m.config, AbsoluteFile f), ext)
let (outDir, _, _) = splitFile(outfile)
if not outDir.isEmpty:
createDir(outDir)
discard writeRopeIfNotEqual(genHeader() & code, outfile)
for obj, content in items(globals.classes):
genClass(m.config, obj, content, ext)
let outFile = m.config.prepareToWriteOutput()
discard writeRopeIfNotEqual(genHeader() & code, outFile)
proc myOpen(graph: ModuleGraph; s: PSym): PPassContext =
result = newModule(graph, s)

7
compiler/layouter.nim
View File

@@ -56,12 +56,13 @@ proc openEmitter*(em: var Emitter, cache: IdentCache;
em.lastLineNumber = 1
proc closeEmitter*(em: var Emitter) =
if fileExists(em.config.outFile) and readFile(em.config.outFile.string) == em.content:
let outFile = em.config.absOutFile
if fileExists(outFile) and readFile(outFile.string) == em.content:
discard "do nothing, see #9499"
return
var f = llStreamOpen(em.config.outFile, fmWrite)
var f = llStreamOpen(outFile, fmWrite)
if f == nil:
rawMessage(em.config, errGenerated, "cannot open file: " & em.config.outFile.string)
rawMessage(em.config, errGenerated, "cannot open file: " & outFile.string)
return
f.llStreamWrite em.content
llStreamClose(f)

42
compiler/main.nim
View File

@@ -33,8 +33,9 @@ proc semanticPasses(g: ModuleGraph) =
registerPass g, verbosePass
registerPass g, semPass
proc writeDepsFile(g: ModuleGraph; project: AbsoluteFile) =
let f = open(changeFileExt(project, "deps").string, fmWrite)
proc writeDepsFile(g: ModuleGraph) =
let fname = g.config.nimcacheDir / RelativeFile(g.config.projectName & ".deps")
let f = open(fname.string, fmWrite)
for m in g.modules:
if m != nil:
f.writeLine(toFullPath(g.config, m.position.FileIndex))
@@ -48,7 +49,7 @@ proc commandGenDepend(graph: ModuleGraph) =
registerPass(graph, gendependPass)
compileProject(graph)
let project = graph.config.projectFull
writeDepsFile(graph, project)
writeDepsFile(graph)
generateDot(graph, project)
execExternalProgram(graph.config, "dot -Tpng -o" &
changeFileExt(project, "png").string &
@@ -62,6 +63,7 @@ proc commandCheck(graph: ModuleGraph) =
when not defined(leanCompiler):
proc commandDoc2(graph: ModuleGraph; json: bool) =
graph.config.outDir = AbsoluteDir(graph.config.outDir / graph.config.outFile)
graph.config.errorMax = high(int) # do not stop after first error
semanticPasses(graph)
if json: registerPass(graph, docgen2JsonPass)
@@ -71,6 +73,16 @@ when not defined(leanCompiler):
proc commandCompileToC(graph: ModuleGraph) =
let conf = graph.config
if conf.outDir.isEmpty:
conf.outDir = conf.projectPath
if conf.outFile.isEmpty:
let targetName = if optGenDynLib in conf.globalOptions:
platform.OS[conf.target.targetOS].dllFrmt % conf.projectName
else:
conf.projectName & platform.OS[conf.target.targetOS].exeExt
conf.outFile = RelativeFile targetName
extccomp.initVars(conf)
semanticPasses(graph)
registerPass(graph, cgenPass)
@@ -80,11 +92,12 @@ proc commandCompileToC(graph: ModuleGraph) =
return # issue #9933
cgenWriteModules(graph.backend, conf)
if conf.cmd != cmdRun:
let proj = changeFileExt(conf.projectFull, "")
extccomp.callCCompiler(conf, proj)
extccomp.writeJsonBuildInstructions(conf, proj)
extccomp.callCCompiler(conf)
# for now we do not support writing out a .json file with the build instructions when HCR is on
if not conf.hcrOn:
extccomp.writeJsonBuildInstructions(conf)
if optGenScript in graph.config.globalOptions:
writeDepsFile(graph, toGeneratedFile(conf, proj, ""))
writeDepsFile(graph)
proc commandJsonScript(graph: ModuleGraph) =
let proj = changeFileExt(graph.config.projectFull, "")
@@ -93,6 +106,12 @@ proc commandJsonScript(graph: ModuleGraph) =
when not defined(leanCompiler):
proc commandCompileToJS(graph: ModuleGraph) =
let conf = graph.config
if conf.outDir.isEmpty:
conf.outDir = conf.projectPath
if conf.outFile.isEmpty:
conf.outFile = RelativeFile(conf.projectName & ".js")
#incl(gGlobalOptions, optSafeCode)
setTarget(graph.config.target, osJS, cpuJS)
#initDefines()
@@ -102,7 +121,7 @@ when not defined(leanCompiler):
registerPass(graph, JSgenPass)
compileProject(graph)
if optGenScript in graph.config.globalOptions:
writeDepsFile(graph, toGeneratedFile(conf, conf.projectFull, ""))
writeDepsFile(graph)
proc interactivePasses(graph: ModuleGraph) =
initDefines(graph.config.symbols)
@@ -193,6 +212,12 @@ proc mainCommand*(graph: ModuleGraph) =
quit "compiler wasn't built with JS code generator"
else:
conf.cmd = cmdCompileToJS
if conf.hcrOn:
# XXX: At the moment, system.nim cannot be compiled in JS mode
# with "-d:useNimRtl". The HCR option has been processed earlier
# and it has added this define implictly, so we must undo that here.
# A better solution might be to fix system.nim
undefSymbol(conf.symbols, "useNimRtl")
commandCompileToJS(graph)
of "doc0":
when defined(leanCompiler):
@@ -287,6 +312,7 @@ proc mainCommand*(graph: ModuleGraph) =
(key: "project_path", val: %conf.projectFull.string),
(key: "defined_symbols", val: definedSymbols),
(key: "lib_paths", val: %libpaths),
(key: "outdir", val: %conf.outDir.string),
(key: "out", val: %conf.outFile.string),
(key: "nimcache", val: %getNimcacheDir(conf).string),
(key: "hints", val: hints),

2
compiler/modulegraphs.nim
View File

@@ -33,6 +33,7 @@ type
modules*: seq[PSym] ## indexed by int32 fileIdx
packageSyms*: TStrTable
deps*: IntSet # the dependency graph or potentially its transitive closure.
importDeps*: Table[FileIndex, seq[FileIndex]] # explicit import module dependencies
suggestMode*: bool # whether we are in nimsuggest mode or not.
invalidTransitiveClosure: bool
inclToMod*: Table[FileIndex, FileIndex] # mapping of include file to the
@@ -118,6 +119,7 @@ proc newModuleGraph*(cache: IdentCache; config: ConfigRef): ModuleGraph =
result = ModuleGraph()
initStrTable(result.packageSyms)
result.deps = initIntSet()
result.importDeps = initTable[FileIndex, seq[FileIndex]]()
result.modules = @[]
result.importStack = @[]
result.inclToMod = initTable[FileIndex, FileIndex]()

5
compiler/modules.nim
View File

@@ -12,7 +12,7 @@
import
ast, astalgo, magicsys, std / sha1, msgs, cgendata, sigmatch, options,
idents, os, lexer, idgen, passes, syntaxes, llstream, modulegraphs, rod,
lineinfos, pathutils
lineinfos, pathutils, tables
proc resetSystemArtifacts*(g: ModuleGraph) =
magicsys.resetSysTypes(g)
@@ -91,6 +91,9 @@ proc importModule*(graph: ModuleGraph; s: PSym, fileIdx: FileIndex): PSym {.proc
assert graph.config != nil
result = compileModule(graph, fileIdx, {})
graph.addDep(s, fileIdx)
# keep track of import relationships
if graph.config.hcrOn:
graph.importDeps.mgetOrPut(FileIndex(s.position), @[]).add(fileIdx)
#if sfSystemModule in result.flags:
# localError(result.info, errAttemptToRedefine, result.name.s)
# restore the notes for outer module:

19
compiler/nim.nim
View File

@@ -21,9 +21,11 @@ when defined(i386) and defined(windows) and defined(vcc):
import
commands, lexer, condsyms, options, msgs, nversion, nimconf, ropes,
extccomp, strutils, os, osproc, platform, main, parseopt,
nodejs, scriptconfig, idents, modulegraphs, lineinfos, cmdlinehelper,
scriptconfig, idents, modulegraphs, lineinfos, cmdlinehelper,
pathutils
include nodejs
when hasTinyCBackend:
import tccgen
@@ -78,23 +80,10 @@ proc handleCmdLine(cache: IdentCache; conf: ConfigRef) =
if conf.cmd == cmdRun:
tccgen.run(conf.arguments)
if optRun in conf.globalOptions:
var ex = quoteShell conf.absOutFile
if conf.cmd == cmdCompileToJS:
var ex: string
if not conf.outFile.isEmpty:
ex = conf.outFile.prependCurDir.quoteShell
else:
ex = quoteShell(
completeCFilePath(conf, changeFileExt(conf.projectFull, "js").prependCurDir))
execExternalProgram(conf, findNodeJs() & " " & ex & ' ' & conf.arguments)
else:
var binPath: AbsoluteFile
if not conf.outFile.isEmpty:
# If the user specified an outFile path, use that directly.
binPath = conf.outFile.prependCurDir
else:
# Figure out ourselves a valid binary name.
binPath = changeFileExt(conf.projectFull, ExeExt).prependCurDir
var ex = quoteShell(binPath)
execExternalProgram(conf, ex & ' ' & conf.arguments)
when declared(GC_setMaxPause):

1
compiler/nodejs.nim
View File

@@ -6,3 +6,4 @@ proc findNodeJs*(): string =
result = findExe("node")
if result == "":
result = findExe("iojs")

26
compiler/options.nim
View File

@@ -38,7 +38,6 @@ type # please make sure we have under 32 options
# evaluation
optPatterns, # en/disable pattern matching
optMemTracker,
optHotCodeReloading,
optLaxStrings,
optNilSeqs,
optOldAst
@@ -81,6 +80,7 @@ type # please make sure we have under 32 options
optMixedMode # true if some module triggered C++ codegen
optListFullPaths # use full paths in toMsgFilename, toFilename
optNoNimblePath
optHotCodeReloading
optDynlibOverrideAll
TGlobalOptions* = set[TGlobalOption]
@@ -215,7 +215,8 @@ type
packageCache*: StringTableRef
searchPaths*: seq[AbsoluteDir]
lazyPaths*: seq[AbsoluteDir]
outFile*: AbsoluteFile
outFile*: RelativeFile
outDir*: AbsoluteDir
prefixDir*, libpath*, nimcacheDir*: AbsoluteDir
dllOverrides, moduleOverrides*: StringTableRef
projectName*: string # holds a name like 'nim'
@@ -253,6 +254,8 @@ type
severity: Severity) {.closure.}
cppCustomNamespace*: string
proc hcrOn*(conf: ConfigRef): bool = return optHotCodeReloading in conf.globalOptions
template depConfigFields*(fn) {.dirty.} =
fn(target)
fn(options)
@@ -314,7 +317,9 @@ proc newConfigRef*(): ConfigRef =
packageCache: newPackageCache(),
searchPaths: @[],
lazyPaths: @[],
outFile: AbsoluteFile"", prefixDir: AbsoluteDir"",
outFile: RelativeFile"",
outDir: AbsoluteDir"",
prefixDir: AbsoluteDir"",
libpath: AbsoluteDir"", nimcacheDir: AbsoluteDir"",
dllOverrides: newStringTable(modeCaseInsensitive),
moduleOverrides: newStringTable(modeStyleInsensitive),
@@ -430,9 +435,6 @@ const
const oKeepVariableNames* = true
template compilingLib*(conf: ConfigRef): bool =
gGlobalOptions * {optGenGuiApp, optGenDynLib} != {}
proc mainCommandArg*(conf: ConfigRef): string =
## This is intended for commands like check or parse
## which will work on the main project file unless
@@ -452,8 +454,16 @@ proc setConfigVar*(conf: ConfigRef; key, val: string) =
conf.configVars[key] = val
proc getOutFile*(conf: ConfigRef; filename: RelativeFile, ext: string): AbsoluteFile =
if not conf.outFile.isEmpty: result = conf.outFile
else: result = conf.projectPath / changeFileExt(filename, ext)
result = (if conf.outFile.isEmpty: conf.projectPath else: conf.outDir) /
changeFileExt(filename, ext)
proc absOutFile*(conf: ConfigRef): AbsoluteFile =
conf.outDir / conf.outFile
proc prepareToWriteOutput*(conf: ConfigRef): AbsoluteFile =
## Create the output directory and returns a full path to the output file
createDir conf.outDir
return conf.outDir / conf.outFile
proc getPrefixDir*(conf: ConfigRef): AbsoluteDir =
## Gets the prefix dir, usually the parent directory where the binary resides.

4
compiler/passes.nim
View File

@@ -167,9 +167,9 @@ proc processModule*(graph: ModuleGraph; module: PSym, stream: PLLStream): bool {
if graph.stopCompile(): break
var n = parseTopLevelStmt(p)
if n.kind == nkEmpty: break
if sfSystemModule notin module.flags and
if (sfSystemModule notin module.flags and
({sfNoForward, sfReorder} * module.flags != {} or
codeReordering in graph.config.features):
codeReordering in graph.config.features)):
# read everything, no streaming possible
var sl = newNodeI(nkStmtList, n.info)
sl.add n

4
compiler/pathutils.nim
View File

@@ -42,6 +42,10 @@ proc cmpPaths*(x, y: AbsoluteDir): int {.borrow.}
proc createDir*(x: AbsoluteDir) {.borrow.}
proc toAbsoluteDir*(path: string): AbsoluteDir =
result = if path.isAbsolute: AbsoluteDir(path)
else: AbsoluteDir(getCurrentDir() / path)
proc `$`*(x: AnyPath): string = x.string
when true:

6
compiler/pragmas.nim
View File

@@ -21,7 +21,7 @@ const
const
procPragmas* = {FirstCallConv..LastCallConv, wImportc, wExportc, wNodecl,
wMagic, wNosideeffect, wSideeffect, wNoreturn, wDynlib, wHeader,
wCompilerProc, wCore, wProcVar, wDeprecated, wVarargs, wCompileTime, wMerge,
wCompilerProc, wNonReloadable, wCore, wProcVar, wDeprecated, wVarargs, wCompileTime, wMerge,
wBorrow, wExtern, wImportCompilerProc, wThread, wImportCpp, wImportObjC,
wAsmNoStackFrame, wError, wDiscardable, wNoInit, wCodegenDecl,
wGensym, wInject, wRaises, wTags, wLocks, wDelegator, wGcSafe, wOverride,
@@ -31,7 +31,7 @@ const
templatePragmas* = {wImmediate, wDeprecated, wError, wGensym, wInject, wDirty,
wDelegator, wExportNims, wUsed, wPragma}
macroPragmas* = {FirstCallConv..LastCallConv, wImmediate, wImportc, wExportc,
wNodecl, wMagic, wNosideeffect, wCompilerProc, wCore, wDeprecated, wExtern,
wNodecl, wMagic, wNosideeffect, wCompilerProc, wNonReloadable, wCore, wDeprecated, wExtern,
wImportCpp, wImportObjC, wError, wDiscardable, wGensym, wInject, wDelegator,
wExportNims, wUsed}
iteratorPragmas* = {FirstCallConv..LastCallConv, wNosideeffect, wSideeffect,
@@ -888,6 +888,8 @@ proc singlePragma(c: PContext, sym: PSym, n: PNode, i: var int,
cppDefine(c.graph.config, sym.name.s)
recordPragma(c, it, "cppdefine", sym.name.s)
if sfFromGeneric notin sym.flags: markCompilerProc(c, sym)
of wNonReloadable:
sym.flags.incl sfNonReloadable
of wProcVar:
noVal(c, it)
incl(sym.flags, sfProcvar)

5
compiler/semexprs.nim
View File

@@ -2535,19 +2535,22 @@ proc semExpr(c: PContext, n: PNode, flags: TExprFlags = {}): PNode =
result = semExpr(c, buildOverloadedSubscripts(n, getIdent(c.cache, "{}")), flags)
of nkPragmaExpr:
var
expr = n[0]
pragma = n[1]
pragmaName = considerQuotedIdent(c, pragma[0])
flags = flags
finalNodeFlags: TNodeFlags = {}
case whichKeyword(pragmaName)
of wExplain:
flags.incl efExplain
of wExecuteOnReload:
finalNodeFlags.incl nfExecuteOnReload
else:
# what other pragmas are allowed for expressions? `likely`, `unlikely`
invalidPragma(c, n)
result = semExpr(c, n[0], flags)
result.flags.incl finalNodeFlags
of nkPar, nkTupleConstr:
case checkPar(c, n)
of paNone: result = errorNode(c, n)

2
compiler/semfold.nim
View File

@@ -15,8 +15,6 @@ import
nversion, platform, math, msgs, os, condsyms, idents, renderer, types,
commands, magicsys, modulegraphs, strtabs, lineinfos
import system/indexerrors
proc newIntNodeT*(intVal: BiggestInt, n: PNode; g: ModuleGraph): PNode =
case skipTypes(n.typ, abstractVarRange).kind
of tyInt:

16
compiler/sighashes.nim
View File

@@ -260,9 +260,13 @@ proc hashType(c: var MD5Context, t: PType; flags: set[ConsiderFlag]) =
else:
for i in 0..<t.len: c.hashType(t.sons[i], flags)
c &= char(t.callConv)
if CoType notin flags:
if tfNoSideEffect in t.flags: c &= ".noSideEffect"
if tfThread in t.flags: c &= ".thread"
# purity of functions doesn't have to affect the mangling (which is in fact
# problematic for HCR - someone could have cached a pointer to another
# function which changes its purity and suddenly the cached pointer is danglign)
# IMHO anything that doesn't affect the overload resolution shouldn't be part of the mangling...
# if CoType notin flags:
# if tfNoSideEffect in t.flags: c &= ".noSideEffect"
# if tfThread in t.flags: c &= ".thread"
if tfVarargs in t.flags: c &= ".varargs"
of tyArray:
c &= char(t.kind)
@@ -344,6 +348,12 @@ proc hashOwner*(s: PSym): SigHash =
md5Final c, result.Md5Digest
proc sigHash*(s: PSym): SigHash =
if s.kind in routineKinds and s.typ != nil:
result = hashProc(s)
else:
result = hashNonProc(s)
proc idOrSig*(s: PSym, currentModule: string,
sigCollisions: var CountTable[SigHash]): Rope =
if s.kind in routineKinds and s.typ != nil:

18
compiler/transf.nim
View File

@@ -814,8 +814,9 @@ proc transformExceptBranch(c: PTransf, n: PNode): PTransNode =
proc dontInlineConstant(orig, cnst: PNode): bool {.inline.} =
# symbols that expand to a complex constant (array, etc.) should not be
# inlined, unless it's the empty array:
result = orig.kind == nkSym and cnst.kind in {nkCurly, nkPar, nkTupleConstr, nkBracket} and
cnst.len != 0
result = orig.kind == nkSym and
cnst.kind in {nkCurly, nkPar, nkTupleConstr, nkBracket} and
cnst.len != 0
proc commonOptimizations*(g: ModuleGraph; c: PSym, n: PNode): PNode =
result = n
@@ -1023,10 +1024,15 @@ proc transform(c: PTransf, n: PNode): PTransNode =
when false:
if oldDeferAnchor != nil: c.deferAnchor = oldDeferAnchor
var cnst = getConstExpr(c.module, PNode(result), c.graph)
# we inline constants if they are not complex constants:
if cnst != nil and not dontInlineConstant(n, cnst):
result = PTransNode(cnst) # do not miss an optimization
# Constants can be inlined here, but only if they cannot result in a cast
# in the back-end (e.g. var p: pointer = someProc)
let exprIsPointerCast = n.kind in {nkCast, nkConv, nkHiddenStdConv} and
n.typ.kind == tyPointer
if not exprIsPointerCast:
var cnst = getConstExpr(c.module, PNode(result), c.graph)
# we inline constants if they are not complex constants:
if cnst != nil and not dontInlineConstant(n, cnst):
result = PTransNode(cnst) # do not miss an optimization
proc processTransf(c: PTransf, n: PNode, owner: PSym): PNode =
# Note: For interactive mode we cannot call 'passes.skipCodegen' and skip

11
compiler/vm.nim
View File

@@ -11,12 +11,12 @@
## An instruction is 1-3 int32s in memory, it is a register based VM.
import ast except getstr
import system/indexerrors
import
strutils, astalgo, msgs, vmdef, vmgen, nimsets, types, passes,
parser, vmdeps, idents, trees, renderer, options, transf, parseutils,
vmmarshal, gorgeimpl, lineinfos, tables, btrees, macrocacheimpl
vmmarshal, gorgeimpl, lineinfos, tables, btrees, macrocacheimpl,
sighashes
from semfold import leValueConv, ordinalValToString
from evaltempl import evalTemplate
@@ -1492,6 +1492,13 @@ proc rawExecute(c: PCtx, start: int, tos: PStackFrame): TFullReg =
regs[ra].node.strVal = a.sym.name.s
else:
stackTrace(c, tos, pc, errFieldXNotFound & "strVal")
of opcNSigHash:
decodeB(rkNode)
createStr regs[ra]
if regs[rb].node.kind != nkSym:
stackTrace(c, tos, pc, "node is not a symbol")
else:
regs[ra].node.strVal = $sigHash(regs[rb].node.sym)
of opcSlurp:
decodeB(rkNode)
createStr regs[ra]

1
compiler/vmdef.nim
View File

@@ -92,6 +92,7 @@ type
opcNIdent,
opcNGetType,
opcNStrVal,
opcNSigHash,
opcNSetIntVal,
opcNSetFloatVal, opcNSetSymbol, opcNSetIdent, opcNSetType, opcNSetStrVal,

1
compiler/vmgen.nim
View File

@@ -1224,6 +1224,7 @@ proc genMagic(c: PCtx; n: PNode; dest: var TDest; m: TMagic) =
c.freeTemp(tmp)
#genUnaryABC(c, n, dest, opcNGetType)
of mNStrVal: genUnaryABC(c, n, dest, opcNStrVal)
of mNSigHash: genUnaryABC(c, n , dest, opcNSigHash)
of mNSetIntVal:
unused(c, n, dest)
genBinaryStmt(c, n, opcNSetIntVal)

2
compiler/wordrecg.nim
View File

@@ -53,6 +53,7 @@ type
wFastcall, wClosure, wNoconv, wOn, wOff, wChecks, wRangechecks,
wBoundchecks, wOverflowchecks, wNilchecks,
wFloatchecks, wNanChecks, wInfChecks, wMoveChecks,
wNonReloadable, wExecuteOnReload,
wAssertions, wPatterns, wWarnings,
wHints, wOptimization, wRaises, wWrites, wReads, wSize, wEffects, wTags,
wDeadCodeElimUnused, # deprecated, dead code elim always happens
@@ -142,6 +143,7 @@ const
"noconv", "on", "off", "checks", "rangechecks", "boundchecks",
"overflowchecks", "nilchecks",
"floatchecks", "nanchecks", "infchecks", "movechecks",
"nonreloadable", "executeonreload",
"assertions", "patterns", "warnings", "hints",
"optimization", "raises", "writes", "reads", "size", "effects", "tags",

1
doc/advopt.txt
View File

@@ -18,6 +18,7 @@ Advanced commands:
Advanced options:
-o:FILE, --out:FILE set the output filename
--outdir:DIR set the path where the output file will be written
--stdout:on|off output to stdout
--colors:on|off turn compiler messages coloring on|off
--listFullPaths:on|off list full paths in messages

98
doc/nimc.rst
View File

@@ -416,44 +416,92 @@ is raised.
Hot code reloading
------------------
**Note:** At the moment hot code reloading is supported only in
JavaScript projects.
The `hotCodeReloading`:idx: option enables special compilation mode where changes in
the code can be applied automatically to a running program. The code reloading
happens at the granularity of an individual module. When a module is reloaded,
Nim will preserve the state of all global variables which are initialized with
a standard variable declaration in the code. All other top level code will be
executed repeatedly on each reload. If you want to prevent this behavior, you
can guard a block of code with the ``once`` construct:
The `hotCodeReloading`:idx: option enables special compilation mode where
changes in the code can be applied automatically to a running program.
The code reloading happens at the granularity of an individual module.
When a module is reloaded, any newly added global variables will be
initialized, but all other top-level code appearing in the module won't
be re-executed and the state of all existing global variables will be
preserved. One can use the special event handlers ``beforeCodeReload`` and
``afterCodeReload`` to reset the state of a particular variable or to force
the execution of certain statements:
.. code-block:: Nim
var settings = initTable[string, string]()
var
settings = initTable[string, string]()
lastReload: Time
once:
myInit()
for k, v in loadSettings():
settings[k] = v
for k, v in loadSettings():
settings[k] = v
initProgram()
If you want to reset the state of a global variable on each reload, just
re-assign a value anywhere within the top-level code:
afterCodeReload:
lastReload = now()
resetProgramState()
On each code reload, Nim will first execute all `beforeCodeReload`:idx:
handlers registered in the previous version of the program and then all
`afterCodeReload`:idx handlers appearing in the newly loaded code. Please note
that any handlers appearing in modules that weren't reloaded will also be
executed. To prevent this behavior, one can guard the code with the
`hasModuleChanged()`:idx: API:
.. code-block:: Nim
var lastReload: Time
import mydb
lastReload = now()
resetProgramState()
var myCache = initTable[Key, Value]()
**Known limitations:** In the JavaScript target, global variables using the
``codegenDecl`` pragma will be re-initialized on each reload. Please guard the
initialization with a `once` block to work-around this.
afterCodeReload:
if hasModuleChanged(mydb):
resetCache(myCache)
The hot code reloading is based on dynamic library hot swapping in the native
targets and direct manipulation of the global namespace in the JavaScript
target. The Nim compiler does not specify the mechanism for detecting the
conditions when the code must be reloaded. Instead, the program code is
expected to call `performCodeReload()`:idx every time it wishes to reload
its code.
It's expected that most projects will implement the reloading with a suitable
build-system triggered IPC notification mechanism, but a polling solution is
also possible through the provided `hasAnyModuleChanged()`:idx API.
In order to access ``beforeCodeReload``, ``afterCodeReload``, ``hasModuleChanged``
or ``hasAnyModuleChanged`` one must import the `hotcodereloading`:idx module.
**Usage in Native projects:**
Native projects using the hot code reloading option will be implicitly
compiled with the `-d:useNimRtl` option and they will depend on both
the ``nimrtl`` library and the ``nimhcr`` library which implements the
hot code reloading run-time.
All modules of the project will be compiled to separate dynamic link
libraries placed in the ``nimcache`` directory. Please note that during
the execution of the program, the hot code reloading run-time will load
only copies of these libraries in order to not interfere with any newly
issued build commands.
The main module of the program is considered non-reloadable. Please note
that procs from reloadable modules should not appear in the call stack of
program while ``performCodeReload`` is being called. Thus, the main module
is a suitable place for implementing a program loop capable of calling
``performCodeReload``.
Please note that reloading won't be possible when any of the type definitions
in the program has been changed. When closure iterators are used (directly or
through async code), the reloaded refinitions will affect only newly created
instances. Existing iterator instancess will execute their original code to
completion.
**Usage in JavaScript projects:**
Once your code is compiled for hot reloading, you can use a framework such
as `LiveReload <http://livereload.com/>` or `BrowserSync <https://browsersync.io/>`
to implement the actual reloading behavior in your project.
Once your code is compiled for hot reloading, the ``nim-livereload`` NPM
package provides a convenient solution for implementing the actual reloading
in the browser using a framework such as [LiveReload](http://livereload.com/)
or [BrowserSync](https://browsersync.io/).
DynlibOverride

27
lib/core/hotcodereloading.nim Normal file
View File

@@ -0,0 +1,27 @@
when defined(hotcodereloading):
import
macros
template beforeCodeReload*(body: untyped) =
hcrAddEventHandler(true, proc = body) {.executeOnReload.}
template afterCodeReload*(body: untyped) =
hcrAddEventHandler(false, proc = body) {.executeOnReload.}
macro hasModuleChanged*(module: typed): untyped =
if module.kind != nnkSym or module.symKind != nskModule:
error "hasModuleChanged expects a module symbol", module
return newCall(bindSym"hcrHasModuleChanged", newLit(module.signatureHash))
proc hasAnyModuleChanged*(): bool = hcrReloadNeeded()
when not defined(JS):
template performCodeReload* = hcrPerformCodeReload()
else:
template performCodeReload* = discard
else:
template beforeCodeReload*(body: untyped) = discard
template afterCodeReload*(body: untyped) = discard
template hasModuleChanged*(module: typed): bool = false
proc hasAnyModuleChanged*(): bool = false
template performCodeReload*() = discard

7
lib/core/macros.nim
View File

@@ -347,6 +347,13 @@ object
doAssert(dumpTypeImpl(b) == t)
doAssert(dumpTypeImpl(c) == t)
when defined(nimHasSignatureHashInMacro):
proc signatureHash*(n: NimNode): string {.magic: "NSigHash", noSideEffect.}
## Returns a stable identifier derived from the signature of a symbol.
## The signature combines many factors such as the type of the symbol,
## the owning module of the symbol and others. The same identifier is
## used in the back-end to produce the mangled symbol name.
proc getTypeImpl*(n: typedesc): NimNode {.magic: "NGetType", noSideEffect.}
## Version of ``getTypeImpl`` which takes a ``typedesc``.

2
lib/core/strs.nim
View File

@@ -125,7 +125,7 @@ proc cstrToNimstr(str: cstring): NimStringV2 {.compilerRtl.} =
if str == nil: toNimStr(str, 0)
else: toNimStr(str, str.len)
proc nimToCStringConv(s: NimStringV2): cstring {.compilerProc, inline.} =
proc nimToCStringConv(s: NimStringV2): cstring {.compilerProc, nonReloadable, inline.} =
if s.len == 0: result = cstring""
else: result = cstring(unsafeAddr s.p.data)

2
lib/core/typeinfo.nim
View File

@@ -27,8 +27,6 @@
include "system/inclrtl.nim"
include "system/hti.nim"
import system/indexerrors
{.pop.}
type

5
lib/nimbase.h
View File

@@ -354,7 +354,12 @@ typedef NU8 NU;
# endif
#endif
// for now there isn't an easy way for C code to reach the program result
// when hot code reloading is ON - users will have to:
// load the nimhcr.dll, get the hcrGetGlobal proc from there and use it
#ifndef NIM_HOT_CODE_RELOADING
extern NI nim_program_result;
#endif
typedef float NF32;
typedef double NF64;

652
lib/nimhcr.nim Normal file
View File

@@ -0,0 +1,652 @@
#
#
# Nim's Runtime Library
# (c) Copyright 2018 Nim Contributors
#
# See the file "copying.txt", included in this
# distribution, for details about the copyright.
#
## This is the Nim hot code reloading run-time for the native targets.
##
## This minimal dynamic library is not subject to reloading when the
## `hotCodeReloading` build mode is enabled. It's responsible for providing
## a permanent memory location for all globals and procs within a program
## and orchestrating the reloading. For globals, this is easily achieved
## by storing them on the heap. For procs, we produce on the fly simple
## trampolines that can be dynamically overwritten to jump to a different
## target. In the host program, all globals and procs are first registered
## here with ``hcrRegisterGlobal`` and ``hcrRegisterProc`` and then the
## returned permanent locations are used in every reference to these symbols
## onwards.
##
## Detailed description:
##
## When code is compiled with the hotCodeReloading option for native targets
## a couple of things happen for all modules in a project:
## - the useNimRtl option is forced (including when building the HCR runtime too)
## - all modules of a target get built into separate shared libraries
## - the smallest granularity of reloads is modules
## - for each .c (or .cpp) in the corresponding nimcache folder of the project
## a shared object is built with the name of the source file + DLL extension
## - only the main module produces whatever the original project type intends
## (again in nimcache) and is then copied to its original destination
## - linking is done in parallel - just like compilation
## - function calls to functions from the same project go through function pointers:
## - with a few exceptions - see the nonReloadable pragma
## - the forward declarations of the original functions become function
## pointers as static globals with the same names
## - the original function definitions get suffixed with <name>_actual
## - the function pointers get initialized with the address of the corresponding
## function in the DatInit of their module through a call to either hcrRegisterProc
## or hcrGetProc. When being registered, the <name>_actual address is passed to
## hcrRegisterProc and a permanent location is returned and assigned to the pointer.
## This way the implementation (<name>_actual) can change but the address for it
## will be the same - this works by just updating a jump instruction (trampoline).
## For functions from other modules hcrGetProc is used (after they are registered).
## - globals are initialized only once and their state is preserved
## - including locals with the {.global.} pragma
## - their definitions are changed into pointer definitions which are initialized
## in the DatInit() of their module with calls to hcrRegisterGlobal (supplying the
## size of the type that this HCR runtime should allocate) and a bool is returned
## which when true triggers the initialization code for the global (only once).
## Globals from other modules: a global pointer coupled with a hcrGetGlobal call.
## - globals which have already been initialized cannot have their values changed
## by changing their initialization - use a handler or some other mechanism
## - new globals can be introduced when reloading
## - top-level code (global scope) is executed only once - at the first module load
## - the runtime knows every symbol's module owner (globals and procs)
## - both the RTL and HCR shared libraries need to be near the program for execution
## - same folder, in the PATH or LD_LIBRARY_PATH env var, etc (depending on OS)
## - the main module is responsible for initializing the HCR runtime
## - the main module loads the RTL and HCR shared objects
## - after that a call to hcrInit() is done in the main module which triggers
## the loading of all modules the main one imports, and doing that for the
## dependencies of each module recursively. Basically a DFS traversal.
## - then initialization takes place with several passes over all modules:
## - HcrInit - initializes the pointers for HCR procs such as hcrRegisterProc
## - HcrCreateTypeInfos - creates globals which will be referenced in the next pass
## - DatInit - usual dat init + register/get procs and get globals
## - Init - it does the following multiplexed operations:
## - register globals (if already registered - then just retrieve pointer)
## - execute top level scope (only if loaded for the first time)
## - when modules are loaded the originally built shared libraries get copied in
## the same folder and the copies are loaded instead of the original files
## - a module import tree is built in the runtime (and maintained when reloading)
## - hcrPerformCodeReload
## - named `performCodeReload`, requires the hotcodereloading module
## - explicitly called by the user - the current active callstack shouldn't contain
## any functions which are defined in modules that will be reloaded (or crash!).
## The reason is that old dynalic libraries get unloaded.
## Example:
## if A is the main module and it imports B, then only B is reloadable and only
## if when calling hcrPerformCodeReload there is no function defined in B in the
## current active callstack at the point of the call (it has to be done from A)
## - for reloading to take place the user has to have rebuilt parts of the application
## without changes affecting the main module in any way - it shouldn't be rebuilt.
## - to determine what needs to be reloaded the runtime starts traversing the import
## tree from the root and checks the timestamps of the loaded shared objects
## - modules that are no longer referenced are unloaded and cleaned up properly
## - symbols (procs/globals) that have been removed in the code are also cleaned up
## - so changing the init of a global does nothing, but removing it, reloading,
## and then re-introducing it with a new initializer works
## - new modules can be imported, and imports can also be reodereded/removed
## - hcrReloadNeeded() can be used to determine if any module needs reloading
## - named `hasAnyModuleChanged`, requires the hotcodereloading module
## - code in the beforeCodeReload/afterCodeReload handlers is executed on each reload
## - require the hotcodereloading module
## - such handlers can be added and removed
## - before each reload all "beforeCodeReload" handlers are executed and after
## that all handlers (including "after") from the particular module are deleted
## - the order of execution is the same as the order of top-level code execution.
## Example: if A imports B which imports C, then all handlers in C will be executed
## first (from top to bottom) followed by all from B and lastly all from A
## - after the reload all "after" handlers are executed the same way as "before"
## - the handlers for a reloaded module are always removed when reloading and then
## registered when the top-level scope is executed (thanks to `executeOnReload`)
##
## TODO - after first merge in upstream Nim:
##
## - profile
## - build speed with and without hot code reloading - difference should be small
## - runtime degradation of HCR-enabled code - important!!!
## - ARM support for the trampolines
## - investigate:
## - rethink the closure iterators
## - ability to keep old versions of dynamic libraries alive
## - because of async server code
## - perhaps with refcounting of .dlls for unfinished closures
## - linking with static libs
## - all shared objects for each module will (probably) have to link to them
## - state in static libs gets duplicated
## - linking is slow and therefore iteration time suffers
## - have just a single .dll for all .nim files and bulk reload?
## - think about the compile/link/passC/passL/emit/injectStmt pragmas
## - if a passC pragma is introduced (either written or dragged in by a new
## import) the whole command line for compilation changes - for example:
## winlean.nim: {.passC: "-DWIN32_LEAN_AND_MEAN".}
## - play with plugins/dlls/lfIndirect/lfDynamicLib/lfExportLib - shouldn't add an extra '*'
## - everything thread-local related
## - tests
## - add a new travis build matrix entry which builds everything with HCR enabled
## - currently building with useNimRtl is problematic - lots of problems...
## - how to supply the nimrtl/nimhcr shared objects to all test binaries...?
## - think about building to C++ instead of only to C - added type safety
## - run tests through valgrind and the sanitizers! of HUGE importance!
##
## TODO - nice to have cool stuff:
##
## - separate handling of global state for much faster reloading and manipulation
## - imagine sliders in an IDE for tweaking variables
## - perhaps using shared memory
## - multi-dll projects - how everything can be reloaded..?
## - a single HCR instance shared across multiple .dlls
## - instead of having to call hcrPerformCodeReload from a function in each dll
## - which currently renders the main module of each dll not reloadable
## - ability to check with the current callstack if a reload is "legal"
## - if it is in any function which is in a module about to be reloaded ==> error
## - pragma annotations for files - to be excluded from dll shenanigans
## - for such file-global pragmas look at codeReordering or injectStmt
## - how would the initialization order be kept? messy...
## - per function exclude pragmas would be TOO messy and hard...
## - C code calling stable exportc interface of nim code (for bindings)
## - generate proxy functions with the stable names
## - in a non-reloadable part (the main binary) that call the function pointers
## - parameter passing/forwarding - how? use the same trampoline jumping?
## - extracting the dependencies for these stubs/proxies will be hard...
## - changing memory layout of types - detecting this..?
## - implement with registerType() call to HCR runtime...?
## - and checking if a previously registered type matches
## - issue an error
## - or let the user handle this by transferring the state properly
## - perhaps in the before/afterCodeReload handlers
## - optimization: calls to procs within a module (+inlined) to use the _actual versions
## - implement executeOnReload for global vars too - not just statements (and document!)
## - cleanup at shutdown - freeing all globals
##
## TODO - unimportant:
##
## - have a "bad call" trampoline that all no-longer-present functions are routed to call there
## - so the user gets some error msg if he calls a dangling pointer instead of a crash
## - before/afterCodeReload and hasModuleChanged should be accessible only where appropriate
## - nim_program_result is inaccessible in HCR mode from external C code (see nimbase.h)
## - proper .json build file - but the format is different... multiple link commands...
## - avoid registering globals on each loop when using an iterator in global scope
##
## TODO - REPL:
## - proper way (as proposed by Zahary):
## - parse the input code and put everything in global scope except for
## statements with side effects only - those go in afterCodeReload blocks
## - my very hacky idea: just append to a closure iterator the new statements
## followed by a yield statement. So far I can think of 2 problems:
## - import and some other code cannot be written inside of a proc -
## has to be parsed and extracted in the outer scope
## - when new variables are created they are actually locals to the closure
## so the struct for the closure state grows in memory, but it has already
## been allocated when the closure was created with the previous smaller size.
## That would lead to working with memory outside of the initially allocated
## block. Perhaps something can be done about this - some way of re-allocating
## the state and transferring the old...
when not defined(JS) and (defined(hotcodereloading) or
defined(createNimHcr) or
defined(testNimHcr)):
const
dllExt = when defined(windows): "dll"
elif defined(macosx): "dylib"
else: "so"
type
HcrProcGetter* = proc (libHandle: pointer, procName: cstring): pointer {.nimcall.}
HcrGcMarkerProc = proc () {.nimcall.}
HcrModuleInitializer* = proc () {.nimcall.}
when defined(createNimHcr):
when system.appType != "lib":
{.error: "This file has to be compiled as a library!".}
import os, tables, sets, times, strutils, reservedmem, dynlib
template trace(args: varargs[untyped]) =
when defined(testNimHcr) or defined(traceHcr):
echo args
proc sanitize(arg: Time): string =
when defined(testNimHcr): return "<time>"
else: return $arg
proc sanitize(arg: string|cstring): string =
when defined(testNimHcr): return ($arg).splitFile.name.splitFile.name
else: return $arg
{.pragma: nimhcr, compilerProc, exportc, dynlib.}
when hostCPU in ["i386", "amd64"]:
type
ShortJumpInstruction {.packed.} = object
opcode: byte
offset: int32
LongJumpInstruction {.packed.} = object
opcode1: byte
opcode2: byte
offset: int32
absoluteAddr: pointer
proc writeJump(jumpTableEntry: ptr LongJumpInstruction, targetFn: pointer) =
let
jumpFrom = jumpTableEntry.shift(sizeof(ShortJumpInstruction))
jumpDistance = distance(jumpFrom, targetFn)
if abs(jumpDistance) < 0x7fff0000:
let shortJump = cast[ptr ShortJumpInstruction](jumpTableEntry)
shortJump.opcode = 0xE9 # relative jump
shortJump.offset = int32(jumpDistance)
else:
jumpTableEntry.opcode1 = 0xff # indirect absolute jump
jumpTableEntry.opcode2 = 0x25
when hostCPU == "i386":
# on x86 we write the absolute address of the following pointer
jumpTableEntry.offset = cast[int32](addr jumpTableEntry.absoluteAddr)
else:
# on x64, we use a relative address for the same location
jumpTableEntry.offset = 0
jumpTableEntry.absoluteAddr = targetFn
elif hostCPU == "arm":
const jumpSize = 8
elif hostCPU == "arm64":
const jumpSize = 16
const defaultJumpTableSize = case hostCPU
of "i386": 50
of "amd64": 500
else: 50
let jumpTableSizeStr = getEnv("HOT_CODE_RELOADING_JUMP_TABLE_SIZE")
let jumpTableSize = if jumpTableSizeStr.len > 0: parseInt(jumpTableSizeStr)
else: defaultJumpTableSize
# TODO: perhaps keep track of free slots due to removed procs using a free list
var jumpTable = ReservedMemSeq[LongJumpInstruction].init(
memStart = cast[pointer](0x10000000),
maxLen = jumpTableSize * 1024 * 1024 div sizeof(LongJumpInstruction),
accessFlags = memExecReadWrite)
type
ProcSym = object
jump: ptr LongJumpInstruction
gen: int
GlobalVarSym = object
p: pointer
markerProc: HcrGcMarkerProc
gen: int
ModuleDesc = object
procs: Table[string, ProcSym]
globals: Table[string, GlobalVarSym]
imports: seq[string]
handle: LibHandle
hash: string
gen: int
lastModification: Time
handlers: seq[tuple[isBefore: bool, cb: proc ()]]
proc newModuleDesc(): ModuleDesc =
result.procs = initTable[string, ProcSym]()
result.globals = initTable[string, GlobalVarSym]()
result.handle = nil
result.gen = -1
result.lastModification = low(Time)
# the global state necessary for traversing and reloading the module import tree
var modules = initTable[string, ModuleDesc]()
var root: string
var system: string
var mainDatInit: HcrModuleInitializer
var generation = 0
# necessary for queries such as "has module X changed" - contains all but the main module
var hashToModuleMap = initTable[string, string]()
# necessary for registering handlers and keeping them up-to-date
var currentModule: string
# supplied from the main module - used by others to initialize pointers to this runtime
var hcrDynlibHandle: pointer
var getProcAddr: HcrProcGetter
proc hcrRegisterProc*(module: cstring, name: cstring, fn: pointer): pointer {.nimhcr.} =
trace " register proc: ", module.sanitize, " ", name
# Please note: We must allocate a local copy of the strings, because the supplied
# `cstring` will reside in the data segment of a DLL that will be later unloaded.
let name = $name
let module = $module
var jumpTableEntryAddr: ptr LongJumpInstruction
modules[module].procs.withValue(name, p):
trace " update proc: ", name
jumpTableEntryAddr = p.jump
p.gen = generation
do:
let len = jumpTable.len
jumpTable.setLen(len + 1)
jumpTableEntryAddr = addr jumpTable[len]
modules[module].procs[name] = ProcSym(jump: jumpTableEntryAddr, gen: generation)
writeJump jumpTableEntryAddr, fn
return jumpTableEntryAddr
proc hcrGetProc*(module: cstring, name: cstring): pointer {.nimhcr.} =
trace " get proc: ", module.sanitize, " ", name
return modules[$module].procs[$name].jump
proc hcrRegisterGlobal*(module: cstring,
name: cstring,
size: Natural,
gcMarker: HcrGcMarkerProc,
outPtr: ptr pointer): bool {.nimhcr.} =
trace " register global: ", module.sanitize, " ", name
# Please note: We must allocate local copies of the strings, because the supplied
# `cstring` will reside in the data segment of a DLL that will be later unloaded.
# Also using a ptr pointer instead of a var pointer (an output parameter)
# because for the C++ backend var parameters use references and in this use case
# it is not possible to cast an int* (for example) to a void* and then pass it
# to void*& since the casting yields an rvalue and references bind only to lvalues.
let name = $name
let module = $module
modules[module].globals.withValue(name, global):
trace " update global: ", name
outPtr[] = global.p
global.gen = generation
global.markerProc = gcMarker
return false
do:
outPtr[] = alloc0(size)
modules[module].globals[name] = GlobalVarSym(p: outPtr[],
gen: generation,
markerProc: gcMarker)
return true
proc hcrGetGlobal*(module: cstring, name: cstring): pointer {.nimhcr.} =
trace " get global: ", module.sanitize, " ", name
return modules[$module].globals[$name].p
proc getListOfModules(cstringArray: ptr pointer): seq[string] =
var curr = cast[ptr cstring](cstringArray)
while len(curr[]) > 0:
result.add($curr[])
curr = cast[ptr cstring](cast[int64](curr) + sizeof(ptr cstring))
template cleanup(collection, body) =
var toDelete: seq[string]
for name, data in collection.pairs:
if data.gen < generation:
toDelete.add(name)
trace "HCR Cleaning ", astToStr(collection), " :: ", name, " ", data.gen
for name {.inject.} in toDelete:
body
proc cleanupGlobal(module: string, name: string) =
var g: GlobalVarSym
if modules[module].globals.take(name, g):
dealloc g.p
proc cleanupSymbols(module: string) =
cleanup modules[module].globals:
cleanupGlobal(module, name)
cleanup modules[module].procs:
modules[module].procs.del(name)
proc unloadDll(name: string) =
if modules[name].handle != nil:
unloadLib(modules[name].handle)
proc loadDll(name: cstring) {.nimhcr.} =
let name = $name
trace "HCR LOADING: ", name.sanitize
if modules.contains(name):
unloadDll(name)
else:
modules.add(name, newModuleDesc())
let copiedName = name & ".copy." & dllExt
copyFile(name, copiedName)
let lib = loadLib(copiedName)
assert lib != nil
modules[name].handle = lib
modules[name].gen = generation
modules[name].lastModification = getLastModificationTime(name)
# update the list of imports by the module
let getImportsProc = cast[proc (): ptr pointer {.nimcall.}](
checkedSymAddr(lib, "HcrGetImportedModules"))
modules[name].imports = getListOfModules(getImportsProc())
# get the hash of the module
let getHashProc = cast[proc (): cstring {.nimcall.}](
checkedSymAddr(lib, "HcrGetSigHash"))
modules[name].hash = $getHashProc()
hashToModuleMap[modules[name].hash] = name
# Remove handlers for this module if reloading - they will be re-registered.
# In order for them to be re-registered we need to de-register all globals
# that trigger the registering of handlers through calls to hcrAddEventHandler
modules[name].handlers.setLen(0)
proc initHcrData(name: cstring) {.nimhcr.} =
trace "HCR Hcr init: ", name.sanitize
cast[proc (h: pointer, gpa: HcrProcGetter) {.nimcall.}](
checkedSymAddr(modules[$name].handle, "HcrInit000"))(hcrDynlibHandle, getProcAddr)
proc initTypeInfoGlobals(name: cstring) {.nimhcr.} =
trace "HCR TypeInfo globals init: ", name.sanitize
cast[HcrModuleInitializer](checkedSymAddr(modules[$name].handle, "HcrCreateTypeInfos"))()
proc initPointerData(name: cstring) {.nimhcr.} =
trace "HCR Dat init: ", name.sanitize
cast[HcrModuleInitializer](checkedSymAddr(modules[$name].handle, "DatInit000"))()
proc initGlobalScope(name: cstring) {.nimhcr.} =
trace "HCR Init000: ", name.sanitize
# set the currently inited module - necessary for registering the before/after HCR handlers
currentModule = $name
cast[HcrModuleInitializer](checkedSymAddr(modules[$name].handle, "Init000"))()
var modulesToInit: seq[string] = @[]
var allModulesOrderedByDFS: seq[string] = @[]
proc recursiveDiscovery(dlls: seq[string]) =
for curr in dlls:
if modules.contains(curr):
# skip updating modules that have already been updated to the latest generation
if modules[curr].gen >= generation:
trace "HCR SKIP: ", curr.sanitize, " gen is already: ", modules[curr].gen
continue
# skip updating an unmodified module but continue traversing its dependencies
if modules[curr].lastModification >= getLastModificationTime(curr):
trace "HCR SKIP (not modified): ", curr.sanitize, " ", modules[curr].lastModification.sanitize
# update generation so module doesn't get collected
modules[curr].gen = generation
# recurse to imported modules - they might be changed
recursiveDiscovery(modules[curr].imports)
allModulesOrderedByDFS.add(curr)
continue
loadDll(curr)
# first load all dependencies of the current module and init it after that
recursiveDiscovery(modules[curr].imports)
allModulesOrderedByDFS.add(curr)
modulesToInit.add(curr)
proc initModules() =
# first init the pointers to hcr functions and also do the registering of typeinfo globals
for curr in modulesToInit:
initHcrData(curr)
initTypeInfoGlobals(curr)
# for now system always gets fully inited before any other module (including when reloading)
initPointerData(system)
initGlobalScope(system)
# proceed with the DatInit calls - for all modules - including the main one!
for curr in allModulesOrderedByDFS:
if curr != system:
initPointerData(curr)
mainDatInit()
# execute top-level code (in global scope)
for curr in modulesToInit:
if curr != system:
initGlobalScope(curr)
# cleanup old symbols which are gone now
for curr in modulesToInit:
cleanupSymbols(curr)
proc hcrInit*(moduleList: ptr pointer, main, sys: cstring,
datInit: HcrModuleInitializer, handle: pointer, gpa: HcrProcGetter) {.nimhcr.} =
trace "HCR INITING: ", main.sanitize, " gen: ", generation
# initialize globals
root = $main
system = $sys
mainDatInit = datInit
hcrDynlibHandle = handle
getProcAddr = gpa
# the root is already added and we need it because symbols from it will also be registered in the HCR system
modules[root].imports = getListOfModules(moduleList)
modules[root].gen = high(int) # something huge so it doesn't get collected
# recursively initialize all modules
recursiveDiscovery(modules[root].imports)
initModules()
# the next module to be inited will be the root
currentModule = root
proc hcrHasModuleChanged*(moduleHash: string): bool {.nimhcr.} =
let module = hashToModuleMap[moduleHash]
return modules[module].lastModification < getLastModificationTime(module)
proc hcrReloadNeeded*(): bool {.nimhcr.} =
for hash, _ in hashToModuleMap:
if hcrHasModuleChanged(hash):
return true
return false
proc hcrPerformCodeReload*() {.nimhcr.} =
if not hcrReloadNeeded():
trace "HCR - no changes"
return
# We disable the GC during the reload, because the reloading procedures
# will replace type info objects and GC marker procs. This seems to create
# problems when the GC is executed while the reload is underway.
# Future versions of NIMHCR won't use the GC, because all globals and the
# metadata needed to access them will be placed in shared memory, so they
# can be manipulted from external programs without reloading.
GC_disable()
defer: GC_enable()
inc(generation)
trace "HCR RELOADING: ", generation
var traversedHandlerModules = initSet[string]()
proc recursiveExecuteHandlers(isBefore: bool, module: string) =
# do not process an already traversed module
if traversedHandlerModules.containsOrIncl(module): return
traversedHandlerModules.incl module
# first recurse to do a DFS traversal
for curr in modules[module].imports:
recursiveExecuteHandlers(isBefore, curr)
# and then execute the handlers - from leaf modules all the way up to the root module
for curr in modules[module].handlers:
if curr.isBefore == isBefore:
curr.cb()
# first execute the before reload handlers
traversedHandlerModules.clear()
recursiveExecuteHandlers(true, root)
# do the reloading
modulesToInit = @[]
allModulesOrderedByDFS = @[]
recursiveDiscovery(modules[root].imports)
initModules()
# execute the after reload handlers
traversedHandlerModules.clear()
recursiveExecuteHandlers(false, root)
# collecting no longer referenced modules - based on their generation
cleanup modules:
cleanupSymbols(name)
unloadDll(name)
hashToModuleMap.del(modules[name].hash)
modules.del(name)
proc hcrAddEventHandler*(isBefore: bool, cb: proc ()) {.nimhcr.} =
modules[currentModule].handlers.add(
(isBefore: isBefore, cb: cb))
proc hcrAddModule*(module: cstring) {.nimhcr.} =
if not modules.contains($module):
modules.add($module, newModuleDesc())
proc hcrGeneration*(): int {.nimhcr.} =
generation
proc hcrMarkGlobals*() {.nimhcr, nimcall, gcsafe.} =
# This is gcsafe, because it will be registered
# only in the GC of the main thread.
{.gcsafe.}:
for _, module in modules:
for _, global in module.globals:
if global.markerProc != nil:
global.markerProc()
elif defined(hotcodereloading) or defined(testNimHcr):
when not defined(JS):
const
nimhcrLibname = when defined(windows): "nimhcr." & dllExt
elif defined(macosx): "libnimhcr." & dllExt
else: "libnimhcr." & dllExt
{.pragma: nimhcr, compilerProc, importc, dynlib: nimhcrLibname.}
proc hcrRegisterProc*(module: cstring, name: cstring, fn: pointer): pointer {.nimhcr.}
proc hcrGetProc*(module: cstring, name: cstring): pointer {.nimhcr.}
proc hcrRegisterGlobal*(module: cstring, name: cstring, size: Natural,
gcMarker: HcrGcMarkerProc, outPtr: ptr pointer): bool {.nimhcr.}
proc hcrGetGlobal*(module: cstring, name: cstring): pointer {.nimhcr.}
proc hcrInit*(moduleList: ptr pointer,
main, sys: cstring,
datInit: HcrModuleInitializer,
handle: pointer,
gpa: HcrProcGetter) {.nimhcr.}
proc hcrAddModule*(module: cstring) {.nimhcr.}
proc hcrHasModuleChanged*(moduleHash: string): bool {.nimhcr.}
proc hcrReloadNeeded*(): bool {.nimhcr.}
proc hcrPerformCodeReload*() {.nimhcr.}
proc hcrAddEventHandler*(isBefore: bool, cb: proc ()) {.nimhcr.}
proc hcrMarkGlobals*() {.nimhcr, nimcall, gcsafe.}
when declared(nimRegisterGlobalMarker):
nimRegisterGlobalMarker(hcrMarkGlobals)
else:
proc hcrHasModuleChanged*(moduleHash: string): bool =
# TODO
false
proc hcrAddEventHandler*(isBefore: bool, cb: proc ()) =
# TODO
discard

5
lib/nimhcr.nim.cfg Normal file
View File

@@ -0,0 +1,5 @@
--app:lib
--threads:on
-d:useNimRtl
-d:createNimHcr

1
lib/nimrtl.nim.cfg
View File

@@ -1,5 +1,6 @@
# The RTL.dll needs to be compiled with these options!
--app:lib
--threads:on
--define:createNimRtl

2
lib/pure/collections/sharedstrings.nim
View File

@@ -12,8 +12,6 @@
type
UncheckedCharArray = UncheckedArray[char]
import system/indexerrors
type
Buffer = ptr object
refcount: int

2
lib/pure/os.nim
View File

@@ -47,7 +47,7 @@
include "system/inclrtl"
import
strutils, pathnorm, system/indexerrors
strutils, pathnorm
const weirdTarget = defined(nimscript) or defined(js)

241
lib/pure/reservedmem.nim Normal file
View File

@@ -0,0 +1,241 @@
#
#
# Nim's Runtime Library
# (c) Copyright 2015 Nim Contributors
#
# See the file "copying.txt", included in this
# distribution, for details about the copyright.
#
## :Authors: Zahary Karadjov
##
## This module provides utilities for reserving a portions of the
## address space of a program without consuming physical memory.
## It can be used to implement a dynamically resizable buffer that
## is guaranteed to remain in the same memory location. The buffer
## will be able to grow up to the size of the initially reserved
## portion of the address space.
from ospaths import raiseOSError, osLastError
template distance*(lhs, rhs: pointer): int =
cast[int](rhs) - cast[int](lhs)
template shift*(p: pointer, distance: int): pointer =
cast[pointer](cast[int](p) + distance)
type
MemAccessFlags* = int
ReservedMem* = object
memStart: pointer
usedMemEnd: pointer
committedMemEnd: pointer
memEnd: pointer
maxCommittedAndUnusedPages: int
accessFlags: MemAccessFlags
ReservedMemSeq*[T] = object
mem: ReservedMem
when defined(windows):
import winlean
type
SYSTEM_INFO {.final, pure.} = object
u1: uint32
dwPageSize: uint32
lpMinimumApplicationAddress: pointer
lpMaximumApplicationAddress: pointer
dwActiveProcessorMask: ptr uint32
dwNumberOfProcessors: uint32
dwProcessorType: uint32
dwAllocationGranularity: uint32
wProcessorLevel: uint16
wProcessorRevision: uint16
proc getSystemInfo(lpSystemInfo: ptr SYSTEM_INFO) {.stdcall, dynlib: "kernel32", importc: "GetSystemInfo".}
proc getAllocationGranularity: uint =
var sysInfo: SYSTEM_INFO
getSystemInfo(addr sysInfo)
return uint(sysInfo.dwAllocationGranularity)
let allocationGranularity = getAllocationGranularity().int
const
memNoAccess = MemAccessFlags(PAGE_NOACCESS)
memExec* = MemAccessFlags(PAGE_EXECUTE)
memExecRead* = MemAccessFlags(PAGE_EXECUTE_READ)
memExecReadWrite* = MemAccessFlags(PAGE_EXECUTE_READWRITE)
memRead* = MemAccessFlags(PAGE_READONLY)
memReadWrite* = MemAccessFlags(PAGE_READWRITE)
template check(expr) =
let r = expr
if r == cast[type(r)](0):
raiseOSError(osLastError())
else:
import posix
var MAP_ANONYMOUS {.importc: "MAP_ANONYMOUS", header: "<sys/mman.h>".}: cint
var MAP_NORESERVE {.importc: "MAP_NORESERVE", header: "<sys/mman.h>".}: cint
# var MAP_FIXED_NOREPLACE {.importc: "MAP_FIXED_NOREPLACE", header: "<sys/mman.h>".}: cint
var SC_PAGESIZE {.importc: "_SC_PAGESIZE", header: "<unistd.h>".}: cint
let allocationGranularity = sysconf(SC_PAGESIZE)
let
memNoAccess = MemAccessFlags(PROT_NONE)
memExec* = MemAccessFlags(PROT_EXEC)
memExecRead* = MemAccessFlags(PROT_EXEC or PROT_READ)
memExecReadWrite* = MemAccessFlags(PROT_EXEC or PROT_READ or PROT_WRITE)
memRead* = MemAccessFlags(PROT_READ)
memReadWrite* = MemAccessFlags(PROT_READ or PROT_WRITE)
template check(expr) =
if not expr:
raiseOSError(osLastError())
func nextAlignedOffset(n, alignment: int): int =
result = n
let m = n mod alignment
if m != 0: result += alignment - m
when defined(windows):
const
MEM_DECOMMIT = 0x4000
MEM_RESERVE = 0x2000
MEM_COMMIT = 0x1000
proc virtualFree(lpAddress: pointer, dwSize: int,
dwFreeType: int32): cint {.header: "<windows.h>", stdcall,
importc: "VirtualFree".}
proc virtualAlloc(lpAddress: pointer, dwSize: int, flAllocationType,
flProtect: int32): pointer {.
header: "<windows.h>", stdcall, importc: "VirtualAlloc".}
proc init*(T: type ReservedMem,
maxLen: Natural,
initLen: Natural = 0,
initCommitLen = initLen,
memStart = pointer(nil),
accessFlags = memReadWrite,
maxCommittedAndUnusedPages = 3): ReservedMem =
assert initLen <= initCommitLen
let commitSize = nextAlignedOffset(initCommitLen, allocationGranularity)
when defined(windows):
result.memStart = virtualAlloc(memStart, maxLen, MEM_RESERVE, accessFlags.cint)
check result.memStart
if commitSize > 0:
check virtualAlloc(result.memStart, commitSize, MEM_COMMIT, accessFlags.cint)
else:
var allocFlags = MAP_PRIVATE or MAP_ANONYMOUS # or MAP_NORESERVE
# if memStart != nil:
# allocFlags = allocFlags or MAP_FIXED_NOREPLACE
result.memStart = mmap(memStart, maxLen, PROT_NONE, allocFlags, -1, 0)
check result.memStart != MAP_FAILED
if commitSize > 0:
check mprotect(result.memStart, commitSize, cint(accessFlags)) == 0
result.usedMemEnd = result.memStart.shift(initLen)
result.committedMemEnd = result.memStart.shift(commitSize)
result.memEnd = result.memStart.shift(maxLen)
result.accessFlags = accessFlags
result.maxCommittedAndUnusedPages = maxCommittedAndUnusedPages
func len*(m: ReservedMem): int =
distance(m.memStart, m.usedMemEnd)
func commitedLen*(m: ReservedMem): int =
distance(m.memStart, m.committedMemEnd)
func maxLen*(m: ReservedMem): int =
distance(m.memStart, m.memEnd)
proc setLen*(m: var ReservedMem, newLen: int) =
let len = m.len
m.usedMemEnd = m.memStart.shift(newLen)
if newLen > len:
let d = distance(m.committedMemEnd, m.usedMemEnd)
if d > 0:
let commitExtensionSize = nextAlignedOffset(d, allocationGranularity)
when defined(windows):
check virtualAlloc(m.committedMemEnd, commitExtensionSize,
MEM_COMMIT, m.accessFlags.cint)
else:
check mprotect(m.committedMemEnd, commitExtensionSize, m.accessFlags.cint) == 0
else:
let d = distance(m.usedMemEnd, m.committedMemEnd) -
m.maxCommittedAndUnusedPages * allocationGranularity
if d > 0:
let commitSizeShrinkage = nextAlignedOffset(d, allocationGranularity)
let newCommitEnd = m.committedMemEnd.shift(-commitSizeShrinkage)
when defined(windows):
check virtualFree(newCommitEnd, commitSizeShrinkage, MEM_DECOMMIT)
else:
check posix_madvise(newCommitEnd, commitSizeShrinkage,
POSIX_MADV_DONTNEED) == 0
m.committedMemEnd = newCommitEnd
proc init*(SeqType: type ReservedMemSeq,
maxLen: Natural,
initLen: Natural = 0,
initCommitLen: Natural = 0,
memStart = pointer(nil),
accessFlags = memReadWrite,
maxCommittedAndUnusedPages = 3): SeqType =
let elemSize = sizeof(SeqType.T)
result.mem = ReservedMem.init(maxLen * elemSize,
initLen * elemSize,
initCommitLen * elemSize,
memStart, accessFlags,
maxCommittedAndUnusedPages)
func `[]`*[T](s: ReservedMemSeq[T], pos: Natural): lent T =
let elemAddr = s.mem.memStart.shift(pos * sizeof(T))
rangeCheck elemAddr < s.mem.usedMemEnd
result = (cast[ptr T](elemAddr))[]
func `[]`*[T](s: var ReservedMemSeq[T], pos: Natural): var T =
let elemAddr = s.mem.memStart.shift(pos * sizeof(T))
rangeCheck elemAddr < s.mem.usedMemEnd
result = (cast[ptr T](elemAddr))[]
func `[]`*[T](s: ReservedMemSeq[T], rpos: BackwardsIndex): lent T =
return s[int(s.len) - int(rpos)]
func `[]`*[T](s: var ReservedMemSeq[T], rpos: BackwardsIndex): var T =
return s[int(s.len) - int(rpos)]
func len*[T](s: ReservedMemSeq[T]): int =
s.mem.len div sizeof(T)
proc setLen*[T](s: var ReservedMemSeq[T], newLen: int) =
# TODO call destructors
s.mem.setLen(newLen * sizeof(T))
proc add*[T](s: var ReservedMemSeq[T], val: T) =
let len = s.len
s.setLen(len + 1)
s[len] = val
proc pop*[T](s: var ReservedMemSeq[T]): T =
assert s.usedMemEnd != s.memStart
let lastIdx = s.len - 1
result = s[lastIdx]
s.setLen(lastIdx)
func commitedLen*[T](s: ReservedMemSeq[T]): int =
s.mem.commitedLen div sizeof(T)
func maxLen*[T](s: ReservedMemSeq[T]): int =
s.mem.maxLen div sizeof(T)

6
lib/pure/strformat.nim
View File

@@ -285,7 +285,11 @@ macro `&`*(pattern: string): untyped =
var i = 0
let res = genSym(nskVar, "fmtRes")
result = newNimNode(nnkStmtListExpr, lineInfoFrom=pattern)
result.add newVarStmt(res, newCall(bindSym"newStringOfCap", newLit(f.len + count(f, '{')*10)))
# XXX: https://github.com/nim-lang/Nim/issues/8405
# When compiling with -d:useNimRtl, certain procs such as `count` from the strutils
# module are not accessible at compile-time:
let expectedGrowth = when defined(useNimRtl): 0 else: count(f, '{') * 10
result.add newVarStmt(res, newCall(bindSym"newStringOfCap", newLit(f.len + expectedGrowth)))
var strlit = ""
while i < f.len:
if f[i] == '{':

87
lib/pure/strtabs.nim
View File

@@ -143,51 +143,6 @@ template get(t: StringTableRef, key: string) =
else:
raise newException(KeyError, "key not found")
proc `[]=`*(t: StringTableRef, key, val: string) {.
rtlFunc, extern: "nstPut", noSideEffect.}
proc newStringTable*(mode: StringTableMode): StringTableRef {.
rtlFunc, extern: "nst$1".} =
## Creates a new empty string table.
##
## See also:
## * `newStringTable(keyValuePairs) proc
## <#newStringTable,varargs[tuple[string,string]],StringTableMode>`_
new(result)
result.mode = mode
result.counter = 0
newSeq(result.data, startSize)
proc newStringTable*(keyValuePairs: varargs[string],
mode: StringTableMode): StringTableRef {.
rtlFunc, extern: "nst$1WithPairs".} =
## Creates a new string table with given `key, value` string pairs.
##
## `StringTableMode` must be specified.
runnableExamples:
var mytab = newStringTable("key1", "val1", "key2", "val2",
modeCaseInsensitive)
result = newStringTable(mode)
var i = 0
while i < high(keyValuePairs):
result[keyValuePairs[i]] = keyValuePairs[i + 1]
inc(i, 2)
proc newStringTable*(keyValuePairs: varargs[tuple[key, val: string]],
mode: StringTableMode = modeCaseSensitive): StringTableRef {.
rtlFunc, extern: "nst$1WithTableConstr".} =
## Creates a new string table with given `(key, value)` tuple pairs.
##
## The default mode is case sensitive.
runnableExamples:
var
mytab1 = newStringTable({"key1": "val1", "key2": "val2"}, modeCaseInsensitive)
mytab2 = newStringTable([("key3", "val3"), ("key4", "val4")])
result = newStringTable(mode)
for key, val in items(keyValuePairs): result[key] = val
proc len*(t: StringTableRef): int {.rtlFunc, extern: "nst$1".} =
## Returns the number of keys in `t`.
result = t.counter
@@ -292,6 +247,48 @@ proc `[]=`*(t: StringTableRef, key, val: string) {.
rawInsert(t, t.data, key, val)
inc(t.counter)
proc newStringTable*(mode: StringTableMode): StringTableRef {.
rtlFunc, extern: "nst$1".} =
## Creates a new empty string table.
##
## See also:
## * `newStringTable(keyValuePairs) proc
## <#newStringTable,varargs[tuple[string,string]],StringTableMode>`_
new(result)
result.mode = mode
result.counter = 0
newSeq(result.data, startSize)
proc newStringTable*(keyValuePairs: varargs[string],
mode: StringTableMode): StringTableRef {.
rtlFunc, extern: "nst$1WithPairs".} =
## Creates a new string table with given `key, value` string pairs.
##
## `StringTableMode` must be specified.
runnableExamples:
var mytab = newStringTable("key1", "val1", "key2", "val2",
modeCaseInsensitive)
result = newStringTable(mode)
var i = 0
while i < high(keyValuePairs):
result[keyValuePairs[i]] = keyValuePairs[i + 1]
inc(i, 2)
proc newStringTable*(keyValuePairs: varargs[tuple[key, val: string]],
mode: StringTableMode = modeCaseSensitive): StringTableRef {.
rtlFunc, extern: "nst$1WithTableConstr".} =
## Creates a new string table with given `(key, value)` tuple pairs.
##
## The default mode is case sensitive.
runnableExamples:
var
mytab1 = newStringTable({"key1": "val1", "key2": "val2"}, modeCaseInsensitive)
mytab2 = newStringTable([("key3", "val3"), ("key4", "val4")])
result = newStringTable(mode)
for key, val in items(keyValuePairs): result[key] = val
proc raiseFormatException(s: string) =
var e: ref ValueError
new(e)

20
lib/system.nim
View File

@@ -417,6 +417,12 @@ when not defined(niminheritable):
{.pragma: inheritable.}
when not defined(nimunion):
{.pragma: unchecked.}
when not defined(nimHasHotCodeReloading):
{.pragma: nonReloadable.}
when defined(hotCodeReloading):
{.pragma: hcrInline, inline.}
else:
{.pragma: hcrInline.}
# comparison operators:
proc `==`*[Enum: enum](x, y: Enum): bool {.magic: "EqEnum", noSideEffect.}
@@ -1590,7 +1596,7 @@ when defined(nodejs) and not defined(nimscript):
var programResult* {.importc: "process.exitCode".}: int
programResult = 0
else:
var programResult* {.exportc: "nim_program_result".}: int
var programResult* {.compilerproc, exportc: "nim_program_result".}: int
## modify this variable to specify the exit code of the program
## under normal circumstances. When the program is terminated
## prematurely using ``quit``, this value is ignored.
@@ -3802,6 +3808,15 @@ template doAssert*(cond: untyped, msg = "") =
const expr = astToStr(cond)
assertImpl(cond, msg, expr, true)
when compileOption("rangechecks"):
template rangeCheck*(cond) =
## Helper for performing user-defined range checks.
## Such checks will be performed only when the ``rangechecks``
## compile-time option is enabled.
if not cond: sysFatal(RangeError, "range check failed")
else:
template rangeCheck*(cond) = discard
iterator items*[T](a: seq[T]): T {.inline.} =
## iterates over each item of `a`.
var i = 0
@@ -4251,3 +4266,6 @@ export widestrs
import system/io
export io
when not defined(createNimHcr):
include nimhcr

8
lib/system/cgprocs.nim
View File

@@ -13,8 +13,8 @@ type
LibHandle = pointer # private type
ProcAddr = pointer # library loading and loading of procs:
proc nimLoadLibrary(path: string): LibHandle {.compilerproc.}
proc nimUnloadLibrary(lib: LibHandle) {.compilerproc.}
proc nimGetProcAddr(lib: LibHandle, name: cstring): ProcAddr {.compilerproc.}
proc nimLoadLibrary(path: string): LibHandle {.compilerproc, hcrInline, nonReloadable.}
proc nimUnloadLibrary(lib: LibHandle) {.compilerproc, hcrInline, nonReloadable.}
proc nimGetProcAddr(lib: LibHandle, name: cstring): ProcAddr {.compilerproc, hcrInline, nonReloadable.}
proc nimLoadLibraryError(path: string) {.compilerproc, noinline.}
proc nimLoadLibraryError(path: string) {.compilerproc, hcrInline, nonReloadable.}

2
lib/system/chcks.nim
View File

@@ -8,7 +8,7 @@
#
# Implementation of some runtime checks.
import system/indexerrors
include system/indexerrors
proc raiseRangeError(val: BiggestInt) {.compilerproc, noinline.} =
when hostOS == "standalone":

2
lib/system/dyncalls.nim
View File

@@ -33,7 +33,7 @@ proc nimLoadLibraryError(path: string) =
discard MessageBoxA(0, msg[0].addr, nil, 0)
quit(1)
proc procAddrError(name: cstring) {.noinline.} =
proc procAddrError(name: cstring) {.compilerproc, nonReloadable, hcrInline.} =
# carefully written to avoid memory allocation:
cstderr.rawWrite("could not import: ")
cstderr.rawWrite(name)

4
lib/system/excpt.nim
View File

@@ -20,9 +20,9 @@ var
proc c_fwrite(buf: pointer, size, n: csize, f: CFilePtr): cint {.
importc: "fwrite", header: "<stdio.h>".}
proc rawWrite(f: CFilePtr, s: string|cstring) =
proc rawWrite(f: CFilePtr, s: cstring) {.compilerproc, nonreloadable, hcrInline.} =
# we cannot throw an exception here!
discard c_fwrite(cstring(s), 1, s.len, f)
discard c_fwrite(s, 1, s.len, f)
when not defined(windows) or not defined(guiapp):
proc writeToStdErr(msg: cstring) = rawWrite(cstderr, msg)

8
lib/system/gc_common.nim
View File

@@ -446,10 +446,10 @@ proc deallocHeap*(runFinalizers = true; allowGcAfterwards = true) =
type
GlobalMarkerProc = proc () {.nimcall, benign.}
var
globalMarkersLen: int
globalMarkers: array[0..3499, GlobalMarkerProc]
threadLocalMarkersLen: int
threadLocalMarkers: array[0..3499, GlobalMarkerProc]
globalMarkersLen {.exportc.}: int
globalMarkers {.exportc.}: array[0..3499, GlobalMarkerProc]
threadLocalMarkersLen {.exportc.}: int
threadLocalMarkers {.exportc.}: array[0..3499, GlobalMarkerProc]
gHeapidGenerator: int
proc nimRegisterGlobalMarker(markerProc: GlobalMarkerProc) {.compilerProc.} =

6
lib/system/memory.nim
View File

@@ -11,7 +11,7 @@ proc nimCopyMem(dest, source: pointer, size: Natural) {.compilerproc, inline.} =
d[i] = s[i]
inc i
proc nimSetMem(a: pointer, v: cint, size: Natural) {.inline.} =
proc nimSetMem(a: pointer, v: cint, size: Natural) {.nonReloadable, inline.} =
when useLibC:
c_memset(a, v, size)
else:
@@ -22,7 +22,7 @@ proc nimSetMem(a: pointer, v: cint, size: Natural) {.inline.} =
a[i] = v
inc i
proc nimZeroMem(p: pointer, size: Natural) {.compilerproc, inline.} =
proc nimZeroMem(p: pointer, size: Natural) {.compilerproc, nonReloadable, inline.} =
nimSetMem(p, 0, size)
proc nimCmpMem(a, b: pointer, size: Natural): cint {.compilerproc, inline.} =
@@ -37,7 +37,7 @@ proc nimCmpMem(a, b: pointer, size: Natural): cint {.compilerproc, inline.} =
if d != 0: return d
inc i
proc nimCStrLen(a: cstring): csize {.compilerproc, inline.} =
proc nimCStrLen(a: cstring): csize {.compilerproc, nonReloadable, inline.} =
when useLibC:
c_strlen(a)
else:

2
lib/system/sysstr.nim
View File

@@ -82,7 +82,7 @@ proc copyStr(s: NimString, start: int): NimString {.compilerProc.} =
if s == nil: return nil
result = copyStrLast(s, start, s.len-1)
proc nimToCStringConv(s: NimString): cstring {.compilerProc, inline.} =
proc nimToCStringConv(s: NimString): cstring {.compilerProc, nonReloadable, inline.} =
if s == nil or s.len == 0: result = cstring""
else: result = cstring(addr s.data)

4
lib/windows/winlean.nim
View File

@@ -683,6 +683,10 @@ const
FILE_BEGIN* = 0'i32
INVALID_SET_FILE_POINTER* = -1'i32
NO_ERROR* = 0'i32
PAGE_NOACCESS* = 0x01'i32
PAGE_EXECUTE* = 0x10'i32
PAGE_EXECUTE_READ* = 0x20'i32
PAGE_EXECUTE_READWRITE* = 0x40'i32
PAGE_READONLY* = 2'i32
PAGE_READWRITE* = 4'i32
FILE_MAP_READ* = 4'i32

2
nimpretty/nimpretty.nim
View File

@@ -48,7 +48,7 @@ type
proc prettyPrint(infile, outfile: string, opt: PrettyOptions) =
var conf = newConfigRef()
let fileIdx = fileInfoIdx(conf, AbsoluteFile infile)
conf.outFile = AbsoluteFile outfile
conf.outFile = RelativeFile outfile
when defined(nimpretty2):
var p: TParsers
p.parser.em.indWidth = opt.indWidth

44
testament/categories.nim
View File

@@ -129,32 +129,36 @@ proc runBasicDLLTest(c, r: var TResults, cat: Category, options: string) =
else:
""
var test1 = makeTest("lib/nimrtl.nim", options & " --app:lib -d:createNimRtl --threads:on", cat)
var test1 = makeTest("lib/nimrtl.nim", options & " --outdir:tests/dll", cat)
test1.spec.action = actionCompile
testSpec c, test1
var test2 = makeTest("tests/dll/server.nim", options & " --app:lib -d:useNimRtl --threads:on" & rpath, cat)
var test2 = makeTest("tests/dll/server.nim", options & " --threads:on" & rpath, cat)
test2.spec.action = actionCompile
testSpec c, test2
var test3 = makeTest("lib/nimhcr.nim", options & " --outdir:tests/dll" & rpath, cat)
test3.spec.action = actionCompile
testSpec c, test3
when defined(Windows):
# windows looks in the dir of the exe (yay!):
var nimrtlDll = DynlibFormat % "nimrtl"
safeCopyFile("lib" / nimrtlDll, "tests/dll" / nimrtlDll)
else:
# windows looks in the dir of the exe (yay!):
when not defined(Windows):
# posix relies on crappy LD_LIBRARY_PATH (ugh!):
const libpathenv = when defined(haiku):
"LIBRARY_PATH"
else:
"LD_LIBRARY_PATH"
const libpathenv = when defined(haiku): "LIBRARY_PATH"
else: "LD_LIBRARY_PATH"
var libpath = getEnv(libpathenv).string
# Temporarily add the lib directory to LD_LIBRARY_PATH:
putEnv(libpathenv, "tests/dll" & (if libpath.len > 0: ":" & libpath else: ""))
defer: putEnv(libpathenv, libpath)
var nimrtlDll = DynlibFormat % "nimrtl"
safeCopyFile("lib" / nimrtlDll, "tests/dll" / nimrtlDll)
testSpec r, makeTest("tests/dll/client.nim", options & " -d:useNimRtl --threads:on" & rpath,
cat)
testSpec r, makeTest("tests/dll/client.nim", options & " --threads:on" & rpath, cat)
testSpec r, makeTest("tests/dll/nimhcr_unit.nim", options & rpath, cat)
if "boehm" notin options:
# force build required - see the comments in the .nim file for more details
var hcr_integration = makeTest("tests/dll/nimhcr_integration.nim",
options & " --forceBuild --hotCodeReloading:on" & rpath, cat)
hcr_integration.args = prepareTestArgs(hcr_integration.spec.getCmd, hcr_integration.name,
hcr_integration.options, getTestSpecTarget())
testSpec r, hcr_integration
proc dllTests(r: var TResults, cat: Category, options: string) =
# dummy compile result:
@@ -651,17 +655,17 @@ proc runJoinedTest(r: var TResults, cat: Category, testsDir: string) =
var (buf, exitCode) = execCmdEx2(command = compilerPrefix, args = args, options = {poStdErrToStdOut, poUsePath}, input = "",
onStdout = if verboseMegatest: onStdout else: nil)
if exitCode != 0:
echo buf
echo buf.string
quit("megatest compilation failed")
# Could also use onStdout here.
(buf, exitCode) = execCmdEx("./megatest")
if exitCode != 0:
echo buf
echo buf.string
quit("megatest execution failed")
norm buf
writeFile("outputGotten.txt", buf)
norm buf.string
writeFile("outputGotten.txt", buf.string)
var outputExpected = ""
for i, runSpec in specs:
outputExpected.add marker & runSpec.file & "\n"
@@ -669,7 +673,7 @@ proc runJoinedTest(r: var TResults, cat: Category, testsDir: string) =
outputExpected.add '\n'
norm outputExpected
if buf != outputExpected:
if buf.string != outputExpected:
writeFile("outputExpected.txt", outputExpected)
discard execShellCmd("diff -uNdr outputExpected.txt outputGotten.txt")
echo "output different!"

59
testament/tester.nim
View File

@@ -12,7 +12,9 @@
import
parseutils, strutils, pegs, os, osproc, streams, parsecfg, json,
marshal, backend, parseopt, specs, htmlgen, browsers, terminal,
algorithm, compiler/nodejs, times, sets, md5
algorithm, times, sets, md5, sequtils
include compiler/nodejs
var useColors = true
var backendLogging = true
@@ -58,6 +60,7 @@ type
name: string
cat: Category
options: string
args: seq[string]
spec: TSpec
startTime: float
@@ -127,13 +130,17 @@ proc nimcacheDir(filename, options: string, target: TTarget): string =
let hashInput = options & $target
return "nimcache" / (filename & '_' & hashInput.getMD5)
proc callCompiler(cmdTemplate, filename, options: string,
target: TTarget, extraOptions=""): TSpec =
proc prepareTestArgs(cmdTemplate, filename, options: string,
target: TTarget, extraOptions=""): seq[string] =
let nimcache = nimcacheDir(filename, options, target)
let options = options & " " & quoteShell("--nimCache:" & nimcache) & extraOptions
let c = parseCmdLine(cmdTemplate % ["target", targetToCmd[target],
"options", options, "file", filename.quoteShell,
"filedir", filename.getFileDir()])
return parseCmdLine(cmdTemplate % ["target", targetToCmd[target],
"options", options, "file", filename.quoteShell,
"filedir", filename.getFileDir()])
proc callCompiler(cmdTemplate, filename, options: string,
target: TTarget, extraOptions=""): TSpec =
let c = prepareTestArgs(cmdTemplate, filename, options, target, extraOptions)
var p = startProcess(command=c[0], args=c[1 .. ^1],
options={poStdErrToStdOut, poUsePath})
let outp = p.outputStream
@@ -307,10 +314,13 @@ proc cmpMsgs(r: var TResults, expected, given: TSpec, test: TTest, target: TTarg
inc(r.passed)
proc generatedFile(test: TTest, target: TTarget): string =
let (_, name, _) = test.name.splitFile
let ext = targetToExt[target]
result = nimcacheDir(test.name, test.options, target) /
((if target == targetJS: "" else: "compiler_") & name.changeFileExt(ext))
if target == targetJS:
result = test.name.changeFileExt("js")
else:
let (_, name, _) = test.name.splitFile
let ext = targetToExt[target]
result = nimcacheDir(test.name, test.options, target) /
("compiler_" & name.changeFileExt(ext))
proc needsCodegenCheck(spec: TSpec): bool =
result = spec.maxCodeSize > 0 or spec.ccodeCheck.len > 0
@@ -368,6 +378,12 @@ proc compilerOutputTests(test: TTest, target: TTarget, given: var TSpec,
if given.err == reSuccess: inc(r.passed)
r.addResult(test, target, expectedmsg, givenmsg, given.err)
proc getTestSpecTarget(): TTarget =
if getEnv("NIM_COMPILE_TO_CPP", "false").string == "true":
return targetCpp
else:
return targetC
proc testSpec(r: var TResults, test: TTest, targets: set[TTarget] = {}) =
var expected = test.spec
if expected.parseErrors.len > 0:
@@ -384,10 +400,7 @@ proc testSpec(r: var TResults, test: TTest, targets: set[TTarget] = {}) =
expected.targets.incl targets
# still no target specified at all
if expected.targets == {}:
if getEnv("NIM_COMPILE_TO_CPP", "false").string == "true":
expected.targets = {targetCpp}
else:
expected.targets = {targetC}
expected.targets = {getTestSpecTarget()}
for target in expected.targets:
inc(r.total)
if target notin gTargets:
@@ -409,21 +422,15 @@ proc testSpec(r: var TResults, test: TTest, targets: set[TTarget] = {}) =
of actionRun:
# In this branch of code "early return" pattern is clearer than deep
# nested conditionals - the empty rows in between to clarify the "danger"
var given = callCompiler(expected.getCmd, test.name, test.options,
target)
var given = callCompiler(expected.getCmd, test.name, test.options, target)
if given.err != reSuccess:
r.addResult(test, target, "", given.msg, given.err)
continue
let isJsTarget = target == targetJS
var exeFile: string
if isJsTarget:
let file = test.name.lastPathPart.changeFileExt("js")
exeFile = nimcacheDir(test.name, test.options, target) / file
else:
exeFile = changeFileExt(test.name, ExeExt)
var exeFile = changeFileExt(test.name, if isJsTarget: "js" else: ExeExt)
if not existsFile(exeFile):
r.addResult(test, target, expected.output, "executable not found", reExeNotFound)
r.addResult(test, target, expected.output,
"executable not found: " & exeFile, reExeNotFound)
continue
let nodejs = if isJsTarget: findNodeJs() else: ""
@@ -432,10 +439,10 @@ proc testSpec(r: var TResults, test: TTest, targets: set[TTarget] = {}) =
reExeNotFound)
continue
var exeCmd: string
var args: seq[string]
var args = test.args
if isJsTarget:
exeCmd = nodejs
args.add exeFile
args = concat(@[exeFile], args)
else:
exeCmd = exeFile
var (buf, exitCode) = execCmdEx2(exeCmd, args, options = {poStdErrToStdOut}, input = expected.input)

4
tests/dll/nimhcr_0.nim Normal file
View File

@@ -0,0 +1,4 @@
let g_0 = 1000 # new value! but also a "new" global :)
proc getInt*(): int = return g_0

14
tests/dll/nimhcr_0_1.nim Normal file
View File

@@ -0,0 +1,14 @@
import hotcodereloading
let g_0 = 42 # lets start with the ultimate answer
proc getInt*(): int = return g_0
programResult = 0 # should be accessible
beforeCodeReload:
echo " 0: before"
afterCodeReload:
echo " 0: after"

18
tests/dll/nimhcr_0_2.nim Normal file
View File

@@ -0,0 +1,18 @@
import hotcodereloading
import nimhcr_1 # new import!
# global scope for this module was executed when loading the program
# with a previous version which didn't contain this print statement
echo " 0: I SHOULDN'T BE PRINTED!"
var g_0 = 0 # changed value but won't take effect
proc getInt*(): int = return g_0 + g_1 + f_1()
beforeCodeReload:
echo " 0: before - improved!" # changed handlers!
afterCodeReload:
echo " 0: after - improved!"
g_0 = 100 # we cannot change it in its initialization but we can in the 'after' handler!

18
tests/dll/nimhcr_0_3.nim Normal file
View File

@@ -0,0 +1,18 @@
import hotcodereloading
import nimhcr_1
import nimhcr_2 # a new and different import!
proc makeCounter*(): auto =
return iterator: int {.closure.} =
for i in countup(0, 10, 1):
yield i
let c = makeCounter()
afterCodeReload:
echo " 0: after - closure iterator: ", c()
echo " 0: after - closure iterator: ", c()
proc getInt*(): int = return g_1 + g_2.len

19
tests/dll/nimhcr_0_4.nim Normal file
View File

@@ -0,0 +1,19 @@
import hotcodereloading
import nimhcr_1 # only importing 1
let g_0 = 1000 # new value! but also a "new" global :)
proc getInt*(): int = return g_0
proc makeCounter*(): auto =
return iterator: int {.closure.} =
for i in countup(0, 10, 1):
yield i
let c = makeCounter()
afterCodeReload:
echo " 0: after - closure iterator! after reload! does it remember? :", c()
echo " 0: after - closure iterator! after reload! does it remember? :", c()

2
tests/dll/nimhcr_0_5.nim Normal file
View File

@@ -0,0 +1,2 @@
proc getInt*(): int = return 42 # back to the answer...

4
tests/dll/nimhcr_0_6.nim Normal file
View File

@@ -0,0 +1,4 @@
let g_0 = 1000 # new value! but also a "new" global :)
proc getInt*(): int = return g_0

0
tests/dll/nimhcr_1.nim Normal file
View File

51
tests/dll/nimhcr_1_1.nim Normal file
View File

@@ -0,0 +1,51 @@
echo " 1: print me once!"
import hotcodereloading
let g_1* = 8 # devilish!
proc f_1*(): int =
var a {.global.} = 1
a.inc
return a
# all these constructs should compile
let some_glob_1 = 1
echo " 1: ", some_glob_1
if true:
let some_glob_2 = 2
echo " 1: ", some_glob_2
if true:
let some_glob_3 = 3
echo " 1: ", some_glob_3
block:
let some_glob_4 = 4
proc inBlock(num: int) =
echo " 1: ", num
inBlock(some_glob_4)
var counter = 3
while counter > 0:
let some_glob_5 = 5
echo " 1: ", some_glob_5
counter.dec
type
Type1 = object
a: int
b: int
var t = Type1(a: 42, b: 11)
echo " 1: Type1.a:", t.a
type
obj = ref object
dat: int
str: string
proc foo(): (int, obj) = (1, obj(dat: 3, str: "bar"))
let (aa, bb) = foo()
afterCodeReload:
echo aa
echo bb.str

4
tests/dll/nimhcr_1_2.nim Normal file
View File

@@ -0,0 +1,4 @@
import nimhcr_2
proc f_1*(): int = return f_2()

0
tests/dll/nimhcr_1_3.nim Normal file
View File

0
tests/dll/nimhcr_2.nim Normal file
View File

15
tests/dll/nimhcr_2_1.nim Normal file
View File

@@ -0,0 +1,15 @@
import hotcodereloading
type
Type2 = ref object of RootObj
data*: int
let g_2* = @[Type2(data: 2), Type2(data: 3)][1..^1] # should have a length of 1
var a: tuple[str: string, i: int]
a.str = " 2: random string"
echo a.str
beforeCodeReload:
echo " 2: before!"

7
tests/dll/nimhcr_2_2.nim Normal file
View File

@@ -0,0 +1,7 @@
import hotcodereloading
proc f_2*(): int = return 1
afterCodeReload:
echo " 2: after!"

0
tests/dll/nimhcr_2_3.nim Normal file
View File

152
tests/dll/nimhcr_integration.nim Normal file
View File

@@ -0,0 +1,152 @@
discard """
output: '''
main: HELLO!
main: hasAnyModuleChanged? true
main: before
0: after
main: after
The answer is: 1000
main: hasAnyModuleChanged? false
The answer is: 1000
main: hasAnyModuleChanged? true
0: before
main: before
1: print me once!
1: 1
1: 2
1: 3
1: 4
1: 5
1: 5
1: 5
1: Type1.a:42
1
bar
0: after - improved!
main: after
The answer is: 110
main: hasAnyModuleChanged? true
0: before - improved!
main: before
2: random string
1
bar
0: after - closure iterator: 0
0: after - closure iterator: 1
main: after
The answer is: 9
main: hasAnyModuleChanged? true
2: before!
main: before
2: after!
0: after - closure iterator! after reload! does it remember? :2
0: after - closure iterator! after reload! does it remember? :3
main: after
The answer is: 1000
main: hasAnyModuleChanged? true
main: before
main: after
The answer is: 42
done
'''
"""
## This is perhaps the most complex test in the nim test suite - calling the
## compiler on the file itself with the same set or arguments and reloading
## parts of the program at runtime! In the same folder there are a few modules
## with names such as `nimhcr_<number>.nim`. Each of them has a few versions which
## are in the format of `nimhcr_<number>_<version>.nim`. The below code uses the
## `update` proc to say which of the modules should bump its version (and that
## is done by copying `nimhcr_<number>_<version>.nim` onto `nimhcr_<number>.nim`).
## The files should refer to each other (when importing) without the versions.
## A few files can be updated by calling `update` for each of their indexes
## and after that with a single call to `compileReloadExecute` the new version
## of the program will be compiled, reloaded, and the only thing the main module
## calls from `nimhcr_0.nim` (the procedure `getInt` proc) is called for a result.
##
## This test is expected to be executed with arguments - the full nim compiler
## command used for building it - so it can rebuild iself the same way - example:
##
## compiling:
## nim c --hotCodeReloading:on --nimCache:<folder> <this_file>.nim
## executing:
## <this_file>.exe nim c --hotCodeReloading:on --nimCache:<folder> <this_file>.nim
import os, osproc, times, strutils, hotcodereloading
import nimhcr_0 # getInt() - the only thing we continually call from the main module
proc compileReloadExecute() =
# Remove the `--forceBuild` option - is there in the first place because:
# - when `koch test` is ran for the first time the nimcache is empty
# - when each of the variants are built (debug, release after that, different GCs)
# the main executable that gets built into the appropriate nimcache folder
# gets copied to the originally intended destination and is executed
# (this behaviour is only when the --hotCodeReloading option is used).
# - when `koch test` is ran again and the nimcache is full the executable files
# in the nimcache folder aren't relinked and therefore aren't copied to the
# originally intended destination - so when the binary at the intended
# destination is executed - it is actually a remnant from a previous execution.
# That is a problem because it points to shared objects to load from its own
# nimcache folder - the one used for building it - a previous run! And when
# this test changes other modules it references but the main module (this file)
# remains intact - the binary isn't replaced. `--forceBuild` fixes this but has
# to be applied only for the main build - the one done from koch, but when this
# binary triggers rebuilding itself here it shouldn't rebuild the main module -
# that would lead to replacing the main binary executable which is running!
let cmd = commandLineParams()[0..^1].join(" ").replace(" --forceBuild")
let (stdout, exitcode) = execCmdEx(cmd)
if exitcode != 0:
echo "COMPILATION ERROR!"
echo "COMMAND: ", cmd
echo "STDOUT: ", stdout
quit 1
echo "main: hasAnyModuleChanged? ", hasAnyModuleChanged()
performCodeReload()
echo " The answer is: ", getInt()
# there are 3 files and all of them start from their 1st version
var vers = [1, 1, 1]
proc update(file: int) =
proc getfile(mid: string): string =
let (path, _, _) = splitFile(currentSourcePath())
return path & "/nimhcr_" & mid & ".nim"
copyFile(getfile($file & "_" & $vers[file]), getfile($file))
inc vers[file]
beforeCodeReload:
echo "main: before"
afterCodeReload:
echo "main: after"
echo "main: HELLO!"
update 0
compileReloadExecute() # versions are: 1 - -
compileReloadExecute() # no change
update 0
update 1
compileReloadExecute() # versions are: 2 1 -
update 0
update 2
compileReloadExecute() # versions are: 3 1 1
update 0
update 1
update 2
compileReloadExecute() # versions are: 4 2 2
update 0
compileReloadExecute() # versions are: 5 2 2
# final update so there are no git modifications left after everything
# (the last versions are like the first files without a version suffix)
update 0
update 1
update 2
echo "done"

147
tests/dll/nimhcr_unit.nim Normal file
View File

@@ -0,0 +1,147 @@
discard """
output: '''
fastcall_proc implementation #1 10
11
fastcall_proc implementation #2 20
22
fastcall_proc implementation #2 20
22
fastcall_proc implementation #3 30
33
fastcall_proc implementation #3 30
33
fastcall_proc implementation #3 30
33
fastcall_proc implementation #3 40
43
cdecl_proc implementation #1 10
11
cdecl_proc implementation #2 20
22
cdecl_proc implementation #2 20
22
cdecl_proc implementation #3 30
33
cdecl_proc implementation #3 30
33
cdecl_proc implementation #3 30
33
cdecl_proc implementation #3 40
43
stdcall_proc implementation #1 10
11
stdcall_proc implementation #2 20
22
stdcall_proc implementation #2 20
22
stdcall_proc implementation #3 30
33
stdcall_proc implementation #3 30
33
stdcall_proc implementation #3 30
33
stdcall_proc implementation #3 40
43
noconv_proc implementation #1 10
11
noconv_proc implementation #2 20
22
noconv_proc implementation #2 20
22
noconv_proc implementation #3 30
33
noconv_proc implementation #3 30
33
noconv_proc implementation #3 30
33
noconv_proc implementation #3 40
43
inline_proc implementation #1 10
11
inline_proc implementation #2 20
22
inline_proc implementation #2 20
22
inline_proc implementation #3 30
33
inline_proc implementation #3 30
33
inline_proc implementation #3 30
33
inline_proc implementation #3 40
43
'''
"""
import macros
macro carryOutTests(callingConv: untyped): untyped =
let
procName = $callingConv & "_proc"
globalName = $callingConv & "_global"
callingConv = callingConv
p1 = ident(procName & "1")
p2 = ident(procName & "2")
p3 = ident(procName & "3")
g1 = ident(globalName & "1")
g2 = ident(globalName & "2")
result = quote do:
var `g1`: pointer = nil
if hcrRegisterGlobal("dummy_module", `globalName`, sizeof(int), nil, addr `g1`):
cast[ptr int](`g1`)[] = 10
var `g2`: pointer = nil
if hcrRegisterGlobal("dummy_module", `globalName`, sizeof(int), nil, addr `g2`):
cast[ptr int](`g2`)[] = 20
doAssert `g1` == `g2` and cast[ptr int](`g1`)[] == 10
type
F = proc (x: int): int {.placeholder.}
proc `p1`(x: int): int {.placeholder.}=
echo `procName`, " implementation #1 ", x
return x + 1
let fp1 = cast[F](hcrRegisterProc("dummy_module", `procName`, `p1`))
echo fp1(10)
proc `p2`(x: int): int {.placeholder.} =
echo `procName`, " implementation #2 ", x
return x + 2
let fp2 = cast[F](hcrRegisterProc("dummy_module", `procName`, `p2`))
echo fp1(20)
echo fp2(20)
proc `p3`(x: int): int {.placeholder.} =
echo `procName`, " implementation #3 ", x
return x + 3
let fp3 = cast[F](hcrRegisterProc("dummy_module", `procName`, `p3`))
echo fp1(30)
echo fp2(30)
echo fp3(30)
let fp4 = cast[F](hcrGetProc("dummy_module", `procName`))
echo fp4(40)
proc replacePlaceholderPragmas(n: NimNode) =
if n.kind == nnkPragma:
n[0] = callingConv
else:
for i in 0 ..< n.len:
replacePlaceholderPragmas n[i]
replacePlaceholderPragmas result
# echo result.treeRepr
hcrAddModule("dummy_module")
carryOutTests fastcall
carryOutTests cdecl
carryOutTests stdcall
carryOutTests noconv
carryOutTests inline

2
tests/dll/nimhcr_unit.nim.cfg Normal file
View File

@@ -0,0 +1,2 @@
-d:useNimRtl
-d:testNimHcr

10
tests/dll/test_nimhcr_integration.bat Normal file
View File

@@ -0,0 +1,10 @@
set NIM=nim
set NIM_FLAGS=-d:debug
%NIM% c --outdir:"." %NIM_FLAGS% ../../lib/nimrtl.nim
%NIM% c --outdir:"." %NIM_FLAGS% ../../lib/nimhcr.nim
set HCR_FLAGS=--forceBuild --hotCodeReloading:on --nimcache:nimcache %NIM_FLAGS%
%NIM% %HCR_FLAGS% c nimhcr_integration.nim
nimhcr_integration %NIM% %HCR_FLAGS% c nimhcr_integration.nim

17
tests/dll/test_nimhcr_integration.sh Executable file
View File

@@ -0,0 +1,17 @@
#!/bin/bash
set -e
rm -rf nimcache
NIM_FLAGS=${*:- -d:debug}
NIM=nim
$NIM c --outdir:"." $NIM_FLAGS ../../lib/nimrtl.nim
$NIM c --outdir:"." $NIM_FLAGS ../../lib/nimhcr.nim
echo ===== Compiling HCR Integration Test =====
HCR_FLAGS="--forceBuild --hotCodeReloading:on --nimcache:nimcache $NIM_FLAGS"
$NIM $HCR_FLAGS c nimhcr_integration.nim
export LD_LIBRARY_PATH=$(pwd):$LD_LIBRARY_PATH
./nimhcr_integration $NIM $HCR_FLAGS c nimhcr_integration.nim

7
tests/types/tissues_types.nim
View File

@@ -41,6 +41,13 @@ block t5648:
var g = Foo()
g.bar = 3
var
mainPtr1: pointer = main
mainPtr2 = pointer(main)
mainPtr3 = cast[pointer](main)
doAssert mainPtr1 == mainPtr2 and mainPtr2 == mainPtr3
main()
block t7581:

2
tools/kochdocs.nim
View File

@@ -356,7 +356,7 @@ proc buildDocs*(args: string) =
let
a = nimArgs & " " & args
docHackJs = "dochack.js"
docHackJsSource = docHackDir / "nimcache" / docHackJs
docHackJsSource = docHackDir / docHackJs
docHackJsDest = docHtmlOutput / docHackJs
buildJS() # This call generates docHackJsSource
let docup = webUploadOutput / NimVersion