Initial version of the hot-code reloading support for native targets (#10729)

* squashed work by Zahary

* squashing a ton of useful history... otherwise rebasing on top of upstream Nim after commit 82c009a2cb would be impossible.

* Code review changes; Working test suite (without code reloading enabled)

* - documentation
- implemented the HCR test - almost works...
- fix the issue on Unix where for executable targets the source file for the main module of a project in nimcache was being overwritten with the binary itself (and thus the actual source code was lost)
- fixing embedded paths to shared objects on unix (the "lib" prefix was being prepended to the entire path instead of just the filename)
- other fixes
- removing unnecessary includes since that file is already included in chcks.nim which is in turn included in system.nim (and previously was getting imported in chcks.nim but then system.nim improts something... and that breaks HCR (perhaps it could be fixed but it would be nice not to import anything in system))

* fix for clang & C++ - explicitly casting a function pointer to void*
more stable mangling of parameter names when HCR is on
the length of the static arrays in the DatInit functions is now part of the name of the variables, so when they get resized they get also recreated
more stable mangling for inline functions - no longer depends on the module which first used them
work on the new complicated HCR test - turned surprisingly complex - WIP
test now successfully passes even when re-running `koch test` (previously when the nimcache wasn't cold that lead to errors)
better documentation
calling setStackBottomWith for PreMain
passes over the HcrInit/DatInit/Init calls of all modules are now in the proper order (first all of one type, then all of the next). Also typeinfo globals are registered (created) in a single pass before the DatInit pass (because of the way generic instantiations are handled)
Fix the test suite execution on macOs
fix for being able to query the program arguments when using HCR on posix!
other fixes

* Bugfix: Fix a compilation error in C++ mode when a function pointer
is converted to a raw pointer

* basic documentation for the new hot code reloading semantics

* Add change log entry

* Don't re-execute the top-level statements while reloading JS code

* fix a number of tests broken in a recent bugfix

* Review changes

* Added {.executeOnReload.} pragma that indicates top-level statements
  that should be executed on each reload. To make this work, I've modified
  the way the `if (hcr_init_) {...}` guards are produced in the init code.
  This still needs more work as the new guards seem to be inserted within
  the previously generated guards.

  This change also removes the need for `lastRegistedGlobal` in nimhcr.

* Implemented the `signatureHash` magic and the `hasModuleChanged` API
  depending on it (the actual logic is not imlemented yet).

* Add the "hcr" prefix to all HCR-related symbols in the system module.
  Added a new `hotcodereloading` module exporting the high-level API to
  the user.

  Besides being more hygienic, this was also required in order to make
  it possible to use macros in the high-level API. Without the split,
  `system` would have to import `macros`, which was going to produce
  the well-known init problems.

* Attempted to solve the "GC markers problem".

  Crashes were expected with the previous code, because the GC markers
  were compiled as normal procs are registered in the GC. When their
  module is unloaded, dangling pointers will remain in the GC tables.
  To solve this issue, I don't register any GC markers when HCR is on,
  but I add them to the HCR globals metadata and I use a single marker
  registed in nimhcr during the initialization of the system module that
  will be responsible for marking all globals.

* fix a compilation error

* - implemented the hasModuleChanged functionality
- tuples can be returned and broken into different vars in global scope
- added comments for the closnig scopes of the if statements in the init proc
- the new executeOnReload pragma works now!
- other fixes

* finally! fixing this hack in a proper way - declaring the destructor out of line (out of the class body) - we no longer need to forward-declare popCurrentExceptionEx

* Force full module parsing

This is a temporary hack that breaks some tests. I'll investigate
later how these can be fixed.

* tuples are now properly handled when global!

* these comments mess up the codegen in debug mode when $n is not actually a new line (or something like that) - these labels are intended only for GOTO labels anyway...

* "solved" the issue with the .pdb locks on windows when a binary is being debugged and hot code reloading is used at the same time

* fixes after rebasing...

* small fixes for the test

* better handling of globals! no more compiler crashes for locals with the global pragma, also simplified code around loops in global scope which have local vars (actually globals)

* we can now use the global pragma even for ... globals!

* the right output

* lets try those boehm GC tests

* after the test is ran it will be at its starting state - no git modifications

* clarification in the docs

* removed unnecessary line directives for forward declarations of functions - they were causing trouble with hot code reloading when no semantic change propagates to the main module but a line directive got changed and thus the main module had to be recompiled since the .c code had changed

* fixed bug! was inserting duplicate keys into the table and later was removing only 1 copy of all the duplicates (after a few reloads)

* no longer breaking into DatInit code when not supposed to

* fixes after rebasing

* yet more fixes after rebasing

* Update jssys.nim

* Rework the HCR path-handling logic

After reviewing the code more carefully, I've noticed that the old logic
will be broken when the user overrides the '--out:f' compiler option.

Besides fixing this issues, I took the opportunity to implement the
missing '--outdir:d' option.

Other changes:

* ./koch test won't overwrite any HCR and RTL builds located in nim/lib
* HCR and RTL are compiled with --threads:on by default

* Clean up the globals registration logic

* Handle non-flattened top-level stmtlists in JS as well

* The HCR is not supported with the Boehm GC yet

Also fixes some typos and the expected output of the HCR integration test

* The GC marker procs are now properly used as trampolines

* Fix the HCR integration test in release builds

* Fix ./koch tools

* this forward declaration doesn't seem to be necessary, and in fact breaks HCR because a 2nd function pointer is emitted for this externed/rtl func

* the forward declaration I removed in the last commit was actually necessary

* Attempt to make all tests green

* Fix tgenscript

* BAT file for running the HCR integration test on Windows [skip ci]

* Fix the docgen tests

* A final fix for Travis (hopefully)

tests/dll/nimhcr_0.nim

@@ -0,0 +1,4 @@
+
+let g_0 = 1000 # new value! but also a "new" global :)
+
+proc getInt*(): int = return g_0

tests/dll/nimhcr_0_1.nim

@@ -0,0 +1,14 @@
+
+import hotcodereloading
+
+let g_0 = 42 # lets start with the ultimate answer
+
+proc getInt*(): int = return g_0
+
+programResult = 0 # should be accessible
+
+beforeCodeReload:
+  echo "   0: before"
+afterCodeReload:
+  echo "   0: after"
+  

tests/dll/nimhcr_0_2.nim

@@ -0,0 +1,18 @@
+
+import hotcodereloading
+
+import nimhcr_1 # new import!
+
+# global scope for this module was executed when loading the program
+# with a previous version which didn't contain this print statement
+echo "   0: I SHOULDN'T BE PRINTED!"
+
+var g_0 = 0 # changed value but won't take effect
+
+proc getInt*(): int = return g_0 + g_1 + f_1()
+
+beforeCodeReload:
+  echo "   0: before - improved!" # changed handlers!
+afterCodeReload:
+  echo "   0: after - improved!"
+  g_0 = 100 # we cannot change it in its initialization but we can in the 'after' handler!

tests/dll/nimhcr_0_3.nim

@@ -0,0 +1,18 @@
+
+import hotcodereloading
+
+import nimhcr_1
+import nimhcr_2 # a new and different import!
+
+proc makeCounter*(): auto =
+  return iterator: int {.closure.} =
+    for i in countup(0, 10, 1):
+      yield i
+
+let c = makeCounter()
+
+afterCodeReload:
+  echo "   0: after - closure iterator: ", c()
+  echo "   0: after - closure iterator: ", c()
+
+proc getInt*(): int = return g_1 + g_2.len

tests/dll/nimhcr_0_4.nim

@@ -0,0 +1,19 @@
+
+import hotcodereloading
+
+import nimhcr_1 # only importing 1
+
+let g_0 = 1000 # new value! but also a "new" global :)
+
+proc getInt*(): int = return g_0
+
+proc makeCounter*(): auto =
+  return iterator: int {.closure.} =
+    for i in countup(0, 10, 1):
+      yield i
+
+let c = makeCounter()
+
+afterCodeReload:
+  echo "   0: after - closure iterator! after reload! does it remember? :", c()
+  echo "   0: after - closure iterator! after reload! does it remember? :", c()

tests/dll/nimhcr_0_5.nim

@@ -0,0 +1,2 @@
+
+proc getInt*(): int = return 42 # back to the answer...

tests/dll/nimhcr_0_6.nim

@@ -0,0 +1,4 @@
+
+let g_0 = 1000 # new value! but also a "new" global :)
+
+proc getInt*(): int = return g_0

tests/dll/nimhcr_1_1.nim

@@ -0,0 +1,51 @@
+
+echo "   1: print me once!"
+
+import hotcodereloading
+
+let g_1* = 8 # devilish!
+
+proc f_1*(): int =
+  var a {.global.} = 1
+  a.inc
+  return a
+
+
+# all these constructs should compile
+let some_glob_1 = 1
+echo "   1: ", some_glob_1
+if true:
+  let some_glob_2 = 2
+  echo "   1: ", some_glob_2
+  if true:
+    let some_glob_3 = 3
+    echo "   1: ", some_glob_3
+block:
+  let some_glob_4 = 4
+  proc inBlock(num: int) =
+    echo "   1: ", num
+  inBlock(some_glob_4)
+var counter = 3
+while counter > 0:
+  let some_glob_5 = 5
+  echo "   1: ", some_glob_5
+  counter.dec
+
+type
+  Type1 = object
+    a: int
+    b: int
+var t = Type1(a: 42, b: 11)
+echo "   1: Type1.a:", t.a
+
+type
+  obj = ref object
+    dat: int
+    str: string
+
+proc foo(): (int, obj) = (1, obj(dat: 3, str: "bar"))
+
+let (aa, bb) = foo()
+afterCodeReload:
+  echo aa
+  echo bb.str

tests/dll/nimhcr_1_2.nim

@@ -0,0 +1,4 @@
+
+import nimhcr_2
+
+proc f_1*(): int = return f_2()

tests/dll/nimhcr_2_1.nim

@@ -0,0 +1,15 @@
+
+import hotcodereloading
+
+type
+  Type2 = ref object of RootObj
+    data*: int
+
+let g_2* = @[Type2(data: 2), Type2(data: 3)][1..^1] # should have a length of 1
+
+var a: tuple[str: string, i: int]
+a.str = "   2: random string"
+echo a.str
+
+beforeCodeReload:
+  echo "   2: before!"

tests/dll/nimhcr_2_2.nim

@@ -0,0 +1,7 @@
+
+import hotcodereloading
+
+proc f_2*(): int = return 1
+
+afterCodeReload:
+  echo "   2: after!"

tests/dll/nimhcr_integration.nim

@@ -0,0 +1,152 @@
+discard """
+output: '''
+main: HELLO!
+main: hasAnyModuleChanged? true
+main: before
+   0: after
+main: after
+              The answer is: 1000
+main: hasAnyModuleChanged? false
+              The answer is: 1000
+main: hasAnyModuleChanged? true
+   0: before
+main: before
+   1: print me once!
+   1: 1
+   1: 2
+   1: 3
+   1: 4
+   1: 5
+   1: 5
+   1: 5
+   1: Type1.a:42
+1
+bar
+   0: after - improved!
+main: after
+              The answer is: 110
+main: hasAnyModuleChanged? true
+   0: before - improved!
+main: before
+   2: random string
+1
+bar
+   0: after - closure iterator: 0
+   0: after - closure iterator: 1
+main: after
+              The answer is: 9
+main: hasAnyModuleChanged? true
+   2: before!
+main: before
+   2: after!
+   0: after - closure iterator! after reload! does it remember? :2
+   0: after - closure iterator! after reload! does it remember? :3
+main: after
+              The answer is: 1000
+main: hasAnyModuleChanged? true
+main: before
+main: after
+              The answer is: 42
+done
+'''
+"""
+
+## This is perhaps the most complex test in the nim test suite - calling the
+## compiler on the file itself with the same set or arguments and reloading
+## parts of the program at runtime! In the same folder there are a few modules
+## with names such as `nimhcr_<number>.nim`. Each of them has a few versions which
+## are in the format of `nimhcr_<number>_<version>.nim`. The below code uses the
+## `update` proc to say which of the modules should bump its version (and that
+## is done by copying `nimhcr_<number>_<version>.nim` onto `nimhcr_<number>.nim`).
+## The files should refer to each other (when importing) without the versions.
+## A few files can be updated by calling `update` for each of their indexes
+## and after that with a single call to `compileReloadExecute` the new version
+## of the program will be compiled, reloaded, and the only thing the main module
+## calls from `nimhcr_0.nim` (the procedure `getInt` proc) is called for a result.
+##
+## This test is expected to be executed with arguments - the full nim compiler
+## command used for building it - so it can rebuild iself the same way - example:
+##
+## compiling:
+##                   nim c --hotCodeReloading:on --nimCache:<folder> <this_file>.nim
+## executing:
+##   <this_file>.exe nim c --hotCodeReloading:on --nimCache:<folder> <this_file>.nim
+
+import os, osproc, times, strutils, hotcodereloading
+
+import nimhcr_0 # getInt() - the only thing we continually call from the main module
+
+proc compileReloadExecute() =
+  # Remove the `--forceBuild` option - is there in the first place because:
+  # - when `koch test` is ran for the first time the nimcache is empty
+  # - when each of the variants are built (debug, release after that, different GCs)
+  #   the main executable that gets built into the appropriate nimcache folder
+  #   gets copied to the originally intended destination and is executed
+  #   (this behaviour is only when the --hotCodeReloading option is used).
+  # - when `koch test` is ran again and the nimcache is full the executable files
+  #   in the nimcache folder aren't relinked and therefore aren't copied to the
+  #   originally intended destination - so when the binary at the intended
+  #   destination is executed - it is actually a remnant from a previous execution.
+  #   That is a problem because it points to shared objects to load from its own
+  #   nimcache folder - the one used for building it - a previous run! And when
+  #   this test changes other modules it references but the main module (this file)
+  #   remains intact - the binary isn't replaced. `--forceBuild` fixes this but has
+  #   to be applied only for the main build - the one done from koch, but when this
+  #   binary triggers rebuilding itself here it shouldn't rebuild the main module -
+  #   that would lead to replacing the main binary executable which is running!
+  let cmd = commandLineParams()[0..^1].join(" ").replace(" --forceBuild")
+  let (stdout, exitcode) = execCmdEx(cmd)
+  if exitcode != 0:
+    echo "COMPILATION ERROR!"
+    echo "COMMAND: ", cmd
+    echo "STDOUT: ", stdout
+    quit 1
+  echo "main: hasAnyModuleChanged? ", hasAnyModuleChanged()
+  performCodeReload()
+  echo "              The answer is: ", getInt()
+
+# there are 3 files and all of them start from their 1st version
+var vers = [1, 1, 1]
+proc update(file: int) =
+  proc getfile(mid: string): string =
+    let (path, _, _) = splitFile(currentSourcePath())
+    return path & "/nimhcr_" & mid & ".nim"
+  copyFile(getfile($file & "_" & $vers[file]), getfile($file))
+  inc vers[file]
+
+beforeCodeReload:
+  echo "main: before"
+
+afterCodeReload:
+  echo "main: after"
+
+echo "main: HELLO!"
+
+update 0
+compileReloadExecute() # versions are: 1 - -
+
+compileReloadExecute() # no change
+
+update 0
+update 1
+compileReloadExecute() # versions are: 2 1 -
+
+update 0
+update 2
+compileReloadExecute() # versions are: 3 1 1
+
+update 0
+update 1
+update 2
+compileReloadExecute() # versions are: 4 2 2
+
+update 0
+compileReloadExecute() # versions are: 5 2 2
+
+# final update so there are no git modifications left after everything
+# (the last versions are like the first files without a version suffix)
+update 0
+update 1
+update 2
+
+echo "done"

tests/dll/nimhcr_unit.nim

@@ -0,0 +1,147 @@
+discard """
+output: '''
+fastcall_proc implementation #1 10
+11
+fastcall_proc implementation #2 20
+22
+fastcall_proc implementation #2 20
+22
+fastcall_proc implementation #3 30
+33
+fastcall_proc implementation #3 30
+33
+fastcall_proc implementation #3 30
+33
+fastcall_proc implementation #3 40
+43
+cdecl_proc implementation #1 10
+11
+cdecl_proc implementation #2 20
+22
+cdecl_proc implementation #2 20
+22
+cdecl_proc implementation #3 30
+33
+cdecl_proc implementation #3 30
+33
+cdecl_proc implementation #3 30
+33
+cdecl_proc implementation #3 40
+43
+stdcall_proc implementation #1 10
+11
+stdcall_proc implementation #2 20
+22
+stdcall_proc implementation #2 20
+22
+stdcall_proc implementation #3 30
+33
+stdcall_proc implementation #3 30
+33
+stdcall_proc implementation #3 30
+33
+stdcall_proc implementation #3 40
+43
+noconv_proc implementation #1 10
+11
+noconv_proc implementation #2 20
+22
+noconv_proc implementation #2 20
+22
+noconv_proc implementation #3 30
+33
+noconv_proc implementation #3 30
+33
+noconv_proc implementation #3 30
+33
+noconv_proc implementation #3 40
+43
+inline_proc implementation #1 10
+11
+inline_proc implementation #2 20
+22
+inline_proc implementation #2 20
+22
+inline_proc implementation #3 30
+33
+inline_proc implementation #3 30
+33
+inline_proc implementation #3 30
+33
+inline_proc implementation #3 40
+43
+'''
+"""
+
+import macros
+
+macro carryOutTests(callingConv: untyped): untyped =
+  let
+    procName = $callingConv & "_proc"
+    globalName = $callingConv & "_global"
+    callingConv = callingConv
+    p1 = ident(procName & "1")
+    p2 = ident(procName & "2")
+    p3 = ident(procName & "3")
+    g1 = ident(globalName & "1")
+    g2 = ident(globalName & "2")
+
+  result = quote do:
+    var `g1`: pointer = nil
+    if hcrRegisterGlobal("dummy_module", `globalName`, sizeof(int), nil, addr `g1`):
+      cast[ptr int](`g1`)[] = 10
+
+    var `g2`: pointer = nil
+    if hcrRegisterGlobal("dummy_module", `globalName`, sizeof(int), nil, addr `g2`):
+      cast[ptr int](`g2`)[] = 20
+
+    doAssert `g1` == `g2` and cast[ptr int](`g1`)[] == 10
+
+    type
+      F = proc (x: int): int {.placeholder.}
+
+    proc `p1`(x: int): int {.placeholder.}=
+      echo `procName`, " implementation #1 ", x
+      return x + 1
+
+    let fp1 = cast[F](hcrRegisterProc("dummy_module", `procName`, `p1`))
+    echo fp1(10)
+
+    proc `p2`(x: int): int {.placeholder.} =
+      echo `procName`, " implementation #2 ", x
+      return x + 2
+
+    let fp2 = cast[F](hcrRegisterProc("dummy_module", `procName`, `p2`))
+    echo fp1(20)
+    echo fp2(20)
+
+    proc `p3`(x: int): int {.placeholder.} =
+      echo `procName`, " implementation #3 ", x
+      return x + 3
+
+    let fp3 = cast[F](hcrRegisterProc("dummy_module", `procName`, `p3`))
+    echo fp1(30)
+    echo fp2(30)
+    echo fp3(30)
+
+    let fp4 = cast[F](hcrGetProc("dummy_module", `procName`))
+    echo fp4(40)
+
+  proc replacePlaceholderPragmas(n: NimNode) =
+    if n.kind == nnkPragma:
+      n[0] = callingConv
+    else:
+      for i in 0 ..< n.len:
+        replacePlaceholderPragmas n[i]
+
+  replacePlaceholderPragmas result
+  # echo result.treeRepr
+
+hcrAddModule("dummy_module")
+
+carryOutTests fastcall
+carryOutTests cdecl
+carryOutTests stdcall
+carryOutTests noconv
+carryOutTests inline
+

tests/dll/nimhcr_unit.nim.cfg

@@ -0,0 +1,2 @@
+-d:useNimRtl
+-d:testNimHcr

tests/dll/test_nimhcr_integration.bat

@@ -0,0 +1,10 @@
+set NIM=nim
+set NIM_FLAGS=-d:debug
+
+%NIM% c --outdir:"." %NIM_FLAGS% ../../lib/nimrtl.nim
+%NIM% c --outdir:"." %NIM_FLAGS% ../../lib/nimhcr.nim
+
+set HCR_FLAGS=--forceBuild --hotCodeReloading:on --nimcache:nimcache %NIM_FLAGS%
+
+%NIM% %HCR_FLAGS% c nimhcr_integration.nim
+nimhcr_integration %NIM% %HCR_FLAGS% c nimhcr_integration.nim

tests/dll/test_nimhcr_integration.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+set -e
+
+rm -rf nimcache
+
+NIM_FLAGS=${*:- -d:debug}
+NIM=nim
+
+$NIM c --outdir:"." $NIM_FLAGS ../../lib/nimrtl.nim
+$NIM c --outdir:"." $NIM_FLAGS ../../lib/nimhcr.nim
+
+echo ===== Compiling HCR Integration Test =====
+HCR_FLAGS="--forceBuild --hotCodeReloading:on --nimcache:nimcache $NIM_FLAGS"
+$NIM $HCR_FLAGS c nimhcr_integration.nim
+export LD_LIBRARY_PATH=$(pwd):$LD_LIBRARY_PATH
+./nimhcr_integration $NIM $HCR_FLAGS c nimhcr_integration.nim

tests/types/tissues_types.nim

@@ -41,6 +41,13 @@ block t5648:
     var g = Foo()
     g.bar = 3
 
+  var
+    mainPtr1: pointer = main
+    mainPtr2 = pointer(main)
+    mainPtr3 = cast[pointer](main)
+
+  doAssert mainPtr1 == mainPtr2 and mainPtr2 == mainPtr3
+
   main()
 
 block t7581: