Go to file
zah ca4b971bc8 Initial version of the hot-code reloading support for native targets (#10729)
* squashed work by Zahary

* squashing a ton of useful history... otherwise rebasing on top of upstream Nim after commit 82c009a2cb would be impossible.

* Code review changes; Working test suite (without code reloading enabled)

* - documentation
- implemented the HCR test - almost works...
- fix the issue on Unix where for executable targets the source file for the main module of a project in nimcache was being overwritten with the binary itself (and thus the actual source code was lost)
- fixing embedded paths to shared objects on unix (the "lib" prefix was being prepended to the entire path instead of just the filename)
- other fixes
- removing unnecessary includes since that file is already included in chcks.nim which is in turn included in system.nim (and previously was getting imported in chcks.nim but then system.nim improts something... and that breaks HCR (perhaps it could be fixed but it would be nice not to import anything in system))

* fix for clang & C++ - explicitly casting a function pointer to void*
more stable mangling of parameter names when HCR is on
the length of the static arrays in the DatInit functions is now part of the name of the variables, so when they get resized they get also recreated
more stable mangling for inline functions - no longer depends on the module which first used them
work on the new complicated HCR test - turned surprisingly complex - WIP
test now successfully passes even when re-running `koch test` (previously when the nimcache wasn't cold that lead to errors)
better documentation
calling setStackBottomWith for PreMain
passes over the HcrInit/DatInit/Init calls of all modules are now in the proper order (first all of one type, then all of the next). Also typeinfo globals are registered (created) in a single pass before the DatInit pass (because of the way generic instantiations are handled)
Fix the test suite execution on macOs
fix for being able to query the program arguments when using HCR on posix!
other fixes

* Bugfix: Fix a compilation error in C++ mode when a function pointer
is converted to a raw pointer

* basic documentation for the new hot code reloading semantics

* Add change log entry

* Don't re-execute the top-level statements while reloading JS code

* fix a number of tests broken in a recent bugfix

* Review changes

* Added {.executeOnReload.} pragma that indicates top-level statements
  that should be executed on each reload. To make this work, I've modified
  the way the `if (hcr_init_) {...}` guards are produced in the init code.
  This still needs more work as the new guards seem to be inserted within
  the previously generated guards.

  This change also removes the need for `lastRegistedGlobal` in nimhcr.

* Implemented the `signatureHash` magic and the `hasModuleChanged` API
  depending on it (the actual logic is not imlemented yet).

* Add the "hcr" prefix to all HCR-related symbols in the system module.
  Added a new `hotcodereloading` module exporting the high-level API to
  the user.

  Besides being more hygienic, this was also required in order to make
  it possible to use macros in the high-level API. Without the split,
  `system` would have to import `macros`, which was going to produce
  the well-known init problems.

* Attempted to solve the "GC markers problem".

  Crashes were expected with the previous code, because the GC markers
  were compiled as normal procs are registered in the GC. When their
  module is unloaded, dangling pointers will remain in the GC tables.
  To solve this issue, I don't register any GC markers when HCR is on,
  but I add them to the HCR globals metadata and I use a single marker
  registed in nimhcr during the initialization of the system module that
  will be responsible for marking all globals.

* fix a compilation error

* - implemented the hasModuleChanged functionality
- tuples can be returned and broken into different vars in global scope
- added comments for the closnig scopes of the if statements in the init proc
- the new executeOnReload pragma works now!
- other fixes

* finally! fixing this hack in a proper way - declaring the destructor out of line (out of the class body) - we no longer need to forward-declare popCurrentExceptionEx

* Force full module parsing

This is a temporary hack that breaks some tests. I'll investigate
later how these can be fixed.

* tuples are now properly handled when global!

* these comments mess up the codegen in debug mode when $n is not actually a new line (or something like that) - these labels are intended only for GOTO labels anyway...

* "solved" the issue with the .pdb locks on windows when a binary is being debugged and hot code reloading is used at the same time

* fixes after rebasing...

* small fixes for the test

* better handling of globals! no more compiler crashes for locals with the global pragma, also simplified code around loops in global scope which have local vars (actually globals)

* we can now use the global pragma even for ... globals!

* the right output

* lets try those boehm GC tests

* after the test is ran it will be at its starting state - no git modifications

* clarification in the docs

* removed unnecessary line directives for forward declarations of functions - they were causing trouble with hot code reloading when no semantic change propagates to the main module but a line directive got changed and thus the main module had to be recompiled since the .c code had changed

* fixed bug! was inserting duplicate keys into the table and later was removing only 1 copy of all the duplicates (after a few reloads)

* no longer breaking into DatInit code when not supposed to

* fixes after rebasing

* yet more fixes after rebasing

* Update jssys.nim

* Rework the HCR path-handling logic

After reviewing the code more carefully, I've noticed that the old logic
will be broken when the user overrides the '--out:f' compiler option.

Besides fixing this issues, I took the opportunity to implement the
missing '--outdir:d' option.

Other changes:

* ./koch test won't overwrite any HCR and RTL builds located in nim/lib
* HCR and RTL are compiled with --threads:on by default

* Clean up the globals registration logic

* Handle non-flattened top-level stmtlists in JS as well

* The HCR is not supported with the Boehm GC yet

Also fixes some typos and the expected output of the HCR integration test

* The GC marker procs are now properly used as trampolines

* Fix the HCR integration test in release builds

* Fix ./koch tools

* this forward declaration doesn't seem to be necessary, and in fact breaks HCR because a 2nd function pointer is emitted for this externed/rtl func

* the forward declaration I removed in the last commit was actually necessary

* Attempt to make all tests green

* Fix tgenscript

* BAT file for running the HCR integration test on Windows [skip ci]

* Fix the docgen tests

* A final fix for Travis (hopefully)
2019-02-26 15:48:55 +01:00
2019-02-12 22:00:31 +01:00
2018-11-19 09:22:21 +01:00
2018-10-12 17:27:47 +02:00
2018-11-23 11:58:28 +01:00
2019-02-05 14:01:47 +01:00
2019-02-19 17:25:03 +01:00
2017-10-28 09:25:56 +02:00
2018-12-11 21:23:18 +01:00
2019-02-18 08:07:10 +01:00
2019-02-18 08:07:10 +01:00
2019-02-12 22:00:31 +01:00
2018-05-31 01:37:34 +03:00
2019-02-23 11:34:39 +01:00
2015-02-03 09:04:24 +01:00
2019-01-23 11:12:46 +01:00

Nim Build Status

This repository contains the Nim compiler, Nim's stdlib, tools and documentation. For more information about Nim, including downloads and documentation for the latest release, check out Nim's website or bleeding edge docs.

Community

Join the IRC chat Join the Gitter chat Get help View Nim posts on Stack Overflow Follow @nim_lang on Twitter

  • The forum - the best place to ask questions and to discuss Nim.
  • #nim IRC Channel (Freenode) - a place to discuss Nim in real-time. Also where most development decisions get made.
  • Gitter - an additional place to discuss Nim in real-time. There is a bridge between Gitter and the IRC channel.
  • Telegram - an additional place to discuss Nim in real-time. There is the official Telegram channel.
  • Stack Overflow - a popular Q/A site for programming related topics that includes posts about Nim.
  • Github Wiki - Misc user-contributed content.

Compiling

The compiler currently officially supports the following platform and architecture combinations:

  • Windows (Windows XP or greater) - x86 and x86_64
  • Linux (most, if not all, distributions) - x86, x86_64, ppc64 and armv6l
  • Mac OS X (10.04 or greater) - x86, x86_64 and ppc64

More platforms are supported, however they are not tested regularly and they may not be as stable as the above-listed platforms.

Compiling the Nim compiler is quite straightforward if you follow these steps:

First, the C source of an older version of the Nim compiler is needed to bootstrap the latest version because the Nim compiler itself is written in the Nim programming language. Those C sources are available within the nim-lang/csources repository.

Next, to build from source you will need:

  • A C compiler such as gcc 3.x/later or an alternative such as clang, Visual C++ or Intel C++. It is recommended to use gcc 3.x or later.
  • Either git or wget to download the needed source repositories.
  • The build-essential package when using gcc on Ubuntu (and likely other distros as well).

Then, if you are on a *nix system or Windows, the following steps should compile Nim from source using gcc, git and the koch build tool.

Note: The following commands are for the development version of the compiler. For most users, installing the latest stable version is enough. Check out the installation instructions on the website to do so: https://nim-lang.org/install.html.

For package mantainers: see packaging guidelines.

# step 1:
git clone https://github.com/nim-lang/Nim.git
cd Nim

# step 2 (posix) clones `csources.git`, bootstraps Nim compiler and compiles tools
sh build_all.sh

# step 2 (windows)
git clone --depth 1 https://github.com/nim-lang/csources.git

cd csources
# requires `gcc` in your PATH, see also https://nim-lang.org/install_windows.html
build.bat # x86 Windows
build64.bat # x86_64 Windows
cd ..

bin\nim c koch
koch boot -d:release
koch tools # Compile Nimble and other tools
# end of step 2 (windows)

Finally, once you have finished the build steps (on Windows, Mac or Linux) you should add the bin directory to your PATH.

Koch

koch is the build tool used to build various parts of Nim and to generate documentation and the website, among other things. The koch tool can also be used to run the Nim test suite.

Assuming that you added Nim's bin directory to your PATH, you may execute the tests using ./koch tests. The tests take a while to run, but you can run a subset of tests by specifying a category (for example ./koch tests cat async).

For more information on the koch build tool please see the documentation within the doc/koch.rst file.

Nimble

nimble is Nim's package manager. To learn more about it, see the nim-lang/nimble repository.

Contributors

This project exists thanks to all the people who contribute.

Contributing

Backers on Open Collective Sponsors on Open Collective Setup a bounty via Bountysource Donate Bitcoins Open Source Helpers

See detailed contributing guidelines. We welcome all contributions to Nim regardless of how small or large they are. Everything from spelling fixes to new modules to be included in the standard library are welcomed and appreciated. Before you start contributing, you should familiarize yourself with the following repository structure:

  • bin/, build/ - these directories are empty, but are used when Nim is built.
  • compiler/ - the compiler source code. Also includes nimfix, and plugins within compiler/nimfix and compiler/plugins respectively.
  • nimsuggest - the nimsuggest tool that previously lived in the nim-lang/nimsuggest repository.
  • config/ - the configuration for the compiler and documentation generator.
  • doc/ - the documentation files in reStructuredText format.
  • lib/ - the standard library, including:
    • pure/ - modules in the standard library written in pure Nim.
    • impure/ - modules in the standard library written in pure Nim with dependencies written in other languages.
    • wrappers/ - modules which wrap dependencies written in other languages.
  • tests/ - contains categorized tests for the compiler and standard library.
  • tools/ - the tools including niminst and nimweb (mostly invoked via koch).
  • koch.nim - tool used to bootstrap Nim, generate C sources, build the website, and generate the documentation.

If you are not familiar with making a pull request using GitHub and/or git, please read this guide.

Ideally you should make sure that all tests pass before submitting a pull request. However, if you are short on time, you can just run the tests specific to your changes by only running the corresponding categories of tests. Travis CI verifies that all tests pass before allowing the pull request to be accepted, so only running specific tests should be harmless. Integration tests should go in tests/untestable.

If you're looking for ways to contribute, please look at our issue tracker. There are always plenty of issues labelled Easy; these should be a good starting point for an initial contribution to Nim.

You can also help with the development of Nim by making donations. Donations can be made using:

If you have any questions feel free to submit a question on the Nim forum, or via IRC on the #nim channel.

Backers

Thank you to all our backers! [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

You can also see a list of all our sponsors/backers from various payment services on the sponsors page of our website.

License

The compiler and the standard library are licensed under the MIT license, except for some modules which explicitly state otherwise. As a result you may use any compatible license (essentially any license) for your own programs developed with Nim. You are explicitly permitted to develop commercial applications using Nim.

Please read the copying.txt file for more details.

Copyright © 2006-2019 Andreas Rumpf, all rights reserved.

Description
Nim is a statically typed compiled systems programming language. It combines successful concepts from mature languages like Python, Ada and Modula. Its design focuses on efficiency, expressiveness, and elegance (in that order of priority).
Readme 632 MiB
Languages
Nim 96.6%
HTML 1.8%
Python 0.5%
C 0.4%
Shell 0.4%
Other 0.2%