This PR, courtesy of @NagyZoltanPeter
(https://github.com/waku-org/nwaku/pull/3522) adds the ability to track
memory allocations in a program suitable for use with
[heaptrack](https://github.com/KDE/heaptrack).
By passing `-d:heaptrack --debugger:native` to compilation, calls to
heaptrack will be injected when memory is being allocated and released -
unlike `-d:useMalloc` this strategy also works with `refc` and the
default memory pool.
See https://github.com/KDE/heaptrack for usage examples. The resulting
binary needs to be run with `heaptrack` and with the shared
`libheaptrack_preload.so` in the `LD_LIBRARY_PATH`.
fixes#24998
Basically it retraces back to the situation before
https://github.com/nim-lang/Nim/pull/18366 and
https://github.com/nim-lang/Nim/pull/18362, i.e.
```nim
import fuzz/a
import fuzz/a
```
```nim
import fuzz/a
from buzz/a
```
```nim
import fuzz/a except nil
from fuzz/a import addInt
```
All of these cases are now flagged as invalid and triggers a
redefinition error, i.e., each module name importing is treated as
consistent as the symbol definition
kinda annoying for importing/exporting with `when conditions` though
ref https://github.com/nim-lang/Nim/issues/18762https://github.com/nim-lang/Nim/issues/20907
```nim
from std/strutils import toLower
when not defined(js):
from std/strutils import toUpper
```
Lets single threaded applications benefit from tracking foreign cells as
well.
After this, `SmallChunk` technically doesn't need to act as a linked
list anymore I think, gotta investigate that more though.
The likelihood of overflowing `chunk.free` also rises, so to work around
that it might make sense to check `foreignCells` instead of adjusting
free space or replace free with a counter for the local capacity.
For Nim compile I can observe a ~10mb reduction, and smaller ones for
other projects.
Ref: https://github.com/nim-lang/Nim/issues/23788
There was a small leak in the above issue even after fixing the
segfault. The sizes of `free` and `acc` were changed to 32bit because
adding the `foreignCells` field will drastically increase the memory
usage for programs that hold onto memory for a long time if they stay as
64bit.
I have made `realloc` absorb unused adjacent memory, which improves the
performance. I'm investigating whether `deallocOsPages` can be used to
improve memory comsumption.
* test azure
* use exit 1
* try again
* use useSysAssert
* disable i386
* use refc for tlsEmulation on i386
* use refc
* disable i386
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* allocator: catch up with multi-threading techniques
* removed the global thread lock
* more atomics for fun and profit
* added important sysAssert
* stats remain thread local and don't have to be atomic
* undo split chunk optimizations in the hope it makes the CI happy
* adding new system module sysexitprocs and including system exit procedures when registering exit handlers defined in userland
* fixing failing tests and adding initialization guard to handle cases where the module's global init logic isn't invoked first as is the case with some gc implementaions
* js backend shouldn't try to invoke actual system exit procs
* fixing formatting in sysexitprocs.nim
* 256 was too much - my max number of plugins in my engine is 64 and I require two hooks per runtime it looks like with tls emulation turned off, so for my purposes 128 should be sufficient
* so atExit should be enough here, can get rid of all the extra cruft I had added on top since I didn't realize atExit already provided a stack
* done being cute - since newruntime prevents correct cpp codegen for object variants apparently and breaks tests if I try to use std/exitprocs, ddSysExitProc is just going into both modules. Since system doesn't include system/io, polluting system with it doesn't make sense either... at least it is only importc'd when it is required in either module and we don't have to have any weird when defined(nimOwnedEnabled) with a comment explaining why
* I don't care about observable stores
* enforce explicit initializations
* cleaner code for the stdlib
* stdlib: use explicit initializations
* make tests green
* algorithm.nim: set result explicitly
* remove out parameters and bring the PR into a mergable state
* updated the changelog
* Cleanup, remove lib/system/allocators.nim. seqs_v2 and strs_v2 now use
allocShared0 by default.
* Fixed -d:useMalloc allocShared / reallocShared / deallocShared. These now use the alloc/dealloc/realloc implementation that also takes care of zeroing memory at realloc.
* Removed debug printfs
* Removed unpairedEnvAllocs() from tests/destructor/tnewruntime_misc
* More mmdisp cleanups. The shared allocators do not need to zero memory or throw since the regular ones already do that
* Introduced realloc0 and reallocShared0, these procs are now used by
strs_v2 and seqs_v2. This also allowed the -d:useMalloc allocator to
drop the extra header with allocation length.
* Moved strs_v2/seqs_v2 'allocated' flag into 'cap' field
* Added 'getAllocStats()' to get low level alloc/dealloc counters. Enable with -d:allocStats
* *allocShared implementations for boehm and go allocators now depend on the proper *allocImpl procs
* new .gensym implementation
* make astspec test green again
* introduce a --useVersion switch to group compatibility switches
* fixes#10180
* fixes#11494
* fixes#11483
* object constructor fields and named parameters are also not gensym'ed
* disabled broken package
* right shift is now by default sign preserving
* fix hashString and semfold
* enable arithmetic shift right globally for CI
* fix typo
* remove xxx
* use oldShiftRight as flag
* apply feedback
* add changelog entry
