TODO:
- [ ] test writing of .nif files
- [x] implement loading of fields in PType/PSym that might not have been
loaded
- [ ] implement interface logic
- [ ] implement pragma "replays"
- [ ] implement special logic for `converter`
- [ ] implement special logic for `method`
- [ ] test the logic holds up for `export`
- [ ] implement logic to free the memory of PSym/PType if memory
pressure is high
- [ ] implement logic to close memory mapped files if too many are open.
---------
Co-authored-by: demotomohiro <gpuppur@gmail.com>
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>
fixes#25046
```nim
proc makeiter(v: string): iterator(): string =
return iterator(): string =
yield v
# loops
for c in makeiter("test")():
echo "loops ", c
```
becomes
```nim
var temp = makeiter("test")
for c in temp():
echo "loops ", c
```
for closures that might have side effects
fixes #25048
```nim
proc canFormAcycleAux =
of tyObject:
# Inheritance can introduce cyclic types, however this is not relevant
# as the type that is passed to 'new' is statically known!
# er but we use it also for the write barrier ...
if tfFinal notin t.flags:
# damn inheritance may introduce cycles:
result = true
```
It seems that all objects without `tfFinal` in their flags are
registering cycles. It doesn't seem that `Env` can be a cyclic type
because of inheritance since it is not going to be inherited after all
by another `Env` object type
fixes#24863, refs #23787 and #24316
Working off the minimized example, my understanding of the issue is: `n`
captures `r` as `:envP.r1` where `:envP` is the environment of `b`, then
`proc () = n()` does the lambda lifting of `n` again (which isn't done
if the `proc ()` is marked `{.closure.}`, hence the workaround) which
then captures the `:envP` as another field inside the `:envP`, so it
generates `:envP.:envP_2.r1` but the `.:envP_2` field is `nil`, so it
causes a segfault.
The problem is that the capture of `r` in `n` is done inside
`detectCapturedVars` for the surrounding closure iterator: inner procs
are not special cased and traversed as regular nodes, so it thinks it's
inside the iterator and generates a field access of `:envP` freely. The
lambda lifting version of `detectCapturedVars` ignores inner procs and
works off of symbol uses (anonymous iterator and lambda declarations
pretend their symbol is used).
As a naive solution, closure iterators now also ignore inner proc
declarations same as `lambdalifting.detectCapturedVars`, but unlike it
they also don't do anything for the inner proc symbols. Lambdalifting
seems to properly handle the lifted variables but in the worst case we
can also make sure `closureiters.detectCapturedVars` traverses inner
procs by marking every local of the closure iter used in them as needing
lifting (but not doing the lifting). This does not seem necessary for
now so it's not done (was done and reverted in [this
commit](9bb39a9259)),
but regressions are still possible
The first commit reverts the revert of #23787.
The second fixes lambdalifting in convolutedly nested
closures/closureiters. This is considered to be the reason of #24094,
though I can't tell for sure, as I was not able to reproduce #24094 for
complicated but irrelevant reasons. Therefore I ask @jmgomez, @metagn or
anyone who could reproduce it to try it again with this PR.
I would suggest this PR to not be squashed if possible, as the history
is already messy enough.
Some theory behind the lambdalifting fix:
- A closureiter that captures anything outside its body will always have
`:up` in its env. This property is now used as a trigger to lift any
proc that captures such a closureiter.
- Instantiating a closureiter involves filling out its `:up`, which was
previously done incorrectly. The fixed algorithm is to use "current" env
if it is the owner of the iter declaration, or traverse through `:up`s
of env param until the common ancestor is found.
---------
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
refs #24094, soft reverts #23787#23787 turned out to cause issues as described in #24094, but the
changes are still positive, so it is now only enabled if compiling with
`-d:nimOptIters`. Unfortunately the changes are really interwoven with
each other so the checks for this switch in the code are a bit messy,
but searching for `nimOptIters` should give the necessary clues to
remove the switch properly later on.
Locally tested that nimlangserver works but others can also check.
This pr redefines the relation between lambda lifting and closureiter
transformation.
Key takeaways:
- Lambdalifting now has less distinction between closureiters and
regular closures. Namely instead of lifting _all_ closureiter variables,
it lifts only those variables it would also lift for simple closure,
i.e. those not owned by the closure.
- It is now closureiter transformation's responsibility to lift all the
locals that need lifting and are not lifted by lambdalifting. So now we
lift only those locals that appear in more than one state. The rest
remains on stack, yay!
- Closureiter transformation always relies on the closure env param
created by lambdalifting. Special care taken to make lambdalifting
create it even in cases when it's "too early" to lift.
- Environments created by lambdalifting will contain `:state` only for
closureiters, whereas previously any closure env contained it.
IMO this is a more reasonable approach as it simplifies not only
lambdalifting, but transf too (e.g. freshVarsForClosureIters is now gone
for good).
I tried to organize the changes logically by commits, so it might be
easier to review this on per commit basis.
Some ugliness:
- Adding lifting to closureiters transformation I had to repeat this
matching of `return result = value` node. I tried to understand why it
is needed, but that was just another rabbit hole, so I left it for
another time. @Araq your input is welcome.
- In the last commit I've reused currently undocumented `liftLocals`
pragma for symbols so that closureiter transformation will forcefully
lift those even if they don't require lifting otherwise. This is needed
for [yasync](https://github.com/yglukhov/yasync) or else it will be very
sad.
Overall I'm quite happy with the results, I'm seeing some noticeable
code size reductions in my projects. Heavy closureiter/async users,
please give it a go.
fixes#4695
ref https://github.com/nim-lang/Nim/pull/15818
Since `nkState` is only for the main loop state labels and `nkGotoState`
is used only for dispatching the `:state` (since
https://github.com/nim-lang/Nim/pull/7770), it's feasible to rewrite the
loop body into a single case-based dispatcher, which enables support for
JS, VM backend. `nkState` Node is replaced by a label and Node pair and
`nkGotoState` is only used for intermediary processing. Backends only
need to implement `nkBreakState` and `closureIterSetupExc` to support
closure iterators.
pending https://github.com/nim-lang/Nim/pull/23484
<del> I also observed some performance boost for C backend in the
release mode (not in the danger mode though, I suppose the old
implementation is optimized into computed goto in the danger mode)
</del>
allPathsAsgnResult???
* refactoring in preparation for better, simpler name mangling that works with IC flawlessly
* use new disamb field
* see if this makes tests green
* make tests green again
* Fix bug in freshVarForClosureIter. Fixes#18474.
freshVarForClosureIter was returning non-fresh symbols sometimes.
Fixed by making addField return the generated PSym.
* remove discardable
Co-authored-by: Nick Smallbone <nick@smallbone.se>
* reworked ID handling
* the packed AST now has its own ID mechanism
* basic serialization code works
* extract rodfiles to its own module
* rodfiles: store and compare configs
* rodfiles: store dependencies
* store config at the end
* precise dependency tracking
* dependency tracking for rodfiles
* completed loading of PSym, PType, etc
* removed dead code
* bugfix: do not realloc seqs when taking addr into an element
* make IC opt-in for now
* makes tcompilerapi green again
* final cleanups
Co-authored-by: Andy Davidoff <github@andy.disruptek.com>
* refactoring: idents don't need inheritance
* refactoring: adding an IdGenerator (part 1)
* refactoring: adding an IdGenerator (part 2)
* refactoring: adding an IdGenerator (part 3)
* refactoring: adding an IdGenerator (part 4)
* refactoring: adding an IdGenerator (part 5)
* refactoring: adding an IdGenerator (part 5)
* IdGenerator must be a ref type; hello world works again
* make bootstrapping work again
* progress: add back the 'exactReplica' ideas
* added back the missing exactReplica hacks
* make tcompilerapi work again
* make important packages green
* attempt to fix the build for 32 bit machines (probably need a better solution here)
* `nim doc --backend:js|cpp...`
`nim doc --doccmd:'-d:foo --threads:on'`
`nim r --backend:cpp...` (implies --run --usenimcache)
* --usenimcache works with all targets
* --docCmd:skip now skips compiling snippets; 50X speedup for doc/manual.rst
* first implementation of the =trace and =dispose hooks for the cycle collector
* a cycle collector for ARC: progress
* manual: the .acyclic pragma is a thing once again
* gcbench: adaptations for --gc:arc
* enable valgrind tests for the strutils tests
* testament: better valgrind support
* ARC refactoring: growable jumpstacks
* ARC cycle detector: non-recursive algorithm
* moved and renamed core/ files back to system/
* refactoring: --gc:arc vs --gc:orc since 'orc' is even more experimental and we want to ship --gc:arc soonish
