TODO:
- [ ] test writing of .nif files
- [x] implement loading of fields in PType/PSym that might not have been
loaded
- [ ] implement interface logic
- [ ] implement pragma "replays"
- [ ] implement special logic for `converter`
- [ ] implement special logic for `method`
- [ ] test the logic holds up for `export`
- [ ] implement logic to free the memory of PSym/PType if memory
pressure is high
- [ ] implement logic to close memory mapped files if too many are open.
---------
Co-authored-by: demotomohiro <gpuppur@gmail.com>
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>
fixes#24093
transforms
```nim
let a = new array[1000, byte]
block:
for _ in cast[typeof(a)](a)[]:
discard
```
into
```nim
let a = new array[1000, byte]
block:
let temp = cast[typeof(a)](a)
for _ in temp[]:
discard
```
So it keeps the same behavior with the manual version
fixes#7179
```nim
var f = 751.0
echo f.int8
```
In this case, `int8(float)` yields different numbers for different
optimization levels, since float to int conversions are undefined
behaviors. In this PR, it mitigates this problem by conversions to same
size integers before converting to the final type: i.e.
`int8(int64(float))`, which has UB problems but is better than before
refs #24929, partially reverts #23403
Instead of using `Table[ItemId, T]`, the old algorithm is brought back
into `TIdTable[T]` to prevent a performance regression. The inheritance
removal from #23403 still holds, only `ItemId`s are stored.
implements https://github.com/nim-lang/RFCs/issues/557
It inserts defect handing into a bare except branch
```nim
try:
raiseAssert "test"
except:
echo "nope"
```
=>
```nim
try:
raiseAssert "test"
except:
# New behaviov, now well-defined: **never** catches the assert, regardless of panic mode
raiseDefect()
echo "nope"
```
In this way, `except` still catches foreign exceptions, but panics on
`Defect`. Probably when Nim has `except {.foreign.}`, we can extend
`raiseDefect` to foreign exceptions as well. That's supposed to be a
small use case anyway.
`--legacy:noPanicOnExcept` is provided for a transition period.
fixes#24631
[Object
constructors](793baf34ff/compiler/semobjconstr.nim (L462)),
[casts](793baf34ff/compiler/semexprs.nim (L494))
and [type
conversions](793baf34ff/compiler/semexprs.nim (L419))
copy their type nodes verbatim instead of producing semchecked type
nodes. This causes a crash in transf when an untyped expression in the
type node has `nil` type. To deal with this, don't try to transform the
type node in these expressions at all. I couldn't reproduce the problem
with type conversion nodes though so those are unchanged in transf.
fixes#24402
```nim
iterator myPairsInline*[T](twoDarray: seq[seq[T]]): (int, seq[T]) {.inline.} =
for indexValuePair in twoDarray.pairs:
yield indexValuePair
proc innerTestTotalMem() =
var my2dArray: seq[seq[int32]] = @[]
# fill with some data...
for i in 0'i32..100:
var z = @[i, i+1]
my2dArray.add z
for oneDindex, innerArray in myPairsInline(my2dArray):
discard
innerTestTotalMem()
```
In `for oneDindex, innerArray in myPairsInline(my2dArray)`, `oneDindex`
and `innerArray` becomes `cursors` because they satisfy the criterion of
`isSimpleIteratorVar`. On the one hand, it is not correct to have them
point to the temporary generated by tuple unpacking, which left the
memory of the temporary uncleaned up. On the other hand, we don't need
to generate a temporary for a symbol node when unpacking the tuple.
fixes#17958
In `transf`, conversions in subscript expressions are skipped (with
`skipConv`'s rules). This is because array indexing can produce
conversions to the range type that is the array's index type, which
causes a `RangeDefect` rather than an `IndexDefect` (and also
`--rangeChecks` and `--indexChecks` are both considered). However this
causes problems when explicit conversions are used, between types of
different bitsizes, because those also get skipped.
To fix this, we only skip the conversion if:
* it's a hidden (implicit) conversion
* it's a range check conversion (produces `nkChckRange`)
* the subscript is on an array type and the result type of the
conversion has the same bounds as the array index type
And `skipConv` rules also still apply (int/float classification).
Another idea would be to prevent the implicit conversion to the array
index type from being generated. But there is no good way to do this:
matching to the base type instead prevents types like `uint32` from
implicitly converting (i.e. it can convert to `range[0..3]` but not
`int`), and analyzing whether this is an array bound check is easier in
`transf`, since `sigmatch` just produces a type conversion.
The rules for skipping the conversion could also receive some other
tweaks: We could add a rule that changing bitsizes also doesn't skip the
conversion, but this breaks the `uint32` case. We could simplify it to
only removing implicit skips to specifically fix#17958, but this is
wrong in general.
We could also add something like `nkChckIndex` that generates index
errors instead but this is weird when it doesn't have access to the
collection type and it might be overkill.
fixes#13417, fixes#19703
When passing an expression to an `openarray` iterator parameter: If the
expression is a statement list (considered "complex"), it's assigned in
a non-deep-copying way to a temporary variable first, then this variable
is used as a parameter. If it's not a statement list, i.e. a call or a
symbol, the parameter is substituted directly with the given expression.
In the case of calls, this results in the call potentially being
executed more than once, or can cause redefined variables in the
codegen.
To fix this, calls are also considered as "complex" assignments to
openarrays, as long as the return type of the call is not `openarray` as
the generated assignment in that case has issues/is unimplemented
(caused a segfault [here in
datamancer](47ba4d81bf/src/datamancer/dataframe.nim (L1580))).
As for why creating a temporary isn't the default only with exceptions
for things like `nkSym`, the "non-deep-copying" way of assignment
apparently still causes arrays to be copied according to a comment in
the code. I'm not sure to what extent this is true: if it still happens
on ARC/ORC, if it happens for every array length, or if we can fix it by
passing arrays by reference. Otherwise, a more general way to assign to
openarrays might be needed, but I'm not sure if the compiler can easily
do this.
The first commit reverts the revert of #23787.
The second fixes lambdalifting in convolutedly nested
closures/closureiters. This is considered to be the reason of #24094,
though I can't tell for sure, as I was not able to reproduce #24094 for
complicated but irrelevant reasons. Therefore I ask @jmgomez, @metagn or
anyone who could reproduce it to try it again with this PR.
I would suggest this PR to not be squashed if possible, as the history
is already messy enough.
Some theory behind the lambdalifting fix:
- A closureiter that captures anything outside its body will always have
`:up` in its env. This property is now used as a trigger to lift any
proc that captures such a closureiter.
- Instantiating a closureiter involves filling out its `:up`, which was
previously done incorrectly. The fixed algorithm is to use "current" env
if it is the owner of the iter declaration, or traverse through `:up`s
of env param until the common ancestor is found.
---------
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
fixes#24305, refs #23807
Since #23014 `nkHiddenAddr` is produced to fast assign array elements in
iterators. However the array access inside this `nkHiddenAddr` can get
folded at compile time, generating invalid code. In #23807, compile time
folding of regular `addr` expressions was changed to be prevented in
`transf` but `nkHiddenAddr` was not updated alongside it.
The method for preventing folding in `addr` in #23807 was also faulty,
it should only trigger on the immediate child node of the address rather
than all nodes nested inside it. This caused a regression as outlined in
[this
comment](https://github.com/nim-lang/Nim/pull/24322#issuecomment-2419560182).
To fix both issues, `addr` and `nkHiddenAddr` now both shallowly prevent
constant folding for their immediate children.
refs #24094, soft reverts #23787#23787 turned out to cause issues as described in #24094, but the
changes are still positive, so it is now only enabled if compiling with
`-d:nimOptIters`. Unfortunately the changes are really interwoven with
each other so the checks for this switch in the code are a bit messy,
but searching for `nimOptIters` should give the necessary clues to
remove the switch properly later on.
Locally tested that nimlangserver works but others can also check.
This pr redefines the relation between lambda lifting and closureiter
transformation.
Key takeaways:
- Lambdalifting now has less distinction between closureiters and
regular closures. Namely instead of lifting _all_ closureiter variables,
it lifts only those variables it would also lift for simple closure,
i.e. those not owned by the closure.
- It is now closureiter transformation's responsibility to lift all the
locals that need lifting and are not lifted by lambdalifting. So now we
lift only those locals that appear in more than one state. The rest
remains on stack, yay!
- Closureiter transformation always relies on the closure env param
created by lambdalifting. Special care taken to make lambdalifting
create it even in cases when it's "too early" to lift.
- Environments created by lambdalifting will contain `:state` only for
closureiters, whereas previously any closure env contained it.
IMO this is a more reasonable approach as it simplifies not only
lambdalifting, but transf too (e.g. freshVarsForClosureIters is now gone
for good).
I tried to organize the changes logically by commits, so it might be
easier to review this on per commit basis.
Some ugliness:
- Adding lifting to closureiters transformation I had to repeat this
matching of `return result = value` node. I tried to understand why it
is needed, but that was just another rabbit hole, so I left it for
another time. @Araq your input is welcome.
- In the last commit I've reused currently undocumented `liftLocals`
pragma for symbols so that closureiter transformation will forcefully
lift those even if they don't require lifting otherwise. This is needed
for [yasync](https://github.com/yglukhov/yasync) or else it will be very
sad.
Overall I'm quite happy with the results, I'm seeing some noticeable
code size reductions in my projects. Heavy closureiter/async users,
please give it a go.
* refactoring in preparation for better, simpler name mangling that works with IC flawlessly
* use new disamb field
* see if this makes tests green
* make tests green again
* fixes#21540; deref block at transf phase to make injectdestructors function properly
* add a test case
* add one more test
* fixes the type of block
* transform block
* fixes#21306; don't transform yields in the var section when introducing new local vars
* adds `inVarSection` so the var section in the var section is freshed
* use `isIntroducingNewLocalVars` to avoid yield transformations in var sections
* fixes comments
