fixes issue described in https://forum.nim-lang.org/t/12579
In #24065 explicit generic parameter matching was made to fail matches
on arguments with unresolved types in generic contexts (the sigmatch
diff, following #24010), similar to what is done for regular calls since
#22029. However unlike regular calls, a failed match in a generic
context for a standalone explicit generic instantiation did not convert
the expression into one with `tyFromExpr` type, which means it would
error immediately given any unresolved parameter. This is now done to
fix the issue.
For explicit generic instantiations on single non-overloaded symbols, a
successful match is still instantiated. For multiple overloads (i.e.
symchoice), if any of the overloads fail the match, the entire
expression is considered untyped and any instantiations are not used, so
as to not void overloads that would match later. This means even
symchoices without unresolved arguments aren't instantiated, which may
be too restrictive, but it could also be too lenient and we might need
to make symchoice instantiations always untyped. The behavior for
symchoice is not sound anyway given it causes #9997 so this is something
to consider for a redesign.
Diff follows #24276.
(cherry picked from commit 67ad1ae159)
fixes#24378
```nim
type Win = typeof(`body`)
doAssert not supportsCopyMem((int, Win))
```
`semAfterMacroCall` doesn't skip the children aliases types in the tuple
typedesc construction while the normal program seem to skip the aliases
types somewhere
`(int, Win)` is kept as `(int, alias string)` instead of expected `(int,
string)`
(cherry picked from commit 5e56f0a356)
refs #23625, refs #24289
Encountered in #24360 but could not reproduce minimally: overloading on
static parameters can work with the normal compile commands but crash
`nim check`. Static overloading relies on `tryConstExpr` which recovers
from things like `globalError` and fails softly, in this case this can
happen when a variable etc. is not available to evaluate in the VM. But
with `nim check`, the compiler does not throw an exception in this case,
and instead tries to keep generating the entire expression in the VM,
which can cause crashes.
To fix this, when the compiler has no error outputs even on `nim check`,
we raise a global error so that the VM code generation stops early. This
fixes both `tryConstExpr` and speeds up `nim check`, because no error
outputs means we don't need cascading errors.
(cherry picked from commit efd603eb28)
pre-existing issues:
```nim
block:
type
FooObj = object
data: int
Foo = ref ref FooObj
proc delete(self: Foo) =
echo self.data
var s: Foo
new(s, delete)
```
it crashed with arc/orc in 1.6.x and 2.x.x
```nim
block:
type
Foo = ref int
proc delete(self: Foo) =
echo self[]
var s: Foo
new(s, delete)
```
The simple fix is to add a type restriction for the type `T` for arc/orc
versions
```nim
proc new*[T: object](a: var ref T, finalizer: proc (x: T) {.nimcall.})
```
(cherry picked from commit 2af602a5c8)
fixes#13417, fixes#19703
When passing an expression to an `openarray` iterator parameter: If the
expression is a statement list (considered "complex"), it's assigned in
a non-deep-copying way to a temporary variable first, then this variable
is used as a parameter. If it's not a statement list, i.e. a call or a
symbol, the parameter is substituted directly with the given expression.
In the case of calls, this results in the call potentially being
executed more than once, or can cause redefined variables in the
codegen.
To fix this, calls are also considered as "complex" assignments to
openarrays, as long as the return type of the call is not `openarray` as
the generated assignment in that case has issues/is unimplemented
(caused a segfault [here in
datamancer](47ba4d81bf/src/datamancer/dataframe.nim (L1580))).
As for why creating a temporary isn't the default only with exceptions
for things like `nkSym`, the "non-deep-copying" way of assignment
apparently still causes arrays to be copied according to a comment in
the code. I'm not sure to what extent this is true: if it still happens
on ARC/ORC, if it happens for every array length, or if we can fix it by
passing arrays by reference. Otherwise, a more general way to assign to
openarrays might be needed, but I'm not sure if the compiler can easily
do this.
(cherry picked from commit d303c289fa)
fixes#23952
It reorders `type Foo = enum A, B = -1` to `type Foo = enum B = -1, A`
so that `firstOrd` etc. continue to work.
(cherry picked from commit 294b1566e7)
closes https://github.com/nim-lang/RFCs/issues/380, fixes#4773, fixes
#14729, fixes#16755, fixes#18150, fixes#22984, refs #11167 (only some
comments fixed), refs #12620 (needs tiny workaround)
The compiler gains a concept of root "nominal" types (i.e. objects,
enums, distincts, direct `Foo = ref object`s, generic versions of all of
these). Exported top-level routines in the same module as the nominal
types that their parameter types derive from (i.e. with
`var`/`sink`/`typedesc`/generic constraints) are considered attached to
the respective type, as the RFC states. This happens for every argument
regardless of placement.
When a call is overloaded and overload matching starts, for all
arguments in the call that already have a type, we add any operation
with the same name in the scope of the root nominal type of each
argument (if it exists) to the overload match. This also happens as
arguments gradually get typed after every overload match. This restricts
the considered overloads to ones attached to the given arguments, as
well as preventing `untyped` arguments from being forcefully typed due
to unrelated overloads. There are some caveats:
* If no overloads with a name are in scope, type bound ops are not
triggered, i.e. if `foo` is not declared, `foo(x)` will not consider a
type bound op for `x`.
* If overloads in scope do not have enough parameters up to the argument
which needs its type bound op considered, then type bound ops are also
not added. For example, if only `foo()` is in scope, `foo(x)` will not
consider a type bound op for `x`.
In the cases of "generic interfaces" like `hash`, `$`, `items` etc. this
is not really a problem since any code using it will have at least one
typed overload imported. For arbitrary versions of these though, as in
the test case for #12620, a workaround is to declare a temporary
"template" overload that never matches:
```nim
# neither have to be exported, just needed for any use of `foo`:
type Placeholder = object
proc foo(_: Placeholder) = discard
```
I don't know what a "proper" version of this could be, maybe something
to do with the new concepts.
Possible directions:
A limitation with the proposal is that parameters like `a: ref Foo` are
not attached to any type, even if `Foo` is nominal. Fixing this for just
`ptr`/`ref` would be a special case, parameters like `seq[Foo]` would
still not be attached to `Foo`. We could also skip any *structural* type
but this could produce more than one nominal type, i.e. `(Foo, Bar)`
(not that this is hard to implement, it just might be unexpected).
Converters do not use type bound ops, they still need to be in scope to
implicitly convert. But maybe they could also participate in the nominal
type consideration: if `Generic[T] = distinct T` has a converter to `T`,
both `Generic` and `T` can be considered as nominal roots.
The other restriction in the proposal, being in the same scope as the
nominal type, could maybe be worked around by explicitly attaching to
the type, i.e.: `proc foo(x: T) {.attach: T.}`, similar to class
extensions in newer OOP languages. The given type `T` needs to be
obtainable from the type of the given argument `x` however, i.e.
something like `proc foo(x: ref T) {.attach: T.}` doesn't work to fix
the `ref` issue since the compiler never obtains `T` from a given `ref
T` argument. Edit: Since the module is queried now, this is likely not
possible.
---------
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
(cherry picked from commit 2864830941)
fixes#24338
When unrolling each iteration of a `fields` iterator, the compiler only
opens a new scope for semchecking, but doesn't generate a node that
signals to the codegen that a new scope should be created. This causes
issues for reused template instantiations that reuse variable symbols
between each iteration, which causes the codegen to generate multiple
declarations for them in the same scope (regardless of `inject` or
`gensym`). To fix this, we wrap the unrolled iterations in an `if true:
body` node, which both opens a new scope and doesn't interfere with
`break`.
(cherry picked from commit ca5df9ab25)
Running `ctags` on files with quoted symbols (e.g. `$`) would list \`
instead of the full ident. Issue was the result getting reassigned at
the end to a \` instead of appending
(cherry picked from commit 93c24fe1c5)
closes https://github.com/nim-lang/RFCs/issues/554
Adds a symmetric difference operation to the language bitset type. This
maps to a simple `xor` operation on the backend and thus is likely
faster than the current alternatives, namely `(a - b) + (b - a)` or `a +
b - a * b`. The compiler VM implementation of bitsets already
implemented this via `symdiffSets` but it was never used.
The standalone binary operation is added to `setutils`, named
`symmetricDifference` in line with [hash
sets](https://nim-lang.org/docs/sets.html#symmetricDifference%2CHashSet%5BA%5D%2CHashSet%5BA%5D).
An operator version `-+-` and an in-place version like `toggle` as
described in the RFC are also added, implemented as trivial sugar.
(cherry picked from commit ae9287c4f3)
refs #8064, refs #24010
Error messages for standalone explicit generic instantiations are
revamped. Failing standalone explicit generic instantiations now only
error after overloading has finished and resolved to the default `[]`
magic (this means `[]` can now be overloaded for procs but this isn't
necessarily intentional, in #24010 it was documented that it isn't
possible). The error messages for failed instantiations are also no
longer a simple `cannot instantiate: foo` message, instead they now give
the same type mismatch error message as overloads with mismatching
explicit generic parameters.
This is now possible due to the changes in #24010 that delegate all
explicit generic proc instantiations to overload resolution. Old code
that worked around this is now removed. `maybeInstantiateGeneric` could
maybe also be removed in favor of just `explicitGenericSym`, the `result
== n` case is due to `explicitGenericInstError` which is only for niche
cases.
Also, to cause "ambiguous identifier" error messages when the explicit
instantiation is a symchoice and the expression context doesn't allow
symchoices, we semcheck the sym/symchoice created by
`explicitGenericSym` with the given expression flags.
#8064 isn't entirely fixed because the error message is still misleading
for the original case which does `values[1]`, as a consequence of
#12664.
(cherry picked from commit 0a058a6b8f)
The first commit reverts the revert of #23787.
The second fixes lambdalifting in convolutedly nested
closures/closureiters. This is considered to be the reason of #24094,
though I can't tell for sure, as I was not able to reproduce #24094 for
complicated but irrelevant reasons. Therefore I ask @jmgomez, @metagn or
anyone who could reproduce it to try it again with this PR.
I would suggest this PR to not be squashed if possible, as the history
is already messy enough.
Some theory behind the lambdalifting fix:
- A closureiter that captures anything outside its body will always have
`:up` in its env. This property is now used as a trigger to lift any
proc that captures such a closureiter.
- Instantiating a closureiter involves filling out its `:up`, which was
previously done incorrectly. The fixed algorithm is to use "current" env
if it is the owner of the iter declaration, or traverse through `:up`s
of env param until the common ancestor is found.
---------
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
(cherry picked from commit 5fa96ef270)
fixes#24305, refs #23807
Since #23014 `nkHiddenAddr` is produced to fast assign array elements in
iterators. However the array access inside this `nkHiddenAddr` can get
folded at compile time, generating invalid code. In #23807, compile time
folding of regular `addr` expressions was changed to be prevented in
`transf` but `nkHiddenAddr` was not updated alongside it.
The method for preventing folding in `addr` in #23807 was also faulty,
it should only trigger on the immediate child node of the address rather
than all nodes nested inside it. This caused a regression as outlined in
[this
comment](https://github.com/nim-lang/Nim/pull/24322#issuecomment-2419560182).
To fix both issues, `addr` and `nkHiddenAddr` now both shallowly prevent
constant folding for their immediate children.
(cherry picked from commit 52cf7dfde0)
fixes#24319
`byRefLoc` (`mapType`) requires the Loc `a` to have the right type.
Without `lfEnforceDeref`, it produces the wrong type for `deref (var
array)`, which may come from `mitems`.
(cherry picked from commit 0347536ff2)
fixes#18896fixes#20886
```nim
type
PFile {.importc: "FILE*", header: "<stdio.h>".} = distinct pointer
# import C's FILE* type; Nim will treat it as a new pointer type
```
This is an excerpt from the Nim manual. In the old Nim versions, it
produces a void pointer type instead of the `FILE*` type that should
have been generated. Because these C types tend to be opaque and adapt
to changes on different platforms. It might affect the portability of
Nim on various OS, i.e. `csource_v2` cannot build on the apline platform
because of `Time` relies on Nim types instead of the `time_t` type.
ref https://github.com/nim-lang/Nim/pull/18851
(cherry picked from commit 8be82c36c9)
fixes#24258
It uses conditionals to guard against ill formed AST to produce better
error messages, rather than crashing
(cherry picked from commit 8b39b2df7d)
fixes#24296fixes#24295
Templates use `expectedType` for type inference. It's justified that
when templates don't have an actual return type, i.e., `untyped` etc.
When the return type of templates is specified, we should not infer the
type
```nim
template g(): string = ""
let c: cstring = g()
```
In this example, it is not reasonable to annotate the templates
expression with the `cstring` type before the `fitNode` check with its
specified return type.
(cherry picked from commit 80e6b35721)
fixes#24288, refs #23625
Since #23625 "cannot evaluate" errors during VM code generation are
"soft" errors with `nim check`, i.e. the code generation isn't halted
(except for the current proc which `return`s which can cause wrong
codegen) and the expression is still attempted to be evaluated. Now,
these errors signal to the VM that the current generated VM code cannot
be evaluated, and so instead of evaluating, an error node is returned.
This keeps the benefit of the "soft" errors without potentially crashing
the compiler on improperly generated VM code. Although maybe the
compiler might not be able to handle the generated error node in some
cases.
This fixes the chame example in #24288 but this is not tested in CI.
Presumably it or the compiler was doing something like `compiles()` on
code that can't run in the VM.
I would accept nicer ways of tracking non-evaluability than
`c.cannotEval = true` but I tried to keep it as harmless as possible.
(cherry picked from commit def1fea43a)
fixes#24021
The field checking for case object branches at some point generates a
negated set `contains` check for the object discriminator. For enum
types, this tries to generate a complement set and convert to a
`contains` check in that instead. It obtains this type from the type of
the element node in the `contains` check.
`buildProperFieldCheck` creates the element node by changing a field
access expression like `foo.z` into `foo.kind`. In order to do this, it
copies the node `foo.z` and sets the field name in the node to the
symbol `kind`. But when copying the node, the type of the original
`foo.z` is retained. This means that the complement is performed on the
type of the accessed field rather than the type of the discriminator,
which causes problems when the accessed field is also an enum.
To fix this, we properly set the type of the copied node to the type of
the kind field. An alternative is just to make a new node instead.
A lot of text for a single line change, I know, but this part of the
codebase could use more explanation.
(cherry picked from commit 1bebc236bd)
fixes#24087, refs https://forum.nim-lang.org/t/341, refs #14222, refs
#14221
The Nim compiler calls `cl` for linking as well as compilation. This
means that options to the linker have to be passed after a `/link`
argument. But the Nim compiler doesn't include this option normally,
because users may still want to pass non-linker options to `cl` at link
time.
To deal with this, a workaround is used: every single library link
option adds `/link` before it. The linker simply ignores extraneous
`/link` arguments and gives a warning instead, since it's an
unrecognized option to the linker. This is really hacky but otherwise we
need to separate linker arguments into arguments passed either to the
compiler or to the linker at link time, and this behavior wouldn't be
meaningful outside of MSVC.
I can't really test this manually but I did test that the linker ignores
`/link`. I also can't really do more than this, I don't really use MSVC
so I wouldn't know how to navigate it, or how people use it. Ideally
someone who knows more about/uses MSVC can give their input or take
over.
(cherry picked from commit 449106a5a4)
fixes#24269, refs #20095
Instead of checking the package of the *used sym* to determine whether a
stylecheck should trigger, we check the package of the lineinfo instead.
Before #20095 this checked for the current compilation context module
instead which caused issues with generic procs, but the lineinfo should
more closely match the AST.
I figured this might cause issues with includes etc but the foreign
package test specifically tests for an include and passes, so maybe the
package determining logic accounts for this already. This still might
not be the correct logic, I'm not too familiar with the package handling
in the compiler.
Package PRs, both merged:
- json_rpc: https://github.com/status-im/nim-json-rpc/pull/226
- json_serialization:
https://github.com/status-im/nim-json-serialization/pull/99
(cherry picked from commit aaf6c408c6)
fixes#24274
The code in the `if` branch replaces the current destination `d` with a
new one. But the location `d` can be an assignment location, in which
case the provided expression isn't generated. To fix this, don't trigger
this code for when the location already exists. An alternative would be
to call `putIntoDest` in this case as is done below.
(cherry picked from commit 9c85f4fd07)
fixes#24091, refs #24092
Any instantiations resolving to a generic body type now gives an error.
Due to #24092, this does not error in cases like matching against `type
M` in generics because generic body type symbols are just not
instantiated. But this prevents parameters with type `type M` from being
used, although there doesn't seem to be any code which does this. Just
in case such code exists, we still allow `typedesc` types resolving to
generic body types.
(cherry picked from commit 2f904535d0)
fixes#23587
As explained in the issue, `getOrDefault` has a parameter named
`default` that can be a proc after generic instantiation. But the
parameter having a proc type [overrides all other
overloads](f73e03b132/compiler/semexprs.nim (L1203))
including the magic `system.default` overload and causes a compile error
if the proc doesn't match the normal use of `default`. To fix this, the
`result = default(B)` initializer call is removed because it's not
needed, `result` is always set in `getOrDefaultImpl` when a default
value is provided.
This is still a suspicious behavior of the compiler but `tables` working
has a higher priority.
(cherry picked from commit 67ea754b7f)
fixes#18649, refs #24183
Same as in #24183 for templates, we now process pragma nodes in generics
so that macro symbols are captured and the pragma arguments are checked,
but ignoring language pragma keywords.
A difference is that we cannot process call nodes as is, we have to
process their children individually so that the early untyped
macro/template instantiation in generics does not kick in.
(cherry picked from commit d72b848d17)
fixes#24233
Integer literals with type `int` can match `int64` with a generic match.
Normally this would generate an conversion via `isFromIntLit`, but when
it matches with a generic match (`isGeneric`) the node is left alone and
continues to have type `int` (related to #4858, but separate; since
`isFromIntLit > isGeneric` it doesn't propagate). This did not cause
problems on the C backend up to this point because either the compiler
generated a cast when generating the C code or it was implicitly casted
in the C code itself. On the JS backend however, we need to generate
`int64` and `int` values differently, so we copy the integer literal and
give it the matched type now instead.
This is somewhat risky even if CI passes but it's required to make the
times module work without [this
workaround](7dfadb8b4e/lib/pure/times.nim (L219-L238))
on `--jsbigint64:on` (the default).
CI exposed an issue: When matching an int literal to a generic parameter
in a generic instantiation, the literal is only treated like a value if
it has `int literal` type, but if it has the type `int`, it gets
transformed into literally the type `int` (#12664, #13906), which breaks
the tests t14193 and t12938. To deal with this, we don't give it the
type `int` if we are in a generic instantiation and preserve the `int
literal` type.
(cherry picked from commit c73eedfe6e)
fixes#24186
When encountering pragma nodes in templates, if it's a language pragma,
we don't process the name, and only any values if they exist. If it's
not a language pragma, we process the full node. Previously only the
values of colon expressions were processed.
To make this simpler, `whichPragma` is patched to consider bracketed
hint/warning etc pragmas like `{.hint[HintName]: off.}` as being a
pragma of kind `wHint` rather than an invalid pragma which would have to
be checked separately. From looking at the uses of `whichPragma` this
doesn't seem like it would cause problems.
Generics have [the same
problem](a27542195c/compiler/semgnrc.nim (L619))
(causing #18649), but to make it work we need to make sure the
templates/macros don't get evaluated or get evaluated correctly (i.e.
passing the proc node as the final argument), either with #23094 or by
completely disabling template/macro evaluation when processing the
pragma node, which would also cover `{.pragma.}` templates.
(cherry picked from commit 911cef1621)
fixes#24228, refs #22022
As described in
https://github.com/nim-lang/Nim/issues/24228#issuecomment-2392462221,
instantiating generic routines inside `typeof` causes all code inside to
be treated as being in a typeof context, and thus preventing compile
time proc folding, causing issues when code is generated for the
instantiated routine. Now, instantiated generic procs are treated as
never being inside a `typeof` context.
This is probably an arbitrary special case and more issues with the
`typeof` behavior from #22022 are likely. Ideally this behavior would be
removed but it's necessary to accomodate the current [proc `declval` in
the package `stew`](https://github.com/status-im/nim-stew/pull/190), at
least without changes to `compileTime` that would either break other
code (making it not eagerly fold by default) or still require a change
in stew (adding an option to disable the eager folding).
Alternatively we could also make the eager folding opt-in only for
generic compileTime procs so that #22022 breaks nothing whatsoever, but
a universal solution would be better. Edit: Done in #24230 via
experimental switch
(cherry picked from commit ea9811a4d2)
split from #24198
This is a required refactor for the only good solution I've been able to
think of for #4858 etc. Explanation:
---
`sigmatch` currently [disables
bindings](d6a71a1067/compiler/sigmatch.nim (L1956))
(except for binding to other generic parameters) when matching against
constraints of generic parameters. This is so when the constraint is a
general metatype like `seq`, the type matching will not treat all
following uses of `seq` as the type matched against that generic
parameter.
However to solve #4858 etc we need to bind `or` types with a conversion
match to the type they are supposed to be converted to (i.e. matching
`int literal(123)` against `int8 | int16` should bind `int8`[^1], not
`int`). The generic parameter constraint binding needs some way to keep
track of this so that matching `int literal(123)` against `T: int8 |
int16` also binds `T` to `int8`[^1].
The only good way to do this IMO is to generate a new "binding context"
when matching against constraints, then binding the generic param to
what the constraint was bound to in that context (in #24198 this is
restricted to just `or` types & concrete types with convertible matches,
it doesn't work in general).
---
`semtypinst` already does something similar for bindings of generic
invocations using `LayeredIdTable`, so `LayeredIdTable` is now split
into its own module and used in `sigmatch` for type bindings as well,
rather than a single-layer `TypeMapping`. Other modules which act on
`sigmatch`'s binding map are also updated to use this type instead.
The type is also made into an `object` type rather than a `ref object`
to reduce the pointer indirection when embedding it inside
`TCandidate`/`TReplTypeVars`, but only on arc/orc since there are some
weird aliasing bugs on refc/markAndSweep that cause a segfault when
setting a layer to its previous layer. If we want we can also just
remove the conditional compilation altogether and always use `ref
object` at the cost of some performance.
[^1]: `int8` binding here and not `int16` might seem weird, since they
match equally well. But we need to resolve the ambiguity here, in #24012
I tested disallowing ambiguities like this and it broke many packages
that tries to match int literals to things like `int16 | uint16` or
`int8 | int16`. Instead of making these packages stop working I think
it's better we resolve the ambiguity with a rule like "the earliest `or`
branch with the best match, matches". This is the rule used in #24198.
(cherry picked from commit cad8726907)
fixes#18396, fixes#20142
Set types with base types matching less than a generic match (so
subrange matches, conversion matches, int conversion matches) are now
considered mismatching, as their representation is different on the
backends (except VM and JS), causing codegen issues. An exception is
granted for set literal types, which now implicitly convert each element
to the matched base type, so things like `s == {'a', 'b'}` are still
possible where `s` is `set[range['a'..'z']]`. Also every conversion
match in this case is unified under the normal "conversion" match, so a
literal doesn't match one set type better than the other, unless it's
equal.
However `{'a', 'b'} == s` or `{'a', 'b'} - s` etc is now not possible.
when it used to work in the VM. So this is somewhat breaking, and needs
a changelog entry.
(cherry picked from commit 7dfadb8b4e)
