Follow-up to #25506.
As I mentioned there, I was in the middle of an edit, so here it is.
Splitting to a separate doc skipped.
A couple of minor mistakes fixed, some things made a bit more concise
and short.
(cherry picked from commit 72e9bfe0a4)
Adds configurable parser modes to std/parseopt module. **Take two.**
Initially solved the issue of not being able to pass arguments to short
options as you do with most everyday CLI programs, but reading the tests
made me add more features so that some of the behaviour could be changed
and here we are.
**`std/parseopt` now supports three parser modes** via an optional
`mode` parameter in `initOptParser` and `getopt`.
Three modes are provided:
- `NimMode` (default, fully backward compatible),
- `LaxMode` (POSIX-inspired with relaxed short option handling),
- `GnuMode` (stricter GNU-style conventions).
The new modes are marked as experimental in the documentation.
The parser behaviour is controlled by a new `ParserRules` enum, which
provides granular feature flags that modes are built from. This makes it
possible for users with specific requirements to define custom rule sets
by importing private symbols, this is mentioned but clearly marked as
unsupported.
**Backward compatibility:**
The default mode preserves existing behaviour completely, with a single
exception: `allowWhitespaceAfterColon` is deprecated.
Now, `allowWhitespaceAfterColon` doesn't make much sense as a single
tuning knob. The `ParserRule.prSepAllowDelimAfter` controls this now.
As `allowWhitespaceAfterColon` had a default, most calls never mention
it so they will silently migrate to the new `initOptParser` overload. To
cover cases when the proc param was used at call-site, I added an
overload, which modifies the default parser mode to reflect the required
`allowWhitespaceAfterColon` value. Should be all smooth for most users,
except the deprecation warning.
The only thing I think can be classified as the breaking change is a
surprising **bug** of the old parser:
```nim
let p = initOptParser("-n 10 -m20 -k= 30 -40", shortNoVal = {'v'})
# ^-disappears
```
This is with the aforementioned `allowWhitespaceAfterColon` being true
by default, of course. In this case the `30` token is skipped
completely. I don't think that's right, so it's fixed.
Things I still don't like about how the old parser and the new default
mode behave:
1. **Parser behaviour is controlled by an emptiness of two containers**.
This is an interesting approach. It's also made more interesting because
the `shortNoVal`/`longNoVal` control both the namesakes, but *and also
how their opposites (value-taking opts) work*.
---
**Edit:**
2. `shortNoVal` is not mandatory:
```nim
let p = initOptParser(@["-a=foo"], shortNoVal = {'a'})
# Nim, Lax parses as: (cmdShortOption, "a", "foo")
# GnuMode parses as: (cmdShortOption, "a", "=foo")
```
In this case, even though the user specified `a` as no no-val, parser
ignores it, relying only on the syntax to decide the kind of the
argument. This is especially problematic with the modes that don't use
the rule `prShortAllowSep` (GnuMode), in this case the provided input is
twice invalid, regardless of the `shortNoVal`.
With the current parser architecture, parsing it this way **is
inevitable**, though. We don't have any way to signal the error state
detected with the input, so the user is expected to validate the input
for mistakes.
Bundling positional arguments is nonsensical and short option can't use
the separator character, so `[cmd "a", arg "=foo"]` and `[cmd "a", cmd
"=", cmd "f"...]` are both out of the question **and** would complicate
validating, requiring keeping track of a previous argument. Hope I'm
clear enough on the issue.
**Future work:**
1. Looks like the new modes are already usable, but from the discussions
elsewhere it looks like we might want to support special-casing
multi-digit short options (`-XX..`) to allow numerical options greater
than 9. This complicates bundling, though, so requires a bit of thinking
through.
2. Signaling error state?
---------
Co-authored-by: Andreas Rumpf <araq4k@proton.me>
(cherry picked from commit 7c873ca615)
fixes https://github.com/nim-lang/Nim/issues/25457
Small chunks allocate memory in fixed-size cells. Each cell is
positioned at exact multiples of the cell size from the chunk's data
start, which makes it much harder to support alignment
```nim
sysAssert c.size == size, "rawAlloc 6"
if c.freeList == nil:
sysAssert(c.acc.int + smallChunkOverhead() + size <= SmallChunkSize,
"rawAlloc 7")
result = cast[pointer](cast[int](addr(c.data)) +% c.acc.int)
inc(c.acc, size)
```
See also https://github.com/nim-lang/Nim/pull/12926
While using big trunk, each allocation gets its own chunk
(cherry picked from commit 94008531c1)
Pegs raise only their own error, but the forward declaration causes an
unwanted Exception effect
* use strformat which does compile-time analysis of the format string to
avoid exceptions
* also in parsecfg
(cherry picked from commit 92ad98f5d8)
This is a problem on big-endian CPUs because you end up with nimvm
computing something different than Nim proper, so e.g. a const table
won't work.
I also took the liberty to replace a redundant implementation of load4
in murmurHash.
(Thanks to barracuda156 for helping debug this.)
(cherry picked from commit a061f026a8)
fixes#25306
```nim
type
StackTraceEntry* = object ## In debug mode exceptions store the stack trace that led
## to them. A `StackTraceEntry` is a single entry of the
## stack trace.
procname*: cstring ## Name of the proc that is currently executing.
line*: int ## Line number of the proc that is currently executing.
filename*: cstring ## Filename of the proc that is currently executing.
when NimStackTraceMsgs:
frameMsg*: string ## When a stacktrace is generated in a given frame and
## rendered at a later time, we should ensure the stacktrace
## data isn't invalidated; any pointer into PFrame is
## subject to being invalidated so shouldn't be stored.
when defined(nimStackTraceOverride):
programCounter*: uint ## Program counter - will be used to get the rest of the info,
## when `$` is called on this type. We can't use
## "cuintptr_t" in here.
procnameStr*, filenameStr*: string ## GC-ed alternatives to "procname" and "filename"
```
(cherry picked from commit 0ea5f2625c)
`InterlockedCompareExchange64 `(winnt.h) is used instead of gcc atomics
when compiling with MSVC on Windows, but the function signatures are
`InterlockedCompareExchange64(ptr int64, int64, int64)` and
`InterlockedCompareExchange32(ptr int32, int32, int32)` as opposed to
`(ptr T, ptr T, T)` for `__atomic_compare_exchange_n`.
Passing a pointer to the expected value (parameter two) instead of the
value itself causes the comparison to unconditionally fail, with stalls
in threaded code using atomic comparisons.
Fix the function signature for MSVC.
Signed-off-by: Ryan Walklin <ryan@testtoast.com>
(cherry picked from commit 2d0b62aa51)
The forward declarations cause `Exception` to be inferred - also,
`llrecv` is an internal implementation detail and the type of the
received item is controlled by generics, thus the ValueError raised
there seems out of place for the generic api.
(cherry picked from commit 91febf1f4c)
- Removed redundant `len` and `reserved` sets already performed by prior
`rawNewStringNoInit` calls.
- Reuse `appendChar`
- Removed never used `newOwnedString`
- Added internal `toOwnedCopy`
- Documents differences in impls of internal procs used for
`system.string.setLen`:
+ `strs_v2.setLengthStrV2`:
- does not set the terminating zero byte when new length is 0
- does not handle negative new length
+ `sysstr.setLengthStr`:
- sets the terminating zero byte when new length is 0
- bounds negative new length to 0
(cherry picked from commit b539adf829)
Continuation of #25180. This one refactors the sequence routines.
Preparation for extending with new routines.
Mostly removes repeating code to simplify debugging.
Removes:
- `incrSeqV2` superseded by `incrSeqV3`,
- `setLengthSeq` superseded by `setLengthSeqV2`
Note comment on line 338, acknowledging that implementation of
`setLenUninit` from #25022 does zero the new memory in this branch,
having been copied from `setLengthSeqV2`. This PR does not fix this.
(cherry picked from commit 01c084077e)
fixes#25007
```nim
proc setLengthSeqUninit(s: PGenericSeq, typ: PNimType, newLen: int, isTrivial: bool): PGenericSeq {.
compilerRtl.} =
```
In this added function, only the line `zeroMem(dataPointer(result,
elemAlign, elemSize, newLen), (result.len-%newLen) *% elemSize)` is
removed from `proc setLengthSeqV2` when enlarging a sequence.
JS and VM versions simply use `setLen`.
(cherry picked from commit 611b8bbf67)
fixes#19728
don't zero-filling memory for "trivial types" without destructor in
refc. I tested locally with internal apis.
(cherry picked from commit b421d0f8ee)
While `a.push.apply(a, b)` is better for performance than the previous
`a = a.concat(b)` due to the fact that it doesn't create a new array,
there is a pretty big problem with it: depending on the JS engine, if
the second array is too long, it can [cause a
crash](https://tanaikech.github.io/2020/04/20/limitation-of-array.prototype.push.apply-under-v8-for-google-apps-script/)
due to the function `push` taking too many arguments. This has
unfortunately been what the codegen produces since 1.4.0 (commit
707367e1ca).
So string addition is now moved to a compilerproc that just uses a `for`
loop. From what I can tell this is the most compatible and the fastest.
Only potential problem compared to `concat` etc is with aliasing, i.e.
adding an array to itself, but I'm guessing it's enough that the length
from before the iteration is used, since it can only grow. The test
checks for aliased nim strings but I don't know if there's an extra
protection for them.
(cherry picked from commit 839cbeb371)
This PR, courtesy of @NagyZoltanPeter
(https://github.com/waku-org/nwaku/pull/3522) adds the ability to track
memory allocations in a program suitable for use with
[heaptrack](https://github.com/KDE/heaptrack).
By passing `-d:heaptrack --debugger:native` to compilation, calls to
heaptrack will be injected when memory is being allocated and released -
unlike `-d:useMalloc` this strategy also works with `refc` and the
default memory pool.
See https://github.com/KDE/heaptrack for usage examples. The resulting
binary needs to be run with `heaptrack` and with the shared
`libheaptrack_preload.so` in the `LD_LIBRARY_PATH`.
(cherry picked from commit 861ebc0f19)
fixes#25240
> Deque items behavior is not the same on 2.0.16 and 2.2.0
The behavior seems to be caused by the temp introduced for the parameter
`deq.len`, which prevents it from being evaluated multiple times
(cherry picked from commit b7c02e9bad)
POSIX signal has an identical definition to ISO C signal:
https://pubs.opengroup.org/onlinepubs/9799919799/functions/signal.html
```c
void (*signal(int sig, void (*func)(int)))(int);
/* more readably restated by glibc as */
typedef void (*sighandler_t)(int);
sighandler_t signal(int signum, sighandler_t handler);
```
However, std/posix had omitted the function's return value; this fixes
that.
To prevent breaking every single line of code ever that touched this
binding (including mine...), I've also made it discardable.
Additionally, I have noticed that bsd_signal's type signature is wrong -
it should have been identical to signal. But bsd_signal was already
removed in POSIX 2008, and sigaction is the recommended, portable POSIX
signal interface. So I just deleted the bsd_signal binding.
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
(cherry picked from commit 483389d399)
TODO list, copied from the documentation:
- [x] compiler/platform.nim Add os/cpu properties.
- [x] lib/system.nim Add os/cpu to the documentation for system.hostOS
and system.hostCPU.
- [x] ~~compiler/options.nim Add special os/cpu property checks in
isDefined.~~ seems unnecessary; isn't dont for most CPUs
- [x] compiler/installer.ini Add os/cpu to Project.Platforms field.
- [x] lib/system/platforms.nim Add os/cpu.
- [x] ~~std/private/osseps.nim Add os specializations.~~
- [x] ~~lib/pure/distros.nim Add os, package handler.~~
- [x] ~~tools/niminst/makefile.nimf Add os/cpu compiler/linker flags.~~
already done in https://github.com/nim-lang/Nim/pull/20943
- [x] tools/niminst/buildsh.nimf Add os/cpu compiler/linker flags.
For csource:
- [x] have compiler/platform.nim updated
- [x] have compiler/installer.ini updated
- [x] have tools/niminst/buildsh.nimf updated
- [x] have tools/niminst/makefile.nimf updated
- [ ] be backported to the Nim version used by the csources
- [ ] the new csources must be pushed
- [ ] the new csources revision must be updated in
config/build_config.txt
Additionally:
- [x] check relation to https://github.com/nim-lang/Nim/pull/20943
Possible future work:
- Porting Nim to s390x-specific operating systems, notably z/OS
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
(cherry picked from commit f4497c6158)
...introduced by me in #24792. Sorry.
This fix doesn't avoid copying the `restrictedBody` twice in the
generated code but has the benefit of working.
Proper fix needs a detection that can set a const bool for a module
once. `when nimvm` is restricted in use and is difficult to dance
around. Some details in: #12517, #12518, #13038
I might have copied the buggy solution from some discussion and it might
have worked at some point, but it's small excuse.
(cherry picked from commit 6938fce40c)
fixes#24998
Basically it retraces back to the situation before
https://github.com/nim-lang/Nim/pull/18366 and
https://github.com/nim-lang/Nim/pull/18362, i.e.
```nim
import fuzz/a
import fuzz/a
```
```nim
import fuzz/a
from buzz/a
```
```nim
import fuzz/a except nil
from fuzz/a import addInt
```
All of these cases are now flagged as invalid and triggers a
redefinition error, i.e., each module name importing is treated as
consistent as the symbol definition
kinda annoying for importing/exporting with `when conditions` though
ref https://github.com/nim-lang/Nim/issues/18762https://github.com/nim-lang/Nim/issues/20907
```nim
from std/strutils import toLower
when not defined(js):
from std/strutils import toUpper
```
(cherry picked from commit 87ee9c84cb)
Inside a signal handler, you cannot allocate memory because the signal
handler, being implemented with a C
[`signal`](https://en.cppreference.com/w/c/program/signal) call, can be
called _during_ a memory allocation - when that happens, the CTRL-C
handler causes a segfault and/or other inconsistent state.
Similarly, the call can happen from a non-nim thread or inside a C
library function call etc, most of which do not support reentrancy and
therefore cannot be called _from_ a signal handler.
The stack trace facility used in the default handler is unfortunately
beyond fixing without more significant refactoring since it uses
garbage-collected types in its API and implementation.
As an alternative to https://github.com/nim-lang/Nim/pull/25110, this PR
removes the most problematic signal handler, namely the one for SIGINT
(ctrl-c) - SIGINT is special because it's meant to cause a regular
shutdown of the application and crashes during SIGINT handling are both
confusing and, if turned into SIGSEGV, have downstream effects like core
dumps and OS crash reports.
The signal handlers for the various crash scenarios remain as-is - they
may too cause their own crashes but we're already going down in a bad
way, so the harm is more limited - in particular, crashing during a
crash handler corrupts `core`/crash dumps. Users wanting to keep their
core files pristine should continue to use `-d:noSignalHandler` - this
is usually the better option for production applications since they
carry more detail than the Nim stack trace that gets printed.
Finally, the example of a ctrl-c handler performs the same mistake of
calling `echo` which is not well-defined - replace it with an example
that is mostly correct (except maybe for the lack of `volatile` for the
`stop` variable).
(cherry picked from commit 41ce86b577)
Raising exceptions halfway through a memory allocation is undefined
behavior since exceptions themselves require multiple allocations and
the allocator functions are not reentrant.
It is of course also expensive performance-wise to introduce lots of
exception-raising code everywhere since it breaks many optimisations and
bloats the code.
Finally, performing pointer arithmetic with signed integers is incorrect
for example on on a 32-bit systems that allows up to 3gb of address
space for applications (large address extensions) and unnecessary
elsewhere - broadly, stuff inside the memory allocator is generated by
the compiler or controlled by the standard library meaning that
applications should not be forced to pay this price.
If we wanted to check for overflow, the right way would be in the
initial allocation location where both the size and count of objects is
known.
The code is updated to use the same arithmetic operator style as for
refc with unchecked operations rather than disabling overflow checking
wholesale in the allocator module - there are reasons for both, but
going with the existing flow seems like an easier place to start.
(cherry picked from commit 8b9972c8b6)
Also added some documentation to the header.
See: https://forum.nim-lang.org/t/13311
> I did try using the flag, but couldn't get it to work. If I do
-d:nimIoSelector, the defined check passes, but the other code fails to
compile because there is no const named nimIoSelector. It seemed like a
bug to me, do you have a working number compiler invocation?
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
(cherry picked from commit d60e0211bc)
