`CsvParser.open` did not reset parser-specific state when reopening a
parser.
After parsing one input and reusing the same parser for another input,
the previous `row`, `headers`, and `processedRows()` value could remain
visible. In particular, calling `readHeaderRow()` on an empty second
input did not clear the previous headers.
Reset the parser state in `open`:
- clear `row`
- clear `headers`
- reset `currRow` to zero
Signed-off-by: cuishuang <imcusg@gmail.com>
`rotateLeft` and `rotatedLeft` raised `DivByZeroDefect` for empty
containers because their whole-container overloads computed `dist mod
arg.len` before checking for an empty input.
Handle empty inputs explicitly:
- `rotateLeft` returns `0` and leaves the container unchanged.
- `rotatedLeft` returns an empty sequence.
Add a regression test covering both overloads.
The slice overloads are intentionally left unchanged because zero-length
slice semantics need separate consideration.
Signed-off-by: cuishuang <imcusg@gmail.com>
wrapWords(splitLongWords = true) dropped the separator before a word it
had to split, fusing it with the word before it. Flush lastSep first,
the same as the branch that handles a word which fits.
Updates twordwrap's longlongwordRes fixture, which was pinned against
the old behavior at three points where a source line break normalizes to
a synthetic space.
Fixes#26173.
## What breaks today
`encodings.convert` allocates its output buffer as `newString(s.len)`,
i.e. the
same size as the input. Any conversion that expands — which is most
non-ASCII to
UTF-8 — therefore hits `E2BIG` and then **resumes** the same `iconv_t`
where it
stopped.
That is fine for stateless encodings, but the shift state of a stateful
encoding
is not guaranteed to survive a short write, and on macOS it does not.
After the
`E2BIG` the converter behaves as if it were back in the initial state,
so the
tail of an ISO-2022-JP string is emitted as raw bytes:
```nim
import std/[encodings, unicode]
proc repeatedA(n: int): string =
result = "\x1B\x24\x42"
for _ in 0 ..< n: result.add "\x24\x22" # あ
result.add "\x1B\x28\x42"
let c = open("UTF-8", "ISO-2022-JP")
for n in 1 .. 10:
echo n, " chars in -> ", c.convert(repeatedA(n)).runeLen, " chars out"
```
```
6 chars in -> 6 chars out
7 chars in -> 8 chars out <-- wrong
10 chars in -> 12 chars out <-- wrong
```
Nothing is raised. The text is just wrong, and only in the tail, so
short test
strings pass and real data does not. The failure correlates exactly with
whether
the output buffer has to grow:
| chars | input bytes | expected output bytes | growth needed | result |
|---|---|---|---|---|
| 6 | 18 | 18 | no | correct |
| 7 | 20 | 21 | **yes** | corrupted |
EUC-JP, which has the same 2-to-3 byte expansion but no shift state, is
correct
at every length — so it is the statefulness, not the growth ratio, that
matters.
The issue has a raw-C-API reproducer showing the state loss happens
inside
`iconv` and is not an artifact of the Nim string handling.
## What this PR does
Two commits, because they are two separate defects:
**1. `fixes #26173; don't resume iconv after a short write`**
Reset the converter with `iconv(c, nil, nil, nil, nil)` and redo the
whole
conversion into a larger buffer instead of resuming. Adds a regression
test that
fails on `devel` and passes with the fix.
**2. `fix out-of-bounds write in encodings.convert on a full output
buffer`**
The `EILSEQ`/`EINVAL` branch does `dst[0] = src[0]` and `dec(outLen)`
without
checking there is room, so a full output buffer writes one byte past the
end and
underflows `outLen` (a `csize_t`). Guarded with `outLen > 0`, letting
the
buffer-growth path handle the full-buffer case.
This is a latent bug independent of #26173, found while working on it —
happy to
split it into its own PR if that is preferred.
## Testing
`tests/stdlib/tencodings.nim` gains coverage for ISO-2022-JP across the
buffer
growth boundary (1..64 chars, plus a string with several ASCII/JIS state
switches) and a stateless EUC-JP case that also crosses the boundary.
- Fails on `devel` at `tencodings.nim(124)` without the fix
- Passes with the fix under both `--mm:refc` and `--mm:orc`
- Existing assertions in the file are unaffected
Verified on macOS 15 / arm64. The Windows path (`convertWin`) does not
use
`iconv` and is untouched; ISO-2022-JP is already in `nameToCodePage` as
50220, so
the new test exercises that path there too.
Fixes#25200.
`xtensa-esp-elf-gcc` (the ESP-IDF toolchain for the Xtensa
ESP32/ESP32-S2/ESP32-S3) defines `int32_t` as `long int`, exactly like
`arm-none-eabi-gcc` and `riscv32-unknown-elf-gcc` do:
```console
$ xtensa-esp32s3-elf-gcc -dM -E -x c /dev/null | grep -E '__INT32_TYPE__|__SIZEOF_(INT|LONG)__|__xtensa__|__unix__'
#define __SIZEOF_INT__ 4
#define __SIZEOF_LONG__ 4
#define __xtensa__ 1
#define __INT32_TYPE__ long int
```
So with `--cpu:esp`, `NI`/`NI32` become `long int` while `nimAddInt`
still expands to `__builtin_sadd_overflow`, which expects `int*`:
```
error: passing argument 3 of '__builtin_sadd_overflow' from incompatible pointer type [-Wincompatible-pointer-types]
note: expected 'int *' but argument is of type 'NI32 *' {aka 'long int *'}
```
This is a warning on GCC 13 and below, but [a hard error since GCC
14](https://gcc.gnu.org/gcc-14/porting_to.html#incompatible-pointer-types)
— the log in #25200 shows the build failing outright with the
`esp-15.1.0` toolchain.
This applies the same fix that #23835 (arm-none-eabi), #24553 (riscv32)
and #26121 (limiting arm to non-`__unix__`) already applied, now for
Xtensa. The `!defined(__unix__)` guard mirrors #26121: the bare-metal
ESP toolchain does not define `__unix__` (verified above), so it is
unaffected, while a hypothetical `xtensa-*-linux` target keeps the
current `int` behaviour rather than being switched blindly.
### Verification
Compiling Nim-generated C for `--cpu:esp --os:standalone` with
`xtensa-esp32s3-elf-gcc 12.2.0` and `-Werror=incompatible-pointer-types`
(to reproduce the GCC 14+ default):
| `lib/nimbase.h` | exit code | `incompatible pointer` diagnostics |
|---|---|---|
| devel | 1 | 1 |
| this PR | 0 | 0 |
No effect on any other target: the change is inside the `NIM_INTBITS ==
32` branch and is gated on `__xtensa__`.
This PR adds a new Nim compiler command and introduces some improvements
to the docgen suite in general.
1. Adds `nim book`, the new command that takes a directory with
Markdown/ReST files and generates a navigatable, searchable, Nim-first
documentation site.
2. Refactors the default nimdoc.cfg, specifically the part marked with
"needs to be refactored." Code duplication was removed, new overridable
variables were added, quirky logic with the "Group by" switch display
was fixed.
Here's a live demo of a `nim book` produced book:
https://moigagoo.github.io/nim-chronos/
The original mdBook-powered version:
https://status-im.github.io/nim-chronos/
Related to this PR but valuable on their own:
1. `.. include::` directive has received several improvements:
- You can now include code from line to line, merged:
https://github.com/nim-lang/Nim/pull/26130
- You can now include code with syntax highlighting, merged:
https://github.com/nim-lang/Nim/pull/26146
2. `.. admonition::` directive (and its derivatives like `warning`,
`error`, etc.) got new useful functions:
- You can now set a title to your admonitions, open:
https://github.com/nim-lang/Nim/pull/26159
- You can make admonitions collapsible (useful when you need to include
a large chunk if code), open: https://github.com/nim-lang/Nim/pull/26159
Fixes for a few crashes, including a runtime underflow, I stumbled upon
while trying some more _inventive_ patterns.
* `primary()`: unary `*` applied to an operand that can match empty
input (!>.*), 'a'?) now pegError()s at pattern-parse time instead of
AssertionDefect "unreachable" from `*`
* `getCharSet()`: unknown builtin/malformed escapes inside charsets
([^\n], [z-\n]) propagate `tkInvalid` instead of reading
`tok.literal[len-1]` of an empty string (IndexDefect)
* rawMatch pkCapture: `{}` with no previous capture is a no-op instead
of a runtime underflow defect
#23835 tried to do this, but it also switched to `long int` on the GNU
ABI, where it's actually just `int`.
Fixes#26111
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
Specialize `rawAlloc` for alignment (cherry-picked from the other PR).
This cuts the frame of the normal unaligned path down enough to regain
the performance lost from loading the cold page in #26110
Also cleans up `MemRegion` a bit, the regressions are either gone or
were measurement errors.
Issue popped up when using `fromJson` into an array but the JSON passed
is an object
```nim
import std/[jsonutils, json]
let data = parseJson """
{"key": "value"}
"""
var foo: seq[int]
foo.fromJson(data)
echo foo #> @[0]
```
Basically the `setLen` would set the size to be equal to the number of
keys, but `getElems` just returns an empty array if the JSON isn't an
array which lead to it just creating zero'd items in the seq without
letting the user know.
Felt adding the checks was better than just skipping the `setLen` since
it lets the user know that there is a problem with the JSON
fixes#26088
in https://github.com/nim-lang/Nim/pull/25772, `beginStores` requires
`newLen` to be passed for setting up the new length. So `streams.nim`
must make the string length exactly newLen.
Boostrap without linking is (conservatively) 17% faster on my machine.
Test setup runs the compiler compiling itself under ORC in release mode
from a clean cache and no C compilation (`--compileOnly`.)
Multiple samples are gathered before judging a potential optimization.
The test suite passes locally before push and bootstrap is tested with
strict views (mostly as a sanity check).
Tests are fair in the sense that they compile the same git worktree and
produce identical C output.
Refc will see less or no benefit.
Changes:
- Save tree traversals in `considerGenSyms` when no mappings exist
- Inline `maybeSkipDistinct` into `typeRel` for a safe cursor
- Only skip to static when there is a static type to reach in
`paramTypesMatchAux`
- Disable overflow checks for hashes
- Disable bounds checks for `nextIdentIter`
- Use cursor annotation when judged safe
- Use lent annotation for ast accessors
Spiritual companion to #26084
## Fix `globalSymbols` support on POSIX
### Summary
Fix `-d:globalSymbols` on POSIX platforms by defining `RTLD_GLOBAL`
alongside `RTLD_NOW` in `system/dyncalls.nim`.
On Linux and macOS, `RTLD_NOW` is defined locally in `dyncalls.nim`, but
`RTLD_GLOBAL` was not. As a result, enabling `-d:globalSymbols` could
fail because `RTLD_GLOBAL` was undeclared.
This change:
* defines `RTLD_GLOBAL` as `0x100` on Linux,
* defines `RTLD_GLOBAL` as `0x8` on macOS,
* imports `RTLD_GLOBAL` from `<dlfcn.h>` on other POSIX platforms.
These values are consistent with the existing POSIX constants already
used elsewhere in the Nim source tree.
### Motivation
`globalSymbols` is intended to load dynamic libraries with `RTLD_GLOBAL`
so that their exported symbols are available to subsequently loaded
shared libraries.
This is needed, for example, when a dynamically loaded library later
loads a plugin or provider that depends on symbols from the first
library.
Without this fix, `-d:globalSymbols` cannot be used reliably for that
purpose on POSIX systems.
### Testing
Tested on Linux with an AArch64 target.
A program using dynamically loaded OpenSSL libraries and a subsequently
loaded OpenSSL provider failed when the OpenSSL libraries were loaded
with the default local symbol visibility.
Using `RTLD_GLOBAL` made the same program work correctly.
After this change, building the original Nim program with:
```text
-d:globalSymbols
```
successfully loads the OpenSSL libraries with global symbol visibility,
and the provider-based TLS 1.2 and TLS 1.3 tests both pass.
The same behavior was also independently reproduced using direct
`dlopen()` / `dlsym()` calls:
```text
RTLD_LOCAL -> TLS 1.2 failed
RTLD_GLOBAL -> TLS 1.2 passed
```
Signed-off-by: Takeyoshi Kikuchi <kikuchi@centurysys.co.jp>
`nimDecRefIsLast` always performed an atomic decrement. When the biased
count is already zero the destroying thread holds the only reference, so
there is nothing to adjudicate and the read-modify-write can be skipped.
Soundness: a counted reference can only be derived from the location
being destroyed -- which happens-before this destructor unless the
program races on that location -- or from another counted reference,
whose contribution is already in `rc` and therefore forces the slow
path. Observing zero proves no other thread holds a reference and that
none can appear. This relies on `--mm:atomicArc` having no collector;
ORC and YRC mutate `rc` from a participant that holds no counted
reference at all, so the fast path is deliberately not enabled for them.
The slow path keeps deciding on the value its own RMW returned. That is
what separates this from nim-lang/threading#45, where the "who frees"
role was decided from a separate load and the RMW result was discarded,
so the role could be dropped by every participant at once.
gcbench, -d:danger, median of 21 pinned runs:
--mm:arc (non-atomic RC) 0.1310
--mm:atomicArc 0.1742
--mm:atomicArc + this 0.1330
-23.7%, closing 95% of the gap to non-atomic reference counting. gcbench
builds its trees with `sink` parameters, so it performs almost no
incRefs and the whole atomicArc penalty is decRef traffic. The worst
case -- a decrement that always sees rc > 0, so the load never pays off
-- measures +1.1%.
`-d:nimNoAtomicArcFastPath` restores the previous code path.
`strbasics.add` uses `copymem` when available. CT conditional
scaffolding mirrors the same from system.
Follow-up to #15951
Compile-time test for `strbasics.add` undiscarded and passes, though
pending bug #15952 is still open.
Fixes#26027.
Map the single-order compare-exchange failure ordering from `release` to
`relaxed` and from `acquire-release` to `acquire`. Apply the mapping to
the trivial and non-trivial strong and weak overloads, and correct the
explicit-order test cases.
Tested `tests/stdlib/concurrency/tatomics.nim` across C/C++, refc/orc,
and native/C++ atomics (8 combinations). Also verified the original GCC
16.1 assertion reproducer.
`toStrLit()` uses `repr()` internally, which forwards quotes and messes
with dashes in the output. Let's use `newStrLitNode()` directly instead.
GitHub: fixes https://github.com/nim-lang/Nim/issues/26039
Fix proposed by GPT 5.6 Sol.
close#26016
Potentially close#22510
No concrete proof for the second one, though the described behavior
matches and the step count explains why it's so difficult to find a
repro.
These functions are unused and never exposed publically - along with it,
get rid of `GC_Strategy` - although it's possible someone could use this
`enum` for their own code it seems unlikely.
I was updating nim to 2.2.10 in nixpkgs
https://github.com/NixOS/nixpkgs/pull/538469
and without this change the fail build, because add(string, cstring) is
not defined. I think it's caused by a change of position of the includes
in system, but with this it works fine.
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
The emscripten branch cast the descriptor address to the value type
EmscriptenMMapBlock instead of the pointer alias PEmscriptenMMapBlock,
so osAllocPages stored realSize/realPointer in a discarded local and
osDeallocPages reinterpreted the address integer as the descriptor
instead of dereferencing it -- calling munmap() with garbage that fails,
so freed pages are never returned. Freed huge chunks are also dropped
from the free list, leaking permanently. Affects wasm32 and wasm64.
Cast to PEmscriptenMMapBlock so both accesses go through memory.
This PR implements `expandSymlink` on Windows with POSIX readlink
semantics: it expands exactly one hop and returns the stored link target
without resolving the full chain.
The main design question was whether Windows symlink expansion should be
built on path-finalization APIs such as `GetFinalPathNameByHandleW`, or
on direct reparse-point inspection. Current `expandSymlink` is a
single-hop "what target is stored in this link object?" operation and
most of other ways to resolve symlinks on Windows actually try to answer
the "final true file location" question in various slightly-incompatible
ways.
The full final-path resolution on Windows is substantially more complex
than readlink and is planned as a follow-up.
## Implementation choice
Implements Windows `expandSymlink` by:
- opening the path with `FILE_FLAG_OPEN_REPARSE_POINT`
- calling `DeviceIoControl(FSCTL_GET_REPARSE_POINT)`
- parsing the reparse payload for `IO_REPARSE_TAG_SYMLINK` and
`IO_REPARSE_TAG_MOUNT_POINT`
- decoding the UTF-16 slice referenced by the payload
- returning the stored target
This is the right primitive for the API:
- does not depend on whole-path finalization
- works for both symlinks and junctions
- matches the existing Linux behaviour
`widestrs` changes allow using WideCString views without temporary
allocations.
Windows prohibits symlink creation without admin rights, so,
unfortunately, the tests are conditionally skipped by default. Manually
running `testament` in an admin console is required.
## Behaviour:
- One hop only
- Relative symlink targets are returned unchanged
- Absolute Windows targets are converted from stored NT-style prefixes
to usable Win32 forms when applicable
- Non-links, malformed payloads, and unsupported reparse tags raise
`OSError`
## Future work
Path canonicalization, i.e. "final true file location". Which is, BTW,
different from `absolutePath`, which works on paths only and doesn't hit
the underlying FS. So this needs to be an API extension.
I'd like to follow-up with this when I sort through the docs, for now
you can resolve symlinks in a loop.
---------
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
Companion to readRawData whose pointer stays valid across moves/copies
of the string. Under --strings:sso it promotes a small inline string to
its heap representation; under refc/v2 the data is already heap-resident
so it aliases readRawData.
Uniform `var string` signature on every backend so code can prepare for
--strings:sso without `when declared`.
This pull request allows setting `--cpu:wasm64`, allowing wasm64 as a
first class target. This avoids having to set `-cpu:riscv64` as a
workaround. Sane defaults for the emscripten toolchain are also
provided.
This PR adds 3 modes to `typeof` to specify how to handle type modifiers
`var`, `sink` and `lent`.
- typeOfModCompatible
Remove or keep type modifiers in the same way as old typeof. That means
keep `sink` but remove `var` and `lent`.
- typeOfModRemoveModifier
Remove type modifiers.
- typeOfModKeepModifier
Keep type modifiers.
Related to https://github.com/nim-lang/Nim/pull/25779https://github.com/nim-lang/Nim/issues/25786
## Summary
Fixes#19782.
The `?` operator in `std/uri` was silently overwriting any query string
already present in the URI. This PR makes it append instead — which
matches the docstring ("Concatenates the query parameters") and the
natural expectation when chaining operations.
**Before:**
```nim
let u = parseUri("https://example.com/foo?existing=1") ? {"bar": "qux"}
echo $u # https://example.com/foo?bar=qux (existing=1 lost)
```
**After:**
```nim
let u = parseUri("https://example.com/foo?existing=1") ? {"bar": "qux"}
echo $u # https://example.com/foo?existing=1&bar=qux
```
## Changes
- `lib/pure/uri.nim`: fix `?` to append with `&` when a query string
already exists; add example to `runnableExamples`
- `tests/stdlib/turi.nim`: two new test cases (append to existing query,
empty params preserve existing)
- `changelog.md`: entry under Standard library changes
## Notes
I work with Claude as a co-processor. I'm 56, came to programming late,
and this is genuinely how I learn and contribute. I understand what I'm
submitting, but I didn't write it alone. If your project prefers
human-only contributions, just say so and I'll close without friction.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: n0madgang <14005836+n0madgang@users.noreply.github.com>
