`CsvParser.open` did not reset parser-specific state when reopening a
parser.
After parsing one input and reusing the same parser for another input,
the previous `row`, `headers`, and `processedRows()` value could remain
visible. In particular, calling `readHeaderRow()` on an empty second
input did not clear the previous headers.
Reset the parser state in `open`:
- clear `row`
- clear `headers`
- reset `currRow` to zero
Signed-off-by: cuishuang <imcusg@gmail.com>
`rotateLeft` and `rotatedLeft` raised `DivByZeroDefect` for empty
containers because their whole-container overloads computed `dist mod
arg.len` before checking for an empty input.
Handle empty inputs explicitly:
- `rotateLeft` returns `0` and leaves the container unchanged.
- `rotatedLeft` returns an empty sequence.
Add a regression test covering both overloads.
The slice overloads are intentionally left unchanged because zero-length
slice semantics need separate consideration.
Signed-off-by: cuishuang <imcusg@gmail.com>
wrapWords(splitLongWords = true) dropped the separator before a word it
had to split, fusing it with the word before it. Flush lastSep first,
the same as the branch that handles a word which fits.
Updates twordwrap's longlongwordRes fixture, which was pinned against
the old behavior at three points where a source line break normalizes to
a synthetic space.
Fixes#26173.
## What breaks today
`encodings.convert` allocates its output buffer as `newString(s.len)`,
i.e. the
same size as the input. Any conversion that expands — which is most
non-ASCII to
UTF-8 — therefore hits `E2BIG` and then **resumes** the same `iconv_t`
where it
stopped.
That is fine for stateless encodings, but the shift state of a stateful
encoding
is not guaranteed to survive a short write, and on macOS it does not.
After the
`E2BIG` the converter behaves as if it were back in the initial state,
so the
tail of an ISO-2022-JP string is emitted as raw bytes:
```nim
import std/[encodings, unicode]
proc repeatedA(n: int): string =
result = "\x1B\x24\x42"
for _ in 0 ..< n: result.add "\x24\x22" # あ
result.add "\x1B\x28\x42"
let c = open("UTF-8", "ISO-2022-JP")
for n in 1 .. 10:
echo n, " chars in -> ", c.convert(repeatedA(n)).runeLen, " chars out"
```
```
6 chars in -> 6 chars out
7 chars in -> 8 chars out <-- wrong
10 chars in -> 12 chars out <-- wrong
```
Nothing is raised. The text is just wrong, and only in the tail, so
short test
strings pass and real data does not. The failure correlates exactly with
whether
the output buffer has to grow:
| chars | input bytes | expected output bytes | growth needed | result |
|---|---|---|---|---|
| 6 | 18 | 18 | no | correct |
| 7 | 20 | 21 | **yes** | corrupted |
EUC-JP, which has the same 2-to-3 byte expansion but no shift state, is
correct
at every length — so it is the statefulness, not the growth ratio, that
matters.
The issue has a raw-C-API reproducer showing the state loss happens
inside
`iconv` and is not an artifact of the Nim string handling.
## What this PR does
Two commits, because they are two separate defects:
**1. `fixes #26173; don't resume iconv after a short write`**
Reset the converter with `iconv(c, nil, nil, nil, nil)` and redo the
whole
conversion into a larger buffer instead of resuming. Adds a regression
test that
fails on `devel` and passes with the fix.
**2. `fix out-of-bounds write in encodings.convert on a full output
buffer`**
The `EILSEQ`/`EINVAL` branch does `dst[0] = src[0]` and `dec(outLen)`
without
checking there is room, so a full output buffer writes one byte past the
end and
underflows `outLen` (a `csize_t`). Guarded with `outLen > 0`, letting
the
buffer-growth path handle the full-buffer case.
This is a latent bug independent of #26173, found while working on it —
happy to
split it into its own PR if that is preferred.
## Testing
`tests/stdlib/tencodings.nim` gains coverage for ISO-2022-JP across the
buffer
growth boundary (1..64 chars, plus a string with several ASCII/JIS state
switches) and a stateless EUC-JP case that also crosses the boundary.
- Fails on `devel` at `tencodings.nim(124)` without the fix
- Passes with the fix under both `--mm:refc` and `--mm:orc`
- Existing assertions in the file are unaffected
Verified on macOS 15 / arm64. The Windows path (`convertWin`) does not
use
`iconv` and is untouched; ISO-2022-JP is already in `nameToCodePage` as
50220, so
the new test exercises that path there too.
This PR adds a new Nim compiler command and introduces some improvements
to the docgen suite in general.
1. Adds `nim book`, the new command that takes a directory with
Markdown/ReST files and generates a navigatable, searchable, Nim-first
documentation site.
2. Refactors the default nimdoc.cfg, specifically the part marked with
"needs to be refactored." Code duplication was removed, new overridable
variables were added, quirky logic with the "Group by" switch display
was fixed.
Here's a live demo of a `nim book` produced book:
https://moigagoo.github.io/nim-chronos/
The original mdBook-powered version:
https://status-im.github.io/nim-chronos/
Related to this PR but valuable on their own:
1. `.. include::` directive has received several improvements:
- You can now include code from line to line, merged:
https://github.com/nim-lang/Nim/pull/26130
- You can now include code with syntax highlighting, merged:
https://github.com/nim-lang/Nim/pull/26146
2. `.. admonition::` directive (and its derivatives like `warning`,
`error`, etc.) got new useful functions:
- You can now set a title to your admonitions, open:
https://github.com/nim-lang/Nim/pull/26159
- You can make admonitions collapsible (useful when you need to include
a large chunk if code), open: https://github.com/nim-lang/Nim/pull/26159
Fixes for a few crashes, including a runtime underflow, I stumbled upon
while trying some more _inventive_ patterns.
* `primary()`: unary `*` applied to an operand that can match empty
input (!>.*), 'a'?) now pegError()s at pattern-parse time instead of
AssertionDefect "unreachable" from `*`
* `getCharSet()`: unknown builtin/malformed escapes inside charsets
([^\n], [z-\n]) propagate `tkInvalid` instead of reading
`tok.literal[len-1]` of an empty string (IndexDefect)
* rawMatch pkCapture: `{}` with no previous capture is a no-op instead
of a runtime underflow defect
Issue popped up when using `fromJson` into an array but the JSON passed
is an object
```nim
import std/[jsonutils, json]
let data = parseJson """
{"key": "value"}
"""
var foo: seq[int]
foo.fromJson(data)
echo foo #> @[0]
```
Basically the `setLen` would set the size to be equal to the number of
keys, but `getElems` just returns an empty array if the JSON isn't an
array which lead to it just creating zero'd items in the seq without
letting the user know.
Felt adding the checks was better than just skipping the `setLen` since
it lets the user know that there is a problem with the JSON
fixes#26088
in https://github.com/nim-lang/Nim/pull/25772, `beginStores` requires
`newLen` to be passed for setting up the new length. So `streams.nim`
must make the string length exactly newLen.
`strbasics.add` uses `copymem` when available. CT conditional
scaffolding mirrors the same from system.
Follow-up to #15951
Compile-time test for `strbasics.add` undiscarded and passes, though
pending bug #15952 is still open.
Fixes#26027.
Map the single-order compare-exchange failure ordering from `release` to
`relaxed` and from `acquire-release` to `acquire`. Apply the mapping to
the trivial and non-trivial strong and weak overloads, and correct the
explicit-order test cases.
Tested `tests/stdlib/concurrency/tatomics.nim` across C/C++, refc/orc,
and native/C++ atomics (8 combinations). Also verified the original GCC
16.1 assertion reproducer.
`toStrLit()` uses `repr()` internally, which forwards quotes and messes
with dashes in the output. Let's use `newStrLitNode()` directly instead.
GitHub: fixes https://github.com/nim-lang/Nim/issues/26039
This PR implements `expandSymlink` on Windows with POSIX readlink
semantics: it expands exactly one hop and returns the stored link target
without resolving the full chain.
The main design question was whether Windows symlink expansion should be
built on path-finalization APIs such as `GetFinalPathNameByHandleW`, or
on direct reparse-point inspection. Current `expandSymlink` is a
single-hop "what target is stored in this link object?" operation and
most of other ways to resolve symlinks on Windows actually try to answer
the "final true file location" question in various slightly-incompatible
ways.
The full final-path resolution on Windows is substantially more complex
than readlink and is planned as a follow-up.
## Implementation choice
Implements Windows `expandSymlink` by:
- opening the path with `FILE_FLAG_OPEN_REPARSE_POINT`
- calling `DeviceIoControl(FSCTL_GET_REPARSE_POINT)`
- parsing the reparse payload for `IO_REPARSE_TAG_SYMLINK` and
`IO_REPARSE_TAG_MOUNT_POINT`
- decoding the UTF-16 slice referenced by the payload
- returning the stored target
This is the right primitive for the API:
- does not depend on whole-path finalization
- works for both symlinks and junctions
- matches the existing Linux behaviour
`widestrs` changes allow using WideCString views without temporary
allocations.
Windows prohibits symlink creation without admin rights, so,
unfortunately, the tests are conditionally skipped by default. Manually
running `testament` in an admin console is required.
## Behaviour:
- One hop only
- Relative symlink targets are returned unchanged
- Absolute Windows targets are converted from stored NT-style prefixes
to usable Win32 forms when applicable
- Non-links, malformed payloads, and unsupported reparse tags raise
`OSError`
## Future work
Path canonicalization, i.e. "final true file location". Which is, BTW,
different from `absolutePath`, which works on paths only and doesn't hit
the underlying FS. So this needs to be an API extension.
I'd like to follow-up with this when I sort through the docs, for now
you can resolve symlinks in a loop.
---------
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
## Summary
Fixes#19782.
The `?` operator in `std/uri` was silently overwriting any query string
already present in the URI. This PR makes it append instead — which
matches the docstring ("Concatenates the query parameters") and the
natural expectation when chaining operations.
**Before:**
```nim
let u = parseUri("https://example.com/foo?existing=1") ? {"bar": "qux"}
echo $u # https://example.com/foo?bar=qux (existing=1 lost)
```
**After:**
```nim
let u = parseUri("https://example.com/foo?existing=1") ? {"bar": "qux"}
echo $u # https://example.com/foo?existing=1&bar=qux
```
## Changes
- `lib/pure/uri.nim`: fix `?` to append with `&` when a query string
already exists; add example to `runnableExamples`
- `tests/stdlib/turi.nim`: two new test cases (append to existing query,
empty params preserve existing)
- `changelog.md`: entry under Standard library changes
## Notes
I work with Claude as a co-processor. I'm 56, came to programming late,
and this is genuinely how I learn and contribute. I understand what I'm
submitting, but I didn't write it alone. If your project prefers
human-only contributions, just say so and I'll close without friction.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: n0madgang <14005836+n0madgang@users.noreply.github.com>
Fixes#18583.
## Problem
Several stdlib collection types compute the separator for `$` using
`result.len > 1`, where `result` starts as the opening bracket (`"["` or
`"{"`). This breaks when a collection element type has a `$` that
returns an empty string: `result.len` stays at 1 after the first item
contributes nothing, so the separator is never inserted for subsequent
items.
```nim
import std/deques
type Test = object
proc `$`(x: Test): string = ""
echo [Test(), Test()].toDeque # prints [] — expected [, ]
```
## Fix
Replace the length check with an explicit `first` flag in all affected
modules: `deques`, `heapqueue`, `lists`, `critbits`, and `strtabs`.
## Tests
Regression tests added to `tdeques`, `theapqueue`, and `tlists` using a
local type whose `$` returns `""`. All three test files pass with `nim c
-r`.
## Notes
I work with Claude as a co-processor. I'm 56, came to programming late,
and this is genuinely how I learn and contribute. I understand what I'm
submitting, but I didn't write it alone. If your project prefers
human-only contributions, just say so and I'll close without friction.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Fixes `std/pegs` lexing for bare UTF-8 terminals such as `\i café`.
- The lexer previously stopped at the first non-ASCII byte, so
`pkTerminalIgnoreCase` never saw the full term despite its rune-aware
`fastRuneAt`/`toLower` matching.
- This now keeps non-ASCII bytes in identifier-style terminals while
ASCII non-ident characters still terminate the symbol.
## Behavior
Before: `match("CAFÉ", peg"\i café")` failed because the terminal was
lexed as `caf`.
After: `match("CAFÉ", peg"\i café")`, `match("Café", peg"\i café")`, and
`findAll` over mixed-case occurrences pass.
`std/pegs` documents `useUnicode = true` as proper UTF-8 support, and
quoted terminals already preserved the same bytes; this makes bare
terminals consistent with that path.
I did not find an existing relevant issue or PR in searches for
pegs/unicode/utf8/getSymbol/pkTerminalIgnoreCase.
fixes#25821
This pull request includes a minor bug fix in the lexer and adds new
test cases for string formatting with binary operators in interpolated
expressions.
Lexer bug fix:
* Fixed an off-by-one error in the unary minus detection logic in the
`rawGetTok` procedure in `lexer.nim`, ensuring that the start-of-buffer
condition is correctly checked.
Testing improvements:
* Added tests to `tstrformat.nim` to verify that binary operators (such
as subtraction) work correctly inside interpolated string expressions
using both `&` and `fmt`.
`setLenUninit(string)` was broken on the legacy refc backend when
growing within existing spare capacity.
`setLengthStrUninit` in `lib/system/sysstr.nim` only updated len when it
had to reallocate or when shrinking.
If oldLen < newLen <= capacity, it returned early without finalizing:
```nim
var s = newStringOfCap(10)
s.add("abc")
s.setLenUninit(6)
doAssert s.len == 6 # used to fail, len stayed 3
```
This escaped `tests/stdlib/tstring.nim` because the testing routine
`checkSetLenUninit` mostly resizes strings created at **exact**
length/capacity, so growth usually took the reallocating branch.
The new regression test covers the missing edge case.
So sorry for catching this only on the day of the stable release! In my
defense, the original PR hung in limbo for quite a while and it didn't
spend enough time in devel after the merge.
Adds `system.setLenUninit` for the `string` type. Allows setting length
without initializing new memory on growth.
- Required for a follow-up to #15951
- Accompanies #22767 (ref #19727) but for strings
- Expands `stdlib/tstring` with tests for `setLen` and `setLenUninit`
---------
Co-authored-by: Andreas Rumpf <araq4k@proton.me>
Fixes bug #25674.
`replace` read `s[i+1]` for a CRLF pair without ensuring `i+1 <
s.len()`, so a value ending in a lone `\\c` (quoted in `writeConfig`)
raised `IndexDefect`.
- Fix: only treat `\\c\\l` when the following character exists.
- Test: `tests/stdlib/tparsecfg.nim` block bug #25674 — fails before
fix, passes after.
1. A trailing `$` at the end of a replacement string could read out of
bounds via `how[i + 1]`; this now raises `ValueError` instead.
2. Numeric capture parsing used `id += (id * 10) + digit` instead of `id
= (id * 10) + digit`, so multi-digit refs were parsed incorrectly (e.g.
`$12` resolved as capture 13 instead of 12).
4. Unterminated named replacement syntax (e.g. `${foo)` is now rejected
with ValueError instead of being accepted and parsed inconsistently.
Found and fixed by GPT 5.3 Codex.
fixes#25205fixes#14873
```nim
type
SysLockObj {.importc: "pthread_mutex_t", pure, final,
header: """#include <sys/types.h>
#include <pthread.h>""", byref.} = object
when defined(linux) and defined(amd64):
abi: array[40 div sizeof(clong), clong]
```
Before this PR, in refc, `resetLoc` generates field assignments for each
fields of `importc` object. But the field `abi` is not a genuine field,
which doesn't exits in the struct. We could use `zeroMem` to reset the
memory if not leave it alone
- Added support for SOCKS5h (h for proxy-side DNS resolving) to
httpclient
- Deprecated `auth` arguments for `newProxy` constructors, for auth to
be embedded in the url.
Unfortunately `http://example.com` is not currently reachable from
github CI, so the tests fail there for a few days already, I'm not sure
what can be done here.
fixes#7179
```nim
var f = 751.0
echo f.int8
```
In this case, `int8(float)` yields different numbers for different
optimization levels, since float to int conversions are undefined
behaviors. In this PR, it mitigates this problem by conversions to same
size integers before converting to the final type: i.e.
`int8(int64(float))`, which has UB problems but is better than before
fixes#25007
```nim
proc setLengthSeqUninit(s: PGenericSeq, typ: PNimType, newLen: int, isTrivial: bool): PGenericSeq {.
compilerRtl.} =
```
In this added function, only the line `zeroMem(dataPointer(result,
elemAlign, elemSize, newLen), (result.len-%newLen) *% elemSize)` is
removed from `proc setLengthSeqV2` when enlarging a sequence.
JS and VM versions simply use `setLen`.
Multiple replacements based on character sets in a single pass. Useful
for string sanitation. Follows existing `multiReplace` semantics.
Note: initially copied the substring version logic with a `while` and a
named block break, but Godbolt showed it had produced slightly larger
assembly using higher registers than the final version.
- [x] Tests
- [x] changelog.md
Without this fix, trying to use `scanTuple` in a generic proc imported
from a different module fails to compile (`undeclared identifier:
'scanf'`):
```nim
# module.nim
import std/strscans
proc scan*[T](s: string): (bool, string) =
s.scanTuple("$+")
```
```nim
# main.nim
import ./module
echo scan[int]("foo")
```
Workaround is to `export scanf` in `module.nim` or `import std/strscans`
in `main.nim`.
fixes#24673
The problem is that there is no way to distinguish `cint`, `cint`, etc
ctypes with Nim types. So `when T is cint | clong | clonglong:` is true
for types derived from `int`, `int32` and `int64`. In this PR, it fixes
the branch to avoid erros for `Natural`
