mirror of
https://github.com/nim-lang/Nim.git
synced 2026-09-17 10:34:53 +00:00
39ee604025
Fixes #26173. ## What breaks today `encodings.convert` allocates its output buffer as `newString(s.len)`, i.e. the same size as the input. Any conversion that expands — which is most non-ASCII to UTF-8 — therefore hits `E2BIG` and then **resumes** the same `iconv_t` where it stopped. That is fine for stateless encodings, but the shift state of a stateful encoding is not guaranteed to survive a short write, and on macOS it does not. After the `E2BIG` the converter behaves as if it were back in the initial state, so the tail of an ISO-2022-JP string is emitted as raw bytes: ```nim import std/[encodings, unicode] proc repeatedA(n: int): string = result = "\x1B\x24\x42" for _ in 0 ..< n: result.add "\x24\x22" # あ result.add "\x1B\x28\x42" let c = open("UTF-8", "ISO-2022-JP") for n in 1 .. 10: echo n, " chars in -> ", c.convert(repeatedA(n)).runeLen, " chars out" ``` ``` 6 chars in -> 6 chars out 7 chars in -> 8 chars out <-- wrong 10 chars in -> 12 chars out <-- wrong ``` Nothing is raised. The text is just wrong, and only in the tail, so short test strings pass and real data does not. The failure correlates exactly with whether the output buffer has to grow: | chars | input bytes | expected output bytes | growth needed | result | |---|---|---|---|---| | 6 | 18 | 18 | no | correct | | 7 | 20 | 21 | **yes** | corrupted | EUC-JP, which has the same 2-to-3 byte expansion but no shift state, is correct at every length — so it is the statefulness, not the growth ratio, that matters. The issue has a raw-C-API reproducer showing the state loss happens inside `iconv` and is not an artifact of the Nim string handling. ## What this PR does Two commits, because they are two separate defects: **1. `fixes #26173; don't resume iconv after a short write`** Reset the converter with `iconv(c, nil, nil, nil, nil)` and redo the whole conversion into a larger buffer instead of resuming. Adds a regression test that fails on `devel` and passes with the fix. **2. `fix out-of-bounds write in encodings.convert on a full output buffer`** The `EILSEQ`/`EINVAL` branch does `dst[0] = src[0]` and `dec(outLen)` without checking there is room, so a full output buffer writes one byte past the end and underflows `outLen` (a `csize_t`). Guarded with `outLen > 0`, letting the buffer-growth path handle the full-buffer case. This is a latent bug independent of #26173, found while working on it — happy to split it into its own PR if that is preferred. ## Testing `tests/stdlib/tencodings.nim` gains coverage for ISO-2022-JP across the buffer growth boundary (1..64 chars, plus a string with several ASCII/JIS state switches) and a stateless EUC-JP case that also crosses the boundary. - Fails on `devel` at `tencodings.nim(124)` without the fix - Passes with the fix under both `--mm:refc` and `--mm:orc` - Existing assertions in the file are unaffected Verified on macOS 15 / arm64. The Windows path (`convertWin`) does not use `iconv` and is untouched; ISO-2022-JP is already in `nameToCodePage` as 50220, so the new test exercises that path there too.
This directory contains the test cases.
Each test must have a filename of the form: t*.nim
Note: Testament is only aware of tests under a directory (eg tests/foo/) and will ignore
top-level tests like tests/tbar.nim.
Specs
Each test can contain a spec in a discard """ ... """ block.
Check out the parseSpec procedure in the specs module for a full and reliable reference
action
Specifies what action this test should take.
Default: run
Options:
compile- compiles the module and fails the test if compilations fails.run- compiles and runs the module, fails the test if compilation or execution of test code fails.reject- compiles the module and fails the test if compilation succeeds.
There are certain spec keys that imply run, including output and
outputsub.
Categories
Each folder under this directory represents a test category, which can be
tested by running koch tests pcat <category> (or cat to avoid parallel
testing, which is slower).
The folder dll contains simple DLL tests.
The folder realtimeGC contains a test for validating that the realtime GC
can run properly without linking against the nimrtl.dll/so.