mirror of
https://github.com/nim-lang/Nim.git
synced 2026-09-17 02:32:11 +00:00
39ee604025
Fixes #26173. ## What breaks today `encodings.convert` allocates its output buffer as `newString(s.len)`, i.e. the same size as the input. Any conversion that expands — which is most non-ASCII to UTF-8 — therefore hits `E2BIG` and then **resumes** the same `iconv_t` where it stopped. That is fine for stateless encodings, but the shift state of a stateful encoding is not guaranteed to survive a short write, and on macOS it does not. After the `E2BIG` the converter behaves as if it were back in the initial state, so the tail of an ISO-2022-JP string is emitted as raw bytes: ```nim import std/[encodings, unicode] proc repeatedA(n: int): string = result = "\x1B\x24\x42" for _ in 0 ..< n: result.add "\x24\x22" # あ result.add "\x1B\x28\x42" let c = open("UTF-8", "ISO-2022-JP") for n in 1 .. 10: echo n, " chars in -> ", c.convert(repeatedA(n)).runeLen, " chars out" ``` ``` 6 chars in -> 6 chars out 7 chars in -> 8 chars out <-- wrong 10 chars in -> 12 chars out <-- wrong ``` Nothing is raised. The text is just wrong, and only in the tail, so short test strings pass and real data does not. The failure correlates exactly with whether the output buffer has to grow: | chars | input bytes | expected output bytes | growth needed | result | |---|---|---|---|---| | 6 | 18 | 18 | no | correct | | 7 | 20 | 21 | **yes** | corrupted | EUC-JP, which has the same 2-to-3 byte expansion but no shift state, is correct at every length — so it is the statefulness, not the growth ratio, that matters. The issue has a raw-C-API reproducer showing the state loss happens inside `iconv` and is not an artifact of the Nim string handling. ## What this PR does Two commits, because they are two separate defects: **1. `fixes #26173; don't resume iconv after a short write`** Reset the converter with `iconv(c, nil, nil, nil, nil)` and redo the whole conversion into a larger buffer instead of resuming. Adds a regression test that fails on `devel` and passes with the fix. **2. `fix out-of-bounds write in encodings.convert on a full output buffer`** The `EILSEQ`/`EINVAL` branch does `dst[0] = src[0]` and `dec(outLen)` without checking there is room, so a full output buffer writes one byte past the end and underflows `outLen` (a `csize_t`). Guarded with `outLen > 0`, letting the buffer-growth path handle the full-buffer case. This is a latent bug independent of #26173, found while working on it — happy to split it into its own PR if that is preferred. ## Testing `tests/stdlib/tencodings.nim` gains coverage for ISO-2022-JP across the buffer growth boundary (1..64 chars, plus a string with several ASCII/JIS state switches) and a stateless EUC-JP case that also crosses the boundary. - Fails on `devel` at `tencodings.nim(124)` without the fix - Passes with the fix under both `--mm:refc` and `--mm:orc` - Existing assertions in the file are unaffected Verified on macOS 15 / arm64. The Windows path (`convertWin`) does not use `iconv` and is untouched; ISO-2022-JP is already in `nameToCodePage` as 50220, so the new test exercises that path there too.