39ee604025
Fixes #26173. ## What breaks today `encodings.convert` allocates its output buffer as `newString(s.len)`, i.e. the same size as the input. Any conversion that expands — which is most non-ASCII to UTF-8 — therefore hits `E2BIG` and then **resumes** the same `iconv_t` where it stopped. That is fine for stateless encodings, but the shift state of a stateful encoding is not guaranteed to survive a short write, and on macOS it does not. After the `E2BIG` the converter behaves as if it were back in the initial state, so the tail of an ISO-2022-JP string is emitted as raw bytes: ```nim import std/[encodings, unicode] proc repeatedA(n: int): string = result = "\x1B\x24\x42" for _ in 0 ..< n: result.add "\x24\x22" # あ result.add "\x1B\x28\x42" let c = open("UTF-8", "ISO-2022-JP") for n in 1 .. 10: echo n, " chars in -> ", c.convert(repeatedA(n)).runeLen, " chars out" ``` ``` 6 chars in -> 6 chars out 7 chars in -> 8 chars out <-- wrong 10 chars in -> 12 chars out <-- wrong ``` Nothing is raised. The text is just wrong, and only in the tail, so short test strings pass and real data does not. The failure correlates exactly with whether the output buffer has to grow: | chars | input bytes | expected output bytes | growth needed | result | |---|---|---|---|---| | 6 | 18 | 18 | no | correct | | 7 | 20 | 21 | **yes** | corrupted | EUC-JP, which has the same 2-to-3 byte expansion but no shift state, is correct at every length — so it is the statefulness, not the growth ratio, that matters. The issue has a raw-C-API reproducer showing the state loss happens inside `iconv` and is not an artifact of the Nim string handling. ## What this PR does Two commits, because they are two separate defects: **1. `fixes #26173; don't resume iconv after a short write`** Reset the converter with `iconv(c, nil, nil, nil, nil)` and redo the whole conversion into a larger buffer instead of resuming. Adds a regression test that fails on `devel` and passes with the fix. **2. `fix out-of-bounds write in encodings.convert on a full output buffer`** The `EILSEQ`/`EINVAL` branch does `dst[0] = src[0]` and `dec(outLen)` without checking there is room, so a full output buffer writes one byte past the end and underflows `outLen` (a `csize_t`). Guarded with `outLen > 0`, letting the buffer-growth path handle the full-buffer case. This is a latent bug independent of #26173, found while working on it — happy to split it into its own PR if that is preferred. ## Testing `tests/stdlib/tencodings.nim` gains coverage for ISO-2022-JP across the buffer growth boundary (1..64 chars, plus a string with several ASCII/JIS state switches) and a stateless EUC-JP case that also crosses the boundary. - Fails on `devel` at `tencodings.nim(124)` without the fix - Passes with the fix under both `--mm:refc` and `--mm:orc` - Existing assertions in the file are unaffected Verified on macOS 15 / arm64. The Windows path (`convertWin`) does not use `iconv` and is untouched; ISO-2022-JP is already in `nameToCodePage` as 50220, so the new test exercises that path there too.
Nim
This repository contains the Nim compiler, Nim's stdlib, tools, and documentation. For more information about Nim, including downloads and documentation for the latest release, check out Nim's website or bleeding edge docs.
Community
- The forum - the best place to ask questions and to discuss Nim.
- #nim IRC Channel (Libera Chat) - a place to discuss Nim in real-time. Also where most development decisions get made.
- Discord - an additional place to discuss Nim in real-time. Most channels there are bridged to IRC.
- Gitter - an additional place to discuss Nim in real-time. There is a bridge between Gitter and the IRC channel.
- Matrix - the main room to discuss Nim in real-time. Matrix space contains a list of rooms, most of them are bridged to IRC.
- Telegram - an additional place to discuss Nim in real-time. There is the official Telegram channel. Not bridged to IRC.
- Stack Overflow - a popular Q/A site for programming related topics that includes posts about Nim.
- GitHub Wiki - Misc user-contributed content.
Compiling
The compiler currently officially supports the following platform and architecture combinations:
| Operating System | Architectures Supported |
|---|---|
| Windows (Windows XP or greater) | x86 and x86_64 |
| Linux (most distributions) | x86, x86_64, ppc64, and armv6l |
| Mac OS X (10.4 or greater) | x86, x86_64, ppc64, and Apple Silicon (ARM64) |
More platforms are supported, however, they are not tested regularly and they may not be as stable as the above-listed platforms.
Compiling the Nim compiler is quite straightforward if you follow these steps:
First, the C source of an older version of the Nim compiler is needed to
bootstrap the latest version because the Nim compiler itself is written in the
Nim programming language. Those C sources are available within the
nim-lang/csources_v3 repository.
Next, to build from source you will need:
- A C compiler such as
gcc6.x/later or an alternative such asclang,Visual C++orIntel C++. It is recommended to usegcc6.x or later. - Either
gitorwgetto download the needed source repositories. - The
build-essentialpackage when usinggccon Ubuntu (and likely other distros as well). - On Windows MinGW 4.3.0 (GCC 8.10) is the minimum recommended compiler.
- Nim hosts a known working MinGW distribution:
Windows Note: Cygwin and similar POSIX runtime environments are not supported.
Then, if you are on a *nix system or Windows, the following steps should compile
Nim from source using gcc, git, and the koch build tool.
Note: The following commands are for the development version of the compiler. For most users, installing the latest stable version is enough. Check out the installation instructions on the website to do so: https://nim-lang.org/install.html.
For package maintainers: see packaging guidelines.
First, get Nim from GitHub:
git clone https://github.com/nim-lang/Nim.git
cd Nim
Next, run the appropriate build shell script for your platform:
build_all.sh(Linux, Mac)build_all.bat(Windows)
Finally, once you have finished the build steps (on Windows, Mac, or Linux) you
should add the bin directory to your PATH.
See also bootstrapping the compiler.
See also reproducible builds.
Koch
koch is the build tool used to build various parts of Nim and to generate
documentation and the website, among other things. The koch tool can also
be used to run the Nim test suite.
Assuming that you added Nim's bin directory to your PATH, you may execute
the tests using ./koch tests. The tests take a while to run, but you
can run a subset of tests by specifying a category (for example
./koch tests cat async).
For more information on the koch build tool please see the documentation
within the doc/koch.md file.
Nimble
nimble is Nim's package manager. To learn more about it, see the
nim-lang/nimble repository.
Contributors
This project exists thanks to all the people who contribute.
Contributing
See detailed contributing guidelines. We welcome all contributions to Nim regardless of how small or large they are. Everything from spelling fixes to new modules to be included in the standard library are welcomed and appreciated. Before you start contributing, you should familiarize yourself with the following repository structure:
bin/,build/- these directories are empty, but are used when Nim is built.compiler/- the compiler source code. Also includes plugins withincompiler/plugins.nimsuggest- the nimsuggest tool that previously lived in thenim-lang/nimsuggestrepository.config/- the configuration for the compiler and documentation generator.doc/- the documentation files in reStructuredText format.lib/- the standard library, including:pure/- modules in the standard library written in pure Nim.impure/- modules in the standard library written in pure Nim with dependencies written in other languages.wrappers/- modules that wrap dependencies written in other languages.
tests/- contains categorized tests for the compiler and standard library.tools/- the tools includingniminst(mostly invoked viakoch).koch.nim- the tool used to bootstrap Nim, generate C sources, build the website, and generate the documentation.
If you are not familiar with making a pull request using GitHub and/or git, please read this guide.
Ideally, you should make sure that all tests pass before submitting a pull request.
However, if you are short on time, you can just run the tests specific to your
changes by only running the corresponding categories of tests. CI verifies
that all tests pass before allowing the pull request to be accepted, so only
running specific tests should be harmless.
Integration tests should go in tests/untestable.
If you're looking for ways to contribute, please look at our issue tracker.
There are always plenty of issues labeled Easy; these should
be a good starting point for an initial contribution to Nim.
You can also help with the development of Nim by making donations. Donations can be made using:
If you have any questions feel free to submit a question on the Nim forum, or via IRC on the #nim channel.
Backers
Thank you to all our backers! [Become a backer]
Sponsors
Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]
You can also see a list of all our sponsors/backers from various payment services on the sponsors page of our website.
License
The compiler and the standard library are licensed under the MIT license, except for some modules which explicitly state otherwise. As a result, you may use any compatible license (essentially any license) for your own programs developed with Nim. You are explicitly permitted to develop commercial applications using Nim.
Please read the copying.txt file for more details.
Copyright © 2006-2026 Andreas Rumpf, all rights reserved.