On windows, `HANDLE` type values are converted to `syncio.FileHandle` in
`lib/std/syncio.nim`, `lib/pure/memfiles.nim` and `lib/pure/osproc.nim`.
`HANDLE` type is `void *` on Windows and its size is larger then `cint`.
https://learn.microsoft.com/en-us/windows/win32/winprog/windows-data-types
This PR change `syncio.FileHandle` type so that converting `HANDLE` type
to `syncio.FileHandle` doesn't lose bits.
We can keep `FileHandle` unchanged and change some of parameter/return
type from `FileHandle` to an type same size to `HANDLE`, but it is
breaking change.
in https://forum.nim-lang.org/t/12463 which seems reasonable enough to
me.
stdlib has several dozens of places doing result.setLen now, but if
`experimental:strictDefs` move sot on by default and there is no easy
way to locally suppress the warning we can revisit this.
Fix non-exported `setFileSize` to take optional `oldSize` to (on posix)
shrink differently than it grows (`ftruncate` not `posix_fallocate`)
since it makes sense to assume the higher address space has already been
allocated there and include the old file size in the `proc resize` call.
Also, do not even try `setFileSize` in the first place unless the `open`
itself works by moving the call into the `if newFileSize != -1` branch.
Just cosmetics, also improve some old 2011 comments, note a logic diff
for callers using both `mappedSize` & `newFileSize` from windows branch
in case someone wants to fix that & simplify code formatting a little.
* Add general purpose `setFileSize` (unexported for now). Use to simplify
`memfiles.open` as well as make robust (via hard allocation, not merely
`ftruncate` address space allocation) on systems with `posix_fallocate`.
As part of this, fix a bad `closeHandle` return check bug on Windows and
add `MemFile.resize` for Windows now that setFileSize makes that easier.
* Adapt existing test to exercise newly portable `MemFile.resize`.
* Since Apple has never provided `posix_fallocate`, provide a fallback.
This is presently written in terms of `ftruncate`, but it can be
improved to use `F_PREALLOCATE` instead, as mentioned in a comment.
* system refactor, move out 600 lines
* compilation, slice, backwardsindex, misc_num moved out of system
* some procs/types moved into arithmetics, basic_types
* system no longer depends on syncio
* some procs moved around to fit with their surroundings
* make exceptions an import, old ops to misc_num
* move instantiationInfo back
* move back nim version, fix windows echo
* include compilation
* better docs for imported modules, fix unsigned ops
also remove ze, ze64, toU8, toU16, toU32 with nimPreviewSlimSystem
* fix terminal
* workaround IC test & weird csize bug, changelog
* move NimMajor etc back to compilation, rebase for CI
* try ic fix
* form single `indices`, slim out TaintedString, try fix IC
* fix CI, update changelog, addQuitProc
* fix CI
* try fix CI
* actually fix CI finally hopefully
* Update lib/system/compilation.nim
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
* update kochdocs
* hopefully fix csize uses for slimsystem
* fix tquit
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
* make more standard libraries work with `nimPreviewSlimSystem`
* typo
* part two
* Delete specutils.nim
* fixes more tests
* more fixes
* fixes tests
* fixes three more tests
* add formatfloat import
* fix
* last
* Implement Pandoc Markdown concise link extension
This implements https://github.com/nim-lang/Nim/issues/20127.
Besides reference to headings we also support doing references
to Nim symbols inside Nim modules.
Markdown:
```
Some heading
------------
Ref. [Some heading].
```
Nim:
```
proc someFunction*() ...
... ## Ref. [someFunction]
```
This is substitution for RST syntax like `` `target`_ ``.
All 3 syntax variants of extension from Pandoc Markdown are supported:
`[target]`, `[target][]`, `[description][target]`.
This PR also fixes clashes in existing files, particularly
conflicts with RST footnote feature, which does not work with
this PR (but there is a plan to adopt a popular [Markdown footnote
extension](https://pandoc.org/MANUAL.html#footnotes) to make footnotes work).
Also the PR fixes a bug that Markdown links did not work when `[...]`
section had a line break.
The implementation is straightforward since link resolution did not
change w.r.t. RST implementation, it's almost only about new syntax
addition. The only essential difference is a possibility to add a custom
link description: form `[description][target]` which does not have an
RST equivalent.
* fix nim 1.0 gotcha
* move io out of system
* fix tests
* fix tests
* next step
* rename to syncio
* rename
* fix nimscript
* comma
* fix
* fix parts of errors
* good for now
* fix test
* io: make file descriptors non-inheritable by default
This prevents file descriptors/handles leakage to child processes
that might cause issues like running out of file descriptors, or potential
security issues like leaking a file descriptor to a restricted file.
While this breaks backward compatibility, I'm rather certain that not
many programs (if any) actually make use of this implementation detail.
A new API `setInheritable` is provided for the few that actually want to
use this functionality.
* io: disable inheritance at file creation time for supported platforms
Some platforms provide extension to fopen-family of functions to allow
for disabling descriptor inheritance atomically during File creation.
This guards against possible leaks when a child process is spawned
before we managed to disable the file descriptor inheritance
(ie. in a multi-threaded program).
* net, nativesockets: make sockets non inheritable by default
With this commit, sockets will no longer leak to child processes when
you don't want it to. Should solves a lot of "address in use" that might
occur when your server has just restarted.
All APIs that create sockets in these modules now expose a `inheritable`
flag that allow users to toggle inheritance for the resulting sockets.
An implementation of `setInheritance()` is also provided for SocketHandle.
While atomically disabling inheritance at creation time is supported on
Windows, it's only implemented by native winsock2, which is too much for
now. This support can be implemented in a future patch.
* posix: add F_DUPFD_CLOEXEC
This command duplicates file descriptor with close-on-exec flag set.
Defined in POSIX.1-2008.
* ioselectors_kqueue: don't leak file descriptors
File descriptors internally used by ioselectors on BSD/OSX are now
shielded from leakage.
* posix: add O_CLOEXEC
This flag allows file descriptors to be open() with close-on-exec flag
set atomically.
This flag is specified in POSIX.1-2008
* tfdleak: test for selectors leakage
Also simplified the test by using handle-type agnostic APIs to test for
validity.
* ioselectors_epoll: mark all fd created close-on-exec
File descriptors from ioselectors should no longer leaks on Linux.
* tfdleak: don't check for selector leakage on Windows
The getFd proc for ioselectors_select returns a hardcoded -1
* io: add NoInheritFlag at compile time
* io: add support for ioctl-based close-on-exec
This allows for the flag to be set/unset in one syscall. While the
performance gains might be negliable, we have one less failure point
to deal with.
* tfdleak: add a test for setInheritable
* stdlib: add nimInheritHandles to restore old behaviors
* memfiles: make file handle not inheritable by default for posix
* io: setInheritable now operates on OS file handle
On Windows, the native handle is the only thing that's inheritable, thus
we can assume that users of this function will already have the handle
available to them. This also allows users to pass down file descriptors
from memfiles on Windows with ease, should that be desired.
With this, nativesockets.setInheritable can be made much simpler.
* changelog: clarify
* nativesockets: document setInheritable return value
* posix_utils: atomically disable fd inheritance for mkstemp
* Allow an escape hatch for platform specific flags (of which there are many,
for example MAP_POPULATE itself is a Linux-only thing, not other Unix).
Continue with same defaults as before in this commit, but that really
should be changed to *not* include MAP_POPULATE. While pre-faulting
all the pages can be useful sometimes *if* you know you're going to
access all the data, it is highly unlikely to be what users expect the
default to be. For some things all that up front work is 1000s of
times slower than being lazy/on-demand/only ever faulting parts of the
file. Even the MAP_POPULATE fan who originally in 2014 committed to
this file defaulted it to off (but turned it always-on as a "temporary"
work around for some long since gone/mutated compiler issue).
Anyway, at least something like this `mapFlags` gives users the ability
to override the poor default choice or activate any other idiosyncratic
platform-specific features.
* Use simple, efficient default flags, but also accept whatever the open/mapMem
caller specifies. Save flags in MemFile so they can be used in `resize`.
This field should not need exporting like the others -- callers can always
save whatever values they pass -- but we include a cautionary comment in
case anyone ever asks for a `*` there, as well as for documentation.
Also make documentation for ``mapFlags`` in open more likely to inspire care.
Updating accessors are also provided since the idea of this change is to
allow "updating" operations external to the module which are by their very
nature closely tied to module internals (as well as to OS interface details).
operations (like eg., resizing a file and re-mapping) do not need
to worry about race conditions of re-opened paths, renamed parent
directories and that sort of thing. Operating directly on already
open handles is both safer and more efficient than relying upon the
stability of filesystem paths.
* Add MemMapFileStream
* Added tests
* Fixed bug in memfiles (zero index for string)
* Added flush to changelog
* Attempt to fix Win's nuances
* Fix attempt to fix
* Continue...
* And again...
* Reworked tests (all for win on Win)
* Fixes in flush (Win)
* Replace fn vars to consts
* Added the attempts parameter to the flush
* Replace while to for
* Move to memfiles
* Use Natural instead of uint
* Better error messages for append mode. Handle specific cases.
* Fix 2 problems. First, 0 is a valid fd on Unix (easily gotten if user first
closes all fds and then starts using memfiles). Use -1 instead for an invalid
fd. Second, it is best practice to conserve open fds on Unix and file handles
on Windows. These handles are not needed unless the user wants to remap the
memory with ``mapMem`` (or a hypothetical future ``proc resize``). Adding a
new bool param ``allowRemap=false`` to ``memfiles.open`` solves this cleanly
in a "mostly" backward compatible way. This is only "mostly" because the
default ``false`` case does not keep unneeded resources allocated, but that
most sensible default means that any ``mapMem`` callers need to fix all their
open calls to have allowRemap=true, as this PR also does for tmemfiles2.nim.
* Include backwards compatibility note.
