* docgen: implement cross-document links
Fully implements https://github.com/nim-lang/RFCs/issues/125
Follow-up of: https://github.com/nim-lang/Nim/pull/18642 (for internal links)
and https://github.com/nim-lang/Nim/issues/20127.
Overview
--------
Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).
Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).
See `doc/docgen.md` for full description.
Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).
Performance note
----------------
Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering https://github.com/nim-lang/RFCs/issues/478.
So every `.nim`/`.md` file is processed for `.idx` first).
But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:
* After: 0.59 s.
* Before: 0.59 s.
So Nim compiler works so slow that doc part basically does not matter :-)
Testing
-------
1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
by adding appropriate ``importdoc``
Implementation note
-------------------
Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.
`.idx` file format changed:
* fields are not escaped in most cases because we need original
strings for referencing, not HTML ones
(the exception is linkTitle for titles and headings).
Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
it always allows distinguish Nim/markup entries, titles/headings, etc.
`rstgen` does not rely any more (in most cases) on ad-hoc logic
to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
`module: funcName()`, now it's `proc funcName()`.
(This format is also propagated to `theindex.html` and search results,
I kept it that way since I like it more though it's discussible.)
This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
"keyword" is original, "linkTitle" is HTML one
* fix paths on Windows + more clear code
* Update compiler/docgen.nim
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
* Handle .md and .nim paths uniformly in findRefFile
* handle titles better + more comments
* don't allow markup overwrite index title for .nim files
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
* system refactor, move out 600 lines
* compilation, slice, backwardsindex, misc_num moved out of system
* some procs/types moved into arithmetics, basic_types
* system no longer depends on syncio
* some procs moved around to fit with their surroundings
* make exceptions an import, old ops to misc_num
* move instantiationInfo back
* move back nim version, fix windows echo
* include compilation
* better docs for imported modules, fix unsigned ops
also remove ze, ze64, toU8, toU16, toU32 with nimPreviewSlimSystem
* fix terminal
* workaround IC test & weird csize bug, changelog
* move NimMajor etc back to compilation, rebase for CI
* try ic fix
* form single `indices`, slim out TaintedString, try fix IC
* fix CI, update changelog, addQuitProc
* fix CI
* try fix CI
* actually fix CI finally hopefully
* Update lib/system/compilation.nim
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
* update kochdocs
* hopefully fix csize uses for slimsystem
* fix tquit
Co-authored-by: ringabout <43030857+ringabout@users.noreply.github.com>
- add additional parameters parsing (other implementations will just
ignore them). E.g. if in RST we have:
.. code:: nim
:test: "nim c $1"
...
then in Markdown that will be:
```nim test="nim c $1"
...
```
- implement Markdown interpretation of additional indentation which is
less than 4 spaces (>=4 spaces is a code block but it's not
implemented yet). RST interpretes it as quoted block, for Markdown it's
just normal paragraphs.
- add separate `md2html` and `md2tex` commands. This is to separate
Markdown behavior in cases when it diverges w.r.t. RST significantly —
most conspicously like in the case of additional indentation above, and
also currently the contradicting inline rule of Markdown is also turned
on only in `md2html` and `md2tex`. **Rationale:** mixing Markdown and
RST arbitrarily is a way to nowhere, we need to provide a way to fix the
particular behavior. Note that still all commands have **both** Markdown
and RST features **enabled**. In this PR `*.nim` files can be processed
only in Markdown mode, while `md2html` is for `*.md` files and
`rst2html` for `*.rst` files.
- rename `*.rst` files to `.*md` as our current default behavior is
already Markdown-ish
- convert code blocks in `docgen.rst` to Markdown style as an example.
Other code blocks will be converted in the follow-up PRs
- fix indentation inside Markdown code blocks — additional indentation
is preserved there
- allow more than 3 backticks open/close blocks (tildas \~ are still not
allowed to avoid conflict with RST adornment headings) see also
https://github.com/nim-lang/RFCs/issues/355
- better error messages
- (other) fix a bug that admonitions cannot be used in sandbox mode; fix
annoying warning on line 2711
* Remove define nimExperimentalAsyncjsThen for std/asyncjs.then and std/jsfetch
* Remove define nimExperimentalAsyncjsThen for std/asyncjs.then and std/jsfetch
* Remove define nimExperimentalAsyncjsThen for std/asyncjs.then and std/jsfetch
* Remove define nimExperimentalAsyncjsThen for std/asyncjs.then and std/jsfetch
* move io out of system
* fix tests
* fix tests
* next step
* rename to syncio
* rename
* fix nimscript
* comma
* fix
* fix parts of errors
* good for now
* fix test
* Fixed `strictFuncs` support for `std/pegs`
Enabled `std/pegs` in the `strictFuncs` import test.
Fixes#18057Fixes#16892
See #18111
* Rebased from `devel`
* Conditionally compile `std/pegs` in `koch`
This is for supporting `csources` bootstrap.
Co-authored-by: quantimnot <quantimnot@users.noreply.github.com>
* improve test coverage for isolation
* a bit better
* rename channels to channels_builtin
* follow up #17276
* fix
* Update lib/std/private/jsutils.nim
* Changes for FreeRTOS/LwIP Port for the ESP32 (ESP-IDF)
Adding FreeRTOS/LwIP to compiler:
* adding freertos option
* dyncalls for freertos
* add freertos to posix os list
* adding lwip option
Setting up networking FreeRTOS/LwIP Port:
* setting up lwip network for freertos
* fixing posix / networking for freertos
* disable setInheritable for freerots
* using lwip for net control items
* Fix builds by ignoring lib/posix/posix_freertos_consts.nim similar to lib/posix/posix_other_consts.nim
* koch: add --localdocs to allow building only local docs
This flag also make koch doc use the passed arguments when building
the offline docs.
This is useful when generating nightlies as we would want to use
--doccmd:skip and also skipping a pass of docgen speed things up
drastically (for non-native targets).
This flag superseded the undocumented --docslocal.
* kochdocs: filter google analytics code from the arg list instead
This commit introduce a small PEG expression to filter out the google
analytics code before building local docs when --localdocs is not
specified. This lets us keep any arguments unrelated to google analytics
when building local docs, useful for use with --doccmd:skip
* update doc CI filter to include the files mostly likely to require doc rebuild
* remove code duplication in ./config/nimdoc.cfg; show link to compiler docs, various fixes
* walkDirRecFilter, factor nativeToUnixPath workaround
* glob for getRst2html
* docslocal: 40s to build all docs
* revert code dedup in github actions which did not work alas...
* fixups
* posix: add full Haiku support
This commit provides a posix_haiku derived from posix_other, with types
following Haiku's definition. This fixes cases where the compiler
generates type check for the wrong types (ie. checks where generated for
an int-derived type but it's actually implemented as an uint instead).
* tools/kochdocs: welcome posix_haiku to the blacklist
