* docgen: implement cross-document links
Fully implements https://github.com/nim-lang/RFCs/issues/125
Follow-up of: https://github.com/nim-lang/Nim/pull/18642 (for internal links)
and https://github.com/nim-lang/Nim/issues/20127.
Overview
--------
Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).
Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).
See `doc/docgen.md` for full description.
Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).
Performance note
----------------
Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering https://github.com/nim-lang/RFCs/issues/478.
So every `.nim`/`.md` file is processed for `.idx` first).
But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:
* After: 0.59 s.
* Before: 0.59 s.
So Nim compiler works so slow that doc part basically does not matter :-)
Testing
-------
1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
by adding appropriate ``importdoc``
Implementation note
-------------------
Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.
`.idx` file format changed:
* fields are not escaped in most cases because we need original
strings for referencing, not HTML ones
(the exception is linkTitle for titles and headings).
Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
it always allows distinguish Nim/markup entries, titles/headings, etc.
`rstgen` does not rely any more (in most cases) on ad-hoc logic
to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
`module: funcName()`, now it's `proc funcName()`.
(This format is also propagated to `theindex.html` and search results,
I kept it that way since I like it more though it's discussible.)
This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
"keyword" is original, "linkTitle" is HTML one
* fix paths on Windows + more clear code
* Update compiler/docgen.nim
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
* Handle .md and .nim paths uniformly in findRefFile
* handle titles better + more comments
* don't allow markup overwrite index title for .nim files
Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
* Make rstgen work with gcsafe
Co-authored-by: Danil Yarantsev <tiberiumk12@gmail.com>
* add tests and fixes
* if nimHasWarningAsError
Co-authored-by: Danil Yarantsev <tiberiumk12@gmail.com>
The strutils `delete` func with signature
func delete*(s: var string, first, last: int)
was deprecated in adba5eb45e, in favor of one with signature
func delete*(s: var string, slice: Slice[int])
However, a few procedures still used the deprecated form. This commit
updates them, resolving these deprecation warnings:
rstgen.nim(766, 12) Warning: use `delete(s, first..last)`; delete is deprecated [Deprecated]
strutils.nim(1651, 19) Warning: use `delete(s, first..last)`; delete is deprecated [Deprecated]
strutils.nim(1679, 7) Warning: use `delete(s, first..last)`; delete is deprecated [Deprecated]
strutils.nim(2472, 7) Warning: use `delete(s, first..last)`; delete is deprecated [Deprecated]
Before this commit:
- `trimZeros` called `s.delete(i+1, i)` for an input that lacks a
trailing zero (like "1.23").
- `removePrefix*(s: var string, prefix: string)` called
`s.delete(0, -1)` when the prefix was the empty string.
which did not modify `s`, nor raise an error. But the newer slice
`delete` raises an `IndexDefect` when the start of the slice is greater
than the end, so we avoid calling the new `delete` for such a case.
Recall that exceptions inheriting from `system.Defect` are not tracked
with the `.raises: []` exception tracking mechanism [1], so this commit
does not break existing code like:
proc foo {.raises: [].} =
var s = "abc1.20"
s.removePrefix("abc")
s.trimZeros()
doAssert s == "1.2"
The `strutils.delete` deprecation was motivated by a problem with
`system.delete` [2][3]:
`system.delete` had surprising behavior when the index passed to it
was out of bounds (it would delete the last entry then). Compile
with `-d:nimStrictDelete` so that an index error is produced
instead.
Be aware however that your code might depend on this quirky behavior
so a review process is required on your part before you can use
`-d:nimStrictDelete`. To make this review easier, use the
`-d:nimAuditDelete` switch, which pretends that `system.delete` is
deprecated so that it is easier to see where it was used in your
code.
`-d:nimStrictDelete` will become the default in upcoming versions.
A similar deprecation happened with `sequtils.delete` [4], but that
deprecated form is already not used in this repo.
[1] https://github.com/nim-lang/Nim/blob/2dec69fe5aa6/doc/manual.md#exception-tracking
[2] https://github.com/nim-lang/Nim/blob/2dec69fe5aa6/changelogs/changelog_1_6_0.md#system
[3] https://github.com/nim-lang/Nim/commit/92cb76571432
[4] https://github.com/nim-lang/Nim/commit/1d6863a7899f
* Implement Pandoc Markdown concise link extension
This implements https://github.com/nim-lang/Nim/issues/20127.
Besides reference to headings we also support doing references
to Nim symbols inside Nim modules.
Markdown:
```
Some heading
------------
Ref. [Some heading].
```
Nim:
```
proc someFunction*() ...
... ## Ref. [someFunction]
```
This is substitution for RST syntax like `` `target`_ ``.
All 3 syntax variants of extension from Pandoc Markdown are supported:
`[target]`, `[target][]`, `[description][target]`.
This PR also fixes clashes in existing files, particularly
conflicts with RST footnote feature, which does not work with
this PR (but there is a plan to adopt a popular [Markdown footnote
extension](https://pandoc.org/MANUAL.html#footnotes) to make footnotes work).
Also the PR fixes a bug that Markdown links did not work when `[...]`
section had a line break.
The implementation is straightforward since link resolution did not
change w.r.t. RST implementation, it's almost only about new syntax
addition. The only essential difference is a possibility to add a custom
link description: form `[description][target]` which does not have an
RST equivalent.
* fix nim 1.0 gotcha
Fix links to subheader when TOC is present
It was observed (in https://github.com/nim-lang/Nim/pull/20112)
that links to 2nd- (and subsequent) -level headings
fail if TOC is present, e.g.:
```nim
.. contents::
Type relations
==============
Convertible relation
--------------------
Ref. `Convertible relation`_
```
The problem here is that links are resolved in `rst.nim` but later
`rstgen.nim` fixes ("fixes") anchors to make them unique so that
TOC always works (if e.g. there was another sub-section like
"Convertible relation").
The solution implemented in this PR is to move that fix-up of anchors
into `rst.nim`, so that link resolution could know final anchors.
The bug seems to be added in https://github.com/nim-lang/Nim/pull/2332
in 2015, that is it is present in Nim 1.0.
Fixes silent disappearance of Markdown (pseudo-)link when it's detected as
unsafe protocol. Now it will be converted to plain text in spirit of
[the specification](https://spec.commonmark.org/0.30/#links).
For that sake the check for protocol is added to rst.nim also.
* fixes a possible 'javascript:' protocol exploit [backport:1.0]
* add tests
* Update tests/stdlib/trstgen.nim
* add the same logic for hyperlinks
* move the logic into a proc
Co-authored-by: narimiran <narimiran@disroot.org>
* fix bullet/enumarated lists with many blank lines
* fix enumerated list parsing
* fix parse failure when next line after list empty
* implement arbitrary start of enumerator
* check that enumerators are in order
* remove redundant start=x if x=1 or a
* add some doc on implemented features
* update start in rst_examples.rst
* allow upper-case letters + more docs
