mirror of
https://github.com/nim-lang/Nim.git
synced 2026-04-06 07:38:24 +00:00
nimdoc: fix char literal tokenization (#25576)
This fixes highlighter's tokenization of char literals inside
parentheses and brackets.
The Nim syntax highlighter in `docutils/highlite.nim` incorrectly
tokenizes character literals that appear after punctuation characters,
such as all kinds of brackets.
For `echo('v', "hello")`, the tokenizer treated the first `'` as
punctuation because the preceding token was punctuation `(`. As a
result, the second `'` (after `v`) was interpreted as the start of a
character literal and the literal incorrectly extended to the end of the
line.
See other examples in the screenshot:
<img width="508" height="266" alt="Screenshot 2026-03-04 at 16-09-06
_y_test"
src="https://github.com/user-attachments/assets/94d991ae-79d2-4208-a046-6ed4ddcb5c34"
/>
This regression originates from a condition added in PR #23015 that
prevented opening a `gtCharLit` token when the previous token kind was
punctuation. Nim syntax allows character literals after punctuation such
as `(`, `[`, `{`, `:`, `;`, or `,`, of course. The only case mentioned
in the manual explicitly that actually requires special handling is
stroped proc declaration for literals (see the [last paragraph
here](https://nim-lang.github.io/Nim/manual.html#lexical-analysis-character-literals)):
```nim
proc `'customLiteral`(s: string)
```
This PR narrows the conditional to not entering charlit only after
backticks.
This commit is contained in:
@@ -326,7 +326,8 @@ proc nimNextToken(g: var GeneralTokenizer, keywords: openArray[string] = @[]) =
|
||||
pos = nimNumber(g, pos)
|
||||
of '\'':
|
||||
inc(pos)
|
||||
if g.kind != gtPunctuation:
|
||||
let followsBacktick = pos >= 2 and g.buf[pos - 2] == '`'
|
||||
if not followsBacktick:
|
||||
g.kind = gtCharLit
|
||||
while true:
|
||||
case g.buf[pos]
|
||||
@@ -338,6 +339,8 @@ proc nimNextToken(g: var GeneralTokenizer, keywords: openArray[string] = @[]) =
|
||||
of '\\':
|
||||
inc(pos, 2)
|
||||
else: inc(pos)
|
||||
else:
|
||||
g.kind = gtPunctuation
|
||||
of '\"':
|
||||
inc(pos)
|
||||
if (g.buf[pos] == '\"') and (g.buf[pos + 1] == '\"'):
|
||||
|
||||
Reference in New Issue
Block a user