Problem: many FileType autocommands assume curbuf is the same as the target
buffer; this can cause &syntax to be restored for the wrong buffer in some cases
when TSHighlighter:destroy is called.
Solution: run nvim_exec_autocmds in the context of the target buffer via
nvim_buf_call.
(cherry picked from commit 3ec63cdab8)
Problem: Error occurs if window is invalid in the middle of parsing.
Solution: Check if window is valid in parsing.
- Error
```
vim.schedule callback: ...im/share/nvim/runtime/lua/vim/treesitter/highlighter.lua:485: Invalid window id: 1037
stack traceback:
[C]: in function 'nvim__redraw'
...im/share/nvim/runtime/lua/vim/treesitter/highlighter.lua:485: in function 'cb'
...m/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:494: in function '_run_async_callbacks'
...m/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:550: in function <...m/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:529>
```
- Reproduce script
```lua
local bufnr = vim.api.nvim_create_buf(false, true)
local many_lines = vim.fn["repeat"]({ "local test = 'a'" }, 100000)
vim.api.nvim_buf_set_lines(bufnr, 0, -1, false, many_lines)
local window = vim.api.nvim_open_win(bufnr, true, {
relative = "editor",
row = 0,
col = 0,
width = 10,
height = 10,
})
vim.bo.filetype = "lua"
vim.schedule(function()
vim.api.nvim_win_close(window, true)
end)
```
(cherry picked from commit 58d85cd03d)
**Problem:** There is a lot of distracting highlight flickering when
editing a buffer with multiple open windows. This is because the
parsing/highlighting state is shared across all windows.
**Solution:** Greatly reduce flicker in window splits by scoping the
highlighter state object and the `parsing` state object to each
individual window, so there is no cross-window interference.
This commit will eliminate flicker while editing one open window. It
works by querying previously calculated trees for highlight marks, in
the case that we are still asynchronously parsing in `on_win`.
It will not fully solve the case of flicker when there are multiple open
windows, since the parser will drop previously parsed injection trees
whenever it receives a call to parse a different region of the buffer.
This will require a refactor of `languagetree.lua`.
(cherry picked from commit 8d75910ef9)
Problem: Spliced conceal_lines marks after changing the buffer text are
left valid, concealing lines that shouldn't be.
Solution: Set the `invalidate` extmark property.
(cherry picked from commit 9274532615)
Problem: _on_conceal_line callbacks are not invoked if callback has not
let Nvim know it wants to receive them. But this may change on
factors other than what is currently checked (changed buffer).
Solution: Forego this optimization, callback is still guarded behind
'conceallevel'.
TSHighlighter now places marks for conceal_lines metadata. A new
internal decor provider callback _on_conceal_line was added that
instructs the highlighter to place conceal_lines marks whenever the
editor needs to know whether a line is concealed. The bundled markdown
queries use conceal_lines metadata to conceal code block fence lines.
**Problem:** An erroneous query in the treesitter highlighter gives a
deluge of errors that makes the editor almost unusable.
**Solution:** Detach the highlighter after an error is detected, so that
it only gets displayed once (per highlighter instance).
Problem: `TSHighlighter:destroy()` causes double-processing of the modeline
and failure of `b:undo_ftplugin`.
Solution: Disable modeline in `TSHighlighter:destroy()` by setting `modeline=false`
if executing `syntaxset` autocommands for the `FileType` event.
Co-authored-by: Daniel Petrovic <daniel.petrovic@ebcont.com>
**Problem:** `vim.treesitter.get_parser()` and `vim.treesitter.start()`
both parse the tree before returning it. This is problematic because if
this is a sync parse, it will stall the editor on large files. If it is
an async parse, the functions return stale trees.
**Solution:** Remove this parsing side effect and leave it to the user
to parse the returned trees, either synchronously or asynchronously.
**Problem:** Parsing can be slow for large files, and it is a blocking
operation which can be disruptive and annoying.
**Solution:** Provide a function for asynchronous parsing, which accepts
a callback to be run after parsing completes.
Co-authored-by: Lewis Russell <lewis6991@gmail.com>
Co-authored-by: Luuk van Baal <luukvbaal@gmail.com>
Co-authored-by: VanaIgr <vanaigranov@gmail.com>
Problem: Not possible to only set a "redraw later" type with
nvim__redraw, which seems to be desired for the
treesitter highlighter.
Solution: Do not update the screen when "flush" is explicitly set to
false and only redraw later types are present. In that case,
do not call ui_flush() either.
Problem: nvim__redraw's "range" marks a buffer range for redraw, and subsequently
flushes the UI without updating the windows containing that buffer.
Solution: Implicitly update the screen, unless specified otherwise.
Only update the screen with the last call of the treesitter
on_changedtree() callback.
Problem: Treesitter highlighter implements an on_bytes callback that
just re-marks a buffer range for redraw. The edit that
prompted the callback will already have done that.
Solution: Remove redundant on_bytes callback from the treesitter
highlighter module.
Problem:
- `vim.highlight` module does not follow `:help dev-name-common`, which
documents the name for "highlight" as "hl".
- Shorter names are usually preferred.
Solution:
Rename `vim.highlight` to `vim.hl`.
This is not a breaking change until 2.0 (or maybe never).
Problem:
Error when calling vim.treesitter.start() and vim.treesitter.stop() in
init.lua.
Solution:
Ensure syntaxset augroup exists after loading synload.vim.
Problem: Treesitter highlighter clears the already populated highlight
state when performing spell checking while drawing a
smoothscrolled topline.
Solution: Save and restore the highlight state in the highlighter's
_on_spell_nav callback.
Experimental and subject to future changes.
Add a way to redraw certain elements that are not redrawn while Nvim is waiting
for input, or currently have no API to do so. This API covers all that can be
done with the :redraw* commands, in addition to the following new features:
- Immediately move the cursor to a (non-current) window.
- Target a specific window or buffer to mark for redraw.
- Mark a buffer range for redraw (replaces nvim__buf_redraw_range()).
- Redraw the 'statuscolumn'.
Using -1 as the initial value can cause the pattern offset to become
negative, which in turn results in a negative subpriority, which fails
validation in nvim_buf_set_extmark.
Tree-sitter queries can add URLs to a capture using the `#set!`
directive, e.g.
(inline_link
(link_text) @text.reference
(link_destination) @text.uri
(#set! @text.reference "url" @text.uri))
The pattern above is included by default in the `markdown_inline`
highlight query so that users with supporting terminals will see
hyperlinks. For now, this creates a hyperlink for *all* Markdown URLs of
the pattern [link text](link url), even if `link url` does not contain
a valid protocol (e.g. if `link url` is a path to a file). We may wish to
change this in the future to only linkify when the URL has a valid
protocol scheme, but for now we delegate handling this to the terminal
emulator.
In order to support directives which reference other nodes, the
highlighter must be updated to use `iter_matches` rather than
`iter_captures`. The former provides the `match` table which maps
capture IDs to nodes. However, this has its own challenges:
- `iter_matches` does not guarantee the order in which patterns are
iterated matches the order in the query file. So we must enforce
ordering manually using "subpriorities" (#27131). The pattern index of
each match dictates the extmark's subpriority.
- When injections are used, the highlighter contains multiple trees. The
pattern indices of each tree must be offset relative to the maximum
pattern index from all previous trees to ensure that extmarks appear
in the correct order.
- The `iter_captures` implementation currently has a bug where the
"match" table is only returned for the first capture within a pattern
(see #27274). This bug means that `#set!` directives in a query
apply only to the first capture within a pattern. Unfortunately, many
queries in the wild have come to depend on this behavior.
`iter_matches` does not share this flaw, so switching to
`iter_matches` exposed bugs in existing highlight queries. These
queries have been updated in this repo, but may still need to be
updated by users. The `#set!` directive applies to the _entire_ query
pattern when used without a capture argument. To make `#set!`
apply only to a single capture, the capture must be given as an
argument.
- Added `@inlinedoc` so single use Lua types can be inlined into the
functions docs. E.g.
```lua
--- @class myopts
--- @inlinedoc
---
--- Documentation for some field
--- @field somefield integer
--- @param opts myOpts
function foo(opts)
end
```
Will be rendered as
```
foo(opts)
Parameters:
- {opts} (table) Object with the fields:
- somefield (integer) Documentation
for some field
```
- Marked many classes with with `@nodoc` or `(private)`.
We can eventually introduce these when we want to.
Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
- `TSQuery`: userdata object for parsed query.
- `vim.treesitter.Query`: renamed from `Query`.
- Add a new field `lang`.
- `TSQueryInfo`:
- Move to `vim/treesitter/_meta.lua`, because C code owns it.
- Correct typing for `patterns`, should be a map from `integer`
(pattern_id) to `(integer|string)[][]` (list of predicates or
directives).
- `vim.treesitter.QueryInfo` is added.
- This currently has the same structure as `TSQueryInfo` (exported
from C code).
- Document the fields (see `TSQuery:inspect`).
- Add typing for `vim._ts_parse_query()`.
- Remove some unused fields
- Prefix classes with `vim.`
- Move around some functions so the query stuff is at the top.
- Improve type hints
- Rework how hl_cache is implemented
Problem:
Treesitter highlighter's on_line was iterating all the parsed trees,
which can be quite a lot when injection is used. This may slow down
scrolling and cursor movement in big files with many comment injections
(e.g., lsp/_meta/protocol.lua).
Solution:
In on_win, collect trees inside the visible range, and use them in
on_line.
NOTE:
This optimization depends on the correctness of on_win's botline_guess
parameter (i.e., it's always greater than or equal to the line numbers
passed to on_line). The documentation does not guarantee this, but I
have never noticed a problem so far.
Problem
---
If a highlighter query returns a significant number of predicate
non-matches, the highlighter will scan well past the end of the window.
Solution
---
In the iterator returned from `iter_captures`, accept an optional
parameter `end_line`. If no parameter provided, the behavior is
unchanged, hence this is a non-invasive tweak.
Fixes: #25113nvim-treesitter/nvim-treesitter#5057
Problem:
* The guessed botline might be smaller than the actual botline e.g. when
there are folds and the user is typing in insert mode. This may result
in incorrect treesitter highlights for injections.
* botline can be larger than the last line number of the buffer, which
results in errors when placing extmarks.
Solution:
* Take a more conservative approximation. I am not sure if it is
sufficient to guarantee correctness, but it seems to be good enough
for the case mentioned above.
* Clamp it to the last line number.
Co-authored-by: Lewis Russell <me@lewisr.dev>
Problem:
Treesitter highlighting is slow for large files with lots of injections.
Solution:
Only parse injections we are going to render during a redraw cycle.
---
- `LanguageTree:parse()` will no longer parse injections by default and
now requires an explicit range argument to be passed.
- `TSHighlighter` now parses injections incrementally during on_win
callbacks for the line range being rendered.
- Plugins which require certain injections to be parsed must run
`parser:parse({ start_row, end_row })` before using the tree.
* docs(lua): teach lua2dox how to table
* docs(lua): teach gen_vimdoc.py about local functions
No more need to mark local functions with @private
* docs(lua): mention @nodoc and @meta in dev-lua-doc
* fixup!
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
---------
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
When injections are added or removed make sure to:
- invoke 'changedtree' callbacks for when new trees are added.
- invoke 'changedtree' callbacks for when trees are invalidated
- redraw regions when languagetree children are removed
Problem:
Codebase inconsistently binds vim.api onto a or api.
Solution:
Use api everywhere. a as an identifier is too short to have at the
module level.
Problem:
Help tags like vim.treesitter.language.add() are confusing because
`vim.treesitter.language` is (thankfully) not a user-facing module.
Solution:
Ignore the "fstem" when generating "treesitter" tags.
