Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
Query patterns can contain quantifiers (e.g. (foo)+ @bar), so a single
capture can map to multiple nodes. The iter_matches API can not handle
this situation because the match table incorrectly maps capture indices
to a single node instead of to an array of nodes.
The match table should be updated to map capture indices to an array of
nodes. However, this is a massively breaking change, so must be done
with a proper deprecation period.
`iter_matches`, `add_predicate` and `add_directive` must opt-in to the
correct behavior for backward compatibility. This is done with a new
"all" option. This option will become the default and removed after the
0.10 release.
Co-authored-by: Christian Clason <c.clason@uni-graz.at>
Co-authored-by: MDeiml <matthias@deiml.net>
Co-authored-by: Gregory Anders <greg@gpanders.com>
Document that the `start` and `stop` parameters in
`Query:iter_captures()` and `Query:iter_matches()` are optional.
The tree-sitter lib has been bumped up to 0.20.9, so we also no longer
need "Requires treesitter >= 0.20.9".
- `TSQuery`: userdata object for parsed query.
- `vim.treesitter.Query`: renamed from `Query`.
- Add a new field `lang`.
- `TSQueryInfo`:
- Move to `vim/treesitter/_meta.lua`, because C code owns it.
- Correct typing for `patterns`, should be a map from `integer`
(pattern_id) to `(integer|string)[][]` (list of predicates or
directives).
- `vim.treesitter.QueryInfo` is added.
- This currently has the same structure as `TSQueryInfo` (exported
from C code).
- Document the fields (see `TSQuery:inspect`).
- Add typing for `vim._ts_parse_query()`.
Problem:
- `:InspectTree` was showing node ranges in 1-based indexing, i.e., in
vim cursor position (lnum, col). However, treesitter API adopts
0-based indexing to represent ranges (Range4). This can often be
confusing for developers and plugin authors when debugging code
written with treesiter APIs.
Solution:
- Change to 0-based indexing from 1-based indexing to show node ranges
in `:InspectTree`.
- Note: To make things not complicated, we do not provide an option or
keymap to configure which indexing mode to use.
The benefit of this is that users only pay for what they use. If e.g.
only `vim.lsp.buf_get_clients()` is called then they don't need to load
all modules under `vim.lsp` which could lead to significant startuptime
saving.
Also `vim.lsp.module` is a bit nicer to user compared to
`require("vim.lsp.module")`.
This isn't used for some nested modules such as `filetype` as it breaks
tests with error messages such as "attempt to index field 'detect'".
It's not entirely certain the reason for this, but it is likely it is
due to filetype being precompiled which would imply deferred loading
isn't needed for performance reasons.
Problem: Parsed language annotations can be random garbage so
`nvim_get_runtime_file` throws an error.
Solution: Validate that `alias` is a valid language name before trying
to find a parser for it.
Improve error messages for `:InspectTree`, when no parsers are available
for the current buffer and filetype. We can show more informative and
helpful error message for users (e.g., which lang was searched for):
```
... No parser available for the given buffer:
+... no parser for 'custom_ft' language, see :help treesitter-parsers
```
Also improve the relevant docs for *treesitter-parsers*.
Problem:
A region managed by an injected parser may shrink after re-running the
injection query. If the updated region goes out of the range to be
parsed, then the corresponding tree will remain outdated, possibly
retaining the nodes that shouldn't exist anymore. This results in
outdated highlights.
Solution:
Re-parse an invalid tree if its region intersects the range to be
parsed.
- Remove some unused fields
- Prefix classes with `vim.`
- Move around some functions so the query stuff is at the top.
- Improve type hints
- Rework how hl_cache is implemented
Problem:
Treesitter highlighter's on_line was iterating all the parsed trees,
which can be quite a lot when injection is used. This may slow down
scrolling and cursor movement in big files with many comment injections
(e.g., lsp/_meta/protocol.lua).
Solution:
In on_win, collect trees inside the visible range, and use them in
on_line.
NOTE:
This optimization depends on the correctness of on_win's botline_guess
parameter (i.e., it's always greater than or equal to the line numbers
passed to on_line). The documentation does not guarantee this, but I
have never noticed a problem so far.
* Collect on_bytes and flush at the invocation of the scheduled callback
to take account of commands that triggers multiple on_bytes.
* More accurately track movement of folds so that foldexpr returns
reasonable values even when the scheduled computation is not run yet.
* Start computing folds from the line above (+ foldminlines) the changed
lines to handle the folds that are removed due to the size limit.
* Shrink folds that end at the line at which another fold starts to
assign proper level to that line.
* Use level '=' for lines that are not computed yet.
Problem:
`LanguageTree:for_each_tree` calls itself for child nodes, so when we
calls `for_each_tree` inside `for_each_tree`, this quickly leads to
exponential tree calls.
Solution:
Use `pairs(child:trees())` directly in this case, as we don't need the
extra callback for each children, this is already handled from the outer
`for_each_tree` call
When first opened, the tree-sitter inspector traverses all of the nodes
in the buffer to calculate an array of nodes. This traversal is done
only once, and _all_ nodes (both named and anonymous) are included.
Toggling anonymous nodes in the inspector only changes how the tree is
drawn in the buffer, but does not affect the underlying data structure
at all.
When the buffer is traversed and the list of nodes is calculated, we
don't know whether or not anonymous nodes will be displayed in the
inspector or not. Thus, we cannot determine during traversal where to
put closing parentheses. Instead, this must be done when drawing.
When we draw, the tree structure has been flatted into a single array,
so we lose parent-child relationships that would otherwise make
determining the number of closing parentheses straightforward. However,
we can instead rely on the fact that a delta between the depth of a node
and the depth of the successive node _must_ mean that more closing
parentheses are required:
(foo
(bar)
(baz) ↑
│
└ (bar) and (baz) have different depths, so (bar) must have an
extra closing parenthesis
This does not depend on whether or not anonymous nodes are displayed and
so works in both cases.
Problem: Only injections under the top level tree are found.
Solution: Iterate through all trees to find injections. When two
injections are contained within the same node in the parent tree, prefer
the injection with the larger byte length.
When parsing with a range, languagetree looks up injections and adds
them if needed. This explicitly invalidates parser, making `is_valid`
report `false` both when including and excluding children.
This is an attempt to describe desired behaviour of `is_valid` in tests,
with what ended up being a single line change to satisfy them.
Problem: Visual highlight is inconsistent on a folded line with
treesitter foldtext.
Solution: Don't added Folded highlight as it is already in background.
This is incorrect in the following scenario:
1. The language tree is Lua > Vim > Lua.
2. An edit simultaneously wipes out the `_regions` of all nodes, while
taking the Vim injection off-screen.
3. The Vim injection is not re-parsed, so the child Lua `_regions` is
still `nil`.
4. The child Lua is assumed, incorrectly, to occupy the whole document.
5. This causes the injections to be parsed again, resulting in Lua > Vim
> Lua > Vim.
6. Now, by the same process, Vim ends up with its range assumed over the
whole document. Now the parse is broken and results in broken
highlighting and poor performance.
It should be fine to instead treat an unparsed node as occupying
nothing (i.e. effectively non-existent). Since, either:
- The parent was just parsed, hence defining `_regions`
- The parent was not just parsed, in which case this node doesn't need
to be parsed either.
Also, the name `has_regions` is confusing; it seems to simply
mean the opposite of "root" or "full_document". However, this PR does
not touch it.
Memoizes a function, using a custom function to hash the arguments.
Private for now until:
- There are other places in the codebase that could benefit from this
(e.g. LSP), but might require other changes to accommodate.
- Invalidation of the cache needs to be controllable. Using weak tables
is an acceptable invalidation policy, but it shouldn't be the only
one.
- I don't think the story around `hash_fn` is completely thought out. We
may be able to have a good default hash_fn by hashing each argument,
so basically a better 'concat'.
Problem:
With incremental injection parsing, injected languages' parsers parse
only the relevant regions and stores the result in _trees with the index
of the corresponding region. Therefore, there can be holes in _trees.
Solution:
* Use generic table functions where appropriate.
* Fix type annotations and docs.
Problem:
It doesn't make much sense to flatten each region (= list of ranges).
This coincidentally worked for region with a single range.
Solution:
Custom function for combining regions.
Problem
---
If a highlighter query returns a significant number of predicate
non-matches, the highlighter will scan well past the end of the window.
Solution
---
In the iterator returned from `iter_captures`, accept an optional
parameter `end_line`. If no parameter provided, the behavior is
unchanged, hence this is a non-invasive tweak.
Fixes: #25113nvim-treesitter/nvim-treesitter#5057
