TSHighlighter now places marks for conceal_lines metadata. A new
internal decor provider callback _on_conceal_line was added that
instructs the highlighter to place conceal_lines marks whenever the
editor needs to know whether a line is concealed. The bundled markdown
queries use conceal_lines metadata to conceal code block fence lines.
Lua coroutines can yield across non-coroutine function boundaries,
meaning that we don't need to wrap each helper function in a coroutine
and resume it within `_parse()`. If we just have them yield when
appropriate, this will be caught by the top level `_parse()` coroutine,
and resuming the `_parse()` will resume from the position in the helper
function where we yielded last.
Problem: It's difficult to navigate large structured text files (vim
help, checkhealth, Markdown).
Solution: Support `gO` for table of contents and `]]`/`[[` for moving
between headings for all these filetypes using treesitter queries.
Refactor: colorization of highlight groups is moved to the `help` ftplugin
while headings-related functionality is implemented in a private
`vim.treesitter` module for possible future use for other filetypes.
**Problem:** Currently, parsing is asynchronous, but it involves a
(sometimes lengthy) step which finds all injection ranges for a tree by
iterating over that language's injection queries. This causes edits in
large files to be extremely slow, and also causes a long stutter during
the initial parse of a large file.
**Solution:** Break up the injection query iteration over multiple event
loop iterations.
**Problem:** An erroneous query in the treesitter highlighter gives a
deluge of errors that makes the editor almost unusable.
**Solution:** Detach the highlighter after an error is detected, so that
it only gets displayed once (per highlighter instance).
Problem: Computing fold levels for an empty buffer (somehow) breaks the
parser state, resulting in a broken highlighter and foldexpr.
Cached foldexpr parser is invalid after filetype has changed.
Solution: Avoid computing fold levels for empty buffer.
Clear cached foldinfos upon `FileType`.
We need to add a separate variable to keep track of this information,
since we cannot read the length of the valid regions table itself, since
it has holes.
Now `:InspectTree` will show missing nodes as e.g. `(MISSING identifier)`
or `(MISSING ";")` rather than just `(identifier)` or `";"`. This is
doable because the `MISSING` keyword is now valid query syntax.
Co-authored-by: Christian Clason <c.clason@uni-graz.at>
When given, only that range will be checked for validity rather than the
entire tree. This is used in the highlighter to save CPU cycles since we
only need to parse a certain region at a time anyway.
This means that all work previously done by a `_parse()` iteration will
be kept in future iterations. This prevents it from running indefinitely
in some cases where the file is very large and there are 2+ injections.
Problem:
When running an initial parse, parse() returns an empty table rather
than an actual range. In `languagetree.lua`, we manually check if
a parse was incremental to determine the changed parse region.
Solution:
- Always return a range (in the C side) from parse().
- Simplify the language tree code a bit.
- Logger no longer shows empty ranges on the initial parse.
This simplifies some logic in `languagetree.lua`, removing the need for
`_has_regions`, and removing side effects in `:included_regions()`.
Before:
- Edit is made which sets `_regions = nil`
- Upon the next call to `included_regions()` (usually right after we
marked `_regions` as `nil` due to an `_iter_regions()` call), if
`_regions` is nil, we repopulate the table (as long as the tree
actually has regions)
After:
- Edit is made which resets `_regions` if it exists
- `included_regions()` no longer needs to perform this logic itself, and
also no longer needs to read a `_has_regions` variable
Problem: `TSHighlighter:destroy()` causes double-processing of the modeline
and failure of `b:undo_ftplugin`.
Solution: Disable modeline in `TSHighlighter:destroy()` by setting `modeline=false`
if executing `syntaxset` autocommands for the `FileType` event.
Co-authored-by: Daniel Petrovic <daniel.petrovic@ebcont.com>
**Problem:** In the case where the user sets the treesitter foldexpr upon startup in
their `init.lua`, the fold info will be calculated before the parser has
been loaded in, meaning folds will be properly calculated until edits or
`:e`.
**Solution:** Refresh fold information upon `VimEnter` as a sanity check
to ensure that a parser really doesn't exist before always returning
`'0'` in the foldexpr.
**Problem:** Currently, if users want to efficiently disable injections,
they have to delete the injection query files at their runtime path.
This is because we only check for existence of the files before running
the query over the entire buffer.
**Solution:** Check for existence of query files, *and* that those files
actually have captures. This will allow users to just comment out
existing queries (or better yet, just add their own injection query to
`~/.config/nvim` which contains only comments) to disable running the
query over the entire buffer (a potentially slow operation)
Problem:
`yx` uses "y" prefix, which shadows a builtin operator.
Solution:
Use `g=` (in the form of `g==` currently), drawing from precedent of
CTRL-= and 'tpope/vim-scriptease'.
**Problem:** The treesitter `foldexpr` runs synchronous parses to
calculate fold levels, which eliminates async parsing performance in the
highlighter.
**Solution:** Migrate the `foldexpr` to also calculate and apply fold
levels asynchronously.
**Problem:** `vim.treesitter.get_parser()` and `vim.treesitter.start()`
both parse the tree before returning it. This is problematic because if
this is a sync parse, it will stall the editor on large files. If it is
an async parse, the functions return stale trees.
**Solution:** Remove this parsing side effect and leave it to the user
to parse the returned trees, either synchronously or asynchronously.
**Problem:** Parsing can be slow for large files, and it is a blocking
operation which can be disruptive and annoying.
**Solution:** Provide a function for asynchronous parsing, which accepts
a callback to be run after parsing completes.
Co-authored-by: Lewis Russell <lewis6991@gmail.com>
Co-authored-by: Luuk van Baal <luukvbaal@gmail.com>
Co-authored-by: VanaIgr <vanaigranov@gmail.com>
**Problem:** Query parsing uses a weak cache which is invalidated
frequently
**Solution:** Make the cache strong, and invalidate it manually when
necessary (that is, when `rtp` is changed or `query.set()` is called)
Co-authored-by: Christian Clason <c.clason@uni-graz.at>
**Problem:** The treesitter `foldexpr` calls `get_parser()` for each
line in the buffer when calculating folds. This can be incredibly slow
for buffers where a parser cannot be found (because the result is not
cached), and exponentially more so when the user has many
`runtimepath`s.
**Solution:** Only fetch the parser when it is needed; that is, only
when initializing fold data for a buffer.
Co-authored-by: Jongwook Choi <wookayin@gmail.com>
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
Problem: autocmd to refresh folds always uses the current buffer if the
option type is local. However, the current buffer may not have a parser,
and thus the assert that checks for a parser could fail.
Solution: check if the foldinfo contains the buffer, and only refresh if
so.
Problem:
char-wise folding for `#trim!` ranges are improperly calculated for nodes that
end at column 0, due to the way `get_node_text` works.
Solution:
Add the blank line that `get_node_text` removes for for nodes ending at column
0. Also properly set column positions when performing linewise trims.
