This commit changes the `offset!` directive so that instead of setting a
`metadata.range` value for the entire pattern, it will set a
`metadata.offset` value. This offset will be applied to the range only
in `vim.treesitter.get_range()`, rather than at directive application
time. This allows the offset to be applied to any and all nodes captured
by the given pattern, and removes the requirement that `#offset!` be
applied to only a single node.
The downside of this change is that plugins which read from
`metadata.range` may be thrown off course, but such plugins should
prefer `vim.treesitter.get_range()` when retrieving ranges anyway.
Note that `#trim!` still sets `metadata.range`, and
`vim.treesitter.get_range()` still reads from `metadata.range`, if it
exists.
Simplify the logic for retrieving the injection ranges for the language
tree. The trees are now also sorted by starting position, regardless of
whether they are part of a combined injection or not. This would be
helpful if ranges are ever to be stored in an interval tree or other
kind of sorted tree structure.
When given, only that range will be checked for validity rather than the
entire tree. This is used in the highlighter to save CPU cycles since we
only need to parse a certain region at a time anyway.
**Problem:** `vim.treesitter.get_parser()` and `vim.treesitter.start()`
both parse the tree before returning it. This is problematic because if
this is a sync parse, it will stall the editor on large files. If it is
an async parse, the functions return stale trees.
**Solution:** Remove this parsing side effect and leave it to the user
to parse the returned trees, either synchronously or asynchronously.
**Problem:** Parsing can be slow for large files, and it is a blocking
operation which can be disruptive and annoying.
**Solution:** Provide a function for asynchronous parsing, which accepts
a callback to be run after parsing completes.
Co-authored-by: Lewis Russell <lewis6991@gmail.com>
Co-authored-by: Luuk van Baal <luukvbaal@gmail.com>
Co-authored-by: VanaIgr <vanaigranov@gmail.com>
Problem:
char-wise folding for `#trim!` ranges are improperly calculated for nodes that
end at column 0, due to the way `get_node_text` works.
Solution:
Add the blank line that `get_node_text` removes for for nodes ending at column
0. Also properly set column positions when performing linewise trims.
This commit also implements more generic trimming, acting on all
whitespace (charwise) rather than just empty lines.
It will unblock
https://github.com/nvim-treesitter/nvim-treesitter/pull/3442 and allow
for properly concealing markdown bullet markers regardless of indent
width, e.g.
**Problems:**
- `vim.treesitter.language.inspect()` returns duplicate
symbol names, sometimes up to 6 of one kind in the case of `markdown`
- The list-like `symbols` table can have holes and is thus not even a
valid msgpack table anyway, mentioned in a test
**Solution:** Return symbols as a map, rather than a list, where field
names are the names of the symbol. The boolean value associated with the
field encodes whether or not the symbol is named.
Note that anonymous nodes are surrounded with double quotes (`"`) to
prevent potential collisions with named counterparts that have the same
identifier.
**Problem:** `vim.treesitter.get_parser` will throw an error if no parser
can be found.
- This means the caller is responsible for wrapping it in a `pcall`,
which is easy to forget
- It also makes it slightly harder to potentially memoize `get_parser`
in the future
- It's a bit unintuitive since many other `get_*` style functions
conventionally return `nil` if no object is found (e.g. `get_node`,
`get_lang`, `query.get`, etc.)
**Solution:** Return `nil` if no parser can be found or created
- This requires a function signature change, and some new assertions in
places where the parser will always (or should always) be found.
- This commit starts by making this change internally, since it is
breaking. Eventually it will be rolled out to the public API.
For context, see https://github.com/neovim/neovim/pull/24738. Before
that PR, Nvim did not correctly handle captures with quantifiers. That
PR made the correct behavior opt-in to minimize breaking changes, with
the intention that the correct behavior would eventually become the
default. Users can still opt-in to the old (incorrect) behavior for now,
but this option will eventually be removed completely.
BREAKING CHANGE: Any plugin which uses `Query:iter_matches()` must
update their call sites to expect an array of nodes in the `match`
table, rather than a single node.
Problem:
Tests have lots of exec_lua calls which input blocks of code
provided as unformatted strings.
Solution:
Teach exec_lua how to handle functions.
Specifically, functions that are run in the context of the test runner
are put in module `test/testutil.lua` while the functions that are run
in the context of the test session are put in
`test/functional/testnvim.lua`.
Closes https://github.com/neovim/neovim/issues/27004.
Move test cases that are more about treesitter query API rather than
parser API or LanguageTree out of "treesitter/parser_spec", and collect
them in another test suite "treesitter/query_spec".
General refactoring, including:
- Improve whitespace and indentation
- Prefix captures with `@`
- Add more comments on `iter_capture()` tests
- Move `test_query` up closer to the fixture source string
No behavioral changes are made.
Query patterns can contain quantifiers (e.g. (foo)+ @bar), so a single
capture can map to multiple nodes. The iter_matches API can not handle
this situation because the match table incorrectly maps capture indices
to a single node instead of to an array of nodes.
The match table should be updated to map capture indices to an array of
nodes. However, this is a massively breaking change, so must be done
with a proper deprecation period.
`iter_matches`, `add_predicate` and `add_directive` must opt-in to the
correct behavior for backward compatibility. This is done with a new
"all" option. This option will become the default and removed after the
0.10 release.
Co-authored-by: Christian Clason <c.clason@uni-graz.at>
Co-authored-by: MDeiml <matthias@deiml.net>
Co-authored-by: Gregory Anders <greg@gpanders.com>
Problem:
A region managed by an injected parser may shrink after re-running the
injection query. If the updated region goes out of the range to be
parsed, then the corresponding tree will remain outdated, possibly
retaining the nodes that shouldn't exist anymore. This results in
outdated highlights.
Solution:
Re-parse an invalid tree if its region intersects the range to be
parsed.
When parsing with a range, languagetree looks up injections and adds
them if needed. This explicitly invalidates parser, making `is_valid`
report `false` both when including and excluding children.
This is an attempt to describe desired behaviour of `is_valid` in tests,
with what ended up being a single line change to satisfy them.
Problem:
Treesitter highlighting is slow for large files with lots of injections.
Solution:
Only parse injections we are going to render during a redraw cycle.
---
- `LanguageTree:parse()` will no longer parse injections by default and
now requires an explicit range argument to be passed.
- `TSHighlighter` now parses injections incrementally during on_win
callbacks for the line range being rendered.
- Plugins which require certain injections to be parsed must run
`parser:parse({ start_row, end_row })` before using the tree.
* feat(treesitter): add injection language fallback
Problem: injection languages are often specified via aliases (e.g.,
filetype or in upper case), requiring custom directives.
Solution: include lookup logic (try as parser name, then filetype, then
lowercase) in LanguageTree itself and remove `#inject-language`
directive.
Co-authored-by: Lewis Russell <me@lewisr.dev>
When an injection has not set include children, make sure not to add
the injection if no ranges are determined.
This could happen when there is an injection with a child that has the
same range as itself. e.g. consider this Makefile snippet
```make
foo:
$(VAR)
```
Line 2 has an injection for bash and a make variable reference. If
include-children isn't set (default), then there is no range on line 2
to inject since the variable reference needs to be excluded.
This caused the language tree to return an empty range, which the parser
now interprets to mean the full buffer. This caused makefiles to have
completely broken highlighting.
Problem: When using treesitter foldexpr,
* :diffput/get open diff folds, and
* folds are not updated in other windows that contain the updated
buffer.
Solution: Update folds in all windows that contain the updated buffer
and use expr foldmethod.
Problem: Treesitter fold is not updated if treesitter hightlight is not
active. More precisely, updating folds requires `LanguageTree:parse()`.
Solution: Call `parse()` before computing folds and compute folds when
lines are added/removed.
This doesn't guarantee correctness of the folds, because some changes
that don't add/remove line won't update the folds even if they should
(e.g. adding pair of braces). But it is good enough for most cases,
while not introducing big overhead.
Also, if highlighting is active, it is likely that
`TSHighlighter._on_buf` already ran `parse()` (or vice versa).
