Problem: `has-ancestor?` is O(n²) for the depth of the tree since it iterates over each of the node's ancestors (bottom-up), and each ancestor takes O(n) time.
This happens because tree-sitter's nodes don't store their parent nodes, and the tree is searched (top-down) each time a new parent is requested.
Solution: Make use of new `ts_node_child_containing_descendant()` in tree-sitter v0.22.6 (which is now the minimum required version) to rewrite the `has-ancestor?` predicate in C to become O(n).
For a sample file, decreases the time taken by `has-ancestor?` from 360ms to 6ms.
Instead of painfully messing with timing to determine if queries were
reparsed, we can simply keep a counter next to the call to ts_query_new
Also memoization had a hidden dependency on the garbage collection of
the the key, a hash value which never is kept around in memory. this was
done intentionally as the hash does not capture all relevant state for the
query (external included files) even if actual query objects still
would be reachable in memory. To make the test fully deterministic in
CI, we explicitly control GC.
* fix(treesitter): enforce lowercase language names
Problem: On case-insensitive file systems (e.g., macOS), `has_parser`
will return `true` for uppercase aliases, which will then try to inject
the uppercase language unsuccessfully.
Solution: Enforce and assume parser names to be lowercase when
resolving language names.
Specifically, functions that are run in the context of the test runner
are put in module `test/testutil.lua` while the functions that are run
in the context of the test session are put in
`test/functional/testnvim.lua`.
Closes https://github.com/neovim/neovim/issues/27004.
Move test cases that are more about treesitter query API rather than
parser API or LanguageTree out of "treesitter/parser_spec", and collect
them in another test suite "treesitter/query_spec".
General refactoring, including:
- Improve whitespace and indentation
- Prefix captures with `@`
- Add more comments on `iter_capture()` tests
- Move `test_query` up closer to the fixture source string
No behavioral changes are made.
Tree-sitter queries can add URLs to a capture using the `#set!`
directive, e.g.
(inline_link
(link_text) @text.reference
(link_destination) @text.uri
(#set! @text.reference "url" @text.uri))
The pattern above is included by default in the `markdown_inline`
highlight query so that users with supporting terminals will see
hyperlinks. For now, this creates a hyperlink for *all* Markdown URLs of
the pattern [link text](link url), even if `link url` does not contain
a valid protocol (e.g. if `link url` is a path to a file). We may wish to
change this in the future to only linkify when the URL has a valid
protocol scheme, but for now we delegate handling this to the terminal
emulator.
In order to support directives which reference other nodes, the
highlighter must be updated to use `iter_matches` rather than
`iter_captures`. The former provides the `match` table which maps
capture IDs to nodes. However, this has its own challenges:
- `iter_matches` does not guarantee the order in which patterns are
iterated matches the order in the query file. So we must enforce
ordering manually using "subpriorities" (#27131). The pattern index of
each match dictates the extmark's subpriority.
- When injections are used, the highlighter contains multiple trees. The
pattern indices of each tree must be offset relative to the maximum
pattern index from all previous trees to ensure that extmarks appear
in the correct order.
- The `iter_captures` implementation currently has a bug where the
"match" table is only returned for the first capture within a pattern
(see #27274). This bug means that `#set!` directives in a query
apply only to the first capture within a pattern. Unfortunately, many
queries in the wild have come to depend on this behavior.
`iter_matches` does not share this flaw, so switching to
`iter_matches` exposed bugs in existing highlight queries. These
queries have been updated in this repo, but may still need to be
updated by users. The `#set!` directive applies to the _entire_ query
pattern when used without a capture argument. To make `#set!`
apply only to a single capture, the capture must be given as an
argument.
Query patterns can contain quantifiers (e.g. (foo)+ @bar), so a single
capture can map to multiple nodes. The iter_matches API can not handle
this situation because the match table incorrectly maps capture indices
to a single node instead of to an array of nodes.
The match table should be updated to map capture indices to an array of
nodes. However, this is a massively breaking change, so must be done
with a proper deprecation period.
`iter_matches`, `add_predicate` and `add_directive` must opt-in to the
correct behavior for backward compatibility. This is done with a new
"all" option. This option will become the default and removed after the
0.10 release.
Co-authored-by: Christian Clason <c.clason@uni-graz.at>
Co-authored-by: MDeiml <matthias@deiml.net>
Co-authored-by: Gregory Anders <greg@gpanders.com>
* use `Special` as default for `@markup.*`, especially `@markup.raw` and
`@markup.math` (`@markup` itself is never used)
* use `Structure` for `@markup.environment`
* highlight all of `@markup.link` as Underlined (otherwise concealed
links are invisible)
Problem: Sharing queries with upstream and Helix is difficult due to
different capture names.
Solution: Define and document a new set of standard captures that
matches tree-sitter "standard captures" (where defined) and is closer to
Helix' Atom-style nested groups.
This is a breaking change for colorschemes that defined highlights based
on the old captures. On the other hand, the default colorscheme now
defines links for all standard captures (not just those used in bundled
queries), improving the out-of-the-box experience.
Problem:
A region managed by an injected parser may shrink after re-running the
injection query. If the updated region goes out of the range to be
parsed, then the corresponding tree will remain outdated, possibly
retaining the nodes that shouldn't exist anymore. This results in
outdated highlights.
Solution:
Re-parse an invalid tree if its region intersects the range to be
parsed.
This is the command invoked repeatedly to make the changes:
:%s/^\(.*\)|\%(\*\(\d\+\)\)\?$\n\1|\%(\*\(\d\+\)\)\?$/\=submatch(1)..'|*'..(max([str2nr(submatch(2)),1])+max([str2nr(submatch(3)),1]))/g
PROBLEM: `vim.treesitter.get_node()` does not recognize the `lang` in
the option table. This option was used in somewhere else, for instance,
`vim.treesitter.dev` (for `inspect_tree`) but was never implemented.
SOLUTION: Make `get_node()` correctly use `opts.lang` when getting a
treesitter parser.
When parsing with a range, languagetree looks up injections and adds
them if needed. This explicitly invalidates parser, making `is_valid`
report `false` both when including and excluding children.
This is an attempt to describe desired behaviour of `is_valid` in tests,
with what ended up being a single line change to satisfy them.
Problem: Visual highlight is inconsistent on a folded line with
treesitter foldtext.
Solution: Don't added Folded highlight as it is already in background.
This is incorrect in the following scenario:
1. The language tree is Lua > Vim > Lua.
2. An edit simultaneously wipes out the `_regions` of all nodes, while
taking the Vim injection off-screen.
3. The Vim injection is not re-parsed, so the child Lua `_regions` is
still `nil`.
4. The child Lua is assumed, incorrectly, to occupy the whole document.
5. This causes the injections to be parsed again, resulting in Lua > Vim
> Lua > Vim.
6. Now, by the same process, Vim ends up with its range assumed over the
whole document. Now the parse is broken and results in broken
highlighting and poor performance.
It should be fine to instead treat an unparsed node as occupying
nothing (i.e. effectively non-existent). Since, either:
- The parent was just parsed, hence defining `_regions`
- The parent was not just parsed, in which case this node doesn't need
to be parsed either.
Also, the name `has_regions` is confusing; it seems to simply
mean the opposite of "root" or "full_document". However, this PR does
not touch it.
Unfortunately the gc=false objects can refer to a dangling tree if the
gc=true tree was freed first. This reuses the same tree object as the
node itself is keeping alive via the uservalue of the node userdata.
(wrapped in a table due to lua 5.1 restrictions)
Problem: `push_tree`, every time its called for the same TSTree with
`do_copy=false` argument, creates a new userdata for it. Each userdata,
when garbage collected, frees the same TSTree C object.
Solution: Add flag to userdata, which indicates, should C object,
which userdata points to, be freed, when userdata is garbage collected.
