This partially reverts 0b8a72b739,
that is unreverts 15e77a56b7
"priority" is an internal neovim concept which does not occur in shared
queries. Ideally a single priority space should eventually be enough
for our needs. But as we don't want to poke at the usages of
priorities right now in the wider ecosystem,
introduce the "subpriorities" so that treesitter code can distinguish
highlights of the same priorities with different tree nesting depth.
This mainly affects `injection.combined` as parent-tree nodes might appear
in the middle of child-tree nodes which otherwise is not possible.
Continuing the work of #31400
That PR allowed the provider to be invoked multiple times per line.
We want only to do that when there actually is more data later on the
line. Additionally, we want to skip over lines which contain no new
highlight items. The TS query cursor already tells us what the next
position with more data is, so there is no need to reinvoke the range
callback before that.
NB: this removes the double buffering introduced in #32619 which
is funtamentally incompatible with this (nvim core is supposed to keep
track of long ranges by itself, without requiring a callback reinvoke
blitz). Need to adjust the priorities some other way to fix the same issue.
Problem: The document of nvim_buf_get_extmark currently lacks the
following:
- "details" directory: nvim_buf_get_extmarks() allows an option details
to get a "details" directory in result, but it doesn't mention where
that "details" directory is, what fields does it have.
Solution: Add docs for "details" directory in nvim_buf_get_extmarks()
- Problem: It's not clear for new plugin developers that `:help` uses
a help-tags file for searching the docs, generated by `:helptags`.
- Solution: Hint to the |:helptags| docs for regenerating the tags
file for their freshly written documentation.
Co-authored-by: Yochem van Rosmalen <git@yochem.nl>
These are not needed after #35129 but making uncrustify still play nice
with them was a bit tricky.
Unfortunately `uncrustify --update-config-with-doc` breaks strings
with backslashes. This issue has been reported upstream,
and in the meanwhile auto-update on every single run has been disabled.
- Render Lua types in api.txt.
- Added `DictAs(name)` API type which acts the same as `Dict` (no parens)
when generating the dispatchers, but acts the same as `Dict(name)`
when generating docs.
- Added `Tuple(...)` API type which is the treated the as `Array` for
generating the dispatchers, but is used to document richer types.
- Added `Enum(...)` API type to better document enums
- Improve typing of some API functions.
- Improve c_grammar to properly parse API types and replace string pattern
logic in the parsers.
- Removed all the hardcoded type overrides in gen_eval_files.lua
Problem: When iterating in reverse with {start} > {end} in
`nvim_buf_get_extmarks()`, marks that overlap {start} and are
greater than {end} are included in the return value twice.
Marks that overlap {end} and do not overlap {start} are not
not included in the return value at all. Marks are not
actually returned in a meaningful "traversal order".
Solution: Rather than actually iterating in reverse, (also possible but
requires convoluted conditions and would require fetching
overlapping marks for both the {start} and {end} position,
while still ending up with non-traversal ordered marks),
iterate normally and reverse the return value.
TSHighlighter now places marks for conceal_lines metadata. A new
internal decor provider callback _on_conceal_line was added that
instructs the highlighter to place conceal_lines marks whenever the
editor needs to know whether a line is concealed. The bundled markdown
queries use conceal_lines metadata to conceal code block fence lines.
Problem: Right aligned virtual text can cover up buffer text if virtual
text is too long
Solution: An additional option for `virt_text_pos` called
`eol_right_align` has been added to truncate virtual text if it would
have otherwise covered up buffer text. This ensures the virtual text
extends no further left than EOL.
This has been possible in the "backend" for a while but
API was missing.
Followup: we will need a `details2=true` mode for `nvim_get_hl_id_by_name`
to return information in a way forward compatible with even further
enhancements.
This was kept for a while as it was a useful short hand and initially
matched what highlights what actually properly implemented. But now
|vim.hl.range()| is a better high-level shorthand with full support for
native multi-line ranges.
Problem: Documentation for "hl_group" in nvim_buf_set_extmark() is
unclear. "hl_group" in nvim_echo() does not accept
highlight group id.
Solution: Move documentation for highlight group name/id to first
mention of hl_group. Update nvim_echo() to accept highlight
group id.
In the api_info() output:
:new|put =map(filter(api_info().functions, '!has_key(v:val,''deprecated_since'')'), 'v:val')
...
{'return_type': 'ArrayOf(Integer, 2)', 'name': 'nvim_win_get_position', 'method': v:true, 'parameters': [['Window', 'window']], 'since': 1}
The `ArrayOf(Integer, 2)` return type didn't break clients when we added
it, which is evidence that clients don't use the `return_type` field,
thus renaming Dictionary => Dict in api_info() is not (in practice)
a breaking change.
Use the grapheme break algorithm from utf8proc to support grapheme
clusters from recent unicode versions.
Handle variant selector VS16 turning some codepoints into double-width
emoji. This means we need to use ptr2cells rather than char2cells when
possible.
Problem:
The nvim_win_xx_ns function family introduced in ba0370b1d7
needs more bake-time. Currently it's narrowly defined for windows, but
other scopes ("buffer") and features are likely in the future.
Solution:
- Rename the API with double-underscore to mark it as EXPERIMENTAL.
TODO/FUTURE:
- Rename and change the signature to support more than just "window"
scope, and for other flexibility.
- Open question: we could choose either:
- "store scopes on namespaces", or
- "store namespaces on scopes (w:/b:/…)"
This avoids redraw when adding/removing an empty namespace for a window.
This also avoids marktree traversal when clearing a namespace that has
already been cleared, which is added as a benchmark.
This reverts commit 15e77a56b7.
Subpriorities were added in https://github.com/neovim/neovim/pull/27131
as a mechanism for enforcing query order when using iter_matches in the
Tree-sitter highlighter. However, iter_matches proved to have too many
complications to use in the highlighter so we eventually reverted back
to using iter_captures (https://github.com/neovim/neovim/pull/27901).
Thus, subpriorities are no longer needed and can be removed.
A lot of functions in move.c only worked for curwin, alternatively
took a `wp` arg but still only work if that happens to be curwin.
Refactor those that are needed for update_topline(wp) to work
for any window.
fixes#27723fixes#27720
Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
As only a few API functions make use of explicit freeing of the return
value, make it opt-in instead. The arena is always present under the
hood, so `Arena *arena` arg now doesn't mean anything other than getting
access to this arena. Also it is in principle possible to return an
allocated value while still using the arena as scratch space for other
stuff (unlikely, but there no reason to not allow it).
Currently having two separate memory strategies for API return values is
a bit unnecessary, and mostly a consequence of converting the hot spot
cases which needed it first. But there is really no downside to using
arena everywhere (which implies also directly using strings which are
allocated earlier or even statically, without copy).
There only restriction is we need to know the size of arrays in advance,
but this info can often be passed on from some earlier stage if it is
missing.
This collects some "small" cases. The more complex stuff will get a PR
each.
Implement api_keydict_to_dict as the complement to api_dict_to_keydict
Fix a conversion error when nvim_get_win_config gets called from lua,
where Float values "x" and "y" didn't get converted to lua numbers.
The "priority" field of extmarks can be used to set priorities of
extmarks which dictates which highlight group a range will actually have
when there are multiple extmarks applied. However, when multiple
extmarks have the same priority, the only way to enforce an actual
priority is through the order in which the extmarks are set.
It is not always possible or desirable to set extmarks in a specific
order, however, so we add a new "subpriority" field that explicitly
enforces the ordering of extmarks that have the same priority.
For now this will be used only to enforce priority of treesitter
highlights. A single node in a treesitter tree may match multiple
captures, in which case that node will have multiple extmarks set. The
order in which captures are returned from the treesitter API is not
_necessarily_ in the same order they are defined in a query file, so we
use the new subpriority field to force that ordering.
For now subpriorites are not documented and are not meant to be used by
external code, and it only applies to ephemeral extmarks. We indicate
the "private" nature of subpriorities by prefixing the field name with
an "_".
