Problem: LspNotify autocmds were not being triggered for didChange
requests when being used during undo/redo (and possibly other) actions.
autocmds are blocked when calling on_lines() callbacks while doing the
undo/redo action.
Solution: Defer firing the autocmd until after the action is complete.
This is closer to what existed before, but now there's a check in the
deferred function to only fire the autocmd if the client is still
active and the buffer is still attached, if applicable.
Problem: LspNotify never passed a buffer when executing the autocmds, so
buffer-local LspNotify autocmd subscriptions didn't have the correct buf
in the event metadata. It was also wrapped in a schedule() so the actual
autocmd was delayed until after the event loop.
This could result in the wrong buffer receiving the notification if
multiple LspNotify autocmds with buffer filters were added. Only the
"latest" one would actually receive non-buffer-filtered autocmds, not
the matching one. It also caused listeners to receive the notification
"out of sync" with when the notification is actually sent. If a buffer
is being deleted (which fires a textDocument/didClose notification), the
notification is scheduled and fired after the buffer is already gone.
Solution: For LSP notifications that pertain to a particular buffer, set
it when executing the LspNotify autocmds so the callback functions that
are filtered on that buffer will get the correct notifications and the
metadata buf field will be correct. Additionally, there is no need to
wrap the LspNotify callback in vim.schedule when it can be called inline
when the notification to the rpc server is fired.
This is tested by removing now-unnecessary autocmds from semantic tokens
(InsertEnter and BufWinEnter should no longer be necessary now that
requests are fired by LspNotify). Without this fix, simply modifying a
buffer doesn't actually trigger LspNotify correctly, and the test for
that fails.
Problem: When multiline semantic token support was introduced, the loop
that finds the end line for a particular token didn't sanitize the token
length sent back by the LSP server. If the server returned an overflowed
length (near uint32 max), neovim would burn cpu and loop for an
extremely long time while trying to find the "end line" represented by
the massively large token, causing neovim to seemingly hang.
Solution: Stop looping once the calculated end_line reaches the actual
last line of the buffer.
Fixes#36257
Problem: The refactor to use Capability left around some cruft and
semi-broken configuration for debounce.
Solution: Clean up now-unnecessary helper methods and simplify
deprecated ones to pass through to the non-deprecated ones. `debounce`
now defaults to 200 for all buffers but is overridable via the
deprecated start() method, which continues to take the max value
specified for any client attached to the buffer.
If we wish to expose changing the debounce in a non-deprecated way, we
will need to consider a "configuration" function, or even a bespoke
method to set the debounce time on the main metaclass (or provide
options to override for a particular buffer). General configuration of
specific LSP features is an as-of-yet unsolved problem.
Problem: The semantic token module is using its own debounce timer for
the buffer on_lines event. If its internal debounce is shorter than the
changetracking module's debounce, it's possible for semantic token
requests to fire for changed buffers before the textDocument/didChange
notification is sent to the server.
Solution: Trigger semantic token requests from the LspNotify autocmd
when the method is the didChange or didOpen notifications, which
enforces a strict happens-before relationship for the sync change
notification followed by a semantic token request.
Note: There is still an internal debounce mechanism in the semantic
token module to handle other debouncing needs specific to its
functionality, such as debouncing server refresh notifications and
handling WinScrolled events when using range requests.
Problem:
In autocmd examples, using "args" as the event-object name is vague and
may be confused with a user-command.
Solution:
Use "ev" as the conventional event-object name.
The wait added in #37853 doesn't seem to do anything as request is sent
immediately on InsertLeave, and the number 4 also seems wrong. Instead,
the actual cause for the flakiness that the feed() (and hence the buffer
change) may arrive before the scheduled initialization of capabilities,
causing there be only only one textDocument/semanticTokens/full request
instead of two.
From the LSP Spec:
> There are two uses cases where it can be beneficial to only compute
> semantic tokens for a visible range:
>
> - for faster rendering of the tokens in the user interface when a user
> opens a file. In this use case, servers should also implement the
> textDocument/semanticTokens/full request as well to allow for flicker
> free scrolling and semantic coloring of a minimap.
> - if computing semantic tokens for a full document is too expensive,
> servers can only provide a range call. In this case, the client might
> not render a minimap correctly or might even decide to not show any
> semantic tokens at all.
This commit unifies the usage of range and full/delta requests as
recommended by the LSP spec and aligns neovim with the way other LSP
clients use these request types for semantic tokens.
When a server supports range requests, neovim will simultaneously send a
range request and a full/delta request when first opening a file, and
will continue to issue range requests until a full response is
processed. At that point, range requests cease and full (or delta)
requests are used going forward. The range request should allow servers
to return a result faster for quicker highlighting of the file while it
works on the potentially more expensive full result. If a server decides
the full result is too expensive, it can just error out that request,
and neovim will continue to use range requests.
This commit also fixes and cleans up some other things:
- gen_lsp: registrationMethod or registrationOptions imply dynamic
registration support
- move autocmd creation/deletion to on_attach/on_detach
- debounce requests due to server refresh notifications
- fix off by one issue in tokens_to_ranges() iteration
* cache all tokens from various range requests for a given document
version
- all new token highlights are merged with previous highlights to
maintain order and the "marked" property
- this allows the tokens to stop flickering once they've loaded once
per document version
* abandon the processing coroutine if the request_id has changed instead
of relying only on the document version
- this will improve efficiency if a new range request is made while a
previous one was processing its result
* apply new highlights from processing coroutine directly to the current
result when the version hasn't changed
- this allows new highlights to be immediately drawable once they've
processed instead of waiting for the whole response to be processed
at once
* rpc layer was changed to provide the request ID back in success
callbacks, which is then provided as a request_id field on the handler
context to lsp handlers
Problem:
Nvim supports `textDocument/semanticTokens/full` and `…/full/delta`
already, but most servers don't support `…/full/delta` and Nvim will try
to request and process full semantic tokens response on every buffer
change. Even though the request is debounced, there is noticeable lag if
the token response is large (in a big file).
Solution:
Support `textDocument/semanticTokens/range`, which requests semantic
tokens for visible screen only.
Before calling "attach" a screen object is just a dummy container for
(row, col) values whose purpose is to be sent as part of the "attach"
function call anyway.
Just create the screen in an attached state directly. Keep the complete
(row, col, options) config together. It is still completely valid to
later detach and re-attach as needed, including to another session.
This reduces the number of nil checks around buf_versions usage
Test changes were lifted from 5c33815
Co-authored-by: Mathias Fussenegger <f.mathias@zignar.net>
* Revert "fix(lsp): account for changedtick version gap on modified reset (#29170)"
This reverts commit 2e6d295f79.
* Revert "refactor(lsp): replace util.buf_versions with changedtick (#28943)"
This reverts commit 5c33815448.
`lsp.util.buf_versions` was already derived from changedtick (`on_lines`
from `buf_attach` synced the version)
As far as I can tell there is no need to keep track of the state in a
separate table.
Specifically, functions that are run in the context of the test runner
are put in module `test/testutil.lua` while the functions that are run
in the context of the test session are put in
`test/functional/testnvim.lua`.
Closes https://github.com/neovim/neovim/issues/27004.
`exec_lua` makes code slighly harder to read, so it's beneficial to
remove it in cases where it's possible or convenient.
Not all `exec_lua` calls should be removed even if the test passes as it
changes the semantics of the test even if it happens to pass.
From https://github.com/neovim/neovim/pull/28155#discussion_r1548185779:
"Note for tests like this, which fundamentally are about conversion, you
end up changing what conversion you are testing. Even if the result
happens to be same (as they often are, as we like the rules to be
consistent if possible), you are now testing the RPC conversion rules
instead of the vim script to in-process lua conversion rules."
From https://github.com/neovim/neovim/pull/28155#discussion_r1548190152:
"A test like this specifies that the cursor is valid immediately and not
after a separate cycle of normal (or an other input-processing) mode."
Added the following LSP semantic token types to be linked to highlight
groups by default:
* @lsp.type.event
* @lsp.type.keyword
* @lsp.type.modifier
* @lsp.type.number
* @lsp.type.operator
* @lsp.type.regexp
* @lsp.type.string
This is the command invoked repeatedly to make the changes:
:%s/^\(.*\)|\%(\*\(\d\+\)\)\?$\n\1|\%(\*\(\d\+\)\)\?$/\=submatch(1)..'|*'..(max([str2nr(submatch(2)),1])+max([str2nr(submatch(3)),1]))/g
feat(lsp)!: change semantic token highlighting
Change the default highlights used, and add more highlights per token.
Add an LspTokenUpdate event and a highlight_token function.
:Inspect now shows any highlights applied by token highlighting rules,
default or user-defined.
BREAKING CHANGE: change the default highlight groups used by semantic
token highlighting.
redraw! redraws the entire screen instead of just the windows with
the buffer which were actually changed.
I considered trying to calculating the range for the delta
but it looks tricky. Could a follow-up.
Apply semantic token modifiers as separate extmarks with corresponding
highlight groups (e.g., `@readonly`). This is a low-effort PR to enable
the most common use cases (applying, e.g., italics or backgrounds on top
of type highlights; language-specific fallbacks like `@global.lua` are
also available). This can be replaced by more complicated selector-style
themes later on.
Instead of testing for every possible modifier type, only test bits up
to the highest set in the token array. Saves many bit ops and
comparisons when there are no modifiers or when the highest set bit is a
lower bit than the highest possible in the legend on average.
Can be further simplified when non-luaJIT gets the full bit module (see #21222)
The spec indicates that the response may be `null`, but it doesn't
really say what a `null` response means. Since neovim raises an error if
the response is `null`, I figured that ignoring it would be the safest
bet.
Co-authored-by: Mathias Fussenegger <f.mathias@zignar.net>
1. The algorithm for applying edits was slightly incorrect. It needs to
preserve the original token list as the edits are applied instead of
mutating it as it iterates. From the spec:
Semantic token edits behave conceptually like text edits on
documents: if an edit description consists of n edits all n edits are
based on the same state Sm of the number array. They will move the
number array from state Sm to Sm+1.
2. Schedule the semantic token engine start() call in the
client._on_attach() function so that users who schedule_wrap() their
config.on_attach() functions (like nvim-lspconfig does) can still
disable semantic tokens by deleting the semanticTokensProvider from
their server capabilities.
* credit to @smolck and @theHamsta for their contributions in laying the
groundwork for this feature and for their work on some of the helper
utility functions and tests
