Problem: Every LSP capability that sends a request on a document change
to update its state was using its own buffer-local autocmd to do so.
That means there is a separate autocmd per buffer per active
capability/feature, and the actual code inside the autocmd callback was
virtually identical. The same problem occurred for capabilities that had
components to draw on the screen that had their own decoration providers.
Solution: Introduce a central `LspNotify` autocmd and decoration
provider in the capability module itself. New base class methods
`on_close` and `on_change` that take a client_id have been provided, for
`didClose` and `didOpen`/`didChange` notifications respectively. New
base class method `on_win` that takes topline and botline has been
provided to add extmarks or perform other work when lines of a buffer
are being drawn.
The autocmd callback loops through each active capability instance for
the buffer and calls a corresponding method with the triggering
client_id if it is currently attached to the buffer that triggered the
autocmd. The decoration provider just calls on_win for all active
capabilities on the buffer.
This slightly tweaks folding range to make use of these new client-id
specific callbacks.
Problem: The codelens LSP module was using its own raw buffer events and
its own debounce mechanism for refreshing code lens in attached buffers.
Solution: Switch the module to using the LspNotify autocmd events.
LspNotify fires just after document versions are synced with the server
and provides a built in debounce mechanism for changes.
Additionally, this fixes some bugs with the previous implementation:
1. The workspace/codeLens/refresh handler re-requested codelens for all
buffers but when the response came back, it forced an extra redraw
after clearing the work the handler had just done.
2. Document synchronization was reworked to be more resilient to
multiple clients providing codelens for a single buffer. The latest
document version is now separately tracked per client (and per
client's lenses per row) instead of for the buffer as a whole. This
allows the on_win() function to properly redraw all codelens even
when different clients' responses for a particular document version
come back at different times.
Problem:
Inlay hints used separate global and per-buffer bufstates tables and
bespoke global autocmds for managing the inlay hint state across buffers
and clients, duplicating the lifecycle logic already provided by the
Capability framework. This caused inconsistencies in how client state
was handled and inlay hint state lifecycle was managed compared to other
LSP features.
Solution:
Replace the ad-hoc bufstate tracking and global autocmds in
vim.lsp.inlay_hint with a proper InlayHint subclass of Capability.
This also refactors the way inlay hint state is managed and fixes bugs I
found while doing this:
1. For each line with inlay hints, the list of the hints along with
whether they have been applied is stored in a current result on the
client state. This allows the on_win decorator to clear all inlay
hints for an old document version once, and then re-add the new
version's hints line-by-line as they are drawn to the screen,
modeling the semantic tokens module.
2. It fixes problems with mixing results from multiple clients attached
to the buffer by fully moving each client's state to its own table.
Previously, only the most recent document version used to populate a
line's inlay hints was stored, but there was no distinction for which
client the hints may have come from. (Fixes#36318)
3. It fixes the workspace/inlayHint/refresh server->client notification
behavior. Previously it would only re-request inlay hints for buffers
currently displayed in a window but would not invalidate them in
non-displayed buffers (or provide any mechanism for those buffers to
re-request at a later time). Model semantic token module here again
by invalidating all buffers, and adding a BufWinEnter autocmd to
refresh hints.
4. Add a mechanism to cancel in-flight requests if a new request for a
newer document version is made before the last one returned
5. Handle stale results by simply dropping them.
Problem: reset_timer() was being called without checking for whether the
client state for the client_id still existed. debounce_request() starts
a timer that defers a call to send_request() which then calls
reset_timer(). If the timer fires after the client_state is erased, then
the deferred function attempts to dereference the timer on a nil client
state.
Solution: Change reset_timer to take a state directly so it can't be nil
and move the reset_timer() call inside a guard that ensures state
exists. Additionally, reset a client's timer when the client detaches so
it doesn't become dangling.
Problem: A previous refactor removed the BufWinEnter autocmd that
initiated a token request. When an LSP server sends a refresh
notification, then buffers that aren't shown in any window lost their
only trigger to request new tokens.
Solution: Add the BufWinEnter autocmd back which simply requests tokens
for all clients attached to the buffer.
Problem: LspNotify will fire for any attached client. If there's at
least two clients where one has semantic tokens enabled and one that
doesn't, the disabled one will get the LspNotify requests but won't have
a client state.
Solution: Only process LspNotify autocmds if there's a client state to
act on.
Fixes#40448
fix(lsp): define autocmds for capabilities in new(), not on_attach()
Problem: Defining autocmds in on_attach() caused issues when multiple
clients provide the same capability for a buffer. Each attaching client
would "replace" the previously defined one since they are all identical.
Then the first one to detach clears them out and any remaining attached
clients would no longer trigger the autocmd for the capability. Further,
the semantic tokens module itself didn't quite work with multiple
clients since any LspNotify (from any client) would send a token request
to all attached clients with no differentiation and the debounce timer
was shared across all clients.
Solution: Always define the buffer-local autocmds in the capability's
`new()` function, and don't mess with the autocmds in on_attach or
on_detach. The capability framework itself will clear the autocmds when
the last client detaches. Also, refactor a bit of the semantic tokens
module so that the various methods take a specific client_id to perform
the work on, and split out timers so each client has its own.
Problem:
Diagnostic tracking used a separate bufstates table and manual
LspDetach/LspNotify autocmd management via _enable()/_refresh(),
duplicating the lifecycle logic already provided by the Capability
framework. This caused inconsistencies in how client attach/detach and
buffer teardown were handled compared to other LSP features.
Solution:
Replace the ad-hoc bufstate tracking and _enable/_refresh pattern in
vim.lsp.diagnostic with a proper Diagnostics subclass of Capability.
This cleans up a few random places in the main lsp module and client
module that were poking the diagnostics. It also fixes some pre-existing
bugs and inconsistencies that were discovered:
- Refresh diagnostics immediately on attach instead of lazily by the
first didOpen/didChange notification
- Fix Capability.active lookup in M.enable() to key by it_bufnr instead
of the filter bufnr
- Set lsp defaults before calling the _text_document_did_open_handler in
Client:on_attach() so defaults are there before any lsp notification
occurs
- Log (and return early) on any error from a diagnostic request result
instead of only returning early for server cancelled errors
Problem: LspNotify never passed a buffer when executing the autocmds, so
buffer-local LspNotify autocmd subscriptions didn't have the correct buf
in the event metadata. It was also wrapped in a schedule() so the actual
autocmd was delayed until after the event loop.
This could result in the wrong buffer receiving the notification if
multiple LspNotify autocmds with buffer filters were added. Only the
"latest" one would actually receive non-buffer-filtered autocmds, not
the matching one. It also caused listeners to receive the notification
"out of sync" with when the notification is actually sent. If a buffer
is being deleted (which fires a textDocument/didClose notification), the
notification is scheduled and fired after the buffer is already gone.
Solution: For LSP notifications that pertain to a particular buffer, set
it when executing the LspNotify autocmds so the callback functions that
are filtered on that buffer will get the correct notifications and the
metadata buf field will be correct. Additionally, there is no need to
wrap the LspNotify callback in vim.schedule when it can be called inline
when the notification to the rpc server is fired.
This is tested by removing now-unnecessary autocmds from semantic tokens
(InsertEnter and BufWinEnter should no longer be necessary now that
requests are fired by LspNotify). Without this fix, simply modifying a
buffer doesn't actually trigger LspNotify correctly, and the test for
that fails.
Problem: When multiline semantic token support was introduced, the loop
that finds the end line for a particular token didn't sanitize the token
length sent back by the LSP server. If the server returned an overflowed
length (near uint32 max), neovim would burn cpu and loop for an
extremely long time while trying to find the "end line" represented by
the massively large token, causing neovim to seemingly hang.
Solution: Stop looping once the calculated end_line reaches the actual
last line of the buffer.
Fixes#36257
Problem: The refactor to use Capability left around some cruft and
semi-broken configuration for debounce.
Solution: Clean up now-unnecessary helper methods and simplify
deprecated ones to pass through to the non-deprecated ones. `debounce`
now defaults to 200 for all buffers but is overridable via the
deprecated start() method, which continues to take the max value
specified for any client attached to the buffer.
If we wish to expose changing the debounce in a non-deprecated way, we
will need to consider a "configuration" function, or even a bespoke
method to set the debounce time on the main metaclass (or provide
options to override for a particular buffer). General configuration of
specific LSP features is an as-of-yet unsolved problem.
Problem: The semantic token module is using its own debounce timer for
the buffer on_lines event. If its internal debounce is shorter than the
changetracking module's debounce, it's possible for semantic token
requests to fire for changed buffers before the textDocument/didChange
notification is sent to the server.
Solution: Trigger semantic token requests from the LspNotify autocmd
when the method is the didChange or didOpen notifications, which
enforces a strict happens-before relationship for the sync change
notification followed by a semantic token request.
Note: There is still an internal debounce mechanism in the semantic
token module to handle other debouncing needs specific to its
functionality, such as debouncing server refresh notifications and
handling WinScrolled events when using range requests.
Problem:
Flickering may occur when paging up/down in big files, as ranges for semantic
tokens are requested. This happens with LSP servers like gopls which return
"/full" semantic tokens if the file is too big, where we fall back to
viewport-range token retrievals.
Solution:
Broaden the requested ranges to one viewport of "overscan" on each side plus
some padding if possible:
(viewport_topline - viewport_height)..(viewport_botline + viewport_height)
Problem: Redrawing when a loaded buffer is not shown in any window on
the current tabpage.
Solution: Check that buffer is shown in a (normal) window before redrawing.
Problem:
`nvim_create_autocmd` is too verbose and its `callback` requires extra
"nesting".
Solution:
Introduce `nvim_on`. Start using it internally. Then we can get a feel
for how it should look before making it public.
Problem:
`:help dev-name-common` states that "buf" should be used instead of
"buffer" but there are cases where buffer is mentioned in the lua API.
Solution:
- Rename occurrences of "buffer" to "buf" for consistency with the
documentation.
- Support (but deprecate) "buffer" for backwards compatibility.
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
Problem
The format of LSP log messages is inconsistent; some include underscores, while others are not logged at all.
Solution
Standardize log recording and unify the log message prefixes with the module names.
Problem:
In autocmd examples, using "args" as the event-object name is vague and
may be confused with a user-command.
Solution:
Use "ev" as the conventional event-object name.
From the LSP Spec:
> There are two uses cases where it can be beneficial to only compute
> semantic tokens for a visible range:
>
> - for faster rendering of the tokens in the user interface when a user
> opens a file. In this use case, servers should also implement the
> textDocument/semanticTokens/full request as well to allow for flicker
> free scrolling and semantic coloring of a minimap.
> - if computing semantic tokens for a full document is too expensive,
> servers can only provide a range call. In this case, the client might
> not render a minimap correctly or might even decide to not show any
> semantic tokens at all.
This commit unifies the usage of range and full/delta requests as
recommended by the LSP spec and aligns neovim with the way other LSP
clients use these request types for semantic tokens.
When a server supports range requests, neovim will simultaneously send a
range request and a full/delta request when first opening a file, and
will continue to issue range requests until a full response is
processed. At that point, range requests cease and full (or delta)
requests are used going forward. The range request should allow servers
to return a result faster for quicker highlighting of the file while it
works on the potentially more expensive full result. If a server decides
the full result is too expensive, it can just error out that request,
and neovim will continue to use range requests.
This commit also fixes and cleans up some other things:
- gen_lsp: registrationMethod or registrationOptions imply dynamic
registration support
- move autocmd creation/deletion to on_attach/on_detach
- debounce requests due to server refresh notifications
- fix off by one issue in tokens_to_ranges() iteration
* cache all tokens from various range requests for a given document
version
- all new token highlights are merged with previous highlights to
maintain order and the "marked" property
- this allows the tokens to stop flickering once they've loaded once
per document version
* abandon the processing coroutine if the request_id has changed instead
of relying only on the document version
- this will improve efficiency if a new range request is made while a
previous one was processing its result
* apply new highlights from processing coroutine directly to the current
result when the version hasn't changed
- this allows new highlights to be immediately drawable once they've
processed instead of waiting for the whole response to be processed
at once
* rpc layer was changed to provide the request ID back in success
callbacks, which is then provided as a request_id field on the handler
context to lsp handlers
By simplifying the way range is supported, we can fix a couple issues as
well as making it less complex and more efficient:
* For non-range LSP servers, don't send requests on WinScrolled. The
semantic tokens module has been reworked to only send one active
request at a time, as it was before range support was added. If range
is not supported, then send_request() only fires if there's been a
change to the buffer's document version.
* Cache the server's support of range and delta requests when attaching
to a buffer to save the lookup on each request.
* Range requests always use the visible window, so just use that for the
`range` param when sending requests when range is supported by the
server. This reduces the API surface area of send_request().
* Debounce the WinScrolled autocmd requests in the same the way requests
are debounced when the buffer contents are changing. Should allow
scrolling via mouse wheel or holding down "j" or "k" work a bit
smoother.
The previous iteration of range support allowed multiple active requests
to be in progress simultaneously. However, a bug was preventing any but
the most recent request to actually apply to the client's highlighting
state so that complexity was unused. It was effectively only using one
active request at a time but was just using range requests on
WinScrolled events instead of a full (or delta) request when the
document version changed.
Problem:
Our LSP type system didnt have a concept of RegistrationMethods, this is where the method to dynamically register for a capability is sent to a different method endpoint then is used to call it. Eg `textDocument/semanticTokens` rather than the specific full/range/delta methods
Solution:
Extended generator to create `vim.lsp.protocol.Methods.Registration` with these registration methods. Also extend `_request_name_to_client_capability` to cover these methods. Adjust typing to suit
Allow the `request` parameter in `tokens_to_ranges` to be `nil` and
update version checking logic accordingly. This prevents errors when
the request is not present and improves robustness of semantic token
handling.
Problem:
Nvim supports `textDocument/semanticTokens/full` and `…/full/delta`
already, but most servers don't support `…/full/delta` and Nvim will try
to request and process full semantic tokens response on every buffer
change. Even though the request is debounced, there is noticeable lag if
the token response is large (in a big file).
Solution:
Support `textDocument/semanticTokens/range`, which requests semantic
tokens for visible screen only.
Problem:
Closes#31453
Solution:
Introduce `vim.lsp.Capability`, which may serve as the base class for
all LSP features that require caching data. it
- was created if there is at least one client that supports the specific method;
- was destroyed if all clients that support the method were detached.
- Apply the refactor for `folding_range.lua` and `semantic_tokens.lua`.
- Show active features in :checkhealth.
Future:
I found that these features that are expected to be refactored by
`vim.lsp.Capability` have one characteristic in common: they all send
LSP requests once the document is modified. The following code is
different, but they are all for this purpose.
- semantic tokens:
fb8dba413f/runtime/lua/vim/lsp/semantic_tokens.lua (L192-L198)
- inlay hints, folding ranges, document color
fb8dba413f/runtime/lua/vim/lsp/inlay_hint.lua (L250-L266)
I think I can sum up this characteristic as the need to keep certain
data synchronized with the latest version computed by the server.
I believe we can handle this at the `vim.lsp.Capability` level, and
I think it will be very useful.
Therefore, my next step is to implement LSP request sending and data
synchronization on `vim.lsp.Capability`, rather than limiting it to the
current create/destroy data approach.
Problem: There is no check for buffer validity before processing
semantic tokens response. This can lead to `Invalid buffer id` error
if processing request takes a long time and the buffer is wiped out.
For example, this can happen after by accident navigating to a buffer
from different project which leads to first loading project's
workspace and *then* processing semantic tokens. During that time
a buffer can be wiped out, as navigation to it was by accident.
Solution: Add extra check for buffer validity before processing semantic
tokens response.
* deprecate old signatures
* move to new str_byteindex/str_utfindex signature
* use single-underscore name (double-underscore is reserved for Lua itself)
