Problem: LspNotify will fire for any attached client. If there's at
least two clients where one has semantic tokens enabled and one that
doesn't, the disabled one will get the LspNotify requests but won't have
a client state.
Solution: Only process LspNotify autocmds if there's a client state to
act on.
Fixes#40448
fix(lsp): define autocmds for capabilities in new(), not on_attach()
Problem: Defining autocmds in on_attach() caused issues when multiple
clients provide the same capability for a buffer. Each attaching client
would "replace" the previously defined one since they are all identical.
Then the first one to detach clears them out and any remaining attached
clients would no longer trigger the autocmd for the capability. Further,
the semantic tokens module itself didn't quite work with multiple
clients since any LspNotify (from any client) would send a token request
to all attached clients with no differentiation and the debounce timer
was shared across all clients.
Solution: Always define the buffer-local autocmds in the capability's
`new()` function, and don't mess with the autocmds in on_attach or
on_detach. The capability framework itself will clear the autocmds when
the last client detaches. Also, refactor a bit of the semantic tokens
module so that the various methods take a specific client_id to perform
the work on, and split out timers so each client has its own.
Problem:
Diagnostic tracking used a separate bufstates table and manual
LspDetach/LspNotify autocmd management via _enable()/_refresh(),
duplicating the lifecycle logic already provided by the Capability
framework. This caused inconsistencies in how client attach/detach and
buffer teardown were handled compared to other LSP features.
Solution:
Replace the ad-hoc bufstate tracking and _enable/_refresh pattern in
vim.lsp.diagnostic with a proper Diagnostics subclass of Capability.
This cleans up a few random places in the main lsp module and client
module that were poking the diagnostics. It also fixes some pre-existing
bugs and inconsistencies that were discovered:
- Refresh diagnostics immediately on attach instead of lazily by the
first didOpen/didChange notification
- Fix Capability.active lookup in M.enable() to key by it_bufnr instead
of the filter bufnr
- Set lsp defaults before calling the _text_document_did_open_handler in
Client:on_attach() so defaults are there before any lsp notification
occurs
- Log (and return early) on any error from a diagnostic request result
instead of only returning early for server cancelled errors
Problem: LspNotify never passed a buffer when executing the autocmds, so
buffer-local LspNotify autocmd subscriptions didn't have the correct buf
in the event metadata. It was also wrapped in a schedule() so the actual
autocmd was delayed until after the event loop.
This could result in the wrong buffer receiving the notification if
multiple LspNotify autocmds with buffer filters were added. Only the
"latest" one would actually receive non-buffer-filtered autocmds, not
the matching one. It also caused listeners to receive the notification
"out of sync" with when the notification is actually sent. If a buffer
is being deleted (which fires a textDocument/didClose notification), the
notification is scheduled and fired after the buffer is already gone.
Solution: For LSP notifications that pertain to a particular buffer, set
it when executing the LspNotify autocmds so the callback functions that
are filtered on that buffer will get the correct notifications and the
metadata buf field will be correct. Additionally, there is no need to
wrap the LspNotify callback in vim.schedule when it can be called inline
when the notification to the rpc server is fired.
This is tested by removing now-unnecessary autocmds from semantic tokens
(InsertEnter and BufWinEnter should no longer be necessary now that
requests are fired by LspNotify). Without this fix, simply modifying a
buffer doesn't actually trigger LspNotify correctly, and the test for
that fails.
Problem: When multiline semantic token support was introduced, the loop
that finds the end line for a particular token didn't sanitize the token
length sent back by the LSP server. If the server returned an overflowed
length (near uint32 max), neovim would burn cpu and loop for an
extremely long time while trying to find the "end line" represented by
the massively large token, causing neovim to seemingly hang.
Solution: Stop looping once the calculated end_line reaches the actual
last line of the buffer.
Fixes#36257
Problem: The refactor to use Capability left around some cruft and
semi-broken configuration for debounce.
Solution: Clean up now-unnecessary helper methods and simplify
deprecated ones to pass through to the non-deprecated ones. `debounce`
now defaults to 200 for all buffers but is overridable via the
deprecated start() method, which continues to take the max value
specified for any client attached to the buffer.
If we wish to expose changing the debounce in a non-deprecated way, we
will need to consider a "configuration" function, or even a bespoke
method to set the debounce time on the main metaclass (or provide
options to override for a particular buffer). General configuration of
specific LSP features is an as-of-yet unsolved problem.
Problem: The semantic token module is using its own debounce timer for
the buffer on_lines event. If its internal debounce is shorter than the
changetracking module's debounce, it's possible for semantic token
requests to fire for changed buffers before the textDocument/didChange
notification is sent to the server.
Solution: Trigger semantic token requests from the LspNotify autocmd
when the method is the didChange or didOpen notifications, which
enforces a strict happens-before relationship for the sync change
notification followed by a semantic token request.
Note: There is still an internal debounce mechanism in the semantic
token module to handle other debouncing needs specific to its
functionality, such as debouncing server refresh notifications and
handling WinScrolled events when using range requests.
Problem:
Flickering may occur when paging up/down in big files, as ranges for semantic
tokens are requested. This happens with LSP servers like gopls which return
"/full" semantic tokens if the file is too big, where we fall back to
viewport-range token retrievals.
Solution:
Broaden the requested ranges to one viewport of "overscan" on each side plus
some padding if possible:
(viewport_topline - viewport_height)..(viewport_botline + viewport_height)
Problem: Redrawing when a loaded buffer is not shown in any window on
the current tabpage.
Solution: Check that buffer is shown in a (normal) window before redrawing.
Problem:
`nvim_create_autocmd` is too verbose and its `callback` requires extra
"nesting".
Solution:
Introduce `nvim_on`. Start using it internally. Then we can get a feel
for how it should look before making it public.
Problem:
`:help dev-name-common` states that "buf" should be used instead of
"buffer" but there are cases where buffer is mentioned in the lua API.
Solution:
- Rename occurrences of "buffer" to "buf" for consistency with the
documentation.
- Support (but deprecate) "buffer" for backwards compatibility.
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
Problem
The format of LSP log messages is inconsistent; some include underscores, while others are not logged at all.
Solution
Standardize log recording and unify the log message prefixes with the module names.
Problem:
In autocmd examples, using "args" as the event-object name is vague and
may be confused with a user-command.
Solution:
Use "ev" as the conventional event-object name.
From the LSP Spec:
> There are two uses cases where it can be beneficial to only compute
> semantic tokens for a visible range:
>
> - for faster rendering of the tokens in the user interface when a user
> opens a file. In this use case, servers should also implement the
> textDocument/semanticTokens/full request as well to allow for flicker
> free scrolling and semantic coloring of a minimap.
> - if computing semantic tokens for a full document is too expensive,
> servers can only provide a range call. In this case, the client might
> not render a minimap correctly or might even decide to not show any
> semantic tokens at all.
This commit unifies the usage of range and full/delta requests as
recommended by the LSP spec and aligns neovim with the way other LSP
clients use these request types for semantic tokens.
When a server supports range requests, neovim will simultaneously send a
range request and a full/delta request when first opening a file, and
will continue to issue range requests until a full response is
processed. At that point, range requests cease and full (or delta)
requests are used going forward. The range request should allow servers
to return a result faster for quicker highlighting of the file while it
works on the potentially more expensive full result. If a server decides
the full result is too expensive, it can just error out that request,
and neovim will continue to use range requests.
This commit also fixes and cleans up some other things:
- gen_lsp: registrationMethod or registrationOptions imply dynamic
registration support
- move autocmd creation/deletion to on_attach/on_detach
- debounce requests due to server refresh notifications
- fix off by one issue in tokens_to_ranges() iteration
* cache all tokens from various range requests for a given document
version
- all new token highlights are merged with previous highlights to
maintain order and the "marked" property
- this allows the tokens to stop flickering once they've loaded once
per document version
* abandon the processing coroutine if the request_id has changed instead
of relying only on the document version
- this will improve efficiency if a new range request is made while a
previous one was processing its result
* apply new highlights from processing coroutine directly to the current
result when the version hasn't changed
- this allows new highlights to be immediately drawable once they've
processed instead of waiting for the whole response to be processed
at once
* rpc layer was changed to provide the request ID back in success
callbacks, which is then provided as a request_id field on the handler
context to lsp handlers
By simplifying the way range is supported, we can fix a couple issues as
well as making it less complex and more efficient:
* For non-range LSP servers, don't send requests on WinScrolled. The
semantic tokens module has been reworked to only send one active
request at a time, as it was before range support was added. If range
is not supported, then send_request() only fires if there's been a
change to the buffer's document version.
* Cache the server's support of range and delta requests when attaching
to a buffer to save the lookup on each request.
* Range requests always use the visible window, so just use that for the
`range` param when sending requests when range is supported by the
server. This reduces the API surface area of send_request().
* Debounce the WinScrolled autocmd requests in the same the way requests
are debounced when the buffer contents are changing. Should allow
scrolling via mouse wheel or holding down "j" or "k" work a bit
smoother.
The previous iteration of range support allowed multiple active requests
to be in progress simultaneously. However, a bug was preventing any but
the most recent request to actually apply to the client's highlighting
state so that complexity was unused. It was effectively only using one
active request at a time but was just using range requests on
WinScrolled events instead of a full (or delta) request when the
document version changed.
Problem:
Our LSP type system didnt have a concept of RegistrationMethods, this is where the method to dynamically register for a capability is sent to a different method endpoint then is used to call it. Eg `textDocument/semanticTokens` rather than the specific full/range/delta methods
Solution:
Extended generator to create `vim.lsp.protocol.Methods.Registration` with these registration methods. Also extend `_request_name_to_client_capability` to cover these methods. Adjust typing to suit
Allow the `request` parameter in `tokens_to_ranges` to be `nil` and
update version checking logic accordingly. This prevents errors when
the request is not present and improves robustness of semantic token
handling.
Problem:
Nvim supports `textDocument/semanticTokens/full` and `…/full/delta`
already, but most servers don't support `…/full/delta` and Nvim will try
to request and process full semantic tokens response on every buffer
change. Even though the request is debounced, there is noticeable lag if
the token response is large (in a big file).
Solution:
Support `textDocument/semanticTokens/range`, which requests semantic
tokens for visible screen only.
Problem:
Closes#31453
Solution:
Introduce `vim.lsp.Capability`, which may serve as the base class for
all LSP features that require caching data. it
- was created if there is at least one client that supports the specific method;
- was destroyed if all clients that support the method were detached.
- Apply the refactor for `folding_range.lua` and `semantic_tokens.lua`.
- Show active features in :checkhealth.
Future:
I found that these features that are expected to be refactored by
`vim.lsp.Capability` have one characteristic in common: they all send
LSP requests once the document is modified. The following code is
different, but they are all for this purpose.
- semantic tokens:
fb8dba413f/runtime/lua/vim/lsp/semantic_tokens.lua (L192-L198)
- inlay hints, folding ranges, document color
fb8dba413f/runtime/lua/vim/lsp/inlay_hint.lua (L250-L266)
I think I can sum up this characteristic as the need to keep certain
data synchronized with the latest version computed by the server.
I believe we can handle this at the `vim.lsp.Capability` level, and
I think it will be very useful.
Therefore, my next step is to implement LSP request sending and data
synchronization on `vim.lsp.Capability`, rather than limiting it to the
current create/destroy data approach.
Problem: There is no check for buffer validity before processing
semantic tokens response. This can lead to `Invalid buffer id` error
if processing request takes a long time and the buffer is wiped out.
For example, this can happen after by accident navigating to a buffer
from different project which leads to first loading project's
workspace and *then* processing semantic tokens. During that time
a buffer can be wiped out, as navigation to it was by accident.
Solution: Add extra check for buffer validity before processing semantic
tokens response.
* deprecate old signatures
* move to new str_byteindex/str_utfindex signature
* use single-underscore name (double-underscore is reserved for Lua itself)
Problem:
- `vim.highlight` module does not follow `:help dev-name-common`, which
documents the name for "highlight" as "hl".
- Shorter names are usually preferred.
Solution:
Rename `vim.highlight` to `vim.hl`.
This is not a breaking change until 2.0 (or maybe never).
Problem:
str_byteindex_enc could return an error if the index was longer than the
lline length. This was handled in each of the calls to it individually
Solution:
* Fix the call at the source level so that if the index is higher than
the line length, line length is returned as per LSP specification
* Remove pcalls on str_byteindex_enc calls. No longer needed now that
str_byteindex_enc has a bounds check.
This reduces the number of nil checks around buf_versions usage
Test changes were lifted from 5c33815
Co-authored-by: Mathias Fussenegger <f.mathias@zignar.net>
