These are not needed after #35129 but making uncrustify still play nice
with them was a bit tricky.
Unfortunately `uncrustify --update-config-with-doc` breaks strings
with backslashes. This issue has been reported upstream,
and in the meanwhile auto-update on every single run has been disabled.
Problem: fuzzy-matching can be improved
Solution: Implement a better fuzzy matching algorithm
(Girish Palya)
Replace fuzzy matching algorithm with improved fzy-based implementation
The
[current](https://www.forrestthewoods.com/blog/reverse_engineering_sublime_texts_fuzzy_match/)
fuzzy matching algorithm has several accuracy issues:
* It struggles with CamelCase
* It fails to prioritize matches at the beginning of strings, often
ranking middle matches higher.
After evaluating alternatives (see my comments
[here](https://github.com/vim/vim/issues/17531#issuecomment-3112046897)
and
[here](https://github.com/vim/vim/issues/17531#issuecomment-3121593900)),
I chose to adopt the [fzy](https://github.com/jhawthorn/fzy) algorithm,
which:
* Resolves the aforementioned issues.
* Performs better.
Implementation details
This version is based on the original fzy
[algorithm](https://github.com/jhawthorn/fzy/blob/master/src/match.c),
with one key enhancement: **multibyte character support**.
* The original implementation supports only ASCII.
* This patch replaces ascii lookup tables with function calls, making it
compatible with multibyte character sets.
* Core logic (`match_row()` and `match_positions()`) remains faithful to
the original, but now operates on codepoints rather than single-byte
characters.
Performance
Tested against a dataset of **90,000 Linux kernel filenames**. Results
(in milliseconds) show a **\~2x performance improvement** over the
current fuzzy matching algorithm.
```
Search String Current Algo FZY Algo
-------------------------------------------------
init 131.759 66.916
main 83.688 40.861
sig 98.348 39.699
index 109.222 30.738
ab 72.222 44.357
cd 83.036 54.739
a 58.94 62.242
b 43.612 43.442
c 64.39 67.442
k 40.585 36.371
z 34.708 22.781
w 38.033 30.109
cpa 82.596 38.116
arz 84.251 23.964
zzzz 35.823 22.75
dimag 110.686 29.646
xa 43.188 29.199
nha 73.953 31.001
nedax 94.775 29.568
dbue 79.846 25.902
fp 46.826 31.641
tr 90.951 55.883
kw 38.875 23.194
rp 101.575 55.775
kkkkkkkkkkkkkkkkkkkkkkkkkkkkk 48.519 30.921
```
```vim
vim9script
var haystack = readfile('/Users/gp/linux.files')
var needles = ['init', 'main', 'sig', 'index', 'ab', 'cd', 'a', 'b',
'c', 'k',
'z', 'w', 'cpa', 'arz', 'zzzz', 'dimag', 'xa', 'nha', 'nedax',
'dbue',
'fp', 'tr', 'kw', 'rp', 'kkkkkkkkkkkkkkkkkkkkkkkkkkkkk']
for needle in needles
var start = reltime()
var tmp = matchfuzzy(haystack, needle)
echom $'{needle}' (start->reltime()->reltimefloat() * 1000)
endfor
```
Additional changes
* Removed the "camelcase" option from both matchfuzzy() and
matchfuzzypos(), as it's now obsolete with the improved algorithm.
related: neovim/neovim#34101fixesvim/vim#17531closes: vim/vim#179007e0df5eee9
Co-authored-by: Girish Palya <girishji@gmail.com>
Problem: Autocompletion slow with include- and tag-completion
Solution: Refactor ins_compl_interrupted() to also check for timeout,
further refactor code to skip outputting message when
performing autocompletion (Girish Palya).
Running `vim *` in `vim/src` was slower than expected when
'autocomplete' was enabled. Include-file and tag-file completion
sources were not subject to the timeout check, causing unnecessary
delays.
So apply the timeout check to these sources as well, improving
autocompletion responsiveness, refactor find_pattern_in_path() to take
an additional "silent" argument, to suppress any messages.
closes: vim/vim#1796659e1d7f353
Co-authored-by: Girish Palya <girishji@gmail.com>
Problem: The maximum search count uses a hard-coded value of 99
(Andres Monge, Joschua Kesper)
Solution: Make it configurable using the 'maxsearchcount' option.
related: vim/vim#8855fixes: vim/vim#17527closes: vim/vim#17695b7b7fa04bf
Co-authored-by: Christian Brabandt <cb@256bit.org>
Problem: ext_messages is implemented to mimic the message grid
implementation w.r.t. scrolling messages, clearing scrolled
messages, hit-enter-prompts and replacing a previous message.
Meanwhile, an ext_messages UI may not be implemented in a way
where these events are wanted. Moreover, correctness of these
events even assuming a "scrolled message" implementation
depends on fragile "currently visible messages" global state,
which already isn't correct after a previous message was
supposed to have been overwritten (because that should not only
happen when `msg_scroll == false`).
Solution: - No longer attempt to keep track of the currently visible
messages: remove the `msg_ext(_history)_visible` variables.
UIs may remove messages pre-emptively (timer based), or never
show messages that don't fit a certain area in the first place.
- No longer emit the `msg(_history)_clear` events to clear
"scrolled" messages. This opens up the `msg_clear` event to
be emitted when messages should actually be cleared (e.g.
when the screen is cleared). May also be useful to emit before
the first message in an event loop cycle as a hint to the UI
that it is a new batch of messages (vim._extui currently
schedules an event to determine that).
- Set `replace_last` explicitly at the few callsites that want
this to be set to true to replace an incomplete status message.
- Don't store a "keep" message to be re-emitted.
Problem: negative matchfuzzy scores although there is a match
(Maxim Kim)
Solution: reset the score if a match has been found but the score is
negative (Girish Palya)
The fuzzy algorithm may miss some matches in long strings due to recursion
limits. As a result, the score can end up negative even when matches exist.
In such cases, reset the score to ensure it is non-negative.
fixes: #vim/vim#17449
closes: vim/vim#17469328332b0b0
Co-authored-by: Girish Palya <girishji@gmail.com>
Problem: search_stat not reset when pattern differs in case
(tahzibijafar)
Solution: use STRNCMP instead of MB_STRNICMP macro
There was a long standing todo comment, that using MB_STRNICMP is wrong.
So let's change it to STRNCMP() instead. Even if it not handle
multi-byte characters correctly, then Vim will rather recompute the
search stat, instead of re-using the old (and possibly wrong) value.
fixes: vim/vim#17312closes: vim/vim#17314670d0c1468
Co-authored-by: Christian Brabandt <cb@256bit.org>
Problem: Currently, 'smartcase' is respected when completing keywords
using <C-N>, <C-P>, <C-X><C-N>, and <C-X><C-P>. However, when
a user continues typing and the completion menu is filtered
using cached matches, 'smartcase' is not applied. This leads
to poor-quality or irrelevant completion suggestions, as shown
in the example below.
Solution: When filtering cached completion items after typing additional
characters, apply case-sensitive comparison if 'smartcase' is
enabled and the typed pattern includes uppercase characters.
This ensures consistent and expected completion behavior.
(Girish Palya)
closes: vim/vim#17271dc314053e1
Co-authored-by: Girish Palya <girishji@gmail.com>
Problem:
When searching for the next pattern via n/N/*/#, cursor
moves to cmdline, perceived as "flicker".
Solution:
Can ui_busy_start() and ui_busy_stop().
Problem: chain complete does not work when 'cot' includes fuzzy
and 'completefuzzycollect' collects wrong next word.
(Konfekt)
Solution: compl_startpos is not set correctly, remove next word check
in search_for_fuzzy_match (glepnir).
fixesvim/vim#17131fixesvim/vim#16942closes: vim/vim#17136cfe502c575
Co-authored-by: glepnir <glephunter@gmail.com>
Problem: Strange error with type for matchfuzzy() "camelcase".
Solution: Show the error "Invalid value for argument camelcase" instead
of "Invalid argument: camelcase" (zeertzjq).
Note that using tv_get_string() will lead to confusion, as when the
value cannot be converted to a string tv_get_string() will also give an
error about that, but "camelcase" takes a boolean, not a string. Also
don't use tv_get_string() for the "limit" argument above.
closes: vim/vim#16926c4815c157b
Problem: When searching for "Cur", CamelCase matches like "lCursor" score
higher than exact prefix matches like Cursor, which is
counter-intuitive (Maxim Kim).
Solution: Add a 'camelcase' option to matchfuzzy() that lets users disable
CamelCase bonuses when needed, making prefix matches rank higher.
(glepnir)
fixes: vim/vim#16504closes: vim/vim#1679728e40a7b55
Co-authored-by: glepnir <glephunter@gmail.com>
Problem: The current_pos.col was incorrectly updated to the length of
the matching text. This will cause the next search to start
from the wrong position.
Solution: current_pos has already been updated in search_str_in_line and
does not need to be changed (glepnir)
closes: vim/vim#169415753084042
Co-authored-by: glepnir <glephunter@gmail.com>
Problem: endless loop with completefuzzycollect and no match found
Solution: move pointer to line end and break loop
closes: vim/vim#16820dd42b05f8a
Co-authored-by: glepnir <glephunter@gmail.com>
Problem: potential out-of-memory issue in search.c
Solution: improve situation and refactor search.c slightly
(John Marriott)
- In function update_search_stat():
add a check for a theoretical null pointer reference, set and remember
the length of lastpat, remove the three calls to STRLEN() and use the
various string's associated lengths instead, add a check for an
out-of-memory condition.
- In function search_for_fuzz_match():
remove a call to strnsave() and thus avoid having to add a check for
an out-of-memory condition, also replace the call to STRLEN() by
ml_get_buf_len().
closes: vim/vim#16689b79fa3d9c8
Co-authored-by: John Marriott <basilisk@internode.on.net>
Problem: fuzzy completion does not work with default completion
Solution: Make it work (glepnir)
closes: vim/vim#151938159fb18a9
Cherry-pick insexpand.c changes from patch 9.1.0608.
N/A patch:
vim-patch:9.1.0632: MS-Windows: Compiler Warnings
Co-authored-by: glepnir <glephunter@gmail.com>
Problem: potential out-of-memory issue in search.c
Solution: improve situation and refactor search.c slightly
(John Marriott)
- In function update_search_stat():
add a check for a theoretical null pointer reference, set and remember
the length of lastpat, remove the three calls to STRLEN() and use the
various string's associated lengths instead, add a check for an
out-of-memory condition.
- In function search_for_fuzz_match():
remove a call to strnsave() and thus avoid having to add a check for
an out-of-memory condition, also replace the call to STRLEN() by
ml_get_buf_len().
closes: vim/vim#16689b79fa3d9c8
Co-authored-by: John Marriott <basilisk@internode.on.net>
Problem: fuzzy-matching does not prefer full match
(Maxim Kim)
Solution: add additional score for a full match
(glepnir)
fixes: vim/vim#15654closes: vim/vim#163005a04999a74
Problem: Option metadata like list of valid values for an option and
option flags are not listed in the `options.lua` file and are instead
manually defined in C, which means option metadata is split between
several places.
Solution: Put metadata such as list of valid values for an option and
option flags in `options.lua`, and autogenerate the corresponding C
variables and enums.
Supersedes #28659
Co-authored-by: glepnir <glephunter@gmail.com>
Problem: Highlight group id is not propagated to the end of the message call
stack, where ext_messages are emitted.
Solution: Refactor message functions to pass along highlight group id
instead of attr id.
while the implementation is not tied to screen chars, it is a reasonable
expectation to support the same size. If nvim is able to display a
multibyte character, it will accept the same character as input,
including in normal mode commands like r{char}
Problem: too many strlen() calls in ex_getln.c
Solution: refactor the code to reduce the number of strlen() calls
(John Marriott)
closes: vim/vim#15809ccf8907570
Co-authored-by: John Marriott <basilisk@internode.on.net>
Use the grapheme break algorithm from utf8proc to support grapheme
clusters from recent unicode versions.
Handle variant selector VS16 turning some codepoints into double-width
emoji. This means we need to use ptr2cells rather than char2cells when
possible.
Problem: Code related to findfile() is spread out.
Solution: Put findfile() related code into a new source file. (Yegappan
Lakshmanan, closesvim/vim#3934)
5fd0f5052f
Keep functions related to wildcard expansion in path.c, as in Vim they
are now spread out among multiple files, which isn't really ideal.
This also makes shada reading slightly faster due to avoiding
some copying and allocation.
Use keysets to drive decoding of msgpack maps for shada entries.
Problem:
Variables are often assigned multiple places in common patterns.
Solution:
Replace these common patterns with different patterns that reduce the
number of assignments.
Use `MAX` and `MIN`:
```c
if (x < y) {
x = y;
}
// -->
x = MAX(x, y);
```
```c
if (x > y) {
x = y;
}
// -->
x = MIN(x, y);
```
Use ternary:
```c
int a;
if (cond) {
a = b;
} els {
a = c;
}
// -->
int a = cond ? b : c;
```
Problem: Current instance of last search pattern not easily spotted.
Solution: Add CurSearch highlighting. (closesvim/vim#10133)
a43993897a
Some code is superseded by later patches that are already ported.
Co-authored-by: LemonBoy <thatlemon@gmail.com>
Problem: fuzzy_match_str_with_pos() does unnecessary list operations.
Solution: Use fuzzy_match() directly (zeertzjq).
closes: vim/vim#149872f95ca9fce
N/A patch:
vim-patch:9.1.0478: potential deref of NULL pointer in fuzzy_match_str_with_pos
Problem: Cannot see matched text in popup menu
Solution: Introduce 2 new highlighting groups: PmenuMatch and
PmenuMatchSel (glepnir)
closes: vim/vim#1469440c1c3317d
Co-authored-by: glepnir <glephunter@gmail.com>
Problem: too many strlen() calls in search.c
Solution: refactor code and remove more strlen() calls,
use explicit variable to remember strlen
(John Marriott)
closes: vim/vim#147968c85a2a49a
Co-authored-by: John Marriott <basilisk@internode.on.net>
Problem: More code can use ml_get_buf_len() instead of STRLEN().
Solution: Change more STRLEN() calls to ml_get_buf_len(). Also do not
set ml_line_textlen in ml_replace_len() if "has_props" is set,
because "len_arg" also includes the size of text properties in
that case. (zeertzjq)
closes: vim/vim#1418394b7c3233e
A lot of functions in move.c only worked for curwin, alternatively
took a `wp` arg but still only work if that happens to be curwin.
Refactor those that are needed for update_topline(wp) to work
for any window.
fixes#27723fixes#27720
Problem: Still a qsort() comparison function fuzzy_match_item_compare()
that returns result of subtraction (after 9.1.0089).
Solution: Use an explicit comparison instead of subtraction.
(zeertzjq)
closes: vim/vim#1400477078276bf
Problem: qsort() comparison functions should be transitive
Solution: Do not subtract values, but rather use explicit comparisons
Improve qsort() comparison functions
There has been a recent report on qsort() causing out-of-bounds read &
write in glibc for non transitive comparison functions
https://www.qualys.com/2024/01/30/qsort.txt
Even so the bug is in glibc's implementation of the qsort() algorithm,
it's bad style to just use substraction for the comparison functions,
which may cause overflow issues and as hinted at in OpenBSD's manual
page for qsort(): "It is almost always an error to use subtraction to
compute the return value of the comparison function."
So check the qsort() comparison functions and change them to be safe.
closes: vim/vim#13980e06e437665
Co-authored-by: Christian Brabandt <cb@256bit.org>
