mirrors/neovim
1
0
Fork 0
mirror of https://github.com/neovim/neovim.git synced 2025-09-27 13:38:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor!: use utf8proc full casefolding

Browse Source 
According to `CaseFolding-15.1.0.txt`, full casefolding should be
preferred over simple casefolding as it's considered to be more correct.
Since utf8proc already provides full casefolding it makes sense to
switch to it. This will also remove a lot of unnecessary build code.

Temporary exceptions are made for two sets characters:

- `ß` will still be considered `ß` (instead of `ss`) as using a full
  casefolding requires interfering with upstream spell files in some
  form.
- `İ` will still be considered `İ` (instead of `i̇`) as using full
  casefolding requires making a value judgement on the "correct"
  behavior. There are two, equally valid case-insensetive comparison for
  this character according to unicode. It is essentially up to the
  implementor to decide which conversion is correct. For this reason it
  might make sense to allow users to decide which conversion should be
  done as an added option to `casemap` in a future PR.
This commit is contained in:
dundargoc
2024-06-18 14:01:20 +02:00
committed by dundargoc
parent 11a6f3c930
commit 328ea02eb7
3 changed files with 25 additions and 1705 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
src/nvim/mbyte.c
View File

@@ -1284,41 +1284,38 @@ bool utf_ambiguous_width(int c)
|| intable(emoji_all, ARRAY_SIZE(emoji_all), c));
}
// Generic conversion function for case operations.
// Return the converted equivalent of "a", which is a UCS-4 character. Use
// the given conversion "table". Uses binary search on "table".
static int utf_convert(int a, const convertStruct *const table, size_t n_items)
{
// indices into table
size_t start = 0;
size_t end = n_items;
while (start < end) {
// need to search further
size_t mid = (end + start) / 2;
if (table[mid].rangeEnd < a) {
start = mid + 1;
} else {
end = mid;
}
}
if (start < n_items
&& table[start].rangeStart <= a
&& a <= table[start].rangeEnd
&& (a - table[start].rangeStart) % table[start].step == 0) {
return a + table[start].offset;
}
return a;
}
// Return the folded-case equivalent of "a", which is a UCS-4 character. Uses
// simple case folding.
// full case folding.
int utf_fold(int a)
{
if (a < 0x80) {
// be fast for ASCII
return a >= 0x41 && a <= 0x5a ? a + 32 : a;
}
return utf_convert(a, foldCase, ARRAY_SIZE(foldCase));
// TODO(dundargoc): utf8proc only does full case folding, which breaks some tests. This is a
// temporary workaround to circumvent failing tests.
//
// (0xdf) ß == ss in full casefolding. Using this however breaks the vim spell tests and the error
// E763 is thrown. This is due to the test spells relying on the vim spell files.
//
// (0x130) İ == i̇ in full casefolding.
if (a == 0xdf || a == 0x130) {
return a;
}
utf8proc_uint8_t input_str[16] = { 0 };
utf8proc_encode_char(a, input_str);
utf8proc_uint8_t *fold_str_utf;
utf8proc_map((utf8proc_uint8_t *)input_str, 0, &fold_str_utf,
UTF8PROC_NULLTERM | UTF8PROC_CASEFOLD);
int fold_codepoint_utf = utf_ptr2char((char *)fold_str_utf);
xfree(fold_str_utf);
return fold_codepoint_utf;
}
// Vim's own character class functions. These exist because many library