This also makes shada reading slightly faster due to avoiding
some copying and allocation.
Use keysets to drive decoding of msgpack maps for shada entries.
If you like it you shouldn't put a ring on it.
This is what _every_ consumer of RStream used anyway, either by calling
rbuffer_reset, or rbuffer_consumed_compact (same as rbuffer_reset
without needing a scratch buffer), or by consuming everything in
each stream_read_cb call directly.
This is a structural refactor with no logical changes, yet. Done in
preparation for simplifying rstream/rbuffer which will require more
state inline in RStream.
The initial idea was to have RStream and WStream as sub-types
symetrically but that doesn't work, as sockets are both reading and
writing. Also there is very little write-specific state to start with,
so the benefit of a separate WStream struct is a lot smaller. Just
document what fields in `Stream` are write specific.
Using a ring buffer for buffered synchronous fileio is just unnecessary
complexity.
- when reading, we always consume the _entire_ buffer before getting
into syscalls. Thus we reset the buffer to its initial position before
when we actually read.
- when writing and buffer is full, we always flush the entire buffer
before starting to buffer again. So we can reset the buffer to its
initial state.
Also no static buffers are needed for writing and skipping. Needing an
extra copy for each write completely defeated the purpose of
a ring buffer (if there had been one)
Problem:
`vim.rpcnotify(0)` and `rpcnotify(0)` are documented as follows:
If {channel} is 0, the event is broadcast to all channels.
But that's not actually true. Channels must call `nvim_subscribe` to
receive "broadcast" events, so it's actually "multicast".
- Assuming there is a use-case for "broadcast", the current model adds
an extra step for broadcasting: all channels need to "subscribe".
- The presence of `nvim_subscribe` is a source of confusion for users,
because its name implies something more generally useful than what it
does.
Presumably the use-case of `nvim_subscribe` is to avoid "noise" on RPC
channels not expected a broadcast notification, and potentially an error
if the channel client reports an unknown event.
Solution:
- Deprecate `nvim_subscribe`/`nvim_unsubscribe`.
- If applications want to multicast, they can keep their own multicast
list. Or they can use `nvim_list_chans()` and `nvim_get_chan_info()`
to enumerate and filter the clients they want to target.
- Always send "broadcast" events to ALL channels. Don't require channels
to "subscribe" to receive broadcasts. This matches the documented
behavior of `rpcnotify()`.
Before, we needed to always pack an entire msgpack_rpc Object to
a continous memory buffer before sending it out to a channel.
But this is generally wasteful. it is better to just flush
whatever is in the buffer and then continue packing to a new buffer.
This is also done for the UI event packer where there are some extra logic
to "finish" of an existing batch of nevents/ncalls. This doesn't really
stop us from flushing the buffer, just that we need to update the state
machine accordingly so the next call to prepare_call() always will
start with a new event (even though the buffer might contain overflow
data from a large event).
As only a few API functions make use of explicit freeing of the return
value, make it opt-in instead. The arena is always present under the
hood, so `Arena *arena` arg now doesn't mean anything other than getting
access to this arena. Also it is in principle possible to return an
allocated value while still using the arena as scratch space for other
stuff (unlikely, but there no reason to not allow it).
Note: kSDItemHeader is something is _written_ by nvim in the shada file
to identify it for debugging purposes outside of nvim. But this data wasn't ever used by
neovim after reading the file back, So I removed the parsing of it for now.
Note: this contains two _temporary_ changes which can be reverted
once the Arena vs no-Arena distinction in API wrappers has been removed.
Both nlua_push_Object and object_to_vim_take_luaref() has been changed
to take the object argument as a pointer. This is not going to be
necessary once these are only used with arena (or not at all) allocated
Objects.
The object_to_vim() variant which leaves luaref untouched might need to
stay for a little longer.
and for return value of nlua_exec/nlua_call_ref, as this uses
the same family of functions.
NB: the handling of luaref:s is a bit of a mess.
add api_luarefs_free_XX functions as a stop-gap as refactoring
luarefs is a can of worms for another PR:s.
as a minor feature/bug-fix, nvim_buf_call and nvim_win_call now preserves
arbitrary return values.
Problem:
rbuffer_consumed assertion fails if Unpacker fails to parse msgpack,
because it doesn't consume bytes on errors
Solution:
Call rbuffer_consumed_compact only if Unpacker isn't closed
Remove `export` pramgas from defs headers as it causes IWYU to believe
that the definitions from the defs headers comes from main header, which
is not what we really want.
FUNC_ATTR_* should only be used in .c files with generated headers.
Defining FUNC_ATTR_* as empty in headers causes misuses of them to be
silently ignored. Instead don't define them by default, and only define
them as empty after a .c file has included its generated header.
We already have an extensive suite of static analysis tools we use,
which causes a fair bit of redundancy as we get duplicate warnings. PVS
is also prone to give false warnings which creates a lot of work to
identify and disable.
refactor: use a more idiomatic loop to iterate over the cells
There are two cases in which the following assertion would fail:
```c
assert(g->icell < g->ncells);
```
1. If `g->ncells = 0`. Update this to be legal.
2. If an EOF is reached while parsing `wrap`. In this case, the unpacker
attempts to resume from `cells`, which is a bug. Create a new state
for parsing `wrap`.
Reference: https://neovim.io/doc/user/ui.html#ui-event-grid_line
Problem: The style guide states that all switch statements that are not conditional on an enum must have a `default` case, but does not give any explicit guideline for switch statements that are conditional on enums. As a result, a `default` case is added in many enum switch statements, even when the switch statement is exhaustive. This is not ideal because it removes the ability to have compiler errors to easily detect unchanged switch statements when a new possible value for an enum is added.
Solution: Add explicit guidelines for switch statements that are conditional on an enum, clarifying that a `default` case is not necessary if the switch statement is exhaustive. Also refactor pre-existing code with unnecessary `default` cases.
Previously, a screen cell would occupy 28+4=32 bytes per cell
as we always made space for up to MAX_MCO+1 codepoints in a cell.
As an example, even a pretty modest 50*80 screen would consume
50*80*2*32 = 256000, i e a quarter megabyte
With the factor of two due to the TUI side buffer, and even more when
using msg_grid and/or ext_multigrid.
This instead stores a 4-byte union of either:
- a valid UTF-8 sequence up to 4 bytes
- an escape char which is invalid UTF-8 (0xFF) plus a 24-bit index to a
glyph cache
This avoids allocating space for huge composed glyphs _upfront_, while
still keeping rendering such glyphs reasonably fast (1 hash table lookup
+ one plain index lookup). If the same large glyphs are using repeatedly
on the screen, this is still a net reduction of memory/cache
consumption. The only case which really gets worse is if you blast
the screen full with crazy emojis and zalgo text and even this case
only leads to 4 extra bytes per char.
When only <= 4-byte glyphs are used, plus the 4-byte attribute code,
i e 8 bytes in total there is a factor of four reduction of memory use.
Memory which will be quite hot in cache as the screen buffer is scanned
over in win_line() buffer text drawing
A slight complication is that the representation depends on host byte
order. I've tested this manually by compling and running this
in qemu-s390x and it works fine. We might add a qemu based solution
to CI at some point.
This involves two redesigns of the map.c implementations:
1. Change of macro style and code organization
The old khash.h and map.c implementation used huge #define blocks with a
lot of backslash line continuations.
This instead uses the "implementation file" .c.h pattern. Such a file is
meant to be included multiple times, with different macros set prior to
inclusion as parameters. we already use this pattern e.g. for
eval/typval_encode.c.h to implement different typval encoders reusing a
similar structure.
We can structure this code into two parts. one that only depends on key
type and is enough to implement sets, and one which depends on both key
and value to implement maps (as a wrapper around sets, with an added
value[] array)
2. Separate the main hash buckets from the key / value arrays
Change the hack buckets to only contain an index into separate key /
value arrays
This is a common pattern in modern, state of the art hashmap
implementations. Even though this leads to one more allocated array, it
is this often is a net reduction of memory consumption. Consider
key+value consuming at least 12 bytes per pair. On average, we will have
twice as many buckets per item.
Thus old implementation:
2*12 = 24 bytes per item
New implementation
1*12 + 2*4 = 20 bytes per item
And the difference gets bigger with larger items.
One might think we have pulled a fast one here, as wouldn't the average size of
the new key/value arrays be 1.5 slots per items due to amortized grows?
But remember, these arrays are fully dense, and thus the accessed memory,
measured in _cache lines_, the unit which actually matters, will be the
fully used memory but just rounded up to the nearest cache line
boundary.
This has some other interesting properties, such as an insert-only
set/map will be fully ordered by insert only. Preserving this ordering
in face of deletions is more tricky tho. As we currently don't use
ordered maps, the "delete" operation maintains compactness of the item
arrays in the simplest way by breaking the ordering. It would be
possible to implement an order-preserving delete although at some cost,
like allowing the items array to become non-dense until the next rehash.
Finally, in face of these two major changes, all code used in khash.h
has been integrated into map.c and friends. Given the heavy edits it
makes no sense to "layer" the code into a vendored and a wrapper part.
Rather, the layered cake follows the specialization depth: code shared
for all maps, code specialized to a key type (and its equivalence
relation), and finally code specialized to value+key type.
Added to support MessagePack-RPC fully compliant clients that do
not return responses in request order.
Although it is currently not an efficient implementation for full
compliance and full compliance cannot be guaranteed, the addition
of the new client type `msgpack-rpc` creates a situation where "if
the client type is `msgpack-rpc`, then backward compatibility is
ignored and full compliance with MessagePack- RPC compliance is
justified even if backward compatibility is ignored if the client
type is `msgpack-rpc`.
