Problem: Marks whose position did not change with the action that
invalidated them (right_gravity = false) are not revalidated
upon undo.
Solution: Remove early return when restoring a marks saved position so
that it is still revalidated. Add "move" guards instead.
Problem: Invalid marks appear to be revalidated multiple times, and
decor is added at the old position for unpaired marks.
Solution: Avoid revalidating already valid marks, and don't use old
position to add to decor for unpaired marks.
This avoids redraw when adding/removing an empty namespace for a window.
This also avoids marktree traversal when clearing a namespace that has
already been cleared, which is added as a benchmark.
Problem: Marks moved by undo may be lost to "b_signcols.count".
Solution: Count signs for each undo object separately instead of
once for the entire undo.
This expands on the global "don't pay for what you don't use" rules for
these special extmark decorations:
- inline virtual text, which needs to be processed in plines.c when we
calculate the size of text on screen
- virtual lines, which are needed when calculating "filler" lines
- signs, with text and/or highlights, both of which needs to be
processed for the entire line already at the beginning of a line.
This adds a count to each node of the marktree, for how many special
marks of each kind can be found in the subtree for this node. This makes
it possible to quickly skip over these extra checks, when working in
regions of the buffer not containing these kind of marks, instead of
before where this could just be skipped if the entire _buffer_
didn't contain such marks.
Problem: Some edge cases to the old (pre-#26406) and current "b_signcols"
structure result in an incorrectly sized "auto" 'signcolumn'.
Solution: * Implement a simpler 'signcolumn' validation strategy by immediately
counting the number of signs in a range upon sign insertion and
deletion. Decrease in performance here but there is a clear path
forward to decreasing this performance hit by moving signs to a
dedicated marktree, or by adding meta-data to the existing
marktree which may be queried more efficiently?
* Also replace "max_count" and keep track of the number of lines with
a certain number of signs. This makes it so that it is no longer
necessary to scan the entire buffer when the maximum number of signs
decreases. This likely makes the commit a net increase in performance.
* To ensure correctness we also have re-initialize the count for an
edited region that spans multiple lines. Such an edit may move the
signs within it. Thus we count and decrement before splicing the
marktree and count and increment after.
Remove `export` pramgas from defs headers as it causes IWYU to believe
that the definitions from the defs headers comes from main header, which
is not what we really want.
Problem: Unpaired marks are invalidated if its column is deleted,
which may just be a "placeholder" column, e.g. for signs.
Solution: Only remove unpaired marks if its entire row is deleted.
Problem: The entire marktree needs to be traversed each time a sign is
removed from the sentinel line.
Solution: Remove sentinel line and instead keep track of the number of
lines that hold up the 'signcolumn' in "max_count". Adjust this
number for added/removed signs, and set it to 0 when the
maximum number of signs on a line changes. Only when
"max_count" is decremented to 0 due to sign removal do we need
to check the entire buffer.
Also replace "invalid_top" and "invalid_bot" with a map of
invalid ranges, further reducing the number of lines to be
checked.
Also improve tree traversal when counting the number of signs.
Instead of looping over the to be checked range and counting
the overlap for each row, keep track of the overlap in an
array and add this to the count.
Problem: Signcolumn width does not increase when ranged sign does not
start at sentinel line.
Solution: Handle paired range of added sign when checking signcols.
Remove the monolithic Decoration struct. Before this change, each extmark
could either represent just a hl_id + priority value as a inline
decoration, or it would take a pointer to this monolitic 112 byte struct
which has to be allocated.
This change separates the decorations into two pieces: DecorSignHighlight
for signs, highlights and simple set-flag decorations (like spell,
ui-watched), and DecorVirtText for virtual text and lines.
The main separation here is whether they are expected to allocate more
memory. Currently this is not really true as sign text has to be an
allocated string, but the plan is to get rid of this eventually (it can
just be an array of two schar_T:s). Further refactors are expected to
improve the representation of each decoration kind individually. The
goal of this particular PR is to get things started by cutting the
Gordian knot which was the monolithic struct Decoration.
Now, each extmark can either contain chained indicies/pointers to
these kinds of objects, or it can fit a subset of DecorSignHighlight
inline.
The point of this change is not only to make decorations smaller in
memory. In fact, the main motivation is to later allow them to grow
_larger_, but on a dynamic, on demand fashion. As a simple example, it
would be possible to augment highlights to take a list of multiple
`hl_group`:s, which then would trivially map to a chain of multiple
DecorSignHighlight entries.
One small feature improvement included with this refactor itself, is
that the restriction that extmarks cannot be removed inside a decoration
provider has been lifted. These are instead safely lifetime extended
on a "to free" list until the current iteration of screen drawing is done.
NB: flags is a mess. but DecorLevel is useless, this slightly less so
Problem: The legacy signlist data structures and associated functions are
redundant since the introduction of extmark signs.
Solution: Store signs defined through the legacy commands in a hashmap, placed
signs in the extmark tree. Replace signlist associated functions.
Usage of the legacy sign commands should yield no change in behavior with the
exception of:
- "orphaned signs" are now always removed when the line it is placed on is
deleted. This used to depend on the value of 'signcolumn'.
- It is no longer possible to place multiple signs with the same identifier
in a single group on multiple lines. This will now move the sign instead.
Moreover, both signs placed through the legacy sign commands and through
|nvim_buf_set_extmark()|:
- Will show up in both |sign-place| and |nvim_buf_get_extmarks()|.
- Are displayed by increasing sign identifier, left to right.
Extmark signs used to be ordered decreasingly as opposed to legacy signs.
We already have an extensive suite of static analysis tools we use,
which causes a fair bit of redundancy as we get duplicate warnings. PVS
is also prone to give false warnings which creates a lot of work to
identify and disable.
Problem: No way to have extmarks automatically removed when the range it
is attached to is deleted.
Solution: Add new 'invalidate' property that will hide a mark when the
entirety of its range is deleted. When "undo_restore" is set
to false, delete the mark from the buffer instead.
It is a design goal of extmarks that they allow precise tracking
of changes across undo/redo, including restore the exact positions
after a do/undo or undo/redo cycle. However this behavior is not useful
for all usecases. Many plugins won't keep marks around for long after
text changes, but uses them more like a cache until some external source
(like LSP semantic highlights) has fully updated to changed text and
then will explicitly readjust/replace extmarks as needed.
Add a "undo_restore" flag which is true by default (matches existing
behavior) but can be set to false to opt-out of this behavior.
Delete dead u_extmark_set() code.
long is 32 bits on windows, while it is 64 bits on other architectures.
This makes the type suboptimal for a codebase meant to be
cross-platform. Replace it with more appropriate integer types.
The removes the previous restriction that nvim_buf_set_extmark()
could not be used to highlight arbitrary multi-line regions
The problem can be summarized as follows: let's assume an extmark with a
hl_group is placed covering the region (5,0) to (50,0) Now, consider
what happens if nvim needs to redraw a window covering the lines 20-30.
It needs to be able to ask the marktree what extmarks cover this region,
even if they don't begin or end here.
Therefore the marktree needs to be augmented with the information covers
a point, not just what marks begin or end there. To do this, we augment
each node with a field "intersect" which is a set the ids of the
marks which overlap this node, but only if it is not part of the set of
any parent. This ensures the number of nodes that need to be explicitly
marked grows only logarithmically with the total number of explicitly
nodes (and thus the number of of overlapping marks).
Thus we can quickly iterate all marks which overlaps any query position
by looking up what leaf node contains that position. Then we only need
to consider all "start" marks within that leaf node, and the "intersect"
set of that node and all its parents.
Now, and the major source of complexity is that the tree restructuring
operations (to ensure that each node has T-1 <= size <= 2*T-1) also need
to update these sets. If a full inner node is split in two, one of the
new parents might start to completely overlap some ranges and its ids
will need to be moved from its children's sets to its own set.
Similarly, if two undersized nodes gets joined into one, it might no
longer completely overlap some ranges, and now the children which do
needs to have the have the ids in its set instead. And then there are
the pivots! Yes the pivot operations when a child gets moved from one
parent to another.
This involves two redesigns of the map.c implementations:
1. Change of macro style and code organization
The old khash.h and map.c implementation used huge #define blocks with a
lot of backslash line continuations.
This instead uses the "implementation file" .c.h pattern. Such a file is
meant to be included multiple times, with different macros set prior to
inclusion as parameters. we already use this pattern e.g. for
eval/typval_encode.c.h to implement different typval encoders reusing a
similar structure.
We can structure this code into two parts. one that only depends on key
type and is enough to implement sets, and one which depends on both key
and value to implement maps (as a wrapper around sets, with an added
value[] array)
2. Separate the main hash buckets from the key / value arrays
Change the hack buckets to only contain an index into separate key /
value arrays
This is a common pattern in modern, state of the art hashmap
implementations. Even though this leads to one more allocated array, it
is this often is a net reduction of memory consumption. Consider
key+value consuming at least 12 bytes per pair. On average, we will have
twice as many buckets per item.
Thus old implementation:
2*12 = 24 bytes per item
New implementation
1*12 + 2*4 = 20 bytes per item
And the difference gets bigger with larger items.
One might think we have pulled a fast one here, as wouldn't the average size of
the new key/value arrays be 1.5 slots per items due to amortized grows?
But remember, these arrays are fully dense, and thus the accessed memory,
measured in _cache lines_, the unit which actually matters, will be the
fully used memory but just rounded up to the nearest cache line
boundary.
This has some other interesting properties, such as an insert-only
set/map will be fully ordered by insert only. Preserving this ordering
in face of deletions is more tricky tho. As we currently don't use
ordered maps, the "delete" operation maintains compactness of the item
arrays in the simplest way by breaking the ordering. It would be
possible to implement an order-preserving delete although at some cost,
like allowing the items array to become non-dense until the next rehash.
Finally, in face of these two major changes, all code used in khash.h
has been integrated into map.c and friends. Given the heavy edits it
makes no sense to "layer" the code into a vendored and a wrapper part.
Rather, the layered cake follows the specialization depth: code shared
for all maps, code specialized to a key type (and its equivalence
relation), and finally code specialized to value+key type.
This reduces the total number of khash_t instantiations from 22 to 8.
Make the khash internal functions take the size of values as a runtime
parameter. This is abstracted with typesafe Map containers which
are still specialized for both key, value type.
Introduce `Set(key)` type for when there is no value.
Refactor shada.c to use Map/Set instead of khash directly.
This requires `map_ref` operation to be more flexible.
Return pointers to both key and value, plus an indicator for new_item.
As a bonus, `map_key` is now redundant.
Instead of Map(cstr_t, FileMarks), use a pointer map as the FileMarks struct is
humongous.
Make `event_strings` actually work like an intern pool instead of wtf it
was doing before.
fix(extmarks): disallow removing extmarks in on_lines callbacks
decor_redraw_start (which runs before decor_providers_invoke_lines) gets
references for the extmarks on a specific line. If these extmarks are
deleted in on_lines callbacks then this results in a heap-use-after-free
error.
Fixes#22801
Problem: Can not get all extmarks in a buffer. Properties are missing
from the details array.
Solution: Allow getting all extmarks in a buffer by supplying a -1
"ns_id". Add missing properties to the details array.
extranges and a bunch of other improvements are coming for 0.10
This gets in some minor surrounding API changes to avoid rebase
conflicts until then.
- decorations will be able to be specific to windows
- adjust deletion API to fit with extranges
Allow Include What You Use to remove unnecessary includes and only
include what is necessary. This helps with reducing compilation times
and makes it easier to visualise which dependencies are actually
required.
Work on https://github.com/neovim/neovim/issues/549, but doesn't close
it since this only works fully for .c files and not headers.
