These are not needed after #35129 but making uncrustify still play nice
with them was a bit tricky.
Unfortunately `uncrustify --update-config-with-doc` breaks strings
with backslashes. This issue has been reported upstream,
and in the meanwhile auto-update on every single run has been disabled.
Problem:
When a function like vim.wait() is used, we continuously drain the main
event queue until it is empty, never stopping for user input. This means
the libuv timer never runs and the terminal never gets refreshed, so
emit_termrequest continously reschedules itself onto the same event
queue, causing an infinite loop.
Solution:
Use a separate "pending" event queue, where events that require a
terminal refresh are temporarily placed. Drain this queue after a
terminal refresh and events are copied back onto the main queue. This
prevents infinite loops since the main event queue will always be able
to properly drain.
loop->children might have been a linked list because used to be
modified in place while looped over. However the loops that exists
rather schedules events to be processed later, outside of the loop,
so this can not happen anymore.
When a linked list is otherwise useful it is better to use
lib/queue_defs.h which defines an _intrusive_ linked list (i e it
doesn't need to do allocations for list items like klist ).
We already have an extensive suite of static analysis tools we use,
which causes a fair bit of redundancy as we get duplicate warnings. PVS
is also prone to give false warnings which creates a lot of work to
identify and disable.
As neovim does have event handling, we are checking for CTRL-C
all the time, not once per second.
Also, do_sleep() reimplements the same loop as
LOOP_PROCESS_EVENTS_UNTIL() already contains internally. Fix the latter
to use the right integer type, so we do not need the extra indirection.
Allow Include What You Use to remove unnecessary includes and only
include what is necessary. This helps with reducing compilation times
and makes it easier to visualise which dependencies are actually
required.
Work on https://github.com/neovim/neovim/issues/549, but doesn't close
it since this only works fully for .c files and not headers.
In the rare case that exit_event is called from process_close_handles,
it stalls waiting for the process to exit (the routine is currently
underway to do just that). This causes `job_spec.lua` to sometimes
stall.
REJECTED IDEAS:
==============================================================
1. Currently `exit_event` is placed on `main_loop.fast_events`. Would the problem
be solved by using `main_loop.events` instead?
- A: Maybe, but it will cause other problems, such as queuing exit_event()
during "Press Enter..." prompt which may result in the event not being
processed, leading to another stall.
2. Can we avoid the timer?
- A: Using a timer is just the easiest way to queue a delayed event without
causing an infinite loop in the queue currently being processed.
3. Can we avoid the new `exit_need_delay` global...
1. by using `process_is_tearing_down` instead?
- A: Can't use `process_is_tearing_down` because its semantics are different.
2. by checking a similar condition as `process_teardown`? f50135a32e/src/nvim/event/process.c (L141-L142)
```
if (!process_is_tearing_down || (kl_empty(main_loop.children) && multiqueue_empty(main_loop.events))) {
uv_timer_start(&main_loop.exit_delay_timer, exit_delay_cb, 0, 0);
return;
}
```
- A: Tried but it did not work (other stalls occurred). Maybe
exit_event() is called from a source other than
process_close_handles() and is delayed, the delayed exit_event() will
be executed before main_loop.events is processed, resulting in an
infinite loop.
- Move uv_stop(), it still causes a "leak" on exit somehow.
- Tenatively restore `UV_RUN_DEFAULT`. It shouldn't hang since we
clobber the handles via `uv_walk((h)=>uv_close(h))`. Although this
still "leaks" on exit, it's faster than the 2-second timeout.
fix#11820fix#7376
- Call uv_stop().
- Restore `uv_loop_close` condition (braindead cosmetic change from
a2efc9cf8b that caused uv_loop_close *not* to be called if
wait=false, sorry).
Not doing `uv_walk(() => uv_close)`: see source comment for explanation.
fix#11820fix#7376
Q: Should we restore use of `UV_RUN_DEFAULT`/`UV_RUN_ONCE` (removed in
a2efc9cf8b)?
A: The while-loop (hopefully) achieves the same purpose while avoiding
a hang.
This avoids having a dummy event to tickle the main loop.
Confirmed using `nvim -u NONE -c 'au FocusGained * q'` in tmux (with
`:set -g focus-events on`): without the flushing it would only exit
after pressing a key.
Moves the flushing done recently in 3626d2107.
`nvim -u NONE -cq` is still working (i.e. consuming the response for the
terminal background query itself), and the flickering mentioned in
3626d2107 is reduced again.
Reverts part of bfb21f3e0 (#7729).
HACK: The cursor does not get repositioned after the paste completes.
Scheduling a dummy event seems to fix it.
Test case:
0. Revert this commit.
1. Paste some text in Normal-mode.
2. Notice the cursor is still in the cmdline area.
ui_bridge:ui_bridge_stop() calls ui_detach_impl() last, so the check for
ui_active() in ui:ui_refresh() doesn't help: tui_main() already freed
the `ui` object.
There is a race between ui_bridge_stop (thread T0) and tui_main (thread T1).
UIBridgeData.stopped could be set while ui_bridge_stop() is in the
middle of loop_poll_events(), which may invoke tui_scheduler() on T0.
The pointers in tui_scheduler() may be invalid by then.
Solution(?): Use the `UI.data` field as a "stopped" flag and check it in
tui_scheduler().
ASAN use-after-free report observed in #7908:
= ==20066==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000000cd0 at pc 0x00000182abed bp 0x7ffe23b07070 sp 0x7ffe23b07068
= READ of size 8 at 0x611000000cd0 thread T0
= 0 0x182abec in tui_scheduler /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:393:23
= 1 0x1876afd in ui_bridge_update_fg /home/travis/build/neovim/neovim/build/src/nvim/auto/ui_events_bridge.generated.h:205:3
= 2 0x186c130 in ui_resize /home/travis/build/neovim/neovim/src/nvim/ui.c:310:3
= 3 0x146b9c2 in screen_resize /home/travis/build/neovim/neovim/src/nvim/screen.c:7483:3
= 4 0x186a6f0 in ui_refresh /home/travis/build/neovim/neovim/src/nvim/ui.c:284:3
= 5 0x186bbe0 in ui_refresh_event /home/travis/build/neovim/neovim/src/nvim/ui.c:297:3
= 6 0xa2219a in multiqueue_process_events /home/travis/build/neovim/neovim/src/nvim/event/multiqueue.c:150:7
= 7 0xa1bd7f in loop_poll_events /home/travis/build/neovim/neovim/src/nvim/event/loop.c:63:3
= 8 0x1872709 in ui_bridge_stop /home/travis/build/neovim/neovim/src/nvim/ui_bridge.c:121:5
= 9 0x1864247 in ui_builtin_stop /home/travis/build/neovim/neovim/src/nvim/ui.c:143:3
= 10 0x1249ec8 in mch_exit /home/travis/build/neovim/neovim/src/nvim/os_unix.c:140:3
= 11 0xe56ba9 in getout /home/travis/build/neovim/neovim/src/nvim/main.c:671:3
= 12 0xfc4c8f in preserve_exit /home/travis/build/neovim/neovim/src/nvim/misc1.c:2653:3
= 13 0x1247c02 in deadly_signal /home/travis/build/neovim/neovim/src/nvim/os/signal.c:137:3
= 14 0x1247921 in on_signal /home/travis/build/neovim/neovim/src/nvim/os/signal.c:162:9
= 15 0xa35618 in signal_event /home/travis/build/neovim/neovim/src/nvim/event/signal.c:47:3
= 16 0xa2219a in multiqueue_process_events /home/travis/build/neovim/neovim/src/nvim/event/multiqueue.c:150:7
= 17 0xa1bd7f in loop_poll_events /home/travis/build/neovim/neovim/src/nvim/event/loop.c:63:3
= 18 0x1237bd6 in input_poll /home/travis/build/neovim/neovim/src/nvim/os/input.c:349:3
= 19 0x123334f in inbuf_poll /home/travis/build/neovim/neovim/src/nvim/os/input.c:372:24
= 20 0x123316d in os_inchar /home/travis/build/neovim/neovim/src/nvim/os/input.c:110:19
= 21 0x170d20e in state_enter /home/travis/build/neovim/neovim/src/nvim/state.c:55:13
= 22 0xbd7441 in command_line_enter /home/travis/build/neovim/neovim/src/nvim/ex_getln.c:384:3
= 23 0xbd0a60 in getcmdline /home/travis/build/neovim/neovim/src/nvim/ex_getln.c:1920:10
= 24 0xbdb365 in getexline /home/travis/build/neovim/neovim/src/nvim/ex_getln.c:2100:10
= 25 0xb00a6b in do_cmdline /home/travis/build/neovim/neovim/src/nvim/ex_docmd.c:528:47
= 26 0x10a7837 in nv_colon /home/travis/build/neovim/neovim/src/nvim/normal.c:4552:18
= 27 0x1091e15 in normal_execute /home/travis/build/neovim/neovim/src/nvim/normal.c:1136:3
= 28 0x170d439 in state_enter /home/travis/build/neovim/neovim/src/nvim/state.c:67:26
= 29 0x104ee14 in normal_enter /home/travis/build/neovim/neovim/src/nvim/normal.c:466:3
= 30 0xe4295c in main /home/travis/build/neovim/neovim/src/nvim/main.c:572:3
= 31 0x2b2ba340bf44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
= 32 0x44d24b in _start (/home/travis/build/neovim/neovim/build/bin/nvim+0x44d24b)
=
= 0x611000000cd0 is located 16 bytes inside of 240-byte region [0x611000000cc0,0x611000000db0)
= freed by thread T1 here:
= 0 0x4ee0e2 in __interceptor_free /local/mnt/workspace/tmp/ubuntu_rel/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:47:3
= 1 0xf4f6d4 in xfree /home/travis/build/neovim/neovim/src/nvim/memory.c:133:3
= 2 0x182a963 in tui_main /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:383:3
= 3 0x18792b0 in ui_thread_run /home/travis/build/neovim/neovim/src/nvim/ui_bridge.c:106:3
= 4 0x2b2ba2697183 in start_thread /build/eglibc-ripdx6/eglibc-2.19/nptl/pthread_create.c:312
=
= previously allocated by thread T0 here:
= 0 0x4ee61a in calloc /local/mnt/workspace/tmp/ubuntu_rel/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:76:3
= 1 0xf4f787 in xcalloc /home/travis/build/neovim/neovim/src/nvim/memory.c:147:15
= 2 0x182000a in tui_start /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:127:12
= 3 0x1863f7c in ui_builtin_start /home/travis/build/neovim/neovim/src/nvim/ui.c:125:3
= 4 0xe41bb9 in main /home/travis/build/neovim/neovim/src/nvim/main.c:457:5
= 5 0x2b2ba340bf44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
=
= Thread T1 created by T0 here:
= 0 0x4d774d in __interceptor_pthread_create /local/mnt/workspace/tmp/ubuntu_rel/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317:3
= 1 0x1aae6b0 in uv_thread_create /home/travis/nvim-deps/build/src/libuv/src/unix/thread.c:75
= 2 0x18217fa in tui_start /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:159:10
= 3 0x1863f7c in ui_builtin_start /home/travis/build/neovim/neovim/src/nvim/ui.c:125:3
= 4 0xe41bb9 in main /home/travis/build/neovim/neovim/src/nvim/main.c:457:5
= 5 0x2b2ba340bf44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
---
Alternative attempt:
commit 6ad9c02491606a0c31e907f38c9931f324327aa5
Author: Justin M. Keyes <justinkz@gmail.com>
Date: Sat Jan 27 15:12:58 2018 +0100
tui: fix use-after-free: swap in empty scheduler
This should make life easier for UIs like VimR which implement their own
in-process bridged UI: they don't need to worry that their `scheduler`
might receive an invalid pointer.
To avoid that, ui_bridge_stopped() swaps in an empty scheduler. Note
that this requires the call to loop_poll_events() to be moved into the
critical section.
diff --git a/src/nvim/ui_bridge.c b/src/nvim/ui_bridge.c
index 779585416f80..491052d19d3b 100644
--- a/src/nvim/ui_bridge.c
+++ b/src/nvim/ui_bridge.c
@@ -93,10 +93,18 @@ UI *ui_bridge_attach(UI *ui, ui_main_fn ui_main, event_scheduler scheduler)
return &rv->bridge;
}
+static void ui_bridge_null_scheduler(Event event, void *d)
+{
+ WLOG("ignoring event (bridge stopped)");
+}
+
void ui_bridge_stopped(UIBridgeData *bridge)
{
uv_mutex_lock(&bridge->mutex);
bridge->stopped = true;
+ // Replace with an empty scheduler, so that the UI internal scheduler does
+ // not get invoked with an invalid pointer. #7922
+ bridge->scheduler = ui_bridge_null_scheduler;
uv_mutex_unlock(&bridge->mutex);
}
@@ -111,14 +119,11 @@ static void ui_bridge_stop(UI *b)
UIBridgeData *bridge = (UIBridgeData *)b;
bool stopped = bridge->stopped = false;
UI_BRIDGE_CALL(b, stop, 1, b);
- for (;;) {
+ while (!stopped) {
uv_mutex_lock(&bridge->mutex);
stopped = bridge->stopped;
- uv_mutex_unlock(&bridge->mutex);
- if (stopped) {
- break;
- }
loop_poll_events(&main_loop, 10); // Process one event (at most).
+ uv_mutex_unlock(&bridge->mutex);
}
uv_thread_join(&bridge->ui_thread);
uv_mutex_destroy(&bridge->mutex);
- Revert timer-based approach.
- Instead, call loop_poll_events() with a timeout in an "active" loop,
to infer that "TUI startup activity has mostly finished", but also to
enforce a mininum time (100 ms) before emitting "enable focus
reporting" termcode. (If TUI startup takes longer than that minimum
time, it's probably a slow environment anyways.)
- Tickle `main_loop` by sending a dummy event. Without this, the
initial "focus-gained" response from the terminal may not get
processed until the user hits a key.
ref #7720
ref #7664
ref #7649
ref #7664
ref 27f9b1c7b0
Asynchronous API functions are served immediately, which means pending
input could change the state of Nvim shortly after an async API function
result is returned.
nvim_get_mode() is different:
- If RPCs are known to be blocked, it responds immediately (without
flushing the input/event queue)
- else it is handled just-in-time before waiting for input, after
pending input was processed. This makes the result more reliable
(but not perfect).
Internally this is handled as a special case, but _semantically_ nothing
has changed: API users never know when input flushes, so this internal
special-case doesn't violate that. As far as API users are concerned,
nvim_get_mode() is just another asynchronous API function.
In all cases nvim_get_mode() never blocks for more than the time it
takes to flush the input/event queue (~µs).
Note: This doesn't address #6166; nvim_get_mode() will provoke #6166 if
e.g. `d` is operator-pending.
Closes#6159
Closes#1234
multiqueue:
- Implement multiqueue_size()
- Rename MultiQueueItem.parent to MultiQueueItem.parent_item, to avoid confusion
with MultiQueue.parent.
`lib/queue.h` implements a basic queue. `event/queue.c` implements
a specialized data structure on top of lib/queue.h; it is not a "normal"
queue.
Rename the specialized multi-level queue implemented in event/queue.c to
"multiqueue", to avoid confusion when reading the code.
Before this change one can eventually notice that "macros (uppercase
symbols) are for the normal queue, lowercase operations are for the
multi-level queue", but that is unnecessary friction for new developers
(or existing developers just visiting this part of the codebase).
Because terminfo_stop() already ran, there is not much reason to wait
for the loop to teardown.
Helped-by: Björn Linse <bjorn.linse@gmail.com>
Helped-by: oni-link <knil.ino@gmail.com>
Closes#4778
References #3541
---
Bug report:
> After pressing `ZZ` I can find two threads freezing, occupying 100% CPU:
|-systemd-+
|-nvim,11567 /home/lz/code/1.rs +set title
| `-{nvim},11574
> 11567 has two threads:
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f7622907780 (LWP 11567) "nvim" 0x00007f76222e66bd in pthread_join () from /usr/lib/libpthread.so.0
2 Thread 0x7f761f5ff700 (LWP 11574) "nvim" 0x00007ffcec9e9c59 in clock_gettime ()
(gdb) thread apply all bt
Thread 2 (Thread 0x7f761f5ff700 (LWP 11574)):
#0 0x00007ffcec9e9c59 in clock_gettime ()
#1 0x00007f76210b9356 in clock_gettime () from /usr/lib/libc.so.6
#2 0x00007f7622513e3c in ?? () from /usr/lib/libuv.so.1
#3 0x00007f7622505e94 in uv_run () from /usr/lib/libuv.so.1
#4 0x00000000004822ed in loop_close (loop=loop@entry=0x7f761f5fe870) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/loop.c:87
#5 0x00000000005a7ec0 in tui_main (bridge=0x7f761f6ac000, ui=0x7f761f69ace0) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/tui/tui.c:234
#6 0x00000000005a9b47 in ui_thread_run (data=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/ui_bridge.c:87
#7 0x00007f7622510d07 in ?? () from /usr/lib/libuv.so.1
#8 0x00007f76222e5474 in start_thread () from /usr/lib/libpthread.so.0
#9 0x00007f76210ac69d in clone () from /usr/lib/libc.so.6
Thread 1 (Thread 0x7f7622907780 (LWP 11567)):
#0 0x00007f76222e66bd in pthread_join () from /usr/lib/libpthread.so.0
#1 0x00007f7622510dae in uv_thread_join () from /usr/lib/libuv.so.1
#2 0x00000000005aac9e in ui_bridge_stop (b=0x7f761f6ac000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/ui_bridge.c:104
#3 0x00000000005a90e4 in ui_builtin_stop () at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/ui.c:91
#4 0x000000000052be09 in mch_exit (r=r@entry=1) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os_unix.c:143
#5 0x00000000004db1cc in getout (exitval=exitval@entry=1) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/main.c:623
#6 0x00000000004fa43c in preserve_exit () at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/misc1.c:2652
#7 0x000000000052b77a in deadly_signal (signum=1) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/signal.c:120
#8 0x000000000052b7cf in on_signal (handle=<optimized out>, signum=<optimized out>, data=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/signal.c:145
#9 0x0000000000484178 in signal_event (argv=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/signal.c:44
#10 0x0000000000483b7f in queue_process_events (queue=0x7f7620417360) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/queue.c:142
#11 0x0000000000482208 in loop_poll_events (loop=0x84dec0 <loop>, ms=ms@entry=4000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/event/loop.c:56
#12 0x000000000052a364 in input_poll (ms=ms@entry=4000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/input.c:325
#13 0x000000000052a3e2 in inbuf_poll (ms=4000) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/input.c:347
#14 0x000000000052a839 in os_inchar (buf=buf@entry=0x0, maxlen=maxlen@entry=0, ms=ms@entry=-1, tb_change_cnt=tb_change_cnt@entry=0) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/os/input.c:107
#15 0x0000000000592eeb in state_enter (s=s@entry=0x7ffcec9d3560) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/state.c:46
#16 0x0000000000508533 in normal_enter (cmdwin=cmdwin@entry=false, noexmode=noexmode@entry=false) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/normal.c:464
#17 0x00000000004dc17e in main (argc=<optimized out>, argv=<optimized out>) at /tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src/nvim/main.c:538
- Implement `loop_schedule` method for queueing events from other threads
- Make `loop_poll_events` `recursive` static variable a field of the Loop
structure
The new event processing architecture changed `jobwait()` semantics: Only one
job is processed at time since process_wait only focuses on one queue.
This fixes the problem with a few changes:
- Allow the event queue polled by `process_wait` to be overriden by a new
argument.
- Allow the parent queue to be overriden with `queue_replace_parent`
- Create a temporary queue that serves as the parent for all jobs passed to
`jobwait()`
- Improve the implementation of deferred/immediate events.
- Use the new queue module to change how/when events are queued/processed by
giving a private queue to each emitter.
- Immediate events(which only exist to break uv_run recursion) are now
represented in the `loop->fast_events` queue.
- Events pushed to child queues are propagated to the event loop main queue and
processed as K_EVENT keys.
- New libuv/pty process abstraction with simplified API and no globals.
- Remove nvim/os/job*. Jobs are now a concept that apply only to programs
spawned by vimscript job* functions.
- Refactor shell.c/channel.c to use the new module, which brings a number of
advantages:
- Simplified API, less code
- No slots in the user job table are used
- Not possible to acidentally receive data from vimscript
- Implement job table in eval.c, which is now a hash table with unilimited job
slots and unique job ids.
- Add event loop abstraction module under src/nvim/event. The
src/nvim/event/loop module replaces src/nvim/os/event
- Remove direct dependency on libuv signal/timer API and use the new abstraction
instead.
- Replace all references to uv_default_loop() by &loop.uv, a new global variable
that wraps libuv main event loop but allows the event loop functions to be
reused in other contexts.
