Previously, when neovim would wrap a line across multiple lines,
terminal emulators could not detect that the lines represent a single
wrapped line as opposed to several separate lines. As a result, many
terminals' selection/copying functionality would treat a wrapped line as
several newline-delimited lines.
Fix this by reenabling a "special trick" from Vim. When a line is
wrapped, write the last character of that line followed by the first
character of the next line to the terminal. This hints to the terminal
that the next line is a continuation of the current line.
Extends the raw_line event with a "wrap" parameter which controls when
to do wrap hinting.
Add ext_newgrid and ext_hlstate extensions. These use predefined
highlights and line-segment based updates, for efficiency and
simplicity.. The ext_hlstate extension in addition allows semantic
identification of builtin and syntax highlights.
Reimplement the old char-based updates in the remote UI layer, for
compatibility. For the moment, this is still the default. The bulitin
TUI uses the new line-based protocol.
cmdline uses curwin cursor position when ext_cmdline is active.
fixes#6974
Before this change, the partial could be freed before the last due
callback got invoked, which caused a use-after-free when the due
callback called the partial.
1. Don't check elapsed time in children_kill_cb(), it's already implied
by the start-time of the timer itself.
2. Restart timer from children_kill_cb() for PTY jobs, to send SIGKILL
after SIGTERM. There is an edge case where SIGKILL might follow
SIGTERM too quickly, if jobstop() is called near the 2-second timer
window. But this edge case is not worth code complication.
Before f31c26f1af the timer was used to try SIGTERM *and* SIGKILL, so
a repeating timer was needed. After f31c26f1af process_stop() sends
SIGTERM immediately, and the timer only sends SIGKILL.
So we don't need a repeating timer.
- Simplifies the logic: don't need to call uv_timer_stop() explicitly.
- Avoids a problem: if process_stop() is called more than once in the
2-second window, the first on_process_exit() would call
uv_timer_stop() which stops the timer for all stopped processes.
children_kill_cb() is racey. One obvious problem is that
process_close_handles() is *queued* by on_process_exit(), so when
children_kill_cb() is invoked, the dead process might still be in the
`loop->children` list. If the OS already reclaimed the dead PID, Nvim
may try to SIGKILL it.
Avoid that by checking `proc->status`.
Vim doesn't have this problem because it doesn't attempt to kill
processes that ignored SIGTERM after a timeout.
closes#8269
XXX: comment at https://stackoverflow.com/q/1173342 :
> Windows recycles PIDs quite fast, you have to be extra careful not
> to kill unrelated processes. These APIs will report PPIDs for long
> dead processes whose PIDs may have been recycled. Check the parent
> start date to make sure it is related to the processes you spawned.
UV_PROCESS_DETACHED compels libuv:uv__process_child_init() to call
setsid() in the child just after fork(). That ensures the process and
its descendants are grouped in a separate session (and process group).
The following jobstart() call correctly groups `sh` and `sleep` in a new
session (and process-group), where `sh` is the "session leader" (and
process-group leader):
:call jobstart(['sh','-c','sleep 60'])
SESN PGRP PID PPID Command
30383 30383 30383 3620 │ ├─ -bash
30383 31432 31432 30383 │ │ └─ nvim -u NORC
30383 31432 31433 30383 │ │ ├─ nvim -u NORC
8105 8105 8105 31432 │ │ └─ sh -c sleep 60
8105 8105 8106 8105 │ │ └─ sleep 60
closes#6530
ref: https://stackoverflow.com/q/1046933
ref: https://unix.stackexchange.com/a/404065
Helped-by: Marco Hinz <mh.codebro+github@gmail.com>
Discussion
------------------------------------------------------------------------
On my linux box before this patch, the termclose_spec.lua:'kills job
trapping SIGTERM' test indirectly causes cmake/busted to wait for 60s.
That's because the test spawns a `sleep 60` descendant process which
hangs around even after nvim exits: nvim killed the parent PID, but not
PGID (process-group), so the grandchild "reparented" to init (PID 1).
Session contains processes (and process-groups) which are logically part
of the same "login session". Process-group is a set of
logically/informally-related processes within a session; for example,
shells assign a process group to each "job". Session IDs and PGIDs both
have type pid_t (like PIDs).
These OS-level mechanisms are, as usual, legacy accidents whose purpose
is upheld by convention and folklore. We can use session-level grouping
(setsid), or we could use process-group-level grouping (setpgid).
Vim uses setsid() if available, otherwise setpgid(0,0).
Windows
------------------------------------------------------------------------
UV_PROCESS_DETACHED on win32 sets CREATE_NEW_PROCESS_GROUP flag.
But uv_kill() does not kill the process-group:
https://github.com/nodejs/node/issues/3617
Ideas:
- Set UV_PROCESS_DETACHED (CREATE_NEW_PROCESS_GROUP), then call
GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, pid)
- Maybe won't work because MSDN says "Only processes that share the
same console as the calling process receive the signal."
https://docs.microsoft.com/en-us/windows/console/generateconsolectrlevent
But CREATE_NEW_PROCESS_GROUP creates a new console ...
ref https://stackoverflow.com/q/1453520
- Group processes within a "job". libuv does that *globally* for
non-detached processes: uv__init_global_job_handle.
- Iterate through CreateToolhelp32Snapshot().
- https://stackoverflow.com/q/1173342
- Vim does this, see terminate_all()
ui_bridge:ui_bridge_stop() calls ui_detach_impl() last, so the check for
ui_active() in ui:ui_refresh() doesn't help: tui_main() already freed
the `ui` object.
There is a race between ui_bridge_stop (thread T0) and tui_main (thread T1).
UIBridgeData.stopped could be set while ui_bridge_stop() is in the
middle of loop_poll_events(), which may invoke tui_scheduler() on T0.
The pointers in tui_scheduler() may be invalid by then.
Solution(?): Use the `UI.data` field as a "stopped" flag and check it in
tui_scheduler().
ASAN use-after-free report observed in #7908:
= ==20066==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000000cd0 at pc 0x00000182abed bp 0x7ffe23b07070 sp 0x7ffe23b07068
= READ of size 8 at 0x611000000cd0 thread T0
= 0 0x182abec in tui_scheduler /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:393:23
= 1 0x1876afd in ui_bridge_update_fg /home/travis/build/neovim/neovim/build/src/nvim/auto/ui_events_bridge.generated.h:205:3
= 2 0x186c130 in ui_resize /home/travis/build/neovim/neovim/src/nvim/ui.c:310:3
= 3 0x146b9c2 in screen_resize /home/travis/build/neovim/neovim/src/nvim/screen.c:7483:3
= 4 0x186a6f0 in ui_refresh /home/travis/build/neovim/neovim/src/nvim/ui.c:284:3
= 5 0x186bbe0 in ui_refresh_event /home/travis/build/neovim/neovim/src/nvim/ui.c:297:3
= 6 0xa2219a in multiqueue_process_events /home/travis/build/neovim/neovim/src/nvim/event/multiqueue.c:150:7
= 7 0xa1bd7f in loop_poll_events /home/travis/build/neovim/neovim/src/nvim/event/loop.c:63:3
= 8 0x1872709 in ui_bridge_stop /home/travis/build/neovim/neovim/src/nvim/ui_bridge.c:121:5
= 9 0x1864247 in ui_builtin_stop /home/travis/build/neovim/neovim/src/nvim/ui.c:143:3
= 10 0x1249ec8 in mch_exit /home/travis/build/neovim/neovim/src/nvim/os_unix.c:140:3
= 11 0xe56ba9 in getout /home/travis/build/neovim/neovim/src/nvim/main.c:671:3
= 12 0xfc4c8f in preserve_exit /home/travis/build/neovim/neovim/src/nvim/misc1.c:2653:3
= 13 0x1247c02 in deadly_signal /home/travis/build/neovim/neovim/src/nvim/os/signal.c:137:3
= 14 0x1247921 in on_signal /home/travis/build/neovim/neovim/src/nvim/os/signal.c:162:9
= 15 0xa35618 in signal_event /home/travis/build/neovim/neovim/src/nvim/event/signal.c:47:3
= 16 0xa2219a in multiqueue_process_events /home/travis/build/neovim/neovim/src/nvim/event/multiqueue.c:150:7
= 17 0xa1bd7f in loop_poll_events /home/travis/build/neovim/neovim/src/nvim/event/loop.c:63:3
= 18 0x1237bd6 in input_poll /home/travis/build/neovim/neovim/src/nvim/os/input.c:349:3
= 19 0x123334f in inbuf_poll /home/travis/build/neovim/neovim/src/nvim/os/input.c:372:24
= 20 0x123316d in os_inchar /home/travis/build/neovim/neovim/src/nvim/os/input.c:110:19
= 21 0x170d20e in state_enter /home/travis/build/neovim/neovim/src/nvim/state.c:55:13
= 22 0xbd7441 in command_line_enter /home/travis/build/neovim/neovim/src/nvim/ex_getln.c:384:3
= 23 0xbd0a60 in getcmdline /home/travis/build/neovim/neovim/src/nvim/ex_getln.c:1920:10
= 24 0xbdb365 in getexline /home/travis/build/neovim/neovim/src/nvim/ex_getln.c:2100:10
= 25 0xb00a6b in do_cmdline /home/travis/build/neovim/neovim/src/nvim/ex_docmd.c:528:47
= 26 0x10a7837 in nv_colon /home/travis/build/neovim/neovim/src/nvim/normal.c:4552:18
= 27 0x1091e15 in normal_execute /home/travis/build/neovim/neovim/src/nvim/normal.c:1136:3
= 28 0x170d439 in state_enter /home/travis/build/neovim/neovim/src/nvim/state.c:67:26
= 29 0x104ee14 in normal_enter /home/travis/build/neovim/neovim/src/nvim/normal.c:466:3
= 30 0xe4295c in main /home/travis/build/neovim/neovim/src/nvim/main.c:572:3
= 31 0x2b2ba340bf44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
= 32 0x44d24b in _start (/home/travis/build/neovim/neovim/build/bin/nvim+0x44d24b)
=
= 0x611000000cd0 is located 16 bytes inside of 240-byte region [0x611000000cc0,0x611000000db0)
= freed by thread T1 here:
= 0 0x4ee0e2 in __interceptor_free /local/mnt/workspace/tmp/ubuntu_rel/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:47:3
= 1 0xf4f6d4 in xfree /home/travis/build/neovim/neovim/src/nvim/memory.c:133:3
= 2 0x182a963 in tui_main /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:383:3
= 3 0x18792b0 in ui_thread_run /home/travis/build/neovim/neovim/src/nvim/ui_bridge.c:106:3
= 4 0x2b2ba2697183 in start_thread /build/eglibc-ripdx6/eglibc-2.19/nptl/pthread_create.c:312
=
= previously allocated by thread T0 here:
= 0 0x4ee61a in calloc /local/mnt/workspace/tmp/ubuntu_rel/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:76:3
= 1 0xf4f787 in xcalloc /home/travis/build/neovim/neovim/src/nvim/memory.c:147:15
= 2 0x182000a in tui_start /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:127:12
= 3 0x1863f7c in ui_builtin_start /home/travis/build/neovim/neovim/src/nvim/ui.c:125:3
= 4 0xe41bb9 in main /home/travis/build/neovim/neovim/src/nvim/main.c:457:5
= 5 0x2b2ba340bf44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
=
= Thread T1 created by T0 here:
= 0 0x4d774d in __interceptor_pthread_create /local/mnt/workspace/tmp/ubuntu_rel/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317:3
= 1 0x1aae6b0 in uv_thread_create /home/travis/nvim-deps/build/src/libuv/src/unix/thread.c:75
= 2 0x18217fa in tui_start /home/travis/build/neovim/neovim/src/nvim/tui/tui.c:159:10
= 3 0x1863f7c in ui_builtin_start /home/travis/build/neovim/neovim/src/nvim/ui.c:125:3
= 4 0xe41bb9 in main /home/travis/build/neovim/neovim/src/nvim/main.c:457:5
= 5 0x2b2ba340bf44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
---
Alternative attempt:
commit 6ad9c02491606a0c31e907f38c9931f324327aa5
Author: Justin M. Keyes <justinkz@gmail.com>
Date: Sat Jan 27 15:12:58 2018 +0100
tui: fix use-after-free: swap in empty scheduler
This should make life easier for UIs like VimR which implement their own
in-process bridged UI: they don't need to worry that their `scheduler`
might receive an invalid pointer.
To avoid that, ui_bridge_stopped() swaps in an empty scheduler. Note
that this requires the call to loop_poll_events() to be moved into the
critical section.
diff --git a/src/nvim/ui_bridge.c b/src/nvim/ui_bridge.c
index 779585416f80..491052d19d3b 100644
--- a/src/nvim/ui_bridge.c
+++ b/src/nvim/ui_bridge.c
@@ -93,10 +93,18 @@ UI *ui_bridge_attach(UI *ui, ui_main_fn ui_main, event_scheduler scheduler)
return &rv->bridge;
}
+static void ui_bridge_null_scheduler(Event event, void *d)
+{
+ WLOG("ignoring event (bridge stopped)");
+}
+
void ui_bridge_stopped(UIBridgeData *bridge)
{
uv_mutex_lock(&bridge->mutex);
bridge->stopped = true;
+ // Replace with an empty scheduler, so that the UI internal scheduler does
+ // not get invoked with an invalid pointer. #7922
+ bridge->scheduler = ui_bridge_null_scheduler;
uv_mutex_unlock(&bridge->mutex);
}
@@ -111,14 +119,11 @@ static void ui_bridge_stop(UI *b)
UIBridgeData *bridge = (UIBridgeData *)b;
bool stopped = bridge->stopped = false;
UI_BRIDGE_CALL(b, stop, 1, b);
- for (;;) {
+ while (!stopped) {
uv_mutex_lock(&bridge->mutex);
stopped = bridge->stopped;
- uv_mutex_unlock(&bridge->mutex);
- if (stopped) {
- break;
- }
loop_poll_events(&main_loop, 10); // Process one event (at most).
+ uv_mutex_unlock(&bridge->mutex);
}
uv_thread_join(&bridge->ui_thread);
uv_mutex_destroy(&bridge->mutex);
- Revert timer-based approach.
- Instead, call loop_poll_events() with a timeout in an "active" loop,
to infer that "TUI startup activity has mostly finished", but also to
enforce a mininum time (100 ms) before emitting "enable focus
reporting" termcode. (If TUI startup takes longer than that minimum
time, it's probably a slow environment anyways.)
- Tickle `main_loop` by sending a dummy event. Without this, the
initial "focus-gained" response from the terminal may not get
processed until the user hits a key.
ref #7720
ref #7664
ref #7649
ref #7664
ref 27f9b1c7b0
ref #7649
ref #766427f9b1c7b0 caused a regression: it uses loop_schedule_deferred() to
defer emitting the "enable focus reporting" termcode. tui_main() never
processes `tui_loop.events` (which loop_schedule_deferred() depends on),
so the event was never actually processed.
But fixing that (by processing `tui_loop.events`) would bring back the
problem 27f9b1c7b0 tried to fix: it still emits the event too soon.
Instead, do a little dance: schedule the event on `main_loop` and then
forward it to `tui_loop`.
NOTE: after this commit, in tmux 2.3 with `focus-events` enabled,
FocusGained is fired on startup and when resuming from suspend.
Using `script` to record the terminal session (and `vterm-dump` to
post-process the result):
BEFORE:
{DECSM 1049}{DECSM 1}{ESC =}
{CUP *}{ED *}{DECSM 2004}{DECSM 1004}{CSI 1,43 r}
{CUP 1,1}
{CUP *}{ED *}{SM 34}{DECSM 25}
{DECRM 25}{CSI 2 q}{CSI 2 q}
{CUP *}{ED *}{LF}
{SGR *}{LS1}{SGR 94}~
...
AFTER:
{CUP *}{ED *}{CSI 1,43 r}
{CUP 1,1}
{CUP *}{ED *}{SM 34}{DECSM 25}
{DECRM 25}{CSI 2 q}{CSI 2 q}
{CUP *}{ED *}{DECSM 2004}{DECSM 1004}{LF}
{SGR *}{LS1}{SGR 94}~
...
Doc for UV_PROCESS_DETACHED in uv.h mentions:
> child process will still keep the parent's event loop alive unless
> the parent process calls uv_unref() on the child's process handle.
ref #3944
Although in_port_t is a typedef for uint16_t, GCC in Ubuntu 12.04
complains about potential loss of data due to converting int to
uint16_t. Since we know this isn't possible, silence the warning to
avoid breaking QB until it gets upgraded to a newer Ubuntu.
When using serverstart("ip.ad.d.r:") to listen on a random port, we need
to abide by getaddrinfo()'s API and pass in a NULL service, rather than
an empty string.
When given an empty string, getaddrinfo() is free to search for a
service by the given name (since the string isn't a number) which will
fail. At least FreeBSD does perform this lookup.
This change implicitly adds IPv6 support.
If the address contains ":", we try to use a TCP socket instead of a Unix domain
socket. Everything in front of the last occurrence of ":" is the hostname and
everything after it the port.
If the hostname lookup fails, we fall back to using a Unix domain socket.
If the port is empty ("localhost:"), a random port will be assigned.
Examples:
NVIM_LISTEN_ADDRESS=localhost:12345 -> TCP (IPv4 or IPv6), port: 12345
NVIM_LISTEN_ADDRESS=localhost: -> TCP (IPv4 or IPv6), port: random (> 1024)
NVIM_LISTEN_ADDRESS=localhost:0 -> TCP (IPv4 or IPv6), port: random (> 1024)
NVIM_LISTEN_ADDRESS=localhost -> Unix domain socket "localhost" in current dir
Asynchronous API functions are served immediately, which means pending
input could change the state of Nvim shortly after an async API function
result is returned.
nvim_get_mode() is different:
- If RPCs are known to be blocked, it responds immediately (without
flushing the input/event queue)
- else it is handled just-in-time before waiting for input, after
pending input was processed. This makes the result more reliable
(but not perfect).
Internally this is handled as a special case, but _semantically_ nothing
has changed: API users never know when input flushes, so this internal
special-case doesn't violate that. As far as API users are concerned,
nvim_get_mode() is just another asynchronous API function.
In all cases nvim_get_mode() never blocks for more than the time it
takes to flush the input/event queue (~µs).
Note: This doesn't address #6166; nvim_get_mode() will provoke #6166 if
e.g. `d` is operator-pending.
Closes#6159
