Coverity warning is a false positive: if rbuffer_read_ptr() returns
NULL then `cnt` is zero.
Revert 76ea97c809 (which caused
the TSan build to hang often--possibly because of the missing ui_flush()).
Instead, modify out_data_append_to_screen() to check for NULL.
ref #6862
rbuffer_read_ptr may return a null
if ptr == null && cnt == 0 && !out_data_decide_throttle(cnt)
then we would have called out_data_append_to_screen(ptr, cnt, eof)
which dereferences the null pointer.
- Establish ERROR log level as "critical". Such errors are rare and will
be valuable when users encounter unusual circumstances.
- Set -DMIN_LOG_LEVEL=3 for release-type builds
Tokenize p_sh if used as default in ex_terminal(). Previously p_sh was
used as the first arg in a list when calling termopen(), this would try
to call an untokenized version of shell, meaning if you had an argument
in 'shell':
set shell=/bin/bash\ --login
the command would fail.
Helped-by: oni-link <knil.ino@gmail.com>
Closes#3999
Instead of managing max_visits, check the time every N visits. This avoids edge
cases where max_visits is large but the chunk frequency slowed down, which would
causing the "..." pulse to show for a very long time. It's more important to
show output at reasonable intervals than to avoid calling os_hrtime().
This ameliorates use-cases like:
:!cat foo.txt
:make
where the user is interested in the last few lines of output.
Try these shell-based ex-commands before/after this commit:
:grep -r '' *
:make
:!yes
:!grep -r '' *
:!git grep ''
:!cat foo
:!echo foo
:!while true; do date; done
:!for i in `seq 1 20000`; do echo XXXXXXXXXX $i; done
In all cases the last few lines of the command should always be shown,
regardless of where throttling was triggered.
Periodically skip :! spam. This is a "cheat" that works for all UIs and greatly
improves responsiveness when :! spams MB or GB of output:
:!yes
:!while true; do date; done
:!git grep ''
:grep -r '' *
After ~10KB of data is seen from a single :! invocation, output will be skipped
for ~1s and three dots "..." will pulse in the bottom-left. Thereafter the
behavior alternates at every:
* 10KB received
* ~1s throttled
This also avoids out-of-memory which could happen with large :! outputs.
Note: This commit does not change the behavior of execute(':!foo').
execute(':!foo') returns the string ':!foo^M', it captures *only* Vim
messages, *not* shell command output. Vim behaves the same way.
Use system('foo') for capturing shell command output.
Closes#1234
Helped-by: oni-link <knil.ino@gmail.com>
Closes#3529Closes#5241
In Vim,
:echo system('cat - &', 'foo')
works because for both system() and :! Vim writes input to a temp file and uses
shell syntax to redirect the file to the backgrounded `cat` (get_cmd_output()
.. make_filter_cmd()).
In Nvim,
:echo system('cat - &', 'foo')
fails because we write the input directly via pipes (shell.c:do_os_system()),
but (per POSIX[1]) backgrounded process input stream is redirected from
/dev/null (unless overridden by shell redirection; supported only by some shells
[2]), so our writes are ignored, the process exits quickly, and if we are
writing data larger than the buffer size we'll see EPIPE.
This still works:
:%w !tee > foo1358.txt &
but this does not:
:%w !tee foo1358.txt &
though it *should* (why doesn't it?) because we still do the temp file dance
in do_bang() .. do_filter().
[1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_03_02
[2] http://unix.stackexchange.com/a/71218
`lib/queue.h` implements a basic queue. `event/queue.c` implements
a specialized data structure on top of lib/queue.h; it is not a "normal"
queue.
Rename the specialized multi-level queue implemented in event/queue.c to
"multiqueue", to avoid confusion when reading the code.
Before this change one can eventually notice that "macros (uppercase
symbols) are for the normal queue, lowercase operations are for the
multi-level queue", but that is unnecessary friction for new developers
(or existing developers just visiting this part of the codebase).
move `call_shell` to misc1.c
Move some fns to state.c
Move some fns to option.c
Move some fns to memline.c
Move `vim_chdir*` fns to file_search.c
Move some fns to new module, bytes.c
Move some fns to fileio.c
Current name is inappropriate for the following reasons:
1. It is often masked by local `loop` variables.
2. It cannot be searched for. There are many `loop` variables where `loop` is
some local variable. There are many cases when “loop” word is used in
a comment.
3. It is in any case bad idea to use a generic name as a name of the global
variable. Best if global has module prefix: this is why it is in `main.h`:
`main_loop` both stands for “a main loop” and “a loop defined in `main.*`”.
Since I have no idea how to list every occurrence of this variable method used
to rename it is “remove it from globals.h, try to compile, fix errors”. Thus if
some occurrence was hidden under false `#if` branch it was not replaced.
The whole stream buffer is now put on screen at once instead of only
data up to the last newline. This has some advantages:
* RBuffer cannot wrap around, so we never forget to output second
half of the buffer.
* Stream data is not delayed anymore, because we don't have to wait for
a newline.
This works by remembering the last used screen column.
Problem: On some systems automatically adding the missing EOL causes
problems. Setting 'binary' has too many side effects.
Solution: Add the 'fixeol' option, default on. (Pavel Samarkin)
34d72d4b6c
The new event processing architecture changed `jobwait()` semantics: Only one
job is processed at time since process_wait only focuses on one queue.
This fixes the problem with a few changes:
- Allow the event queue polled by `process_wait` to be overriden by a new
argument.
- Allow the parent queue to be overriden with `queue_replace_parent`
- Create a temporary queue that serves as the parent for all jobs passed to
`jobwait()`
- Improve the implementation of deferred/immediate events.
- Use the new queue module to change how/when events are queued/processed by
giving a private queue to each emitter.
- Immediate events(which only exist to break uv_run recursion) are now
represented in the `loop->fast_events` queue.
- Events pushed to child queues are propagated to the event loop main queue and
processed as K_EVENT keys.
out_data_cb() can return without emptying the full RBuffer (no NL was
seen). Because the shell output stream is stopped until space in the
Rbuffer is freed up, no more shell output is written.
To prevent this, output the full RBuffer when write_output() did not
write anything.
write_output() can also process the same RBuffer content more than once,
if no NL was seen. To prevent NUL bytes from producing new lines (if
lines are not written to a buffer), translate NUL to SOH(1).
Fixes#2983
- New libuv/pty process abstraction with simplified API and no globals.
- Remove nvim/os/job*. Jobs are now a concept that apply only to programs
spawned by vimscript job* functions.
- Refactor shell.c/channel.c to use the new module, which brings a number of
advantages:
- Simplified API, less code
- No slots in the user job table are used
- Not possible to acidentally receive data from vimscript
- Implement job table in eval.c, which is now a hash table with unilimited job
slots and unique job ids.
- Simplify RStream/WStream API and make it more consistent with libuv.
- Move into the event loop layer(event subdirectory)
- Remove uv_helpers module.
- Simplify job/process internal modules/API.
- Unify RStream and WStream into a single structure. This is necessary because
libuv streams can be readable and writable at the same time(and because the
uv_helpers.c hack to associate multiple streams with libuv handle was removed)
- Make struct definition public, allowing more flexible/simple memory
management by users of the module.
- Adapt channel/job modules to cope with the changes.
- Add event loop abstraction module under src/nvim/event. The
src/nvim/event/loop module replaces src/nvim/os/event
- Remove direct dependency on libuv signal/timer API and use the new abstraction
instead.
- Replace all references to uv_default_loop() by &loop.uv, a new global variable
that wraps libuv main event loop but allows the event loop functions to be
reused in other contexts.
Extract the RBuffer class from rstream.c and reimplement it as a ring buffer,
a more efficient version that doesn't need to relocate memory.
The old rbuffer_read/rbuffer_write interfaces are kept for simple
reading/writing, and the RBUFFER_UNTIL_{FULL,EMPTY} macros are introduced to
hide wrapping logic when more control is required(such as passing the buffer
pointer to a library function that writes directly to the pointer)
Also add a basic infrastructure for writing helper C files that are only
compiled in the unit test library, and use this to write unit tests for RBuffer
which contains some macros that can't be accessed directly by luajit.
Helped-by: oni-link <knil.ino@gmail.com>
Reviewed-by: oni-link <knil.ino@gmail.com>
Reviewed-by: Scott Prager <splinterofchaos@gmail.com>
Reviewed-by: Justin M. Keyes <justinkz@gmail.com>
Reviewed-by: Michael Reed <m.reed@mykolab.com>
Problem : Argument with 'nonnull' attribute passed null @ 203.
Diagnostic : False positive.
Rationale : Problem is supposed to appear when argv[0] is NULL within
do_os_system. But argv is being generated by
shell_build_argv(), which implies argv[0] is the current
value for 'shell' option. Now, option has a non-null
default ($SHELL or "sh"), and, if set by the user, it can
be empty, but not NULL. So, argv[0] can never be NULL.
Resolution : Assert shell_build_argv() postcondition.
For any of these functions, if {cmd} is a string, execute
"&shell &shellcmdflag '{cmd}'", or simply {cmd} if it's a list.
In termopen(), if the 'name' option is not supplied, try to guess using
'{cmd}' (string) or {cmd}[0] (list). Simplify ex_terminal to use the
string form of termopen().
termopen: get name from argument
Convert list_to_argv to tv_to_argv.
Helped-by: Björn Linse <@bfredl>
Helped-by: oni-link <knil.ino@gmail.com>
Helped-by: Thiago de Arruda <@tarruda>
We already use wrappers for allocation, the new `xfree` function is the
equivalent for deallocation and provides a way to fully replace the malloc
implementation used by Neovim.
Even though assuming nvim is busy most times is simpler, it has a problem: A lot
of unnecessary busy_start/busy_stop notifications are sent to the UI. That's
because in the majority of scenarios almost no time is spent between
`event_poll` calls.
This restores the normal behavior which is to call busy_start only when nvim is
going to perform some task that can take a significant amount of time. Also
improve the usage of buffering in the TUI when changing the cursor state.
Switching cursor off is only necessary in two occasions:
- When redrawing to avoid terminal flickering
- When the editor is busy
The first can now be handled by the TUI, so most calls to ui_cursor_off can be
removed from the core.
So, before this commit it was only necessary to switch the cursor off to notify
the user that nvim was running some long operation. Now the cursor_{on,off}
functions have been replaced by busy_{stop,start} which can be handled in a
UI-specific way(turning the cursor off or showing a busy indicator, for
example).
To make things even more simpler, nvim is always busy except when waiting for
user input or other asynchronous events: It automatically switches to a non-busy
state when the event loop is about to be entered for more than 100 milliseconds.
`ui_busy_start` can be called when its not desired to change the busy state in
the event loop (As its now done by functions that perform blocking shell
invocations).
- process spawning was decoupled from the rest of the job control logic. The
goal is reusing it for spawning processes connected to pseudo terminal file
descriptors.
- job_start now receives a JobOptions structure containing all the startup
options.
- Removed term.c, term.h and term_defs.h
- Tests for T_* values were removed. screen.c was simplified as a
consequence(the best strategy for drawing is implemented in the UI layer)
- Redraw functions now call ui.c functions directly. Updates are flushed with
`ui_flush()`
- Removed all termcap options(they now return empty strings for compatibility)
- &term/&ttybuiltin options return a constant value(nvim)
- &t_Co is still available, but it mirrors t_colors directly
- Remove cursor tracking from screen.c and the `screen_start` function. Now the
UI is expected to maintain cursor state across any call, and reset it when
resized.
- Remove unused code
Shell command output was broken in @8a5a8db, which refactored nvim to no longer
switch to cooked mode(linefeeds are processed differently).
Fix the problem by refactoring write_output to accept to extra arguments that
control the flushing behavior and where data will be written to: buffer or
directly to the screen.
Nvim now relies much less on setting terminal mode to cooked mode, remove most
calls to settmode, except for those that happen on startup or when suspending.
Eventually even those will be handled by the UI layer.
