doc: xdg, MAINTAIN.md, channel-id, job control

- tutor: emphasize K

MAINTAIN.md

@@ -0,0 +1,48 @@
+Maintaining the Neovim project
+==============================
+
+Notes on maintaining the Neovim project.
+
+See also: https://github.com/git/git/blob/master/Documentation/howto/maintain-git.txt
+
+Ticket Triage
+-------------
+
+In practice we haven't found a meaningful way to forecast more precisely than
+"next" and "after next". That means there are usually one or two (at most)
+planned milestones:
+
+- Next bugfix-release (1.0.x)
+- Next feature-release (1.x.0)
+
+The forecasting problem might be solved with an explicit priority system (like
+Bram's todo.txt). Meanwhile the Neovim priority system is defined by:
+
+- PRs nearing completion (RDY).
+- Issue labels. E.g. the +plan label increases the ticket's priority merely for
+  having a plan written down: it is _closer to completion_ than tickets without
+  a plan.
+- Comment activity or new information.
+
+Anything that isn't in the next milestone, and doesn't have a RDY PR ... is
+just not something you care very much about, by construction. Post-release you
+can review open issues, but chances are your next milestone is already getting
+full :)
+
+Release Policy
+--------------
+
+The goal is "early and often".
+
+Up to now we use only one branch, the `master` branch.
+
+- If `master` is unstable we don't release.
+- If the last release has a major bug, we:
+  1. Fix the bug on `master`.
+  2. Disable or remove any known risks present on `master`.
+  3. Cut a release from `master`.
+
+This is a bit silly, but it works ok. And it keeps `master` from biting off
+more feature-creep than it can chew.
+
+See also: https://github.com/neovim/neovim/issues/862

runtime/doc/api.txt

@@ -1120,6 +1120,33 @@ nvim_buf_clear_highlight({buffer}, {src_id}, {line_start}, {line_end})
                     {line_end}    End of range of lines to clear (exclusive)
                                   or -1 to clear to end of file.
 
+                                                 *nvim_buf_set_virtual_text()*
+nvim_buf_set_virtual_text({buffer}, {src_id}, {line}, {chunks},
+                          {opts})
+                Set the virtual text (annotation) for a buffer line.
+
+                By default (and currently the only option) the text will be
+                placed after the buffer text. Virtual text will never cause
+                reflow, rather virtual text will be truncated at the end of
+                the screen line. The virtual text will begin after one cell to
+                the right of the ordinary text, this will contain the |lcs-
+                eol| char if set, otherwise just be a space.
+
+                Parameters: ~
+                    {buffer}  Buffer handle
+                    {src_id}  Source group to use or 0 to use a new group, or
+                              -1 for a ungrouped annotation
+                    {line}    Line to annotate with virtual text (zero-
+                              indexed)
+                    {chunks}  A list of [text, hl_group] arrays, each
+                              representing a text chunk with specified
+                              highlight. `hl_group` element can be omitted for
+                              no highlight.
+                    {opts}    Optional parameters. Currently not used.
+
+                Return: ~
+                    The src_id that was used
+
 
 ==============================================================================
 Window Functions                                                  *api-window*

runtime/doc/channel.txt

@@ -32,8 +32,13 @@ Channels support multiple modes or protocols. In the most basic
 mode of operation, raw bytes are read and written to the channel.
 The |rpc| protocol, based on the msgpack-rpc standard, enables nvim and the
 process at the other end to send remote calls and events to each other.
-Additionally, the builtin |terminal-emulator|, is implemented on top of PTY
+The builtin |terminal-emulator| is also implemented on top of PTY channels.
-channels.
+
+Channel Id						*channel-id*
+
+Each channel is identified by an integer id, unique for the life of the
+current Nvim session. Functions like |stdioopen()| return channel ids;
+functions like |chansend()| consume channel ids.
 
 ==============================================================================
 2. Reading and writing raw bytes			      *channel-bytes*
@@ -64,8 +69,8 @@ be raised.
 
 - The arguments passed to the callback function are:
 
-  0: The channel id
+  0: |channel-id|
-  1: the raw data read from the channel, formatted as a |readfile()|-style
+  1: Raw data read from the channel, formatted as a |readfile()|-style
      list. If EOF occured, a single empty string `['']` will be passed in.
      Note that the items in this list do not directly correspond to actual
      lines in the output. See |channel-lines|
@@ -150,9 +155,8 @@ Nvim uses stdin/stdout to interact with the user over the terminal interface
 (TUI). If Nvim is |--headless| the TUI is not started and stdin/stdout can be
 used as a channel. See also |--embed|.
 
-Call |stdioopen()| during |startup| to open the stdio channel as channel-id 1.
+Call |stdioopen()| during |startup| to open the stdio channel as |channel-id| 1.
-Nvim's stderr is always available as channel-id 2 (|v:stderr| to be explicit),
+Nvim's stderr is always available as |v:stderr|, a write-only bytes channel.
-a write-only bytes channel.
 
 Example: >
     func! OnEvent(id, data, event)

runtime/doc/develop.txt

@@ -239,7 +239,14 @@ Example: `nvim_buf_changedtick_event`.
 
 API-CLIENT						*dev-api-client*
 
+Standard Features ~
+
+- Clients should call |nvim_set_client_info()| after connecting, so users and
+  plugins can detect the client by handling the |ChanInfo| event. This
+  avoids the need for special variables or other client hints.
+
 Package Naming ~
+
 API client packages should NOT be named something ambiguous like "neovim" or
 "python-client".  Use "nvim" as a prefix/suffix to some other identifier
 following ecosystem conventions.
@@ -255,10 +262,11 @@ Examples of API-client package names:
         BAD:  neovim
 
 Implementation ~
-Consider using libmpack instead of the msgpack.org C/C++ library. libmpack is
-small, efficient, and C89-compatible. It can be easily inlined in your
-C project source, too.  https://github.com/libmpack/libmpack/
 
+Consider using libmpack instead of the msgpack.org C/C++ library. libmpack is
+small (can be inlined into your C/C++ project) and efficient (no allocations).
+It also implements msgpack-RPC.
+https://github.com/libmpack/libmpack/
 
 EXTERNAL UI 						*dev-ui*
 
@@ -267,7 +275,13 @@ versions of Nvim may add new items to existing events. The API is strongly
 backwards-compatible, but clients must not break if new (optional) fields are
 added to existing events.
 
+Standard Features ~
+
 External UIs are expected to implement these common features:
+
+- Call |nvim_set_client_info()| after connecting, so users and plugins can
+  detect the UI by handling the |ChanInfo| event. This avoids the need for
+  special variables and UI-specific config files (gvimrc, macvimrc, …).
 - Cursor style (shape, color) should conform to the 'guicursor' properties
   delivered with the mode_info_set UI event.
 - Send the ALT/META ("Option" on macOS) key as a |<M-| chord.

runtime/doc/eval.txt

@@ -1817,11 +1817,11 @@ v:shell_error	Result of the last shell command.  When non-zero, the last
 v:statusmsg	Last given status message.  It's allowed to set this variable.
 
 					*v:stderr* *stderr-variable*
-v:stderr	Channel id for stderr. Unlike stdin and stdout (see
+v:stderr	|channel-id| corresponding to stderr. The value is always 2;
-		|stdioopen()|), stderr is always open for writing. This channel
+		use this variable to make your code more descriptive.
-		ID is always 2, but this variable can be used to be explicit.
+		Unlike stdin and stdout (see |stdioopen()|), stderr is always
-		Example: >
+		open for writing. Example: >
-	:call chansend(v:stderr, "something bad happened\n")
+			:call chansend(v:stderr, "error: toaster empty\n")
 <
 					*v:swapname* *swapname-variable*
 v:swapname	Only valid when executing |SwapExists| autocommands: Name of
@@ -5024,11 +5024,10 @@ jobstart({cmd}[, {opts}])				*jobstart()*
 <
 		Returns |job-id| on success, 0 on invalid arguments (or job
 		table is full), -1 if {cmd}[0] or 'shell' is not executable.
-		For communication over the job's stdio, it is represented as a
+		The returned job-id is a valid |channel-id| representing the
-		|channel|, and a channel ID is returned on success. Use
+		job's stdio streams. Use |chansend()| (or |rpcnotify()| and
-		|chansend()| (or |rpcnotify()| and |rpcrequest()| if "rpc" option
+		|rpcrequest()| if "rpc" was enabled) to send data to stdin and
-		was used) to send data to stdin and |chanclose()| to close stdio
+		|chanclose()| to close the streams without stopping the job.
-		streams without stopping the job explicitly.
 
 		See |job-control| and |RPC|.
 
@@ -5082,18 +5081,24 @@ jobstop({id})						*jobstop()*
 		See |job-control|.
 
 jobwait({ids}[, {timeout}])				*jobwait()*
-		Wait for a set of jobs to finish. The {ids} argument is a list
+		Wait for a set of jobs to complete.
-		of |job-id|s to wait for. {timeout} is the maximum number of
+
-		milliseconds to wait. During jobwait(), callbacks for jobs not
+		{ids} is a list of |job-id|s to wait for.
-		in the {ids} list may be invoked. The screen will not redraw
+		{timeout} is the maximum number of milliseconds to wait.
-		unless |:redraw| is invoked by a callback.
+		{timeout} of zero can be used to check if a job-id is valid,
+		without waiting.
+
+		During jobwait() callbacks for jobs not in the {ids} list may
+		be invoked. The screen will not redraw unless |:redraw| is
+		invoked by a callback.
 
 		Returns a list of len({ids}) integers, where each integer is
-		the wait-result of the corresponding job. Each wait-result is:
+		the wait-result of the corresponding job. Each wait-result is
-		    Job exit-code, if the job exited
+		one of the following:
-		    -1 if the wait timed out for the job
+			* Exit-code, if the job exited
-		    -2 if the job was interrupted
+			* -1 if the timeout was exceeded
-		    -3 if the |job-id| is invalid.
+			* -2 if the job was interrupted
+			* -3 if the |job-id| is invalid
 
 join({list} [, {sep}])					*join()*
 		Join the items in {list} together into one String.
@@ -7325,8 +7330,8 @@ stdpath({what})					*stdpath()* *E6100*
 		directories.
 
 		{what}       Type    Description ~
-		cache        String  Cache directory. Useful for plugins
+		cache        String  Cache directory. Arbitrary temporary
-		                     that need temporary files to work.
+		                     storage for plugins, etc.
 		config       String  User configuration directory. The
 		                     |init.vim| is stored here.
 		config_dirs  List    Additional configuration directories.
@@ -7334,6 +7339,9 @@ stdpath({what})					*stdpath()* *E6100*
 		                     is stored here.
 		data_dirs    List    Additional data directories.
 
+		Example: >
+			:echo stdpath("config")
+
 
 str2float({expr})					*str2float()*
 		Convert String {expr} to a Float.  This mostly works the same

runtime/doc/filetype.txt

@@ -152,8 +152,8 @@ file.  It will be overwritten when installing a new version of Vim.
 
 A. If you want to overrule all default file type checks.
    This works by writing one file for each filetype.  The disadvantage is that
-   means there can be many files.  The advantage is that you can simply drop
+   there can be many files.  The advantage is that you can simply drop this 
-   this file in the right directory to make it work.
+   file in the right directory to make it work.
 							*ftdetect*
    1. Create your user runtime directory.  You would normally use the first
       item of the 'runtimepath' option.  Then create the directory "ftdetect"

runtime/doc/job_control.txt

@@ -16,12 +16,14 @@ Concepts
 
 Job Id							*job-id*
 
-When a job starts it is assigned a number, unique for the life of the current
+Each job is identified by an integer id, unique for the life of the current
-Nvim session. Functions like |jobstart()| return job ids. Functions like
+Nvim session. Each job-id is a valid |channel-id|: they share the same "key
+space". Functions like |jobstart()| return job ids; functions like
 |jobsend()|, |jobstop()|, |rpcnotify()|, and |rpcrequest()| take job ids.
 
-The job's stdio streams are represented as a |channel|. It is possible to send
+Job stdio streams form a |channel| which can send and receive raw bytes or
-and recieve raw bytes, or use |msgpack-rpc|.
+|msgpack-rpc| messages.
+
 ==============================================================================
 Usage							*job-control-usage*
 

runtime/doc/nvim.txt

@@ -20,29 +20,24 @@ differences from Vim.
 ==============================================================================
 Transitioning from Vim				*nvim-from-vim*
 
-To start the transition, create init.vim in the correct directory for your
+1. To start the transition, create your |init.vim| (user config) file: >
-platform.
 
-For Linux, macOS and other Unixes, create the file at ~/.config/nvim/init.vim.
+    :call mkdir(stdpath('config'), 'p')
+    :exe 'edit '.stdpath('config').'/init.vim'
 
-For Windows, create the file at %LOCALAPPDATA%\nvim\init.vim. `%LOCALAPPDATA%`
+2. Add these contents to the file: >
-usually expands to `C:\Users\<username>\AppData\Local`.
 
-Note: If your system sets `$XDG_CONFIG_HOME`, use that instead of `~/.config`
-or `%LOCALAPPDATA%` in the paths above. Nvim follows the XDG |base-directories|
-convention.
-
-Next, add these contents to the file:
->
     set runtimepath^=~/.vim runtimepath+=~/.vim/after
     let &packpath = &runtimepath
     source ~/.vimrc
-<
+
+3. Restart Nvim, your existing Vim config will be loaded.
+
 See |provider-python| and |provider-clipboard| for additional software you
 might need to use some features.
 
-Your Vim configuration might not be entirely compatible with Nvim. For a
+Your Vim configuration might not be entirely Nvim-compatible.
-full list of differences between Vim and Nvim see |vim-differences|.
+See |vim-differences| for the full list of changes.
 
 The |'ttymouse'| option, for example, was removed from Nvim (mouse support
 should work without it). If you use the same |vimrc| for Vim and Nvim,

runtime/doc/starting.txt

@@ -699,27 +699,25 @@ greps in the help files) you might be able to use this: >
 4. Suspending						*suspend*
 
 					*iconize* *iconise* *CTRL-Z* *v_CTRL-Z*
-CTRL-Z			Suspend Vim, like ":stop".
+CTRL-Z			Suspend Nvim, like ":stop".
 			Works in Normal and in Visual mode.  In Insert and
 			Command-line mode, the CTRL-Z is inserted as a normal
-			character.  In Visual mode Vim goes back to Normal
+			character.  In Visual mode Nvim goes back to Normal
 			mode.
 
 :sus[pend][!]	or			*:sus* *:suspend* *:st* *:stop*
-:st[op][!]		Suspend Vim.  Vim will continue if you make it the
+:st[op][!]		Suspend Nvim using OS "job control"; it will continue
-			foreground job again.
+			if you make it the foreground job again.  Triggers
-			If the '!' is not given and 'autowrite' is set, every
+			|VimSuspend| before suspending and |VimResume| when
+			resumed.
+			If "!" is not given and 'autowrite' is set, every
 			buffer with changes and a file name is written out.
-			If the '!' is given or 'autowrite' is not set, changed
+			If "!" is given or 'autowrite' is not set, changed
-			buffers are not written, don't forget to bring Vim
+			buffers are not written, don't forget to bring Nvim
 			back to the foreground later!
 
 In the GUI, suspending is implementation-defined.
 
-In X-windows the selection is disowned when Vim suspends.  this means you
-can't paste it in another application (since Vim is going to sleep an attempt
-to get the selection would make the program hang).
-
 ==============================================================================
 5. Exiting						*exiting*
 
@@ -1373,7 +1371,7 @@ file when reading and include:
 9. Standard Paths				*standard-path*
 
 Nvim stores configuration and data in standard locations. Plugins are strongly
-encouraged to follow this pattern also.
+encouraged to follow this pattern also. Use |stdpath()| to get the paths.
 
 						*base-directories* *xdg*
 The "base" (root) directories conform to the XDG Base Directory Specification.
@@ -1381,19 +1379,19 @@ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
 The $XDG_CONFIG_HOME and $XDG_DATA_HOME environment variables are used if they
 exist, otherwise default values (listed below) are used.
 
+CONFIG DIRECTORY (DEFAULT) ~
+                  *$XDG_CONFIG_HOME*            Nvim: stdpath("config")
+    Unix:         ~/.config                   ~/.config/nvim
+    Windows:      ~/AppData/Local             ~/AppData/Local/nvim
+
+DATA DIRECTORY (DEFAULT) ~
+                  *$XDG_DATA_HOME*              Nvim: stdpath("data")
+    Unix:         ~/.local/share              ~/.local/share/nvim
+    Windows:      ~/AppData/Local             ~/AppData/Local/nvim-data
+
 Note: Throughout the user manual these defaults are used as placeholders, e.g.
 "~/.config" is understood to mean "$XDG_CONFIG_HOME or ~/.config".
 
-CONFIG DIRECTORY				*$XDG_CONFIG_HOME*
-          Base                Nvim ~
-Unix:     ~/.config           ~/.config/nvim
-Windows:  ~/AppData/Local     ~/AppData/Local/nvim
-
-DATA DIRECTORY					*$XDG_DATA_HOME*
-          Base                Nvim ~
-Unix:     ~/.local/share      ~/.local/share/nvim
-Windows:  ~/AppData/Local     ~/AppData/Local/nvim-data
-
 LOG FILE					*$NVIM_LOG_FILE*
 Besides 'debug' and 'verbose', Nvim keeps a general log file for internal
 debugging, plugins and RPC clients. >

runtime/doc/vim_diff.txt

@@ -205,19 +205,8 @@ certain features removed/added at compile-time. |feature-compile|
 If a Python interpreter is available on your `$PATH`, |:python| and |:python3|
 are always available and may be used simultaneously. See |provider-python|.
 
-|:!| does not support "interactive" commands. Use |:terminal| instead.
-(GUI Vim has a similar limitation, see ":help gui-pty" in Vim.)
-
-:!start is not special-cased on Windows.
-
-|system()| does not support writing/reading "backgrounded" commands. |E5677|
-
 |:redir| nested in |execute()| works.
 
-Nvim may throttle (skip) messages from shell commands (|:!|, |:grep|, |:make|)
-if there is too much output. No data is lost, this only affects display and
-makes things faster. |:terminal| output is never throttled.
-
 |mkdir()| behaviour changed:
 1. Assuming /tmp/foo does not exist and /tmp can be written to
    mkdir('/tmp/foo/bar', 'p', 0700) will create both /tmp/foo and /tmp/foo/bar 
@@ -323,6 +312,22 @@ Normal commands:
 Options:
   'ttimeout', 'ttimeoutlen' behavior was simplified
 
+Shell:
+  Shell output (|:!|, |:make|, …) is always routed through the UI, so it
+  cannot "mess up" the screen. (You can still use "chansend(v:stderr,…)" if
+  you want to mess up the screen :)
+
+  Nvim throttles (skips) messages from shell commands (|:!|, |:grep|, |:make|)
+  if there is too much output. No data is lost, this only affects display and
+  improves performance. |:terminal| output is never throttled.
+
+  |:!| does not support "interactive" commands. Use |:terminal| instead.
+  (GUI Vim has a similar limitation, see ":help gui-pty" in Vim.)
+
+  :!start is not special-cased on Windows.
+
+  |system()| does not support writing/reading "backgrounded" commands. |E5677|
+
 Startup:
   |-e| and |-es| invoke the same "improved Ex mode" as -E and -Es.
   |-E| and |-Es| reads stdin as text (into buffer 1).

runtime/tutor/en/vim-01-beginner.tutor

@@ -18,9 +18,9 @@ won't be saved. Don't worry about messing things up; just remember that
 pressing [<Esc>](<Esc>) and then [u](u) will undo the latest change.
 
 This tutorial is interactive, and there are a few things you should know.
-Pressing [<Enter>](<Enter>) over text highlighted [like this](holy-grail    )
+- Type [<Enter>](<Enter>) on links [like this](holy-grail    ) to open the linked help section.
-will take you to some relevant help (hopefully), and pressing K over any
+- Or simply type [K](K) on any word to find its documentation!
-word will try to do so too. Sometimes you will be required to modify text like
+- Sometimes you will be required to modify text like
 this here
 Once you have done the changes correctly, the ✗ sign at the left will change
 to ✓. I imagine you can already see how neat Vim can be. ;)

src/nvim/README.md

@@ -23,15 +23,19 @@ Logs
 
 Low-level log messages sink to `$NVIM_LOG_FILE`.
 
-You can use `LOG_CALLSTACK();` anywhere in the source to log the current
+Use `LOG_CALLSTACK()` (Linux only) to log the current stacktrace. To log to an
-stacktrace. To log in an alternate file, e.g. stderr, use
+alternate file (e.g. stderr) use `LOG_CALLSTACK_TO_FILE(FILE*)`.
-`LOG_CALLSTACK_TO_FILE(FILE*)`. (Currently Linux-only.)
 
-UI events are logged at level 0 (`DEBUG_LOG_LEVEL`).
+UI events are logged at DEBUG level (`DEBUG_LOG_LEVEL`).
 
     rm -rf build/
     make CMAKE_EXTRA_FLAGS="-DMIN_LOG_LEVEL=0"
 
+Many log messages have a shared prefix, such as "UI" or "RPC". Use the shell to
+filter the log, e.g. at DEBUG level you might want to exclude UI messages:
+
+    tail -F ~/.local/share/nvim/log | cat -v | stdbuf -o0 grep -v UI | stdbuf -o0 tee -a log
+
 Build with ASAN
 ---------------
 
@@ -276,3 +280,25 @@ Since Nvim inherited its code from Vim, the states are not prepared to receive
 "arbitrary events", so we use a special key to represent those (When a state
 receives an "arbitrary event", it normally doesn't do anything other update the
 screen).
+
+Main loop
+---------
+
+The `Loop` structure (which describes `main_loop`) abstracts multiple queues
+into one loop:
+
+    uv_loop_t uv;
+    MultiQueue *events;
+    MultiQueue *thread_events;
+    MultiQueue *fast_events;
+
+`loop_poll_events` checks `Loop.uv` and `Loop.fast_events` whenever Nvim is
+idle, and also at `os_breakcheck` intervals.
+
+MultiQueue is cool because you can attach throw-away "child queues" trivially.
+For example `do_os_system()` does this (for every spawned process!) to
+automatically route events onto the `main_loop`:
+
+    Process *proc = &uvproc.process;
+    MultiQueue *events = multiqueue_new_child(main_loop.events);
+    proc->events = events;

src/nvim/event/loop.c

@@ -34,6 +34,7 @@ void loop_init(Loop *loop, void *data)
 
 /// Processes one `Loop.uv` event (at most).
 /// Processes all `Loop.fast_events` events.
+/// Does NOT process `Loop.events`, that is an application-specific decision.
 ///
 /// @returns true if `ms` timeout was reached
 bool loop_poll_events(Loop *loop, int ms)

test/README.md

@@ -64,9 +64,9 @@ To run a *specific* functional test:
 
     TEST_FILE=test/functional/foo.lua make functionaltest
 
-To *repeat* a test many times:
+To *repeat* a test:
 
-    .deps/usr/bin/busted --filter 'foo' --repeat 1000 test/functional/ui/foo_spec.lua
+    .deps/usr/bin/busted --lpath='build/?.lua' --filter 'foo' --repeat 1000 test/functional/ui/foo_spec.lua
 
 ### Filter by tag