neovim/runtime/doc/dev_arch.txt

*dev_arch.txt*          Nvim


                            NVIM REFERENCE MANUAL


How to develop Nvim, explanation of modules and subsystems    *dev-arch*

Module-specific details are documented at the top of each module
(`terminal.c`, `undo.c`, …). The top of each major module has (or should have)
an overview in a comment at the top of its file.

The purpose of this document is to give:

1. an overview of how it all fits together
2. how-to guides for common tasks such as:
    - (TODO) deprecating public functions
    - (TODO) adding a new public (API) function or (UI) event

                                  Type |gO| to see the table of contents.

==============================================================================
Filename conventions

The source filenames use extensions to hint about their purpose.

- `*.c`, `*.generated.c` - full C files, with all includes, etc.
- `*.c.h` - parametrized C files, contain all necessary includes, but require
  defining macros before actually using. Example: `typval_encode.c.h`
- `*.h` - full headers, with all includes. Does *not* apply to `*.generated.h`.
- `*.h.generated.h` - exported functions’ declarations.
- `*.c.generated.h` - static functions’ declarations.

==============================================================================
Data structures

- StringBuilder
- kvec or garray.c for dynamic lists / vectors (use StringBuilder for strings)

Use `kvec.h` for most lists. When you absolutely need a linked list, use
`lib/queue_defs.h` which defines an "intrusive" linked list.

Buffer text is stored as a tree of line segments, defined in `src/nvim/memline.c`.
The central idea is found in `ml_find_line`.

Many of the editor concepts are defined as Lua data files:

- Events (autocmds): src/nvim/auevents.lua
- Ex (cmdline) commands: src/nvim/ex_cmds.lua
- Options: src/nvim/options.lua
- Vimscript functions: src/nvim/eval.lua
- v: variables: src/nvim/vvars.lua

==============================================================================
Events                                                            *dev-events*

The events historically called "autocmds", referred to here as "editor events"
or simply "events", are high-level events for use by plugins, user config, and
the Nvim editor. (There is an unrelated, low-level concept defined by the
`event/defs.h#Event` struct, which is just a bag of data passed along the
internal |event-loop|.)

All new editor events must be implemented using `aucmd_defer()` (and where
possible, old events should be migrated to this), so that they are processed
in a predictable manner, which avoids crashes and race conditions. See
`do_markset_autocmd` for an example.

==============================================================================
UI events                                                      *dev-ui-events*

The long-term vision is that UI events are just another type of "editor event"
(formerly known as "autocmds"). There is no real reason that we have separate
types of user-facing or plugin-facing events. Events are events. Their
"transport" is irrelevant and any event should be possible to emit over any
transport (editor or RPC).

Meanwhile the current situation is that UI events are a particular RPC event
packaged in a generic `redraw` notification. They also can be listened to
in-process via |vim.ui_attach()|.

UI events are deferred to UIs, which implies a deepcopy of the UI event data.

The source files most directly involved with UI events are:
1. `src/nvim/ui.*`: calls handler functions of registered UI structs (independent from msgpack-rpc)
2. `src/nvim/api/ui.*`: forwards messages over msgpack-rpc to remote UIs.

UI events are defined in `src/nvim/api/ui_events.in.h` , this file is not
compiled directly, rather it parsed by
`src/nvim/generators/gen_api_ui_events.lua` which autogenerates wrapper
functions used by the source files above. It also generates metadata
accessible as `api_info().ui_events`.

See commit d3a8e9217f39c59dd7762bd22a76b8bd03ca85ff for an example of adding
a new UI event. Remember to bump NVIM_API_LEVEL if it wasn't already during
this development cycle.

Other references:
- |msgpack-rpc|
- |ui|
- https://github.com/neovim/neovim/pull/3246
- https://github.com/neovim/neovim/pull/18375
- https://github.com/neovim/neovim/pull/21605


==============================================================================
API

                                                        *dev-api-fast*
API functions and Vimscript "eval" functions may be marked as |api-fast| which
means they are safe to call in Lua callbacks and other scenarios. A functions
CANNOT be marked as "fast" if could trigger `os_breakcheck()`, which may
"yield" the current execution and start a new execution of code not expecting
this:
- accidentally recursing into a function not expecting this.
- changing (global) state without restoring it before returning to the
  "yielded" callsite.

In practice, this means any code that could trigger `os_breakcheck()` cannot
be "fast". For example, commit 3940c435e405 fixed such a bug with
`nvim__get_runtime` by explicitly disallowing `os_breakcheck()` via the
`EW_NOBREAK` flag.

Common examples of non-fast code: regexp matching, wildcard expansion,
expression evaluation.


==============================================================================
The event-loop                                                    *event-loop*

The internal, low-level, libuv event-loop (|luv-event-loop|) is used to
schedule arbitrary work in a predictable way. One such obvious use-case for
scheduling is deferred editor-events (autocmds). Another example is
|job-control|.

ASYNC EVENT SUPPORT

One of the features Nvim added is the support for handling arbitrary
asynchronous events, which can include:

- RPC requests
- job control callbacks
- timers

Nvim implements this functionality by entering another event loop while
waiting for characters, so instead of: >py

  def state_enter(on_state, data):
    do
      key = readkey()           # Read a key from the user
    while on_state(data, key)   # Invoke callback for the current state

the Nvim program loop is more like: >py

  def state_enter(on_state, data):
    do
      event = read_next_event() # Read an event from the OS
    while on_state(data, event) # Invoke callback for current state

where `event` is something the operating system delivers to us, including (but
not limited to) user input. The `read_next_event()` part is internally
implemented by libuv, the platform layer used by Nvim.

Since Nvim inherited its code from Vim, the states are not prepared to receive
"arbitrary events", so we use a special key to represent those (When a state
receives an "arbitrary event", it normally doesn't do anything other than
update the screen).

MAIN LOOP

The `Loop` structure (which describes `main_loop`) abstracts multiple queues
into one loop: >

    uv_loop_t uv;
    MultiQueue *events;
    MultiQueue *thread_events;
    MultiQueue *fast_events;

`loop_poll_events` checks `Loop.uv` and `Loop.fast_events` whenever Nvim is
idle, and also at `os_breakcheck` intervals.

MultiQueue is cool because you can attach throw-away "child queues" trivially.
For example `do_os_system()` does this (for every spawned process!) to
automatically route events onto the `main_loop`: >

    Process *proc = &uvproc.process;
    MultiQueue *events = multiqueue_new_child(main_loop.events);
    proc->events = events;


NVIM LIFECYCLE

How Nvim processes input.

Consider a typical Vim-like editing session:

01. Vim displays the welcome screen
02. User types: `:`
03. Vim enters command-line mode
04. User types: `edit README.txt<CR>`
05. Vim opens the file and returns to normal mode
06. User types: `G`
07. Vim navigates to the end of the file
09. User types: `5`
10. Vim enters count-pending mode
11. User types: `d`
12. Vim enters operator-pending mode
13. User types: `w`
14. Vim deletes 5 words
15. User types: `g`
16. Vim enters the "g command mode"
17. User types: `g`
18. Vim goes to the beginning of the file
19. User types: `i`
20. Vim enters insert mode
21. User types: `word<ESC>`
22. Vim inserts "word" at the beginning and returns to normal mode

Note that we split user actions into sequences of inputs that change the state
of the editor. While there's no documentation about a "g command mode" (step
16), internally it is implemented similarly to "operator-pending mode".

From this we can see that Vim has the behavior of an input-driven state machine
(more specifically, a pushdown automaton since it requires a stack for
transitioning back from states). Assuming each state has a callback responsible
for handling keys, this pseudocode represents the main program loop: >py

  def state_enter(state_callback, data):
    do
      key = readkey()                 # read a key from the user
    while state_callback(data, key)   # invoke the callback for the current state
<

That is, each state is entered by calling `state_enter` and passing a
state-specific callback and data. Here is a high-level pseudocode for a program
that implements something like the workflow described above: >py

  def main()
    state_enter(normal_state, {}):

  def normal_state(data, key):
    if key == ':':
      state_enter(command_line_state, {})
    elif key == 'i':
      state_enter(insert_state, {})
    elif key == 'd':
      state_enter(delete_operator_state, {})
    elif key == 'g':
      state_enter(g_command_state, {})
    elif is_number(key):
      state_enter(get_operator_count_state, {'count': key})
    elif key == 'G'
      jump_to_eof()
    return true

  def command_line_state(data, key):
    if key == '<cr>':
      if data['input']:
        execute_ex_command(data['input'])
      return false
    elif key == '<esc>'
      return false

    if not data['input']:
      data['input'] = ''

    data['input'] += key
    return true

  def delete_operator_state(data, key):
    count = data['count'] or 1
    if key == 'w':
      delete_word(count)
    elif key == '$':
      delete_to_eol(count)
    return false  # return to normal mode

  def g_command_state(data, key):
    if key == 'g':
      go_top()
    elif key == 'v':
      reselect()
    return false  # return to normal mode

  def get_operator_count_state(data, key):
    if is_number(key):
      data['count'] += key
      return true
    unshift_key(key)  # return key to the input buffer
    state_enter(delete_operator_state, data)
    return false

  def insert_state(data, key):
    if key == '<esc>':
      return false  # exit insert mode
    self_insert(key)
    return true
<

The above gives an idea of how Nvim is organized internally. Some states like
the `g_command_state` or `get_operator_count_state` do not have a dedicated
`state_enter` callback, but are implicitly embedded into other states (this
will change later as we continue the refactoring effort). To start reading the
actual code, here's the recommended order:

1. `state_enter()` function (state.c). This is the actual program loop,
   note that a `VimState` structure is used, which contains function pointers
   for the callback and state data.
2. `main()` function (main.c). After all startup, `normal_enter` is called
   at the end of function to enter normal mode.
3. `normal_enter()` function (normal.c) is a small wrapper for setting
   up the NormalState structure and calling `state_enter`.
4. `normal_check()` function (normal.c) is called before each iteration of
   normal mode.
5. `normal_execute()` function (normal.c) is called when a key is read in normal
   mode.

The basic structure described for normal mode in 3, 4 and 5 is used for other
modes managed by the `state_enter` loop:

- command-line mode: `command_line_{enter,check,execute}()`(`ex_getln.c`)
- insert mode: `insert_{enter,check,execute}()`(`edit.c`)
- terminal mode: `terminal_{enter,execute}()`(`terminal.c`)

IMPORTANT VARIABLES

The current mode is stored in `State`.  The values it can have are `MODE_NORMAL`,
`MODE_INSERT`, `MODE_CMDLINE`, and a few others.

The current window is `curwin`.  The current buffer is `curbuf`.  These point
to structures with the cursor position in the window, option values, the file
name, etc.

All the global variables are declared in `globals.h`.

THE MAIN EVENT-LOOP

The main loop is implemented in state_enter. The basic idea is that Vim waits
for the user to type a character and processes it until another character is
needed.  Thus there are several places where Vim waits for a character to be
typed.  The `vgetc()` function is used for this.  It also handles mapping.

What we consider the "Nvim event loop" is actually a wrapper around `uv_run` to
handle both the `fast_events` queue and possibly (a suitable subset of) deferred
events. Therefore "raw" `vim.uv.run()` is often not enough to "yield" from Lua
plugins; instead they can call `vim.wait(0)`.

Updating the screen is mostly postponed until a command or a sequence of
commands has finished.  The work is done by `update_screen()`, which calls
`win_update()` for every window, which calls `win_line()` for every line.
See the start of [drawscreen.c](drawscreen.c) for more explanations.

COMMAND-LINE MODE

When typing a `:`, `normal_cmd()` will call `getcmdline()` to obtain a line with
an Ex command.  `getcmdline()` calls a loop that will handle each typed
character.  It returns when hitting `<CR>` or `<Esc>` or some other character that
ends the command line mode.

EX COMMANDS

Ex commands are handled by the function `do_cmdline()`.  It does the generic
parsing of the `:` command line and calls `do_one_cmd()` for each separate
command.  It also takes care of while loops.

`do_one_cmd()` parses the range and generic arguments and puts them in the
exarg_t and passes it to the function that handles the command.

The `:` commands are listed in [ex_cmds.lua](ex_cmds.lua).

NORMAL MODE COMMANDS

The Normal mode commands are handled by the `normal_cmd()` function.  It also
handles the optional count and an extra character for some commands.  These
are passed in a `cmdarg_T` to the function that handles the command.

There is a table `nv_cmds` in [normal.c](normal.c) which
lists the first character of every
command.  The second entry of each item is the name of the function that
handles the command.

INSERT MODE COMMANDS

When doing an `i` or `a` command, `normal_cmd()` will call the `edit()` function.
It contains a loop that waits for the next character and handles it.  It
returns when leaving Insert mode.


==============================================================================

vim:tw=78:ts=8:sw=4:et:ft=help:norl: