mirrors/neovim
1
0
Fork 0
mirror of https://github.com/neovim/neovim.git synced 2025-11-15 23:01:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: dev-quickstart, dev-test #36304

Browse Source
This commit is contained in:
Justin M. Keyes
2025-10-24 20:24:51 -04:00
committed by GitHub
parent 737dd83d1f
commit fe4faaf81a
7 changed files with 580 additions and 564 deletions
Download Patch File Download Diff File Expand all files Collapse all files

556
test/README.md
View File

@@ -1,555 +1,3 @@
Tests
=====
## Moved to:
Tests are broadly divided into *unit tests* ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit/)),
*functional tests* ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional/)),
and *old tests* ([test/old/testdir/](https://github.com/neovim/neovim/tree/master/test/old/testdir/)).
- _Unit_ testing is achieved by compiling the tests as a shared library which is
loaded and called by [LuaJit FFI](http://luajit.org/ext_ffi.html).
- _Functional_ tests are driven by RPC, so they do not require LuaJit (as
opposed to Lua).
You can learn the [key concepts of Lua in 15 minutes](http://learnxinyminutes.com/docs/lua/).
Use any existing test as a template to start writing new tests.
Tests are run by `/cmake/RunTests.cmake` file, using `busted` (a Lua test-runner).
For some failures, `.nvimlog` (or `$NVIM_LOG_FILE`) may provide insight.
Depending on the presence of binaries (e.g., `xclip`) some tests will be
ignored. You must compile with libintl to prevent `E319: The command is not
available in this version` errors.
---
- [Running tests](#running-tests)
- [Writing tests](#writing-tests)
- [Lint](#lint)
- [Configuration](#configuration)
---
Layout
======
- `/test/benchmark` : benchmarks
- `/test/functional` : functional tests
- `/test/unit` : unit tests
- `/test/old/testdir` : old tests (from Vim)
- `/test/config` : contains `*.in` files which are transformed into `*.lua`
files using `configure_file` CMake command: this is for accessing CMake
variables in lua tests.
- `/test/includes` : include-files for use by luajit `ffi.cdef` C definitions
parser: normally used to make macros not accessible via this mechanism
accessible the other way.
- `/test/*/preload.lua` : modules preloaded by busted `--helper` option
- `/test/**/testutil.lua` : common utility functions in the context of the test
runner
- `/test/**/testnvim.lua` : common utility functions in the context of the
test session (RPC channel to the Nvim child process created by clear() for each test)
- `/test/*/**/*_spec.lua` : actual tests. Files that do not end with
`_spec.lua` are libraries like `/test/**/testutil.lua`, except that they have
some common topic.
Running tests
=============
Executing Tests
---------------
To run all tests (except "old" tests):
make test
To run only _unit_ tests:
make unittest
To run only _functional_ tests:
make functionaltest
Legacy tests
------------
To run all legacy Vim tests:
make oldtest
To run a *single* legacy test file you can use either:
make oldtest TEST_FILE=test_syntax.vim
or:
make test/old/testdir/test_syntax.vim
- Specify only the test file name, not the full path.
Debugging tests
---------------
- Each test gets a test id which looks like "T123". This also appears in the
log file. Child processes spawned from a test appear in the logs with the
*parent* name followed by "/c". Example:
```
DBG 2022-06-15T18:37:45.226 T57.58016.0 UI: flush
DBG 2022-06-15T18:37:45.226 T57.58016.0 inbuf_poll:442: blocking... events_enabled=0 events_pending=0
DBG 2022-06-15T18:37:45.227 T57.58016.0/c UI: stop
INF 2022-06-15T18:37:45.227 T57.58016.0/c os_exit:595: Nvim exit: 0
DBG 2022-06-15T18:37:45.229 T57.58016.0 read_cb:118: closing Stream (0x7fd5d700ea18): EOF (end of file)
INF 2022-06-15T18:37:45.229 T57.58016.0 on_proc_exit:400: exited: pid=58017 status=0 stoptime=0
```
- You can set `$GDB` to [run functional tests under gdbserver](https://github.com/neovim/neovim/pull/1527):
```sh
GDB=1 TEST_FILE=test/functional/api/buffer_spec.lua TEST_FILTER='nvim_buf_set_text works$' make functionaltest
```
Read more about [filtering tests](#filtering-tests).
Then, in another terminal:
```sh
gdb -ex 'target remote localhost:7777' build/bin/nvim
```
If `$VALGRIND` is also set it will pass `--vgdb=yes` to valgrind instead of
starting gdbserver directly.
See `nvim_argv` in https://github.com/neovim/neovim/blob/master/test/functional/testnvim.lua.
- Hanging tests can happen due to unexpected "press-enter" prompts. The
default screen width is 50 columns. Commands that try to print lines longer
than 50 columns in the command-line, e.g. `:edit very...long...path`, will
trigger the prompt. Try using a shorter path, or `:silent edit`.
- If you can't figure out what is going on, try to visualize the screen. Put
this at the beginning of your test:
```lua
local Screen = require('test.functional.ui.screen')
local screen = Screen.new()
screen:attach()
```
Then put `screen:snapshot_util()` anywhere in your test. See the comments in
`test/functional/ui/screen.lua` for more info.
Debugging Lua test code
-----------------------
Debugging Lua test code is a bit involved. Get your shopping list ready, you'll
need to install and configure:
1. [nvim-dap](https://github.com/mfussenegger/nvim-dap)
2. [local-lua-debugger-vscode](https://github.com/mfussenegger/nvim-dap/wiki/Debug-Adapter-installation#local-lua-debugger-vscode)
3. [nlua](https://github.com/mfussenegger/nlua)
4. [one-small-step-for-vimkind](https://github.com/jbyuki/one-small-step-for-vimkind) (called `osv`)
5. A `nbusted` command in `$PATH`. This command can be a copy of `busted` with
`exec '/usr/bin/lua5.1'"` replaced with `"exec '/usr/bin/nlua'"` (or the
path to your `nlua`)
The setup roughly looks like this:
```
┌─────────────────────────┐
│ nvim used for debugging │◄────┐
└─────────────────────────┘ │
│ │
▼ │
┌─────────────────┐ │
│ local-lua-debug │ │
└─────────────────┘ │
│ │
▼ │
┌─────────┐ │
│ nbusted │ │
└─────────┘ │
│ │
▼ │
┌───────────┐ │
│ test-case │ │
└───────────┘ │
│ │
▼ │
┌────────────────────┐ │
│ nvim test-instance │ │
└────────────────────┘ │
│ ┌─────┐ │
└──►│ osv │─────────────────┘
└─────┘
```
With these installed you can use a configuration like this:
```lua
local dap = require("dap")
local function free_port()
local tcp = vim.loop.new_tcp()
assert(tcp)
tcp:bind('127.0.0.1', 0)
local port = tcp:getsockname().port
tcp:shutdown()
tcp:close()
return port
end
local name = "nvim-test-case" -- arbitrary name
local config = {
name = name,
-- value of type must match the key used in `dap.adapters["local-lua"] = ...` from step 2)
type = "local-lua",
request = "launch",
cwd = "${workspaceFolder}",
program = {
command = "nbusted",
},
args = {
"--ignore-lua",
"--lazy",
"--helper=test/functional/preload.lua",
"--lpath=build/?.lua",
"--lpath=?.lua",
-- path to file to debug, could be replaced with a hardcoded string
function()
return vim.api.nvim_buf_get_name(0)
end,
-- You can filter to specific test-case by adding:
-- '--filter="' .. test_case_name .. '"',
},
env = {
OSV_PORT = free_port
}
}
-- Whenever the config is used it needs to launch a second debug session that attaches to `osv`
-- This makes it possible to step into `exec_lua` code blocks
setmetatable(config, {
__call = function(c)
---@param session dap.Session
dap.listeners.after.event_initialized["nvim_debug"] = function(session)
if session.config.name ~= name then
return
end
dap.listeners.after.event_initialized["nvim_debug"] = nil
vim.defer_fn(function()
dap.run({
name = "attach-osv",
type = "nlua", -- value must match the `dap.adapters` definition key for osv
request = "attach",
port = session.config.env.OSV_PORT,
})
end, 500)
end
return c
end,
})
```
You can either add this configuration to your `dap.configurations.lua` list as
described in `:help dap-configuration` or create it dynamically in a
user-command or function and call it directly via `dap.run(config)`. The latter
is useful if you use tree-sitter to find the test case around a cursor location
with a query like the following and set the `--filter` property to it.
```query
(function_call
name: (identifier) @name (#any-of? @name "describe" "it")
arguments: (arguments
(string) @str
)
)
```
Limitations:
- You need to add the following boilerplate to each spec file where you want to
be able to stop at breakpoints within the test-case code:
```
if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
require("lldebugger").start()
end
```
This is a [local-lua-debugger
limitation](https://github.com/tomblind/local-lua-debugger-vscode?tab=readme-ov-file#busted)
- You cannot step into code of files which get baked into the nvim binary like
the `shared.lua`.
Filtering Tests
---------------
### Filter by name
Tests can be filtered by setting a pattern of test name to `TEST_FILTER` or `TEST_FILTER_OUT`.
``` lua
it('foo api',function()
...
end)
it('bar api',function()
...
end)
```
To run only test with filter name:
TEST_FILTER='foo.*api' make functionaltest
To run all tests except ones matching a filter:
TEST_FILTER_OUT='foo.*api' make functionaltest
### Filter by file
To run a *specific* unit test:
TEST_FILE=test/unit/foo.lua make unittest
or
cmake -E env "TEST_FILE=test/unit/foo.lua" cmake --build build --target unittest
To run a *specific* functional test:
TEST_FILE=test/functional/foo.lua make functionaltest
or
cmake -E env "TEST_FILE=test/functional/foo.lua" cmake --build build --target functionaltest
To *repeat* a test:
BUSTED_ARGS="--repeat=100 --no-keep-going" TEST_FILE=test/functional/foo_spec.lua make functionaltest
or
cmake -E env "TEST_FILE=test/functional/foo_spec.lua" cmake -E env BUSTED_ARGS="--repeat=100 --no-keep-going" cmake --build build --target functionaltest
### Filter by tag
Tests can be "tagged" by adding `#` before a token in the test description.
``` lua
it('#foo bar baz', function()
...
end)
it('#foo another test', function()
...
end)
```
To run only the tagged tests:
TEST_TAG=foo make functionaltest
**NOTE:**
* `TEST_FILE` is not a pattern string like `TEST_TAG` or `TEST_FILTER`. The
given value to `TEST_FILE` must be a path to an existing file.
* Both `TEST_TAG` and `TEST_FILTER` filter tests by the string descriptions
found in `it()` and `describe()`.
Writing tests
=============
Guidelines
----------
- Luajit needs to know about type and constant declarations used in function
prototypes. The
[testutil.lua](https://github.com/neovim/neovim/blob/master/test/unit/testutil.lua)
file automatically parses `types.h`, so types used in the tested functions
could be moved to it to avoid having to rewrite the declarations in the test
files.
- `#define` constants must be rewritten `const` or `enum` so they can be
"visible" to the tests.
- Use **pending()** to skip tests
([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
Do not silently skip the test with `if-else`. If a functional test depends on
some external factor (e.g. the existence of `md5sum` on `$PATH`), *and* you
can't mock or fake the dependency, then skip the test via `pending()` if the
external factor is missing. This ensures that the *total* test-count
(success + fail + error + pending) is the same in all environments.
- *Note:* `pending()` is ignored if it is missing an argument, unless it is
[contained in an `it()` block](https://github.com/neovim/neovim/blob/d21690a66e7eb5ebef18046c7a79ef898966d786/test/functional/ex_cmds/grep_spec.lua#L11).
Provide empty function argument if the `pending()` call is outside `it()`
([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
- Really long `source([=[...]=])` blocks may break Vim's Lua syntax
highlighting. Try `:syntax sync fromstart` to fix it.
Where tests go
--------------
Tests in `/test/unit` and `/test/functional` are divided into groups
by the semantic component they are testing.
- _Unit tests_
([test/unit](https://github.com/neovim/neovim/tree/master/test/unit)) should
match 1-to-1 with the structure of `src/nvim/`, because they are testing
functions directly. E.g. unit-tests for `src/nvim/undo.c` should live in
`test/unit/undo_spec.lua`.
- _Functional tests_
([test/functional](https://github.com/neovim/neovim/tree/master/test/functional))
are higher-level (plugins and user input) than unit tests; they are organized
by concept.
- Try to find an existing `test/functional/*/*_spec.lua` group that makes
sense, before creating a new one.
Fixing tests
============
> Nvim session T123 took 2000 milliseconds to exit
> This indicates a likely problem with the test even if it passed!
This may indicate a leak, because Nvim waits on uv handles before exiting.
Example: https://github.com/neovim/neovim/pull/35768
Lint
====
`make lint` (and `make lintlua`) runs [luacheck](https://github.com/mpeterv/luacheck)
on the test code.
If a luacheck warning must be ignored, specify the warning code. Example:
-- luacheck: ignore 621
http://luacheck.readthedocs.io/en/stable/warnings.html
Ignore the smallest applicable scope (e.g. inside a function, not at the top of
the file).
Configuration
=============
Test behaviour is affected by environment variables. Currently supported
(Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined,
treated as Integer; when defined, treated as String; when defined, treated as
Number; !must be defined to function properly):
- `BUSTED_ARGS` (F) (U): arguments forwarded to `busted`.
- `CC` (U) (S): specifies which C compiler to use to preprocess files.
Currently only compilers with gcc-compatible arguments are supported.
- `GDB` (F) (D): makes nvim instances to be run under `gdbserver`. It will be
accessible on `localhost:7777`: use `gdb build/bin/nvim`, type `target remote
:7777` inside.
- `GDBSERVER_PORT` (F) (I): overrides port used for `GDB`.
- `LOG_DIR` (FU) (S!): specifies where to seek for valgrind and ASAN log files.
- `VALGRIND` (F) (D): makes nvim instances to be run under `valgrind`. Log
files are named `valgrind-%p.log` in this case. Note that non-empty valgrind
log may fail tests. Valgrind arguments may be seen in
`/test/functional/testnvim.lua`. May be used in conjunction with `GDB`.
- `VALGRIND_LOG` (F) (S): overrides valgrind log file name used for `VALGRIND`.
- `TEST_COLORS` (F) (U) (D): enable pretty colors in test runner. Set to true by default.
- `TEST_SKIP_FRAGILE` (F) (D): makes test suite skip some fragile tests.
- `TEST_TIMEOUT` (FU) (I): specifies maximum time, in seconds, before the test
suite run is killed
- `NVIM_LUA_NOTRACK` (F) (D): disable reference counting of Lua objects
- `NVIM_PRG` (F) (S): path to Nvim executable (default: `build/bin/nvim`).
- `NVIM_TEST_MAIN_CDEFS` (U) (1): makes `ffi.cdef` run in main process. This
raises a possibility of bugs due to conflicts in header definitions, despite
the counters, but greatly speeds up unit tests by not requiring `ffi.cdef` to
do parsing of big strings with C definitions.
- `NVIM_TEST_PRINT_I` (U) (1): makes `cimport` print preprocessed, but not yet
filtered through `formatc` headers. Used to debug `formatc`. Printing is done
with the line numbers.
- `NVIM_TEST_PRINT_CDEF` (U) (1): makes `cimport` print final lines which will
be then passed to `ffi.cdef`. Used to debug errors `ffi.cdef` happens to
throw sometimes.
- `NVIM_TEST_PRINT_SYSCALLS` (U) (1): makes it print to stderr when syscall
wrappers are called and what they returned. Used to debug code which makes
unit tests be executed in separate processes.
- `NVIM_TEST_RUN_FAILING_TESTS` (U) (1): makes `itp` run tests which are known
to fail (marked by setting third argument to `true`).
- `NVIM_TEST_CORE_*` (FU) (S): a set of environment variables which specify
where to search for core files. Are supposed to be defined all at once.
- `NVIM_TEST_CORE_GLOB_DIRECTORY` (FU) (S): directory where core files are
located. May be `.`. This directory is then recursively searched for core
files. Note: this variable must be defined for any of the following to have
any effect.
- `NVIM_TEST_CORE_GLOB_RE` (FU) (S): regular expression which must be matched
by core files. E.g. `/core[^/]*$`. May be absent, in which case any file is
considered to be matched.
- `NVIM_TEST_CORE_EXC_RE` (FU) (S): regular expression which excludes certain
directories from searching for core files inside. E.g. use `^/%.deps$` to not
search inside `/.deps`. If absent, nothing is excluded.
- `NVIM_TEST_CORE_DB_CMD` (FU) (S): command to get backtrace out of the
debugger. E.g. `gdb -n -batch -ex "thread apply all bt full"
"$_NVIM_TEST_APP" -c "$_NVIM_TEST_CORE"`. Defaults to the example command.
This debug command may use environment variables `_NVIM_TEST_APP` (path to
application which is being debugged: normally either nvim or luajit) and
`_NVIM_TEST_CORE` (core file to get backtrace from).
- `NVIM_TEST_CORE_RANDOM_SKIP` (FU) (D): makes `check_cores` not check cores
after approximately 90% of the tests. Should be used when finding cores is
too hard for some reason. Normally (on OS X or when
`NVIM_TEST_CORE_GLOB_DIRECTORY` is defined and this variable is not) cores
are checked for after each test.
- `NVIM_TEST_INTEG` (F) (D): enables integration tests that makes real network
calls. By default these tests are skipped. When set to `1`, tests requiring external
HTTP requests (e.g `vim.net.request()`) will be run.
- `NVIM_TEST_RUN_TESTTEST` (U) (1): allows running
`test/unit/testtest_spec.lua` used to check how testing infrastructure works.
- `NVIM_TEST_TRACE_LEVEL` (U) (N): specifies unit tests tracing level:
- `0` disables tracing (the fastest, but you get no data if tests crash and
there no core dump was generated),
- `1` leaves only C function calls and returns in the trace (faster than
recording everything),
- `2` records all function calls, returns and executed Lua source lines.
- `NVIM_TEST_TRACE_ON_ERROR` (U) (1): makes unit tests yield trace on error in
addition to regular error message.
- `NVIM_TEST_MAXTRACE` (U) (N): specifies maximum number of trace lines to
keep. Default is 1024.
- `OSV_PORT`: (F): launches `osv` listening on the given port within nvim test
instances.
- [dev_test.txt](../runtime/doc/dev_test.txt)

2
test/functional/example_spec.lua
View File

@@ -1,5 +1,5 @@
-- To run this test:
-- TEST_FILE=test/functional/example_spec.lua make functionaltest
-- make functionaltest TEST_FILE=test/functional/example_spec.lua
local t = require('test.testutil')
local n = require('test.functional.testnvim')()