diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 346ee1b94e..5b6e29a722 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,6 +4,9 @@ Contributing to Neovim Getting started --------------- +If you are new to the codebase, read [:help dev-quickstart](https://neovim.io/doc/user/dev_tools.html#dev-quickstart) +to see how to run tests and start hacking on the codebase. + If you want to help but don't know where to start, here are some low-risk/isolated tasks: @@ -11,9 +14,9 @@ low-risk/isolated tasks: - Fix bugs found by [Coverity](#coverity). - [Merge a Vim patch] (requires strong familiarity with Vim) - NOTE: read the above link before sending improvements to "runtime files" (anything in `runtime/`). - - Vimscript and documentation files are (mostly) maintained by [Vim], not Nvim. + - *Vimscript* files are (mostly) maintained by [Vim], not Nvim. + - *Lua* files are maintained by *Nvim*. - Nvim's [filetype detection](https://github.com/neovim/neovim/blob/master/runtime/lua/vim/filetype.lua) behavior matches Vim, so changes to filetype detection should be submitted to [Vim] first. - - Lua files are maintained by Nvim. Reporting problems ------------------ @@ -33,9 +36,10 @@ Reporting problems Developer guidelines -------------------- -- Read [:help dev](https://neovim.io/doc/user/develop.html#dev) and [:help dev-doc][dev-doc-guide] if you are working on Nvim core. -- Read [:help dev-ui](https://neovim.io/doc/user/develop.html#dev-ui) if you are developing a UI. -- Read [:help dev-api-client](https://neovim.io/doc/user/develop.html#dev-api-client) if you are developing an API client. +- Read [:help dev-quickstart](https://neovim.io/doc/user/dev_tools.html#dev-quickstart) to see how to run tests and start hacking on the codebase. +- Read [:help dev](https://neovim.io/doc/user/dev.html#dev) and [:help dev-doc][dev-doc-guide] if you are working on Nvim core. +- Read [:help dev-ui](https://neovim.io/doc/user/dev.html#dev-ui) if you are developing a UI. +- Read [:help dev-api-client](https://neovim.io/doc/user/dev.html#dev-api-client) if you are developing an API client. - Install `ninja` for faster builds of Nvim. ```bash sudo apt-get install ninja-build @@ -353,8 +357,8 @@ as context, use the `-W` argument as well. [Merge a Vim patch]: https://neovim.io/doc/user/dev_vimpatch.html [complexity:low]: https://github.com/neovim/neovim/issues?q=is%3Aopen+is%3Aissue+label%3Acomplexity%3Alow [conventional_commits]: https://www.conventionalcommits.org -[dev-doc-guide]: https://neovim.io/doc/user/develop.html#dev-doc -[dev-lua-doc]: https://neovim.io/doc/user/develop.html#dev-lua-doc +[dev-doc-guide]: https://neovim.io/doc/user/dev.html#dev-doc +[dev-lua-doc]: https://neovim.io/doc/user/dev.html#dev-lua-doc [LuaLS]: https://luals.github.io/wiki/annotations/ [gcc-warnings]: https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html [gh]: https://cli.github.com/ diff --git a/runtime/doc/develop.txt b/runtime/doc/dev.txt similarity index 100% rename from runtime/doc/develop.txt rename to runtime/doc/dev.txt diff --git a/runtime/doc/dev_test.txt b/runtime/doc/dev_test.txt new file mode 100644 index 0000000000..48e7e56ccd --- /dev/null +++ b/runtime/doc/dev_test.txt @@ -0,0 +1,519 @@ +*dev_test* Nvim + + + NVIM REFERENCE MANUAL + + +Writing tests for Nvim *dev-test* + + Type |gO| to see the table of contents. + +============================================================================== +Writing tests for Nvim + +Nvim has a powerful yet simple test framework. It's approximately 7x better +than whatever you use at work. + +Each test starts a new Nvim process (10-30ms) which is discarded after the +test finishes. You assert stuff using `t.eq()` and `screen:expect()` (which +automatically waits as needed). That's pretty much it. + +TODO: Expose the test framework as a public interface, for use in 3P plugins: +https://github.com/neovim/neovim/issues/34592 + +============================================================================== +Test framework + +Tests are broadly divided into unit tests (`test/unit/`), functional tests +(`test/functional/`) and old tests (`test/old/testdir/`). + +- Unit testing is achieved by compiling the tests as a shared library which is + loaded and called by [LuaJit FFI](https://luajit.org/ext_ffi.html). +- Functional tests are driven by RPC, so they do not require LuaJit (as + opposed to Lua). They are essentially "integration" tests, they test the + full system. But they are fast. + +You can learn the [Lua concepts 15 minutes](https://learnxinyminutes.com/docs/lua/), +see also |lua-guide|. Use any existing test as a template to start writing new +tests, or see |dev-quickstart|. + +Tests are run by the `/cmake/RunTests.cmake` script using `busted` (a Lua test-runner). +For some failures, `.nvimlog` (or `$NVIM_LOG_FILE`) may provide insight. + +Depending on the presence of binaries (e.g., `xclip`) some tests will be +skipped. + +============================================================================== +Test Layout + +- `/test/benchmark` : benchmarks +- `/test/functional` : functional tests +- `/test/unit` : unit tests +- `/test/old/testdir` : old tests (from Vim) +- `/test/config` : contains `*.in` files which are transformed into `*.lua` + files using `configure_file` CMake command: this is for accessing CMake + variables in Lua tests. +- `/test/includes` : include-files for use by luajit `ffi.cdef` C definitions + parser: normally used to make macros not accessible via this mechanism + accessible the other way. +- `/test/*/preload.lua` : modules preloaded by busted `--helper` option +- `/test/**/testutil.lua` : common utility functions in the context of the test + runner +- `/test/**/testnvim.lua` : common utility functions in the context of the + test session (RPC channel to the Nvim child process created by clear() for each test) +- `/test/*/**/*_spec.lua` : actual tests. Files that do not end with + `_spec.lua` are libraries like `/test/**/testutil.lua`, except that they have + some common topic. + + +============================================================================== +Running tests *dev-run-test* + +EXECUTING TESTS + +To run all tests (except "old" tests): > + make test + +To run only _unit_ tests: > + make unittest + +To run only _functional_ tests: > + make functionaltest + + +LEGACY TESTS + +To run all legacy Vim tests: > + make oldtest + +To run a _single_ legacy test file you can use either: > + # Specify only the test file name, not the full path. + make oldtest TEST_FILE=test_syntax.vim +or: > + make test/old/testdir/test_syntax.vim + + +DEBUGGING TESTS + +- Each test gets a test id which looks like "T123". This also appears in the + log file. Child processes spawned from a test appear in the logs with the + _parent_ name followed by "/c". Example: > + + DBG 2022-06-15T18:37:45.226 T57.58016.0 UI: flush + DBG 2022-06-15T18:37:45.226 T57.58016.0 inbuf_poll:442: blocking... events_enabled=0 events_pending=0 + DBG 2022-06-15T18:37:45.227 T57.58016.0/c UI: stop + INF 2022-06-15T18:37:45.227 T57.58016.0/c os_exit:595: Nvim exit: 0 + DBG 2022-06-15T18:37:45.229 T57.58016.0 read_cb:118: closing Stream (0x7fd5d700ea18): EOF (end of file) + INF 2022-06-15T18:37:45.229 T57.58016.0 on_proc_exit:400: exited: pid=58017 status=0 stoptime=0 + +- You can set `$GDB` to [run functional tests under gdbserver](https://github.com/neovim/neovim/pull/1527): >sh + GDB=1 TEST_FILE=test/functional/api/buffer_spec.lua TEST_FILTER='nvim_buf_set_text works$' make functionaltest +< + Read more about |dev-filter-test|. + + Then, in another terminal: >sh + gdb -ex 'target remote localhost:7777' build/bin/nvim +< + If `$VALGRIND` is also set it will pass `--vgdb=yes` to valgrind instead of + starting gdbserver directly. + + See `nvim_argv` in `test/functional/testnvim.lua`. + +- Hanging tests can happen due to unexpected "press-enter" prompts. The + default screen width is 50 columns. Commands that try to print lines longer + than 50 columns in the command-line, e.g. `:edit very...long...path`, will + trigger the prompt. Try using a shorter path, or `:silent edit`. +- If you can't figure out what is going on, try to visualize the screen. Put + this at the beginning of your test: > + local Screen = require('test.functional.ui.screen') + local screen = Screen.new() + screen:attach() +< Then put `screen:snapshot_util()` anywhere in your test. See the comments in + `test/functional/ui/screen.lua` for more info. + +DEBUGGING LUA TEST CODE + +Debugging Lua test code is a bit involved. Get your shopping list ready, you'll +need to install and configure: + +1. [nvim-dap](https://github.com/mfussenegger/nvim-dap) +2. [local-lua-debugger-vscode](https://github.com/mfussenegger/nvim-dap/wiki/Debug-Adapter-installation#local-lua-debugger-vscode) +3. [nlua](https://github.com/mfussenegger/nlua) +4. [one-small-step-for-vimkind](https://github.com/jbyuki/one-small-step-for-vimkind) (called `osv`) +5. A `nbusted` command in `$PATH`. This command can be a copy of `busted` with + `exec '/usr/bin/lua5.1'"` replaced with `"exec '/usr/bin/nlua'"` (or the + path to your `nlua`) + + +The setup roughly looks like this: > + + ┌─────────────────────────┐ + │ nvim used for debugging │◄────┐ + └─────────────────────────┘ │ + │ │ + ▼ │ + ┌─────────────────┐ │ + │ local-lua-debug │ │ + └─────────────────┘ │ + │ │ + ▼ │ + ┌─────────┐ │ + │ nbusted │ │ + └─────────┘ │ + │ │ + ▼ │ + ┌───────────┐ │ + │ test-case │ │ + └───────────┘ │ + │ │ + ▼ │ + ┌────────────────────┐ │ + │ nvim test-instance │ │ + └────────────────────┘ │ + │ ┌─────┐ │ + └──►│ osv │─────────────────┘ + └─────┘ + + +With these installed you can use a configuration like this: > + + local dap = require("dap") + + + local function free_port() + local tcp = vim.loop.new_tcp() + assert(tcp) + tcp:bind('127.0.0.1', 0) + local port = tcp:getsockname().port + tcp:shutdown() + tcp:close() + return port + end + + + local name = "nvim-test-case" -- arbitrary name + local config = { + name = name, + + -- value of type must match the key used in `dap.adapters["local-lua"] = ...` from step 2) + type = "local-lua", + + request = "launch", + cwd = "${workspaceFolder}", + program = { + command = "nbusted", + }, + args = { + "--ignore-lua", + "--lazy", + "--helper=test/functional/preload.lua", + "--lpath=build/?.lua", + "--lpath=?.lua", + + -- path to file to debug, could be replaced with a hardcoded string + function() + return vim.api.nvim_buf_get_name(0) + end, + + -- You can filter to specific test-case by adding: + -- '--filter="' .. test_case_name .. '"', + }, + env = { + OSV_PORT = free_port + } + } + + -- Whenever the config is used it needs to launch a second debug session that attaches to `osv` + -- This makes it possible to step into `exec_lua` code blocks + setmetatable(config, { + + __call = function(c) + ---@param session dap.Session + dap.listeners.after.event_initialized["nvim_debug"] = function(session) + if session.config.name ~= name then + return + end + dap.listeners.after.event_initialized["nvim_debug"] = nil + vim.defer_fn(function() + dap.run({ + name = "attach-osv", + type = "nlua", -- value must match the `dap.adapters` definition key for osv + request = "attach", + port = session.config.env.OSV_PORT, + }) + end, 500) + end + + return c + end, + }) + + +You can either add this configuration to your `dap.configurations.lua` list as +described in `:help dap-configuration` or create it dynamically in a +user-command or function and call it directly via `dap.run(config)`. The latter +is useful if you use treesitter to find the test case around a cursor location +with a query like the following and set the `--filter` property to it. >query + + (function_call + name: (identifier) @name (#any-of? @name "describe" "it") + arguments: (arguments + (string) @str + ) + ) + +Limitations: + +- You need to add the following boilerplate to each spec file where you want to + be able to stop at breakpoints within the test-case code: > + if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then + require("lldebugger").start() + end +< This is a [local-lua-debugger limitation](https://github.com/tomblind/local-lua-debugger-vscode?tab=readme-ov-file#busted) +- You cannot step into code of files which get baked into the nvim binary like + the `shared.lua`. + + +------------------------------------------------------------------------------ +Filtering tests *dev-filter-test* + +FILTER BY NAME + +Tests can be filtered by setting a pattern of test name to `TEST_FILTER` or `TEST_FILTER_OUT`. > + + it('foo api',function() + ... + end) + it('bar api',function() + ... + end) + +To run only test with filter name: > + TEST_FILTER='foo.*api' make functionaltest + +To run all tests except ones matching a filter: > + TEST_FILTER_OUT='foo.*api' make functionaltest + +FILTER BY FILE + +To run a _specific_ unit test: > + TEST_FILE=test/unit/foo.lua make unittest + +or > + cmake -E env "TEST_FILE=test/unit/foo.lua" cmake --build build --target unittest + +To run a _specific_ functional test: > + TEST_FILE=test/functional/foo.lua make functionaltest + +or > + cmake -E env "TEST_FILE=test/functional/foo.lua" cmake --build build --target functionaltest + +To _repeat_ a test: > + BUSTED_ARGS="--repeat=100 --no-keep-going" TEST_FILE=test/functional/foo_spec.lua make functionaltest + +or > + cmake -E env "TEST_FILE=test/functional/foo_spec.lua" cmake -E env BUSTED_ARGS="--repeat=100 --no-keep-going" cmake --build build --target functionaltest + +FILTER BY TAG + +Tests can be "tagged" by adding `#` before a token in the test description. > + + it('#foo bar baz', function() + ... + end) + it('#foo another test', function() + ... + end) + +To run only the tagged tests: > + TEST_TAG=foo make functionaltest + +NOTE: + +- `TEST_FILE` is not a pattern string like `TEST_TAG` or `TEST_FILTER`. The + given value to `TEST_FILE` must be a path to an existing file. +- Both `TEST_TAG` and `TEST_FILTER` filter tests by the string descriptions + found in `it()` and `describe()`. + + +============================================================================== +Writing tests *dev-write-test* + +GUIDELINES + +- Luajit needs to know about type and constant declarations used in function + prototypes. The + [testutil.lua](https://github.com/neovim/neovim/blob/master/test/unit/testutil.lua) + file automatically parses `types.h`, so types used in the tested functions + could be moved to it to avoid having to rewrite the declarations in the test + files. + - `#define` constants must be rewritten `const` or `enum` so they can be + "visible" to the tests. +- Use `pending()` to skip tests + ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)). + Do not silently skip the test with `if-else`. If a functional test depends on + some external factor (e.g. the existence of `md5sum` on `$PATH`), _and_ you + can't mock or fake the dependency, then skip the test via `pending()` if the + external factor is missing. This ensures that the _total_ test-count + (success + fail + error + pending) is the same in all environments. + - Note: `pending()` is ignored if it is missing an argument, unless it is + [contained in an `it()` block](https://github.com/neovim/neovim/blob/d21690a66e7eb5ebef18046c7a79ef898966d786/test/functional/ex_cmds/grep_spec.lua#L11). + Provide empty function argument if the `pending()` call is outside `it()` + ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)). + +WHERE TESTS GO + +Tests in `/test/unit` and `/test/functional` are divided into groups by the +semantic component they are testing. + +- Unit tests (`test/unit/`) should match 1-to-1 with the structure of + `src/nvim/`, because they are testing functions directly. E.g. unit-tests + for `src/nvim/undo.c` should live in `test/unit/undo_spec.lua`. +- Functional tests (`test/functional/`) are higher-level (plugins, UI, user + input) than unit tests; they are organized by concept. + - Try to find an existing `test/functional/*/*_spec.lua` group that makes + sense, before creating a new one. + + +------------------------------------------------------------------------------ +Fixing tests *dev-fix-test* + +FIXING HARNESS WARNINGS + +> Nvim session T123 took 2000 milliseconds to exit +> This indicates a likely problem with the test even if it passed! + +This may indicate a leak, because Nvim waits on uv handles before exiting. +Example: https://github.com/neovim/neovim/pull/35768 + + +FIXING LINT FAILURES + +`make lint` (and `make lintlua`) runs [LuaLS](https://github.com/LuaLS/lua-language-server/wiki/Annotations) +on the test code. + +If a LuaLS/EmmyLS warning must be ignored, specify the warning code. Example: >lua + + ---@diagnostic disable-next-line: unused-vararg + +https://github.com/LuaLS/lua-language-server/wiki/Annotations#diagnostic + +Ignore the smallest applicable scope (e.g. inside a function, not at the top of +the file). + + +============================================================================== +Configuration *dev-test-config* + +(TODO: clean this up, too many variables and some of them are not used anymore.) + +Test behaviour is affected by environment variables. Currently supported +(Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined, +treated as Integer; when defined, treated as String; when defined, treated as +Number; !must be defined to function properly): + +- `BUSTED_ARGS` (F) (U): arguments forwarded to `busted`. + +- `CC` (U) (S): specifies which C compiler to use to preprocess files. + Currently only compilers with gcc-compatible arguments are supported. + +- `GDB` (F) (D): makes nvim instances to be run under `gdbserver`. It will be + accessible on `localhost:7777`: use `gdb build/bin/nvim`, type `target remote + :7777` inside. + +- `GDBSERVER_PORT` (F) (I): overrides port used for `GDB`. + +- `LOG_DIR` (FU) (S!): specifies where to seek for valgrind and ASAN log files. + +- `VALGRIND` (F) (D): makes nvim instances to be run under `valgrind`. Log + files are named `valgrind-%p.log` in this case. Note that non-empty valgrind + log may fail tests. Valgrind arguments may be seen in + `/test/functional/testnvim.lua`. May be used in conjunction with `GDB`. + +- `VALGRIND_LOG` (F) (S): overrides valgrind log file name used for `VALGRIND`. + +- `TEST_COLORS` (F) (U) (D): enable pretty colors in test runner. Set to true by default. + +- `TEST_SKIP_FRAGILE` (F) (D): makes test suite skip some fragile tests. + +- `TEST_TIMEOUT` (FU) (I): specifies maximum time, in seconds, before the test + suite run is killed + +- `NVIM_LUA_NOTRACK` (F) (D): disable reference counting of Lua objects + +- `NVIM_PRG` (F) (S): path to Nvim executable (default: `build/bin/nvim`). + +- `NVIM_TEST_MAIN_CDEFS` (U) (1): makes `ffi.cdef` run in main process. This + raises a possibility of bugs due to conflicts in header definitions, despite + the counters, but greatly speeds up unit tests by not requiring `ffi.cdef` to + do parsing of big strings with C definitions. + +- `NVIM_TEST_PRINT_I` (U) (1): makes `cimport` print preprocessed, but not yet + filtered through `formatc` headers. Used to debug `formatc`. Printing is done + with the line numbers. + +- `NVIM_TEST_PRINT_CDEF` (U) (1): makes `cimport` print final lines which will + be then passed to `ffi.cdef`. Used to debug errors `ffi.cdef` happens to + throw sometimes. + +- `NVIM_TEST_PRINT_SYSCALLS` (U) (1): makes it print to stderr when syscall + wrappers are called and what they returned. Used to debug code which makes + unit tests be executed in separate processes. + +- `NVIM_TEST_RUN_FAILING_TESTS` (U) (1): makes `itp` run tests which are known + to fail (marked by setting third argument to `true`). + +- `NVIM_TEST_CORE_*` (FU) (S): a set of environment variables which specify + where to search for core files. Are supposed to be defined all at once. + +- `NVIM_TEST_CORE_GLOB_DIRECTORY` (FU) (S): directory where core files are + located. May be `.`. This directory is then recursively searched for core + files. Note: this variable must be defined for any of the following to have + any effect. + +- `NVIM_TEST_CORE_GLOB_RE` (FU) (S): regular expression which must be matched + by core files. E.g. `/core[^/]*$`. May be absent, in which case any file is + considered to be matched. + +- `NVIM_TEST_CORE_EXC_RE` (FU) (S): regular expression which excludes certain + directories from searching for core files inside. E.g. use `^/%.deps$` to not + search inside `/.deps`. If absent, nothing is excluded. + +- `NVIM_TEST_CORE_DB_CMD` (FU) (S): command to get backtrace out of the + debugger. E.g. `gdb -n -batch -ex "thread apply all bt full" + "$_NVIM_TEST_APP" -c "$_NVIM_TEST_CORE"`. Defaults to the example command. + This debug command may use environment variables `_NVIM_TEST_APP` (path to + application which is being debugged: normally either nvim or luajit) and + `_NVIM_TEST_CORE` (core file to get backtrace from). + +- `NVIM_TEST_CORE_RANDOM_SKIP` (FU) (D): makes `check_cores` not check cores + after approximately 90% of the tests. Should be used when finding cores is + too hard for some reason. Normally (on OS X or when + `NVIM_TEST_CORE_GLOB_DIRECTORY` is defined and this variable is not) cores + are checked for after each test. + +- `NVIM_TEST_INTEG` (F) (D): enables integration tests that makes real network + calls. By default these tests are skipped. When set to `1`, tests requiring external + HTTP requests (e.g `vim.net.request()`) will be run. + +- `NVIM_TEST_RUN_TESTTEST` (U) (1): allows running + `test/unit/testtest_spec.lua` used to check how testing infrastructure works. + +- `NVIM_TEST_TRACE_LEVEL` (U) (N): specifies unit tests tracing level: + - `0` disables tracing (the fastest, but you get no data if tests crash and + there no core dump was generated), + - `1` leaves only C function calls and returns in the trace (faster than + recording everything), + - `2` records all function calls, returns and executed Lua source lines. + +- `NVIM_TEST_TRACE_ON_ERROR` (U) (1): makes unit tests yield trace on error in + addition to regular error message. + +- `NVIM_TEST_MAXTRACE` (U) (N): specifies maximum number of trace lines to + keep. Default is 1024. + +- `OSV_PORT`: (F): launches `osv` listening on the given port within nvim test + instances. + + + vim:tw=78:ts=8:sw=4:et:ft=help:norl: diff --git a/runtime/doc/dev_tools.txt b/runtime/doc/dev_tools.txt index 4ca7af2c3a..ffa874012a 100644 --- a/runtime/doc/dev_tools.txt +++ b/runtime/doc/dev_tools.txt @@ -10,6 +10,50 @@ This is for developing or debugging Nvim itself. Type |gO| to see the table of contents. +============================================================================== +Quickstart guide to developing Nvim *dev-quickstart* + +You can start hacking on Nvim in less than 5 minutes: + +1. Ensure you have the build prerequisites from `BUILD.md`. +2. Clone the source repo and "cd" into it: > + git clone https://github.com/neovim/neovim + cd neovim + # (Optional) Build and run Nvim: + make + ./build/bin/nvim +3. Run a single test. We will start with "example_spec.lua", which is a real + test that shows how tests are written: > + make functionaltest TEST_FILE=test/functional/example_spec.lua +4. Notice the `before_each` block in the test file. Because it calls + `clear()`, each `it()` test will start a new Nvim instance. +5. Tests will do stuff in the Nvim instance and make assertions using `eq()`. + Tests that want to check the UI can also use `screen:expect()`. +6. Now make a code change in Nvim itself, then you can see the effects. The + example test does `feed('iline1…')`, so let's make a change to the + insert-mode code, which lives in `src/nvim/edit.c`. In the + `insert_handle_key` function, just after the `normalchar` label, add this + code: > + s->c = 'x'; +7. Then run the "example_spec.lua" test again, and it should fail with + something like this: > + test/functional/example_spec.lua:31: Row 1 did not match. + Expected: + |*line1 | + |*line^2 | + |{0:~ }| + |{0:~ }| + | | + Actual: + |*xine1 | + |*xine^2 | + |{0:~ }| + |{0:~ }| + | | + +You now understand how to modify the codebase, write tests, and run tests. See +|dev-arch| for details about the internal architecture. + ============================================================================== Logs *dev-tools-logs* @@ -485,4 +529,4 @@ For more information: https://github.com/google/sanitizers/wiki/SanitizerCommonF -vim:tw=78:ts=8:et:ft=help:norl: + vim:tw=78:ts=8:sw=4:et:ft=help:norl: diff --git a/src/gen/gen_help_html.lua b/src/gen/gen_help_html.lua index ebd004f448..9e7ef43058 100644 --- a/src/gen/gen_help_html.lua +++ b/src/gen/gen_help_html.lua @@ -91,7 +91,8 @@ local new_layout = { local redirects = { ['api-ui-events'] = 'ui', ['credits'] = 'backers', - ['dev_tools'] = 'debug', + ['dev'] = 'develop', + ['dev-tools'] = 'debug', ['plugins'] = 'editorconfig', ['terminal'] = 'nvim_terminal_emulator', ['tui'] = 'term', diff --git a/test/README.md b/test/README.md index 456f598e71..17de0bccd5 100644 --- a/test/README.md +++ b/test/README.md @@ -1,555 +1,3 @@ -Tests -===== +## Moved to: -Tests are broadly divided into *unit tests* ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit/)), -*functional tests* ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional/)), -and *old tests* ([test/old/testdir/](https://github.com/neovim/neovim/tree/master/test/old/testdir/)). - -- _Unit_ testing is achieved by compiling the tests as a shared library which is - loaded and called by [LuaJit FFI](http://luajit.org/ext_ffi.html). -- _Functional_ tests are driven by RPC, so they do not require LuaJit (as - opposed to Lua). - -You can learn the [key concepts of Lua in 15 minutes](http://learnxinyminutes.com/docs/lua/). -Use any existing test as a template to start writing new tests. - -Tests are run by `/cmake/RunTests.cmake` file, using `busted` (a Lua test-runner). -For some failures, `.nvimlog` (or `$NVIM_LOG_FILE`) may provide insight. - -Depending on the presence of binaries (e.g., `xclip`) some tests will be -ignored. You must compile with libintl to prevent `E319: The command is not -available in this version` errors. - - ---- - -- [Running tests](#running-tests) -- [Writing tests](#writing-tests) -- [Lint](#lint) -- [Configuration](#configuration) - ---- - - -Layout -====== - -- `/test/benchmark` : benchmarks -- `/test/functional` : functional tests -- `/test/unit` : unit tests -- `/test/old/testdir` : old tests (from Vim) -- `/test/config` : contains `*.in` files which are transformed into `*.lua` - files using `configure_file` CMake command: this is for accessing CMake - variables in lua tests. -- `/test/includes` : include-files for use by luajit `ffi.cdef` C definitions - parser: normally used to make macros not accessible via this mechanism - accessible the other way. -- `/test/*/preload.lua` : modules preloaded by busted `--helper` option -- `/test/**/testutil.lua` : common utility functions in the context of the test - runner -- `/test/**/testnvim.lua` : common utility functions in the context of the - test session (RPC channel to the Nvim child process created by clear() for each test) -- `/test/*/**/*_spec.lua` : actual tests. Files that do not end with - `_spec.lua` are libraries like `/test/**/testutil.lua`, except that they have - some common topic. - - -Running tests -============= - -Executing Tests ---------------- - -To run all tests (except "old" tests): - - make test - -To run only _unit_ tests: - - make unittest - -To run only _functional_ tests: - - make functionaltest - - -Legacy tests ------------- - -To run all legacy Vim tests: - - make oldtest - -To run a *single* legacy test file you can use either: - - make oldtest TEST_FILE=test_syntax.vim - -or: - - make test/old/testdir/test_syntax.vim - -- Specify only the test file name, not the full path. - - -Debugging tests ---------------- - -- Each test gets a test id which looks like "T123". This also appears in the - log file. Child processes spawned from a test appear in the logs with the - *parent* name followed by "/c". Example: - ``` - DBG 2022-06-15T18:37:45.226 T57.58016.0 UI: flush - DBG 2022-06-15T18:37:45.226 T57.58016.0 inbuf_poll:442: blocking... events_enabled=0 events_pending=0 - DBG 2022-06-15T18:37:45.227 T57.58016.0/c UI: stop - INF 2022-06-15T18:37:45.227 T57.58016.0/c os_exit:595: Nvim exit: 0 - DBG 2022-06-15T18:37:45.229 T57.58016.0 read_cb:118: closing Stream (0x7fd5d700ea18): EOF (end of file) - INF 2022-06-15T18:37:45.229 T57.58016.0 on_proc_exit:400: exited: pid=58017 status=0 stoptime=0 - ``` -- You can set `$GDB` to [run functional tests under gdbserver](https://github.com/neovim/neovim/pull/1527): - - ```sh - GDB=1 TEST_FILE=test/functional/api/buffer_spec.lua TEST_FILTER='nvim_buf_set_text works$' make functionaltest - ``` - - Read more about [filtering tests](#filtering-tests). - - Then, in another terminal: - - ```sh - gdb -ex 'target remote localhost:7777' build/bin/nvim - ``` - - If `$VALGRIND` is also set it will pass `--vgdb=yes` to valgrind instead of - starting gdbserver directly. - - See `nvim_argv` in https://github.com/neovim/neovim/blob/master/test/functional/testnvim.lua. - -- Hanging tests can happen due to unexpected "press-enter" prompts. The - default screen width is 50 columns. Commands that try to print lines longer - than 50 columns in the command-line, e.g. `:edit very...long...path`, will - trigger the prompt. Try using a shorter path, or `:silent edit`. -- If you can't figure out what is going on, try to visualize the screen. Put - this at the beginning of your test: - ```lua - local Screen = require('test.functional.ui.screen') - local screen = Screen.new() - screen:attach() - ``` - Then put `screen:snapshot_util()` anywhere in your test. See the comments in - `test/functional/ui/screen.lua` for more info. - -Debugging Lua test code ------------------------ - -Debugging Lua test code is a bit involved. Get your shopping list ready, you'll -need to install and configure: - -1. [nvim-dap](https://github.com/mfussenegger/nvim-dap) -2. [local-lua-debugger-vscode](https://github.com/mfussenegger/nvim-dap/wiki/Debug-Adapter-installation#local-lua-debugger-vscode) -3. [nlua](https://github.com/mfussenegger/nlua) -4. [one-small-step-for-vimkind](https://github.com/jbyuki/one-small-step-for-vimkind) (called `osv`) -5. A `nbusted` command in `$PATH`. This command can be a copy of `busted` with - `exec '/usr/bin/lua5.1'"` replaced with `"exec '/usr/bin/nlua'"` (or the - path to your `nlua`) - - -The setup roughly looks like this: - -``` - ┌─────────────────────────┐ - │ nvim used for debugging │◄────┐ - └─────────────────────────┘ │ - │ │ - ▼ │ - ┌─────────────────┐ │ - │ local-lua-debug │ │ - └─────────────────┘ │ - │ │ - ▼ │ - ┌─────────┐ │ - │ nbusted │ │ - └─────────┘ │ - │ │ - ▼ │ - ┌───────────┐ │ - │ test-case │ │ - └───────────┘ │ - │ │ - ▼ │ - ┌────────────────────┐ │ - │ nvim test-instance │ │ - └────────────────────┘ │ - │ ┌─────┐ │ - └──►│ osv │─────────────────┘ - └─────┘ -``` - - -With these installed you can use a configuration like this: - - -```lua -local dap = require("dap") - - -local function free_port() - local tcp = vim.loop.new_tcp() - assert(tcp) - tcp:bind('127.0.0.1', 0) - local port = tcp:getsockname().port - tcp:shutdown() - tcp:close() - return port -end - - -local name = "nvim-test-case" -- arbitrary name -local config = { - name = name, - - -- value of type must match the key used in `dap.adapters["local-lua"] = ...` from step 2) - type = "local-lua", - - request = "launch", - cwd = "${workspaceFolder}", - program = { - command = "nbusted", - }, - args = { - "--ignore-lua", - "--lazy", - "--helper=test/functional/preload.lua", - "--lpath=build/?.lua", - "--lpath=?.lua", - - -- path to file to debug, could be replaced with a hardcoded string - function() - return vim.api.nvim_buf_get_name(0) - end, - - -- You can filter to specific test-case by adding: - -- '--filter="' .. test_case_name .. '"', - }, - env = { - OSV_PORT = free_port - } -} - --- Whenever the config is used it needs to launch a second debug session that attaches to `osv` --- This makes it possible to step into `exec_lua` code blocks -setmetatable(config, { - - __call = function(c) - ---@param session dap.Session - dap.listeners.after.event_initialized["nvim_debug"] = function(session) - if session.config.name ~= name then - return - end - dap.listeners.after.event_initialized["nvim_debug"] = nil - vim.defer_fn(function() - dap.run({ - name = "attach-osv", - type = "nlua", -- value must match the `dap.adapters` definition key for osv - request = "attach", - port = session.config.env.OSV_PORT, - }) - end, 500) - end - - return c - end, -}) - -``` - -You can either add this configuration to your `dap.configurations.lua` list as -described in `:help dap-configuration` or create it dynamically in a -user-command or function and call it directly via `dap.run(config)`. The latter -is useful if you use tree-sitter to find the test case around a cursor location -with a query like the following and set the `--filter` property to it. - -```query -(function_call - name: (identifier) @name (#any-of? @name "describe" "it") - arguments: (arguments - (string) @str - ) -) -``` - -Limitations: - -- You need to add the following boilerplate to each spec file where you want to - be able to stop at breakpoints within the test-case code: - -``` -if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then - require("lldebugger").start() -end -``` - -This is a [local-lua-debugger -limitation](https://github.com/tomblind/local-lua-debugger-vscode?tab=readme-ov-file#busted) - -- You cannot step into code of files which get baked into the nvim binary like - the `shared.lua`. - - -Filtering Tests ---------------- - -### Filter by name - -Tests can be filtered by setting a pattern of test name to `TEST_FILTER` or `TEST_FILTER_OUT`. - -``` lua -it('foo api',function() - ... -end) -it('bar api',function() - ... -end) -``` - -To run only test with filter name: - - TEST_FILTER='foo.*api' make functionaltest - -To run all tests except ones matching a filter: - - TEST_FILTER_OUT='foo.*api' make functionaltest - -### Filter by file - -To run a *specific* unit test: - - TEST_FILE=test/unit/foo.lua make unittest - -or - - cmake -E env "TEST_FILE=test/unit/foo.lua" cmake --build build --target unittest - -To run a *specific* functional test: - - TEST_FILE=test/functional/foo.lua make functionaltest - -or - - cmake -E env "TEST_FILE=test/functional/foo.lua" cmake --build build --target functionaltest - -To *repeat* a test: - - BUSTED_ARGS="--repeat=100 --no-keep-going" TEST_FILE=test/functional/foo_spec.lua make functionaltest - -or - - cmake -E env "TEST_FILE=test/functional/foo_spec.lua" cmake -E env BUSTED_ARGS="--repeat=100 --no-keep-going" cmake --build build --target functionaltest - -### Filter by tag - -Tests can be "tagged" by adding `#` before a token in the test description. - -``` lua -it('#foo bar baz', function() - ... -end) -it('#foo another test', function() - ... -end) -``` - -To run only the tagged tests: - - TEST_TAG=foo make functionaltest - -**NOTE:** - -* `TEST_FILE` is not a pattern string like `TEST_TAG` or `TEST_FILTER`. The - given value to `TEST_FILE` must be a path to an existing file. -* Both `TEST_TAG` and `TEST_FILTER` filter tests by the string descriptions - found in `it()` and `describe()`. - - -Writing tests -============= - -Guidelines ----------- - -- Luajit needs to know about type and constant declarations used in function - prototypes. The - [testutil.lua](https://github.com/neovim/neovim/blob/master/test/unit/testutil.lua) - file automatically parses `types.h`, so types used in the tested functions - could be moved to it to avoid having to rewrite the declarations in the test - files. - - `#define` constants must be rewritten `const` or `enum` so they can be - "visible" to the tests. -- Use **pending()** to skip tests - ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)). - Do not silently skip the test with `if-else`. If a functional test depends on - some external factor (e.g. the existence of `md5sum` on `$PATH`), *and* you - can't mock or fake the dependency, then skip the test via `pending()` if the - external factor is missing. This ensures that the *total* test-count - (success + fail + error + pending) is the same in all environments. - - *Note:* `pending()` is ignored if it is missing an argument, unless it is - [contained in an `it()` block](https://github.com/neovim/neovim/blob/d21690a66e7eb5ebef18046c7a79ef898966d786/test/functional/ex_cmds/grep_spec.lua#L11). - Provide empty function argument if the `pending()` call is outside `it()` - ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)). -- Really long `source([=[...]=])` blocks may break Vim's Lua syntax - highlighting. Try `:syntax sync fromstart` to fix it. - -Where tests go --------------- - -Tests in `/test/unit` and `/test/functional` are divided into groups -by the semantic component they are testing. - -- _Unit tests_ - ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit)) should - match 1-to-1 with the structure of `src/nvim/`, because they are testing - functions directly. E.g. unit-tests for `src/nvim/undo.c` should live in - `test/unit/undo_spec.lua`. -- _Functional tests_ - ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional)) - are higher-level (plugins and user input) than unit tests; they are organized - by concept. - - Try to find an existing `test/functional/*/*_spec.lua` group that makes - sense, before creating a new one. - - -Fixing tests -============ - -> Nvim session T123 took 2000 milliseconds to exit -> This indicates a likely problem with the test even if it passed! - -This may indicate a leak, because Nvim waits on uv handles before exiting. -Example: https://github.com/neovim/neovim/pull/35768 - - -Lint -==== - -`make lint` (and `make lintlua`) runs [luacheck](https://github.com/mpeterv/luacheck) -on the test code. - -If a luacheck warning must be ignored, specify the warning code. Example: - - -- luacheck: ignore 621 - -http://luacheck.readthedocs.io/en/stable/warnings.html - -Ignore the smallest applicable scope (e.g. inside a function, not at the top of -the file). - - -Configuration -============= - -Test behaviour is affected by environment variables. Currently supported -(Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined, -treated as Integer; when defined, treated as String; when defined, treated as -Number; !must be defined to function properly): - -- `BUSTED_ARGS` (F) (U): arguments forwarded to `busted`. - -- `CC` (U) (S): specifies which C compiler to use to preprocess files. - Currently only compilers with gcc-compatible arguments are supported. - -- `GDB` (F) (D): makes nvim instances to be run under `gdbserver`. It will be - accessible on `localhost:7777`: use `gdb build/bin/nvim`, type `target remote - :7777` inside. - -- `GDBSERVER_PORT` (F) (I): overrides port used for `GDB`. - -- `LOG_DIR` (FU) (S!): specifies where to seek for valgrind and ASAN log files. - -- `VALGRIND` (F) (D): makes nvim instances to be run under `valgrind`. Log - files are named `valgrind-%p.log` in this case. Note that non-empty valgrind - log may fail tests. Valgrind arguments may be seen in - `/test/functional/testnvim.lua`. May be used in conjunction with `GDB`. - -- `VALGRIND_LOG` (F) (S): overrides valgrind log file name used for `VALGRIND`. - -- `TEST_COLORS` (F) (U) (D): enable pretty colors in test runner. Set to true by default. - -- `TEST_SKIP_FRAGILE` (F) (D): makes test suite skip some fragile tests. - -- `TEST_TIMEOUT` (FU) (I): specifies maximum time, in seconds, before the test - suite run is killed - -- `NVIM_LUA_NOTRACK` (F) (D): disable reference counting of Lua objects - -- `NVIM_PRG` (F) (S): path to Nvim executable (default: `build/bin/nvim`). - -- `NVIM_TEST_MAIN_CDEFS` (U) (1): makes `ffi.cdef` run in main process. This - raises a possibility of bugs due to conflicts in header definitions, despite - the counters, but greatly speeds up unit tests by not requiring `ffi.cdef` to - do parsing of big strings with C definitions. - -- `NVIM_TEST_PRINT_I` (U) (1): makes `cimport` print preprocessed, but not yet - filtered through `formatc` headers. Used to debug `formatc`. Printing is done - with the line numbers. - -- `NVIM_TEST_PRINT_CDEF` (U) (1): makes `cimport` print final lines which will - be then passed to `ffi.cdef`. Used to debug errors `ffi.cdef` happens to - throw sometimes. - -- `NVIM_TEST_PRINT_SYSCALLS` (U) (1): makes it print to stderr when syscall - wrappers are called and what they returned. Used to debug code which makes - unit tests be executed in separate processes. - -- `NVIM_TEST_RUN_FAILING_TESTS` (U) (1): makes `itp` run tests which are known - to fail (marked by setting third argument to `true`). - -- `NVIM_TEST_CORE_*` (FU) (S): a set of environment variables which specify - where to search for core files. Are supposed to be defined all at once. - -- `NVIM_TEST_CORE_GLOB_DIRECTORY` (FU) (S): directory where core files are - located. May be `.`. This directory is then recursively searched for core - files. Note: this variable must be defined for any of the following to have - any effect. - -- `NVIM_TEST_CORE_GLOB_RE` (FU) (S): regular expression which must be matched - by core files. E.g. `/core[^/]*$`. May be absent, in which case any file is - considered to be matched. - -- `NVIM_TEST_CORE_EXC_RE` (FU) (S): regular expression which excludes certain - directories from searching for core files inside. E.g. use `^/%.deps$` to not - search inside `/.deps`. If absent, nothing is excluded. - -- `NVIM_TEST_CORE_DB_CMD` (FU) (S): command to get backtrace out of the - debugger. E.g. `gdb -n -batch -ex "thread apply all bt full" - "$_NVIM_TEST_APP" -c "$_NVIM_TEST_CORE"`. Defaults to the example command. - This debug command may use environment variables `_NVIM_TEST_APP` (path to - application which is being debugged: normally either nvim or luajit) and - `_NVIM_TEST_CORE` (core file to get backtrace from). - -- `NVIM_TEST_CORE_RANDOM_SKIP` (FU) (D): makes `check_cores` not check cores - after approximately 90% of the tests. Should be used when finding cores is - too hard for some reason. Normally (on OS X or when - `NVIM_TEST_CORE_GLOB_DIRECTORY` is defined and this variable is not) cores - are checked for after each test. - -- `NVIM_TEST_INTEG` (F) (D): enables integration tests that makes real network - calls. By default these tests are skipped. When set to `1`, tests requiring external - HTTP requests (e.g `vim.net.request()`) will be run. - -- `NVIM_TEST_RUN_TESTTEST` (U) (1): allows running - `test/unit/testtest_spec.lua` used to check how testing infrastructure works. - -- `NVIM_TEST_TRACE_LEVEL` (U) (N): specifies unit tests tracing level: - - `0` disables tracing (the fastest, but you get no data if tests crash and - there no core dump was generated), - - `1` leaves only C function calls and returns in the trace (faster than - recording everything), - - `2` records all function calls, returns and executed Lua source lines. - -- `NVIM_TEST_TRACE_ON_ERROR` (U) (1): makes unit tests yield trace on error in - addition to regular error message. - -- `NVIM_TEST_MAXTRACE` (U) (N): specifies maximum number of trace lines to - keep. Default is 1024. - -- `OSV_PORT`: (F): launches `osv` listening on the given port within nvim test - instances. +- [dev_test.txt](../runtime/doc/dev_test.txt) diff --git a/test/functional/example_spec.lua b/test/functional/example_spec.lua index 25554c5a7d..8a39e983d8 100644 --- a/test/functional/example_spec.lua +++ b/test/functional/example_spec.lua @@ -1,5 +1,5 @@ -- To run this test: --- TEST_FILE=test/functional/example_spec.lua make functionaltest +-- make functionaltest TEST_FILE=test/functional/example_spec.lua local t = require('test.testutil') local n = require('test.functional.testnvim')()