File: //home/ubuntu/neovim/test/README.md
Tests
=====
Tests are broadly divided into *unit tests* ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit/)),
*functional tests* ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional/)),
and *old tests* ([test/old/testdir/](https://github.com/neovim/neovim/tree/master/test/old/testdir/)).
- _Unit_ testing is achieved by compiling the tests as a shared library which is
loaded and called by [LuaJit FFI](http://luajit.org/ext_ffi.html).
- _Functional_ tests are driven by RPC, so they do not require LuaJit (as
opposed to Lua).
You can learn the [key concepts of Lua in 15 minutes](http://learnxinyminutes.com/docs/lua/).
Use any existing test as a template to start writing new tests.
Tests are run by `/cmake/RunTests.cmake` file, using `busted` (a Lua test-runner).
For some failures, `.nvimlog` (or `$NVIM_LOG_FILE`) may provide insight.
Depending on the presence of binaries (e.g., `xclip`) some tests will be
ignored. You must compile with libintl to prevent `E319: The command is not
available in this version` errors.
---
- [Running tests](#running-tests)
- [Writing tests](#writing-tests)
- [Lint](#lint)
- [Configuration](#configuration)
---
Layout
======
- `/test/benchmark` : benchmarks
- `/test/functional` : functional tests
- `/test/unit` : unit tests
- `/test/old/testdir` : old tests (from Vim)
- `/test/config` : contains `*.in` files which are transformed into `*.lua`
files using `configure_file` CMake command: this is for accessing CMake
variables in lua tests.
- `/test/includes` : include-files for use by luajit `ffi.cdef` C definitions
parser: normally used to make macros not accessible via this mechanism
accessible the other way.
- `/test/*/preload.lua` : modules preloaded by busted `--helper` option
- `/test/**/testutil.lua` : common utility functions in the context of the test
runner
- `/test/**/testnvim.lua` : common utility functions in the context of the
test session (RPC channel to the Nvim child process created by clear() for each test)
- `/test/*/**/*_spec.lua` : actual tests. Files that do not end with
`_spec.lua` are libraries like `/test/**/testutil.lua`, except that they have
some common topic.
Running tests
=============
Executing Tests
---------------
To run all tests (except "old" tests):
make test
To run only _unit_ tests:
make unittest
To run only _functional_ tests:
make functionaltest
Legacy tests
------------
To run all legacy Vim tests:
make oldtest
To run a *single* legacy test file you can use either:
make oldtest TEST_FILE=test_syntax.vim
or:
make test/old/testdir/test_syntax.vim
- Specify only the test file name, not the full path.
Debugging tests
---------------
- Each test gets a test id which looks like "T123". This also appears in the
log file. Child processes spawned from a test appear in the logs with the
*parent* name followed by "/c". Example:
```
DBG 2022-06-15T18:37:45.226 T57.58016.0 UI: flush
DBG 2022-06-15T18:37:45.226 T57.58016.0 inbuf_poll:442: blocking... events_enabled=0 events_pending=0
DBG 2022-06-15T18:37:45.227 T57.58016.0/c UI: stop
INF 2022-06-15T18:37:45.227 T57.58016.0/c os_exit:595: Nvim exit: 0
DBG 2022-06-15T18:37:45.229 T57.58016.0 read_cb:118: closing Stream (0x7fd5d700ea18): EOF (end of file)
INF 2022-06-15T18:37:45.229 T57.58016.0 on_process_exit:400: exited: pid=58017 status=0 stoptime=0
```
- You can set `$GDB` to [run functional tests under gdbserver](https://github.com/neovim/neovim/pull/1527):
```sh
GDB=1 TEST_FILE=test/functional/api/buffer_spec.lua TEST_FILTER='nvim_buf_set_text works$' make functionaltest
```
Read more about [filtering tests](#filtering-tests).
Then, in another terminal:
```sh
gdb -ex 'target remote localhost:7777' build/bin/nvim
```
If `$VALGRIND` is also set it will pass `--vgdb=yes` to valgrind instead of
starting gdbserver directly.
See `nvim_argv` in https://github.com/neovim/neovim/blob/master/test/functional/testnvim.lua.
- Hanging tests can happen due to unexpected "press-enter" prompts. The
default screen width is 50 columns. Commands that try to print lines longer
than 50 columns in the command-line, e.g. `:edit very...long...path`, will
trigger the prompt. Try using a shorter path, or `:silent edit`.
- If you can't figure out what is going on, try to visualize the screen. Put
this at the beginning of your test:
```lua
local Screen = require('test.functional.ui.screen')
local screen = Screen.new()
screen:attach()
```
Then put `screen:snapshot_util()` anywhere in your test. See the comments in
`test/functional/ui/screen.lua` for more info.
Filtering Tests
---------------
### Filter by name
Tests can be filtered by setting a pattern of test name to `TEST_FILTER` or `TEST_FILTER_OUT`.
``` lua
it('foo api',function()
...
end)
it('bar api',function()
...
end)
```
To run only test with filter name:
TEST_FILTER='foo.*api' make functionaltest
To run all tests except ones matching a filter:
TEST_FILTER_OUT='foo.*api' make functionaltest
### Filter by file
To run a *specific* unit test:
TEST_FILE=test/unit/foo.lua make unittest
or
cmake -E env "TEST_FILE=test/unit/foo.lua" cmake --build build --target unittest
To run a *specific* functional test:
TEST_FILE=test/functional/foo.lua make functionaltest
or
cmake -E env "TEST_FILE=test/functional/foo.lua" cmake --build build --target functionaltest
To *repeat* a test:
BUSTED_ARGS="--repeat=100 --no-keep-going" TEST_FILE=test/functional/foo_spec.lua make functionaltest
or
cmake -E env "TEST_FILE=test/functional/foo_spec.lua" cmake -E env BUSTED_ARGS="--repeat=100 --no-keep-going" cmake --build build --target functionaltest
### Filter by tag
Tests can be "tagged" by adding `#` before a token in the test description.
``` lua
it('#foo bar baz', function()
...
end)
it('#foo another test', function()
...
end)
```
To run only the tagged tests:
TEST_TAG=foo make functionaltest
**NOTE:**
* `TEST_FILE` is not a pattern string like `TEST_TAG` or `TEST_FILTER`. The
given value to `TEST_FILE` must be a path to an existing file.
* Both `TEST_TAG` and `TEST_FILTER` filter tests by the string descriptions
found in `it()` and `describe()`.
Writing tests
=============
Guidelines
----------
- Luajit needs to know about type and constant declarations used in function
prototypes. The
[testutil.lua](https://github.com/neovim/neovim/blob/master/test/unit/testutil.lua)
file automatically parses `types.h`, so types used in the tested functions
could be moved to it to avoid having to rewrite the declarations in the test
files.
- `#define` constants must be rewritten `const` or `enum` so they can be
"visible" to the tests.
- Use **pending()** to skip tests
([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
Do not silently skip the test with `if-else`. If a functional test depends on
some external factor (e.g. the existence of `md5sum` on `$PATH`), *and* you
can't mock or fake the dependency, then skip the test via `pending()` if the
external factor is missing. This ensures that the *total* test-count
(success + fail + error + pending) is the same in all environments.
- *Note:* `pending()` is ignored if it is missing an argument, unless it is
[contained in an `it()` block](https://github.com/neovim/neovim/blob/d21690a66e7eb5ebef18046c7a79ef898966d786/test/functional/ex_cmds/grep_spec.lua#L11).
Provide empty function argument if the `pending()` call is outside `it()`
([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
- Really long `source([=[...]=])` blocks may break Vim's Lua syntax
highlighting. Try `:syntax sync fromstart` to fix it.
Where tests go
--------------
Tests in `/test/unit` and `/test/functional` are divided into groups
by the semantic component they are testing.
- _Unit tests_
([test/unit](https://github.com/neovim/neovim/tree/master/test/unit)) should
match 1-to-1 with the structure of `src/nvim/`, because they are testing
functions directly. E.g. unit-tests for `src/nvim/undo.c` should live in
`test/unit/undo_spec.lua`.
- _Functional tests_
([test/functional](https://github.com/neovim/neovim/tree/master/test/functional))
are higher-level (plugins and user input) than unit tests; they are organized
by concept.
- Try to find an existing `test/functional/*/*_spec.lua` group that makes
sense, before creating a new one.
Lint
====
`make lint` (and `make lualint`) runs [luacheck](https://github.com/mpeterv/luacheck)
on the test code.
If a luacheck warning must be ignored, specify the warning code. Example:
-- luacheck: ignore 621
http://luacheck.readthedocs.io/en/stable/warnings.html
Ignore the smallest applicable scope (e.g. inside a function, not at the top of
the file).
Configuration
=============
Test behaviour is affected by environment variables. Currently supported
(Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined,
treated as Integer; when defined, treated as String; when defined, treated as
Number; !must be defined to function properly):
- `BUSTED_ARGS` (F) (U): arguments forwarded to `busted`.
- `CC` (U) (S): specifies which C compiler to use to preprocess files.
Currently only compilers with gcc-compatible arguments are supported.
- `GDB` (F) (D): makes nvim instances to be run under `gdbserver`. It will be
accessible on `localhost:7777`: use `gdb build/bin/nvim`, type `target remote
:7777` inside.
- `GDBSERVER_PORT` (F) (I): overrides port used for `GDB`.
- `LOG_DIR` (FU) (S!): specifies where to seek for valgrind and ASAN log files.
- `VALGRIND` (F) (D): makes nvim instances to be run under `valgrind`. Log
files are named `valgrind-%p.log` in this case. Note that non-empty valgrind
log may fail tests. Valgrind arguments may be seen in
`/test/functional/testnvim.lua`. May be used in conjunction with `GDB`.
- `VALGRIND_LOG` (F) (S): overrides valgrind log file name used for `VALGRIND`.
- `TEST_COLORS` (F) (U) (D): enable pretty colors in test runner. Set to true by default.
- `TEST_SKIP_FRAGILE` (F) (D): makes test suite skip some fragile tests.
- `TEST_TIMEOUT` (FU) (I): specifies maximum time, in seconds, before the test
suite run is killed
- `NVIM_LUA_NOTRACK` (F) (D): disable reference counting of Lua objects
- `NVIM_PRG` (F) (S): path to Nvim executable (default: `build/bin/nvim`).
- `NVIM_TEST_MAIN_CDEFS` (U) (1): makes `ffi.cdef` run in main process. This
raises a possibility of bugs due to conflicts in header definitions, despite
the counters, but greatly speeds up unit tests by not requiring `ffi.cdef` to
do parsing of big strings with C definitions.
- `NVIM_TEST_PRINT_I` (U) (1): makes `cimport` print preprocessed, but not yet
filtered through `formatc` headers. Used to debug `formatc`. Printing is done
with the line numbers.
- `NVIM_TEST_PRINT_CDEF` (U) (1): makes `cimport` print final lines which will
be then passed to `ffi.cdef`. Used to debug errors `ffi.cdef` happens to
throw sometimes.
- `NVIM_TEST_PRINT_SYSCALLS` (U) (1): makes it print to stderr when syscall
wrappers are called and what they returned. Used to debug code which makes
unit tests be executed in separate processes.
- `NVIM_TEST_RUN_FAILING_TESTS` (U) (1): makes `itp` run tests which are known
to fail (marked by setting third argument to `true`).
- `NVIM_TEST_CORE_*` (FU) (S): a set of environment variables which specify
where to search for core files. Are supposed to be defined all at once.
- `NVIM_TEST_CORE_GLOB_DIRECTORY` (FU) (S): directory where core files are
located. May be `.`. This directory is then recursively searched for core
files. Note: this variable must be defined for any of the following to have
any effect.
- `NVIM_TEST_CORE_GLOB_RE` (FU) (S): regular expression which must be matched
by core files. E.g. `/core[^/]*$`. May be absent, in which case any file is
considered to be matched.
- `NVIM_TEST_CORE_EXC_RE` (FU) (S): regular expression which excludes certain
directories from searching for core files inside. E.g. use `^/%.deps$` to not
search inside `/.deps`. If absent, nothing is excluded.
- `NVIM_TEST_CORE_DB_CMD` (FU) (S): command to get backtrace out of the
debugger. E.g. `gdb -n -batch -ex "thread apply all bt full"
"$_NVIM_TEST_APP" -c "$_NVIM_TEST_CORE"`. Defaults to the example command.
This debug command may use environment variables `_NVIM_TEST_APP` (path to
application which is being debugged: normally either nvim or luajit) and
`_NVIM_TEST_CORE` (core file to get backtrace from).
- `NVIM_TEST_CORE_RANDOM_SKIP` (FU) (D): makes `check_cores` not check cores
after approximately 90% of the tests. Should be used when finding cores is
too hard for some reason. Normally (on OS X or when
`NVIM_TEST_CORE_GLOB_DIRECTORY` is defined and this variable is not) cores
are checked for after each test.
- `NVIM_TEST_RUN_TESTTEST` (U) (1): allows running
`test/unit/testtest_spec.lua` used to check how testing infrastructure works.
- `NVIM_TEST_TRACE_LEVEL` (U) (N): specifies unit tests tracing level:
- `0` disables tracing (the fastest, but you get no data if tests crash and
there no core dump was generated),
- `1` leaves only C function calls and returns in the trace (faster than
recording everything),
- `2` records all function calls, returns and executed Lua source lines.
- `NVIM_TEST_TRACE_ON_ERROR` (U) (1): makes unit tests yield trace on error in
addition to regular error message.
- `NVIM_TEST_MAXTRACE` (U) (N): specifies maximum number of trace lines to
keep. Default is 1024.