417b90a7e5
- add additional parameters parsing (other implementations will just
ignore them). E.g. if in RST we have:
.. code:: nim
:test: "nim c $1"
...
then in Markdown that will be:
```nim test="nim c $1"
...
```
- implement Markdown interpretation of additional indentation which is
less than 4 spaces (>=4 spaces is a code block but it's not
implemented yet). RST interpretes it as quoted block, for Markdown it's
just normal paragraphs.
- add separate `md2html` and `md2tex` commands. This is to separate
Markdown behavior in cases when it diverges w.r.t. RST significantly —
most conspicously like in the case of additional indentation above, and
also currently the contradicting inline rule of Markdown is also turned
on only in `md2html` and `md2tex`. **Rationale:** mixing Markdown and
RST arbitrarily is a way to nowhere, we need to provide a way to fix the
particular behavior. Note that still all commands have **both** Markdown
and RST features **enabled**. In this PR `*.nim` files can be processed
only in Markdown mode, while `md2html` is for `*.md` files and
`rst2html` for `*.rst` files.
- rename `*.rst` files to `.*md` as our current default behavior is
already Markdown-ish
- convert code blocks in `docgen.rst` to Markdown style as an example.
Other code blocks will be converted in the follow-up PRs
- fix indentation inside Markdown code blocks — additional indentation
is preserved there
- allow more than 3 backticks open/close blocks (tildas \~ are still not
allowed to avoid conflict with RST adornment headings) see also
https://github.com/nim-lang/RFCs/issues/355
- better error messages
- (other) fix a bug that admonitions cannot be used in sandbox mode; fix
annoying warning on line 2711
7.6 KiB
=========== Testament
.. default-role:: code .. include:: rstcommon.rst .. contents::
Testament is an advanced automatic unittests runner for Nim tests, is used for the development of Nim itself,
offers process isolation for your tests, it can generate statistics about test cases,
supports multiple targets (C, C++, ObjectiveC, JavaScript, etc),
simulated Dry-Runs <https://en.wikipedia.org/wiki/Dry_run_(testing)>_,
has logging, can generate HTML reports, skip tests from a file, and more,
so can be useful to run your tests, even the most complex ones.
Test files location
By default Testament looks for test files on "./tests/*.nim".
You can overwrite this pattern glob using pattern <glob>:option:.
The default working directory path can be changed using
--directory:"folder/subfolder/":option:.
Testament uses the nim:cmd: compiler on PATH.
You can change that using --nim:"folder/subfolder/nim":option:.
Running JavaScript tests with --targets:"js":option: requires
a working NodeJS on PATH.
Options
--print Also print results to the console
--simulate See what tests would be run but don't run them
(for debugging)
--failing Only show failing/ignored tests
--targets:"c cpp js objc"
Run tests for specified targets (default: c)
--nim:path Use a particular nim executable (default: $PATH/nim)
--directory:dir Change to directory dir before reading the tests
or doing anything else.
--colors:on|off Turn messages coloring on|off.
--backendLogging:on|off Disable or enable backend logging.
By default turned on.
--skipFrom:file Read tests to skip from file - one test per
line, # comments ignored
Running a single test
This is a minimal example to understand the basics, not very useful for production, but easy to understand:
.. code:: console
$ mkdir tests $ echo "assert 42 == 42" > tests/test0.nim $ testament run test0.nim PASS: tests/test0.nim C ( 0.2 sec) $ testament r test0 PASS: tests/test0.nim C ( 0.2 sec)
Running all tests from a directory
.. code:: console
$ testament pattern "tests/*.nim"
To search for tests deeper in a directory, use
.. code:: console
$ testament pattern "tests//*.nim" # one level deeper $ testament pattern "tests//**/*.nim" # two levels deeper
HTML Reports
Generate HTML Reports testresults.html from unittests,
you have to run at least 1 test before generating a report:
.. code:: console
$ testament html
Writing Unitests
Example "template" to edit and write a Testament unittest:
.. code-block:: nim
discard """
# What actions to expect completion on.
# Options:
# "compile": expect successful compilation
# "run": expect successful compilation and execution
# "reject": expect failed compilation. The "reject" action can catch
# {.error.} pragmas but not {.fatal.} pragmas because
# {.fatal.} pragmas guarantee that compilation will be aborted.
action: "run"
# The exit code that the test is expected to return. Typically, the default
# value of 0 is fine. Note that if the test will be run by valgrind, then
# the test will exit with either a code of 0 on success or 1 on failure.
exitcode: 0
# Provide an `output` string to assert that the test prints to standard out
# exatly the expected string. Provide an `outputsub` string to assert that
# the string given here is a substring of the standard out output of the
# test.
output: ""
outputsub: ""
# Whether to sort the output lines before comparing them to the desired
# output.
sortoutput: true
# Each line in the string given here appears in the same order in the
# compiler output, but there may be more lines that appear before, after, or
# in between them.
nimout: '''
a very long, multi-line string'''
# This is the Standard Input the test should take, if any.
input: ""
# Error message the test should print, if any.
errormsg: ""
# Can be run in batch mode, or not.
batchable: true
# Can be run Joined with other tests to run all togheter, or not.
joinable: true
# On Linux 64-bit machines, whether to use Valgrind to check for bad memory
# accesses or memory leaks. On other architectures, the test will be run
# as-is, without Valgrind.
# Options:
# true: run the test with Valgrind
# false: run the without Valgrind
# "leaks": run the test with Valgrind, but do not check for memory leaks
valgrind: false # Can use Valgrind to check for memory leaks, or not (Linux 64Bit only).
# Command the test should use to run. If left out or an empty string is
# provided, the command is taken to be:
# "nim $target --hints:on -d:testing --nimblePath:build/deps/pkgs $options $file"
# You can use the $target, $options, and $file placeholders in your own
# command, too.
cmd: "nim c -r $file"
# Maximum generated temporary intermediate code file size for the test.
maxcodesize: 666
# Timeout seconds to run the test. Fractional values are supported.
timeout: 1.5
# Targets to run the test into (c, cpp, objc, js). Defaults to c.
targets: "c js"
# flags with which to run the test, delimited by `;`
matrix: "; -d:release; -d:caseFoo -d:release"
# Conditions that will skip this test. Use of multiple "disabled" clauses
# is permitted.
disabled: "bsd" # Can disable OSes...
disabled: "win"
disabled: "32bit" # ...or architectures
disabled: "i386"
disabled: "azure" # ...or pipeline runners
disabled: true # ...or can disable the test entirely
""" assert true assert 42 == 42, "Assert error message"
- As you can see the "Spec" is just a
discard """ """. - Spec has sane defaults, so you don't need to provide them all, any simple assert will work just fine.
This is not the full spec of Testament, check the Testament Spec on GitHub, see parseSpec(). <https://github.com/nim-lang/Nim/blob/devel/testament/specs.nim#L238>_Nim itself uses Testament, so there are plenty of test examples. <https://github.com/nim-lang/Nim/tree/devel/tests>_- Has some built-in CI compatibility, like Azure Pipelines, etc.
Testament supports inlined error messages on Unittests, basically comments with the expected error directly on the code. <https://github.com/nim-lang/Nim/blob/9a110047cbe2826b1d4afe63e3a1f5a08422b73f/tests/effects/teffects1.nim>_
Unitests Examples
Expected to fail:
.. code-block:: nim
discard """ errormsg: "undeclared identifier: 'not_defined'" """ assert not_defined == "not_defined", "not_defined is not defined"
Non-Zero exit code:
.. code-block:: nim
discard """ exitcode: 1 """ quit "Non-Zero exit code", 1
Standard output checking:
.. code-block:: nim
discard """
output: '''
0 1 2 3 4 5 '''
""" for i in 0..5: echo i
JavaScript tests:
.. code-block:: nim
discard """ targets: "js" """ when defined(js): import std/jsconsole console.log("My Frontend Project")
Compile-time tests:
.. code-block:: nim
discard """ action: "compile" """ static: assert 9 == 9, "Compile time assert"
Tests without Spec:
.. code-block:: nim
assert 1 == 1
See also:
Unittest <unittest.html>_