Testament now retries a test by a specified amount if it fails in any
way other than an invalid spec. This is to deal with the flaky GC tests
on Windows CI that fail in many different ways, from the linker randomly
erroring, segfaults, etc.
Unfortunately I couldn't do this cleanly in testament's current code.
The proc `addResult`, which is the "final" proc called in a test run's
lifetime, is now wrapped in a proc `finishTest` that returns a bool
`true` if the test failed and has to be retried. This result is
propagated up from `cmpMsgs` and `compilerOutputTests` until it reaches
`testSpecHelper`, which handles these results by recursing if the test
has to be retried. Since calling `testSpecHelper` means "run this test
with one given configuration", this means every single matrix
option/target etc. receive an equal amount of retries each.
The result of `finishTest` is ignored in cases where it's known that it
won't be retried due to passing, being skipped, having an invalid spec
etc. It's also ignored in `testNimblePackages` because it's not
necessary for those specific tests yet and similar retry behavior is
already implemented for part of it.
This was a last resort for the flaky GC tests but they've been a problem
for years at this point, they give us more work to do and turn off
contributors. Ideally GC tests failing should mark as "needs review" in
the CI rather than "failed" but I don't know if Github supports
something like this.
(cherry picked from commit 720d0aee5c)
* [Testament] Extend and document message testing aids
* Enable inline msgs when not reject action.
Eliminates the pain of changing the line and column numbers in `nimout`
or `output` while making changes to the test.
* Enable using inline msgs and nimout together.
Allows ease of inline msgs for the test as well as testing msgs from
other modules.
* Add path separator and test filename variable interpolation in
msgs.
Eases handling path separators in the msgs.
* Add some documentation.
* Fixed lots of broken tests
* Fixed more broken tests
* Support multiple inline messages per a line
* Fix a broken test
* Revert variable substitution in `output`
* Remove uneeded params
* Update doc/testament.md
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Update testament/specs.nim
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Update testament/specs.nim
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Fix indentation
Co-authored-by: quantimnot <quantimnot@users.noreply.github.com>
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* refs #17946; refactor testament test summary, show test duration for failures; increase timeout tchannels
* revert workarounds from https://github.com/nim-lang/Nim/pull/16698 and add allowPrefixMatch optional param to greedyOrderedSubsetLines
* add test
* workaround for yet another testament bug
* add test for reject action
* highlight that a compilation failure is expected
* add the converse test for action=reject
* add trailing newline
* fix wording
* testament: move to root dir (it's not a test)
* osproc: fix process index passed to afterRunEvent for parallel runs
it was passing the index of the process, not index of all commands
* testament: complete file move
* posix_other: Haiku now has spawn.h
This is added per https://dev.haiku-os.org/ticket/13446
* posix_other: Add Haiku specific Dirent members
* cpuinfo: Add an implementation for Haiku
* distros: Add basic Haiku support
* encodings: update Haiku support
* fenv, math: Haiku now provides libm
* times: Add Haiku struct members
* ansi_c, osalloc: Add Haiku constants
* threads: Add Haiku support
* testament: Haiku uses LIBRARY_PATH
* nim.cfg: Update Haiku support
libnetwork should only be linked if network functions are used
* threads: Haiku does not support -pthread switch
* tworkingdir: Haiku's env is in /bin
* posix_other: add SIGKILLTHR for Haiku
* sockets: link with libnetwork on Haiku
* coro: correct ucontext.h location
http://pubs.opengroup.org/onlinepubs/009696699/basedefs/ucontext.h.html
* coro: ucontext backend is not available on Haiku
Haiku doesn't provide the <ucontext.h> header, as it was removed from POSIX
* coro: fix setjmp backend
The compiler does not allow statements after a noreturn function
* nativesockets: Haiku doesn't support AI_V4MAPPED
* system: hostOS can contains "haiku"
* os: add support for Haiku's packagefs
packagefs is read-only, but there are writable holes to the underlying
file system as well
* os: update constant for Haiku
Minor change to help people (e.g. like myself) who are starting out :)
Currently, the readme mentions the use of `./koch tests cat async`, but
the help text for koch doesn't mention it. Helptexts are modified
accordingly.
* Fixes#802, #803 and #3775 - genscript issues
* Test case for genscript
* Test script
* Verify on Linux
* Update categories.nim
* Fix merge
* Improve test framework
* Windows fixes
We accept a new environment variable, NIMTEST_COLOR,
which override the effect of NIMTEST_NO_COLOR.
The environment variable, NIMTEST_COLOR, can be 'never'
or 'always', which set the color output to false or true,
respectively.
