mirror of
https://github.com/nim-lang/Nim.git
synced 2025-12-29 01:14:41 +00:00
720d0aee5c
Testament now retries a test by a specified amount if it fails in any way other than an invalid spec. This is to deal with the flaky GC tests on Windows CI that fail in many different ways, from the linker randomly erroring, segfaults, etc. Unfortunately I couldn't do this cleanly in testament's current code. The proc `addResult`, which is the "final" proc called in a test run's lifetime, is now wrapped in a proc `finishTest` that returns a bool `true` if the test failed and has to be retried. This result is propagated up from `cmpMsgs` and `compilerOutputTests` until it reaches `testSpecHelper`, which handles these results by recursing if the test has to be retried. Since calling `testSpecHelper` means "run this test with one given configuration", this means every single matrix option/target etc. receive an equal amount of retries each. The result of `finishTest` is ignored in cases where it's known that it won't be retried due to passing, being skipped, having an invalid spec etc. It's also ignored in `testNimblePackages` because it's not necessary for those specific tests yet and similar retry behavior is already implemented for part of it. This was a last resort for the flaky GC tests but they've been a problem for years at this point, they give us more work to do and turn off contributors. Ideally GC tests failing should mark as "needs review" in the CI rather than "failed" but I don't know if Github supports something like this.
This directory contains the test cases.
Each test must have a filename of the form: t*.nim
Note: Testament is only aware of tests under a directory (eg tests/foo/) and will ignore
top-level tests like tests/tbar.nim.
Specs
Each test can contain a spec in a discard """ ... """ block.
Check out the parseSpec procedure in the specs module for a full and reliable reference
action
Specifies what action this test should take.
Default: run
Options:
compile- compiles the module and fails the test if compilations fails.run- compiles and runs the module, fails the test if compilation or execution of test code fails.reject- compiles the module and fails the test if compilation succeeds.
There are certain spec keys that imply run, including output and
outputsub.
Categories
Each folder under this directory represents a test category, which can be
tested by running koch tests pcat <category> (or cat to avoid parallel
testing, which is slower).
The folder dll contains simple DLL tests.
The folder realtimeGC contains a test for validating that the realtime GC
can run properly without linking against the nimrtl.dll/so.