follows up #24279
`discard finishTest` was wrong if the test still had a `retries` option:
it would just ignore the result of the test. This is an unlikely mistake
but we safeguard against it by splitting `finishTest` into two, one that
completely ignores the retries option and `finishTestRetryable` which
has to be checked for a retry. This also makes the code look slightly
better.
Testament now retries a test by a specified amount if it fails in any
way other than an invalid spec. This is to deal with the flaky GC tests
on Windows CI that fail in many different ways, from the linker randomly
erroring, segfaults, etc.
Unfortunately I couldn't do this cleanly in testament's current code.
The proc `addResult`, which is the "final" proc called in a test run's
lifetime, is now wrapped in a proc `finishTest` that returns a bool
`true` if the test failed and has to be retried. This result is
propagated up from `cmpMsgs` and `compilerOutputTests` until it reaches
`testSpecHelper`, which handles these results by recursing if the test
has to be retried. Since calling `testSpecHelper` means "run this test
with one given configuration", this means every single matrix
option/target etc. receive an equal amount of retries each.
The result of `finishTest` is ignored in cases where it's known that it
won't be retried due to passing, being skipped, having an invalid spec
etc. It's also ignored in `testNimblePackages` because it's not
necessary for those specific tests yet and similar retry behavior is
already implemented for part of it.
This was a last resort for the flaky GC tests but they've been a problem
for years at this point, they give us more work to do and turn off
contributors. Ideally GC tests failing should mark as "needs review" in
the CI rather than "failed" but I don't know if Github supports
something like this.
This code will crash `check`/`nimsuggest` since the `ra` register is
uninitialised
```nim
import macros
static:
discard parseExpr("'")
```
Now it assigns an empty node so that it has something
Testament changes were so I could properly write a test. It would pass
even with a segfault since it could find the error
* clean up some test categories
* mention exact slice issue
* magics into system
* move trangechecks into overflow
* move tmemory to system
* try fix CI
* try fix CI
* final CI fix
* test azure
* use exit 1
* try again
* use useSysAssert
* disable i386
* use refc for tlsEmulation on i386
* use refc
* disable i386
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* [Testament] Extend and document message testing aids
* Enable inline msgs when not reject action.
Eliminates the pain of changing the line and column numbers in `nimout`
or `output` while making changes to the test.
* Enable using inline msgs and nimout together.
Allows ease of inline msgs for the test as well as testing msgs from
other modules.
* Add path separator and test filename variable interpolation in
msgs.
Eases handling path separators in the msgs.
* Add some documentation.
* Fixed lots of broken tests
* Fixed more broken tests
* Support multiple inline messages per a line
* Fix a broken test
* Revert variable substitution in `output`
* Remove uneeded params
* Update doc/testament.md
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Update testament/specs.nim
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Update testament/specs.nim
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Fix indentation
Co-authored-by: quantimnot <quantimnot@users.noreply.github.com>
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* fixes#17943
* IC: test 'nim check --ic:on' for the full Nim compiler
* Update testament/testament.nim
Co-authored-by: Clyybber <darkmine956@gmail.com>
Co-authored-by: Clyybber <darkmine956@gmail.com>
* refs #17946; refactor testament test summary, show test duration for failures; increase timeout tchannels
* revert workarounds from https://github.com/nim-lang/Nim/pull/16698 and add allowPrefixMatch optional param to greedyOrderedSubsetLines
* add test
* workaround for yet another testament bug
* gitutils: add diffStrings, diffFiles, and use it in testament to compare expected vs gotten
* refactor with createTempDir
* cleanup
* refacotr
* PRTEMP fake test spec changes to show effect of diffStrings
* add runnableExamples for experimental/diff + cross-reference with gitutils
* Revert "PRTEMP fake test spec changes to show effect of diffStrings"
This reverts commit 57dc8d642d.
* Implement https://forum.nim-lang.org/t/7848#50018 with just the same
`SuccessX` hint category, build mode on a separate, final line, and
no change to how the mode is spelled for -d:release/-d:danger.
* Change to add a new BuildMode hint category and keep testament in sync
as per comment.
* Add "--hint:buildmode:off" to `defaultHintsOff`.
* Remove as requested.
* As requested for tests clean up.
* Address code review.
* Address code review.
* Mirror db45642311
* use NIM_TESTAMENT_BATCH for important_packages, improve formatting, code cleanups
* workaround for nimcrypto not having `--path:.`; remove workaround "two are special snowflakes"
* add comment explaining why nimcrypto fails and what packages need to be testable
