Testament now retries a test by a specified amount if it fails in any
way other than an invalid spec. This is to deal with the flaky GC tests
on Windows CI that fail in many different ways, from the linker randomly
erroring, segfaults, etc.
Unfortunately I couldn't do this cleanly in testament's current code.
The proc `addResult`, which is the "final" proc called in a test run's
lifetime, is now wrapped in a proc `finishTest` that returns a bool
`true` if the test failed and has to be retried. This result is
propagated up from `cmpMsgs` and `compilerOutputTests` until it reaches
`testSpecHelper`, which handles these results by recursing if the test
has to be retried. Since calling `testSpecHelper` means "run this test
with one given configuration", this means every single matrix
option/target etc. receive an equal amount of retries each.
The result of `finishTest` is ignored in cases where it's known that it
won't be retried due to passing, being skipped, having an invalid spec
etc. It's also ignored in `testNimblePackages` because it's not
necessary for those specific tests yet and similar retry behavior is
already implemented for part of it.
This was a last resort for the flaky GC tests but they've been a problem
for years at this point, they give us more work to do and turn off
contributors. Ideally GC tests failing should mark as "needs review" in
the CI rather than "failed" but I don't know if Github supports
something like this.
* change `include genode/env` to an import
ref 0b262e9496 (diff-8718bd20d8f61d6638d3d64b19efc31bcd40a6d5be8215b2a1f0b75ed93e8d56)
* fixes comments
* ship a modern nimble with lock files support
* not sure whether the latest nimble has a regression
now I'm trying 0.14.0
* change `pkgs` to `pkgs2` (#21073)
* overwrite problematic packages
* Apply suggestions from code review
* Apply suggestions from code review
* Apply suggestions from code review
* [Testament] Extend and document message testing aids
* Enable inline msgs when not reject action.
Eliminates the pain of changing the line and column numbers in `nimout`
or `output` while making changes to the test.
* Enable using inline msgs and nimout together.
Allows ease of inline msgs for the test as well as testing msgs from
other modules.
* Add path separator and test filename variable interpolation in
msgs.
Eases handling path separators in the msgs.
* Add some documentation.
* Fixed lots of broken tests
* Fixed more broken tests
* Support multiple inline messages per a line
* Fix a broken test
* Revert variable substitution in `output`
* Remove uneeded params
* Update doc/testament.md
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Update testament/specs.nim
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Update testament/specs.nim
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Fix indentation
Co-authored-by: quantimnot <quantimnot@users.noreply.github.com>
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>
* Testament checks host/cpu in compiler/platform for discard spec statement
* Remove duplicated value checks, add comments, add all cpu bit sizes and endianesses
Co-authored-by: quantimnot <quantimnot@users.noreply.github.com>
* Use testament to check Norm test pass
This is what I actually use to test Norm, so it's better to use it.
This should not currently pass. This is expected because this is exactly the problem I want to highlight with this PR. My tests do indeed not pass at the moment.
* Remove clearNimblePath from testament command.
Co-authored-by: flywind <xzsflywind@gmail.com>
* use NIM_TESTAMENT_BATCH for important_packages, improve formatting, code cleanups
* workaround for nimcrypto not having `--path:.`; remove workaround "two are special snowflakes"
* add comment explaining why nimcrypto fails and what packages need to be testable
* trigger valgrind failure on memory leak
* remove non-malloc tests
* remove ORC test
is redundant because we already have an ARC test
* only run valgrind tests on 64-bit Linux
* disable freebsd and openbsd
* Remove tleak_refc
As to not test implementation details (or bug)
* Fix test failures by removing redundant test
Since this tests/shoulfail/tvalgrind.nim was specified here to fail this test
itself fails since it will be skipped on non-linux CI
* Remove test, reason detailed in the previous commit
* Remove redundant disables
* Revert removing disables
* Add and use valgrind: leaks
* Fix
Co-authored-by: Clyybber <darkmine956@gmail.com>
Co-authored-by: n5m
* Working on OpenBSD CI
* Condense steps into 2 steps to make output easier to follow.
* Move up one directory after csources build.
* Remove FreeBSD build manifest and add OpenBSD test ignores for coroutines and hot code reloading.
* If runCI fails, run the test results script.
* Add email trigger for build failure
* Remove .git from repository URL
* Disable SFML test on OpenBSD
* Disable tgetaddrinfo on OpenBSD as only UDP and TCP protocols are supported.
* Remove getFilePermissions as it causes CI test failures with NimScript.
* Set clang as cc in nim.cfg and use gmake to build csources.
* Add getCurrentDir to nimscript.
* Remove duplicate getCurrentDir and check for not weirdTarget.
* Add CI badge for OpenBSD.
* Disable tests which allocate lots of memory for OpenBSD.
* Use `CORO_BACKEND_SETJMP` on OpenBSD rather than ucontext.
* Simplify building of koch
* Disable t8657 on OpenBSD. See issue #13760.
* Fix#12142 - tarray_of_channels fails on OpenBSD
* Disable thhtpclient_ssl and tosprocterminate on OpenBSD. These tests can be enabled at a later date after fixing them.
* Install libffi.
* Set path to libc for openbsd.
* Disable tevalffi for now.
* Remove tevalffi.nim.
* Use ncpuonline sysctl rather than ncpu.
* Disable tacceptcloserace and tasynchttpserver on OpenBSD.
* Enable tacceptcloserace and tasynchttpserver.
* Fix#13775 as suggested by @alaviss - use /bin/cat on OpenBSD rather than /bin/sh.
* Enable test on OpenBSD.
* Disable tflowvar on OpenBSD.
* first implementation of the =trace and =dispose hooks for the cycle collector
* a cycle collector for ARC: progress
* manual: the .acyclic pragma is a thing once again
* gcbench: adaptations for --gc:arc
* enable valgrind tests for the strutils tests
* testament: better valgrind support
* ARC refactoring: growable jumpstacks
* ARC cycle detector: non-recursive algorithm
* moved and renamed core/ files back to system/
* refactoring: --gc:arc vs --gc:orc since 'orc' is even more experimental and we want to ship --gc:arc soonish
* Ref #12103 - adds FreeBSD CI
* Fix getApplFreebsd - length of the string includes the null terminator byte, so minus 1 for result length
* Show last commit in setup task.
* Remove .git from repository URL
* Don't include noisy details showing last commit.
* Add FreeBSD build status badge
* Fix#12182 - disable tconsole on FreeBSD
* Disable tgetaddrinfo on FreebSD as getaddrinfo doesn't support the ICMP protocol.
* Install boehm-gc-threaded
* Use libgc-threaded.so on FreeBSD rather than libgc.so.
* Simplify build failure handling. Update alt text for CI badge.
* Disable test on FreeBSD
* Simplify build config
- use GNU make to build csources
- set PATH variable using the environment key
- remove modification of config to set CC as this is already set
* Install git which seems to be missing from current freebsd images
* Revert change to how path is set
* Add a comment explaining why the length is truncated
* Fix tconsole.
* IC: C codegen is aware of IC
* manual: minor change to make VSCode's RST plugin render it properly
* IC: minor refactoring
* testament: code refactorings
* rodutils: removed dead code
* IC: always build the compiler with the IC feature
* IC: C codegen improvements
* IC: implement the undocumented -d:nimMustCache option for testing purposes
* IC: added first basic tests
* IC: extensive testing of the deserialization feature
* testament: refactoring; better IC tests
* IC: removes 'nimMustCache' flag; readonly does the same
* testament: minor refactoring
* update Nimble version
* testament: removed dead code and imports; IC: added simple test
* IC: progress
