Commit Graph

63 Commits

Author SHA1 Message Date
Feoramund
0ff130d82b Fix ad hoc printf in test runner signal handler 2024-06-02 23:36:04 -04:00
Feoramund
fa29974dab Use Warning log level for reporting memory leaks
Works well with `-define:ODIN_TEST_LOG_LEVEL=warning`.
2024-06-02 23:21:44 -04:00
Feoramund
7764ab2ab0 Prevent test runner deadlock on NetBSD
Add `pthread_testcancel` to `core:sys/unix`
2024-06-02 21:27:36 -04:00
Feoramund
f77ce359ce Be pedantic about not overwriting Odin errors
I was encountering bounds-check error messages being overwritten during
a test, if the test failed for another reason and sent a log message.

The original intent of having this check inside of the above `if` block
was that if a test sent an error message, then it was assumed an
overwrite would be safe, but it's completely possible for a test to fail
for a legitimate reason, then do an unrelated bounds check somewhere
else that would be buried under the animation.

This change will make sure that, no matter what, the progress display
will not trigger a clear if a signal was raised. There's still no
guarantee that bounds-check messages will be printed properly, and it's
best to redirect STDERR.

The only way that can be fixed is if they get a similar hook to
`context.assertion_failure_proc`.
2024-06-02 15:51:26 -04:00
Feoramund
3f1249c27e Tell user about ODIN_TEST_RANDOM_SEED option 2024-06-02 15:34:13 -04:00
Feoramund
d581dbbec5 Keep test runner main thread from using 100% of a CPU core 2024-06-02 14:54:32 -04:00
Feoramund
5e3e958574 Add -define:ODIN_TEST_LOG_LEVEL to set lowest log level 2024-06-02 14:54:32 -04:00
Feoramund
6a9203328b Log thread count at test run start
Provides a helpful info message about the option to change how many
threads are used per run.
2024-06-02 14:54:32 -04:00
Feoramund
5db65aa796 Make it easier to learn about ODIN_TEST_CLIPBOARD 2024-06-02 14:54:32 -04:00
Feoramund
cb00b8022b Add note about SIGSEGV edge case on UNIX-likes 2024-06-02 14:54:32 -04:00
Feoramund
4875f745c8 Remove Windows test runner in favor of libc implementation 2024-06-02 14:54:32 -04:00
Feoramund
45fa9d8148 Expand documentation comment for ODIN_TEST_NAMES 2024-06-02 14:54:32 -04:00
Feoramund
6a5d51f0d6 Use more concise way of satisfying -vet 2024-06-02 14:54:31 -04:00
Feoramund
433ca538bf Be specific about platforms not implementing test runner signal handler 2024-06-02 14:54:31 -04:00
Feoramund
d1723664a7 Catch SIGILL, SIGFPE, SIGSEGV in the test runner 2024-06-02 14:54:31 -04:00
Feoramund
fb37572c4c Rename signal_handler.odin to signal_handler_libc.odin 2024-06-02 14:54:31 -04:00
Feoramund
b74b956fda Remove unneeded import 2024-06-02 14:47:08 -04:00
Feoramund
1f6a6f2cd3 Support deterministic random seeding of tests
Add a new option `ODIN_TEST_RANDOM_SEED` which is picked from the cycle
counter at startup, if it's not specified by the user.

This number is sent to every test in the `T` struct and reset every test
(just in case).
2024-06-02 14:47:07 -04:00
Feoramund
84ad71fdb3 Support ODIN_TEST_PROGRESS_WIDTH=0
This will automatically calculate how wide the progress bars should be
based on the package with the greatest number of tests.

The progress width is now capped to 100.
2024-06-02 14:47:07 -04:00
Feoramund
49fa66370f Report test memory usage only if there's an issue
Adds new option `ODIN_TEST_ALWAYS_REPORT_MEMORY`, for when you always
want to see the memory usage report.
2024-06-02 14:47:07 -04:00
Feoramund
e1a3c0e21d Track memory in the test runner by default 2024-06-02 14:47:07 -04:00
Feoramund
b7e1ae7073 Change test runner options to SCREAMING_SNAKE_CASE
This commit also changes the name of `test_select` to `ODIN_TEST_NAMES`,
to better conform with the already-existing `-test-name:<name>` option.
2024-06-02 14:47:07 -04:00
Feoramund
dcfda195d2 Send terminal control code to STDOUT instead
`STDERR` might be redirected, and this code signals to the terminal to
show the cursor again. Otherwise, the cursor will be invisible.
2024-06-02 14:47:07 -04:00
Feoramund
a1c5bebac7 Fix ANSI redraw eating last log line 2024-06-02 14:47:07 -04:00
Feoramund
dffc3af86c Remove safe_heap_allocator from test runner
I was under the impression that the default `context.allocator` was not
thread-safe, but I've been told that this is not the case.
2024-06-02 14:47:06 -04:00
Feoramund
1afc235359 Use plain sort for internal_tests 2024-06-02 14:47:06 -04:00
Feoramund
87ea4a2652 Temporarily disable Windows-specific test runner
I do not have a Windows machine to test the refactored test runner, and
I am unsure if it would even run correctly on Windows without this
disabled.
2024-06-02 14:34:31 -04:00
Feoramund
d03024088a Remove unneeded code 2024-06-02 14:34:31 -04:00
Feoramund
b6c4dfb68d Refactor the test runner
Changes

- Support multi-threaded testing.
- Support `set_fail_timeout` on all platforms.
- Display an animated progress report.
- Setup all tests with a context logger.
- Give all tests their own separate custom allocators.
- Support tracking test memory usage.
- Display a summary of the failed tests at the end.
- Let users select only specific tests to run.
- Support copying failed tests to the clipboard to run again.
- Support catching SIGINT (CTRL-C) to cancel early.
- Record context in cleanup procs.
- Write all log messages to STDERR for easy redirection.
- Possibly more I've forgotten.

New Options

- `-define:test_threads=N`: Specify thread count.
- `-define:test_thread_memory=B`: Specify initial memory block size in bytes to each thread.
- `-define:test_track_memory=true`: Track the memory usage of individual tests.
- `-define:test_fancy=false`: Disable animated progress report.
- `-define:test_select=package.test_name,...`: Run only select tests.
- `-define:test_clipboard=true`: Copy names of failed tests to the clipboard.
- `-define:test_progress_width=24`: Change the width of the animated progress bars.
2024-06-02 14:34:31 -04:00
gingerBill
6736205723 Merge pull request #3138 from FourteenBrush/master
Add a testing.expectf proc as a way to avoid a tprintf call.
2024-01-29 17:33:29 +00:00
gingerBill
3e7e779abf Replace core:* to base:* where appropriate 2024-01-28 22:18:51 +00:00
FourteenBrush
766d6aa946 Fix typo 2024-01-25 12:05:19 +01:00
FourteenBrush
895ebb95d5 Need to unpack args 2024-01-25 11:44:53 +01:00
FourteenBrush
3a5d80b291 Forgot to include loc param 2024-01-25 10:20:23 +01:00
FourteenBrush
967ccfc7cc Merge branch 'master' of https://github.com/FourteenBrush/Odin 2024-01-25 10:15:25 +01:00
FourteenBrush
712ae1c5ac Add testing.expectf 2024-01-25 10:08:09 +01:00
gingerBill
90ac400ec5 stdcall -> system 2024-01-17 17:25:23 +00:00
ramn
5f53d815d1 fix: make -vet not complain 2023-07-09 00:15:01 +02:00
ramn
7b89f25818 Fix #2637
where testing.expect_value can't compare nils
2023-07-08 23:46:51 +02:00
gingerBill
9b54b99bf6 Use positional and named arguments within the core library 2023-06-21 01:17:05 +01:00
Totoki Kei
5c18cca1ca [core:testing] add global_fail_timeout_thread nil check
Fixed an issue where test would always fail unless set_fail_timeout was called.
2022-06-04 12:21:20 +09:00
gingerBill
ba1930eb01 Update core to use new atomic intrinsics 2022-03-31 00:22:54 +01:00
gingerBill
da1edac56d Enforce -strict-style in CI 2022-01-31 11:29:05 +00:00
gingerBill
0d258e8b55 Make many package time procedures use "contextless" calling convention 2021-10-11 20:08:38 +01:00
gingerBill
3a60bee804 Move assert to implementation file 2021-10-11 20:07:39 +01:00
gingerBill
b67e0002c1 Add testing.expect_value; Improve testing.set_fail_timeout 2021-10-11 20:05:13 +01:00
gingerBill
d23144fd27 Keep -vet happy 2021-10-11 15:30:23 +01:00
gingerBill
e64eb631df Add testing.set_fail_timeout 2021-10-11 15:28:25 +01:00
gingerBill
b2164b5da6 Make the io/conv.odin utilities be #optional_ok 2021-09-29 13:24:42 +01:00
Jeroen van Rijn
97a46f664d testing: Strip ; from test runner. 2021-09-07 14:46:53 +02:00