Problem: Vim tests are slow and flaky at the same time due to reliance
on timeouts which are unreliable.
Solution: improve Vim test performance and reduce flakiness
(Yee Cheng Chin)
A lot of Vim tests currently rely on waiting a specific amount of time
before asserting a condition. This is bad because 1) it is slow, as the
timeout is hardcoded, 2) it's unreliable as a resource-starved runner
may overshoot the timeout. Also, there are a lot of builtin sleep
commands in commonly used utilities like VerifyScreenDump and WaitFor()
which leads to a lot of unnecessary idle time.
Fix these issues by doing the following:
1. Make utilities like VerifyScreenDump and WaitFor use the lowest wait
time possible (1 ms). This essentially turns it into a spin wait. On
fast machines, these will finish very quickly. For existing tests
that had an implicit reliance on the old timeouts (e.g.
VerifyScreenDump had a 50ms wait before), fix the tests to wait that
specific amount explicitly.
2. Fix tests that sleep or wait for long amounts of time to instead
explicitly use a callback mechanism to be notified when a child
terminal job has finished. This allows the test to only take as much
time as possible instead of having to hard code an unreliable
timeout.
With these fixes, tests should 1) completely quickly on fast machines,
and 2) on slow machines they will still run to completion albeit slowly.
Note that previoulsy both were not true. The hardcoded timeouts meant
that on fast machines the tests were mostly idling wasting time, whereas
on slow machines, the timeouts often were not generous enough to allow
them to run to completion.
closes: vim/vim#16615e70587dbdb
Part of shared.vim and test_crash.vim changes only.
Co-authored-by: Yee Cheng Chin <ychin.git@gmail.com>
Problem: tests: still preferring python2 over python3
Solution: prefer Python 3 when picking a Python program in Vim tests,
by checking for the more specific python version first and
only when python3 not found, check for the python binary
(Yee Cheng Chin)
Most OSes have Python 3 mapped to `python3` instead of `python`. Vim
tests should prioritize using that instead of Python 2 in case that is
still installed on the host system.
closes: vim/vim#15986cef8ab2c75
Cherry-pick test changes from patch 8.2.{2824,4684}.
Co-authored-by: Yee Cheng Chin <ychin.git@gmail.com>
Problem: tests: tests may fail on Windows environment
Solution: use shellcmdflag=/D to skip executing autorun from
the registry (Milly)
closes: vim/vim#159004f5681dbdf
Cherry-pick Test_cursorhold_insert_with_timer_interrupt() change from
patch 8.2.1836.
Co-authored-by: Milly <milly.ca@gmail.com>
Problem: MS-Windows: "%T" time format does not appear to work.
Solution: Use "%H:%M:%S" instead.
5fbbec180b
Co-authored-by: Bram Moolenaar <Bram@vim.org>
Problem: A failing flaky test doesn't mention the time.
Solution: Add the time for debugging. Improve error message.
06d32a0c17
Co-authored-by: Bram Moolenaar <Bram@vim.org>
