os.remove() wasn't removing the temporary swap directory which leads to problems when the test is run a second time. That's also the reason why the CI never caught this. os.remove() got replaced by helpers.rmdir().