properly isolate FFI online tests (avoid caching) and reconnect on direct-imap failures - #8565
Conversation
|
First example https://github.com/chatmail/core/actions/runs/30563560871/job/90944100948#step:9:315 is in deltachat-rpc-client tests, but the commit (and the whole PR) is changing legacy python tests. There is no caching in deltachat-rpc-client already, so test_no_markseen_in_team_profile failed for some other reason. I think it was actually this already closed issue: #8446 |
fair enough. But if rpc-tests already do no caching, all the more reason the remove it from FFI based tests as well. I edited the PR description accordingly. Removing the caching actually removes >100 LOCs from the test support code. |
| self.conn.login(user, pw) | ||
| self.select_folder("INBOX") | ||
| return | ||
| except (OSError, ssl.SSLError, imaplib.IMAP4.error): |
There was a problem hiding this comment.
https://docs.python.org/3/library/ssl.html#ssl.SSLError says ssl.SSLError is a subtype of OSError, so if OSError is catched, then there is no need to catch ssl.SSLError as well. Maybe even catch only ConnectionError.
IMAP errors probably should not be ignored, if we managed to connect and IMAP server says e.g. that we cannot login (because the test server has registration disabled, possibly automatically), then retrying forever will likely not help.
There was a problem hiding this comment.
changed it to OSError, IMAP4.abort with a comment.
link2xt
left a comment
There was a problem hiding this comment.
There was also an attempt to remove caching at #5446, apparently reusing the databases saved some time on configuration (requesting autoconfig, logging in with a test session just to add transport), but all the message cleanup with key reuse is likely not worth it.
As we are using a dedicated CI relay instance, we don't need to be careful with creating test addresses. But re-using test addresses in a test running in a single worker can leak events/messages which compromises test isolation. Instead of encountering random flaky tests because let's just not cache addresses to maximize test isolation.
…g to connect on first try also core src/imap.rs retries connecting (2s with backoff) but direct_imap is only used in (CI) tests and we can just keep trying every second until pytest timeout hits.
dc30fd6 to
82f65ec
Compare
|
some measurements for running standard tox CFFI test suite:
it seems runs falls into one of the buckets. pytest durations says all in all the branch should not be noticably slower. running core tests against lxc containers should be noticeably faster, see #7926 |
two commits to fix two test flakyness sources (EDIT: and one cleanup commit removing all caching-related logic from the FFI test plugin).
(EDIT) When account addresses are reused across tests, unexpected events or un-drained messages from previous setup steps/test runs arrive can cause assertion failures. I think it's fine to just create new addresses on CI all the time. Creating 100-200 addresses per test run is worth the guaranteed isolation. Ultimately Postfix on the CI server is free to arbitrary delay messages between mailboxes, and re-using them randomly breaks test isoilation. Better to exclude this failure mode cleanly and avoid caching. RPC-tests also use no caching.
under parallel test load transient socket resets during directimap SSL setup can crash tests during fixture initialization example: :
Just retrying imap connect (like core does itself) is fine.