make db inquiry sequencially - #683
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #683 +/- ##
=====================================
Coverage 78.6% 78.6%
=====================================
Files 41 41
Lines 8998 8998
Branches 1682 1682
=====================================
Hits 7073 7073
Misses 1591 1591
Partials 334 334
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Order the legacy
monitor_requeststable by variable name instead of byrequested_at, so the monitor's merged request queue no longer depends on whichsecond each request landed in.
Changes
TaskTracker.take_monitor_requests—tracking.pyrequested_atdefaults toCURRENT_TIMESTAMP, which resolves to whole seconds.Requests enqueued in the same second therefore tied and fell back to the variable
name, and only requests that happened to straddle a second boundary came back in
arrival order — so the ordering could not deliver the arrival order it looks like
it delivers, and
test_monitor_requests_are_taken_oncefailed under load when theenqueues straddled a tick.
Variable-name order is also what the file queue this table is the legacy half of
already does:
task_status.take_monitor_requestsreads its requests in filenameorder. The monitor concatenates the two, so ordering them the same way is what
makes the merged queue's submission order predictable. The
requested_atcolumnis kept for diagnostics.
New regression test —
tests/unit/test_tracking.pytest_monitor_requests_order_does_not_depend_on_the_clockwrites the twotimestamps a second apart directly rather than waiting for a boundary, so the
case that used to flake is now covered deterministically.
Impact
Only the order in which variables appended mid-run (
moppy-cmorise --append-variables) are submitted as PBS jobs, and only for requests thatstraddle a second boundary. Which variables run, and what they produce, are
unchanged. No schema change.
Verification
ORDER BYchange and passes with it.sleep(1.1)) is nowstable.