fix: make async loop creation thread-safe - #1260
Merged
Carter Tinney (cartertinney) merged 4 commits intoSep 2, 2026
Merged
Conversation
Serialize lazy creation of the shared async client loops so concurrent first access cannot publish different event loops. This preserves Janus queue affinity and prevents handler runners from entering a permanent restart loop. Add concurrent initialization and queue-affinity regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the Janus loop-affinity invariant and why loop state must be checked again after acquiring the creation lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Carter Tinney (cartertinney)
marked this pull request as ready for review
September 2, 2026 15:39
Copilot started reviewing on behalf of
Carter Tinney (cartertinney)
September 2, 2026 15:39
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The concurrency regression test relies on timing and can false-pass against the original race.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds synchronized lazy initialization for shared asynchronous client event loops.
Changes:
- Protects loop creation and cleanup with a shared lock.
- Adds concurrent initialization regression coverage.
- Verifies Janus queue loop affinity.
File summaries
| File | Description |
|---|---|
loop_management.py |
Serializes event-loop creation and cleanup. |
test_loop_management.py |
Tests concurrent first access. |
test_async_inbox.py |
Tests Janus loop affinity. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Coordinate the first two loop-map reads so both worker threads observe the uninitialized state before either can publish a loop. This removes the scheduler-dependent sleep and guarantees the old implementation fails the test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Carter Tinney (cartertinney)
September 2, 2026 17:28
View session
Member
Author
|
/azp run Azure.azure-iot-sdk-python-dps-e2e |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Avishek (avishekpant)
approved these changes
Sep 2, 2026
Carter Tinney (cartertinney)
deleted the
agents/threadsafe-loop-creation
branch
September 2, 2026 20:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause
Python E2E build 163209 timed out in the first async C2D test on Windows/Python 3.10. Message delivery succeeded, but the log showed
CLIENT_INTERNAL_LOOPbeing created twice by concurrent first callers.The loop getters used an unsynchronized check-then-create. After #1258 stopped constructing each Janus queue through the internal loop, the loop was no longer guaranteed to be initialized before the handler runner and feature-enablement paths started concurrently. The queue could bind to the first loop while the second loop replaced the global reference. Every subsequent queue receive then failed Janus's loop-affinity check, and the handler manager's immediate restart behavior turned the failure into a tight restart loop until pytest timed out.
Why this shape
The process-wide loops are intentionally shared by clients to isolate internal work, handler runners, and user coroutine handlers. Moving to per-client loops would require broader ownership and shutdown changes while leaving lazy initialization subject to the same race. Eager creation would unconditionally start three daemon threads. A centralized, double-checked creation lock restores the existing singleton invariant at its owner and keeps the established hot path lock-free.
Why prior review missed it
#1258's reviews and tests focused on Janus resource closure and removing an unnecessary event-loop round trip during queue construction. Its unit tests verified sequential construction, queue operations, and shutdown, and its complete E2E matrix passed. The older loop manager only asserted that repeated sequential calls returned the same loop; it had no concurrent-first-access coverage.
The change therefore exposed a pre-existing race rather than introducing an obviously incorrect Janus operation. It requires two independent first consumers to enter a narrow scheduling window, which did not occur in #1258's runs and appeared later in one Windows job. This PR makes that concurrency contract explicit and deterministic in tests.
Validation