Fix subscription recovery

[Vadim Kovalyov (vadim-kovalyov)]

Problem

SubscriptionProcessor.ProcessPendingSubscriptions() currently drains a client's pending
subscription queue even when ConnectionManager.GetCloudConnection(id) returns
Option.None<ICloudProxy>.

Each dequeued operation is passed to ProcessSubscriptionWithRetry(). For subscriptions that
need a cloud proxy, ProcessSubscription() uses Option.ForEach() or
Option.ForEachAsync(). With Option.None, these calls complete without invoking the cloud
operation and without throwing. The retry policy therefore treats the operation as successful,
while the pending operation has already been removed.

local reconnect
  -> MQTT session restores Methods=true
  -> cloud proxy unavailable during pending processing
  -> SetupCallMethodAsync is skipped as a successful no-op
  -> no upstream method link
  -> IoT Hub returns 404_103
  -> module restart restores service

Fix

• Pending operations are preserved until a proxy is available.
• Recovery retries run independently per client with bounded backoff.
• Overlapping requests trigger another pass instead of being dropped.
• Shutdown now cancels recovery work cleanly.

Tests covererage for missing and failed proxy creation, subscription ordering, client isolation, retry coalescing, overlapping replay, and disposal.

[Copilot]

🟡 Changes recommended

Cloud-proxy creation can remain pending after disposal because its lookup does not observe shutdown cancellation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes subscription recovery when cloud proxies are temporarily unavailable.

Changes:

• Adds per-client retries, bounded backoff, and replay coalescing.
• Preserves pending subscription ordering.
• Cancels recovery work during shutdown and expands test coverage.

File summaries

File	Review
edge-hub/core/test/Microsoft.Azure.Devices.Edge.Hub.Core.Test/SubscriptionProcessorTest.cs	Adds recovery, ordering, isolation, concurrency, and disposal tests.
edge-hub/core/test/Microsoft.Azure.Devices.Edge.Hub.Core.Test/routing/RoutingEdgeHubTest.cs	Verifies subscription processor disposal.
edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs	Disposes Edge Hub during shutdown.
edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/SubscriptionProcessor.cs	Implements recovery and retry logic. Cloud-proxy lookup must observe shutdown cancellation; timeout messaging should describe scheduled retries.
edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/RoutingEdgeHub.cs	Disposes the subscription processor.

Review details

Suppressed comments (1)

edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/SubscriptionProcessor.cs:363

• This timeout is now retried automatically by ProcessSubscriptionsAsync after a backoff, not only when the client reconnects. The current message gives operators an incorrect recovery condition; describe the scheduled retry instead.

                    Log.LogDebug((int)EventIds.ErrorProcessingSubscriptions, ex, Invariant($"Timed out while processing subscriptions for client {id}. Will try again when connected."));

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.