Skip to content

[Cosmos] Backport PPCB fixes to 4.71.3-hotfix - #50279

Open
Abhijeet Mohanty (jeet1995) wants to merge 24 commits into
Azure:hotfix/jeet1995/azure-cosmos_4.71.3-hotfixfrom
jeet1995:release_azure-cosmos-4.71.3-hotfix
Open

[Cosmos] Backport PPCB fixes to 4.71.3-hotfix#50279
Abhijeet Mohanty (jeet1995) wants to merge 24 commits into
Azure:hotfix/jeet1995/azure-cosmos_4.71.3-hotfixfrom
jeet1995:release_azure-cosmos-4.71.3-hotfix

Conversation

@jeet1995

@jeet1995 Abhijeet Mohanty (jeet1995) commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Backports the required PPCB fixes to azure-cosmos 4.71.3-hotfix and adapts its tests and pipelines to the release's older dependency surface and network-isolated hosted agents.

PR branches

  • Base: Azure:hotfix/jeet1995/azure-cosmos_4.71.3-hotfix
  • Base commit: ccbad4b8b7b5727484488f7c002698c23bb5db36
  • Head: jeet1995:release_azure-cosmos-4.71.3-hotfix

Backport inventory

Source Hotfix commit What was backported
#50182 7377cb8136b6 Failback recovery when partition addresses are missing or stale, including focused coverage.
#50158 f74871fd3a3d PPCB diagnostics and failback observability.
#49734 a6bf7a92f34c Publication of partitionLevelCircuitBreakerCfg in diagnostics client configuration.
#49568 7bde221395ab Release-adapted customer-workflow coverage.

Release compatibility

  • Restores PPCB unit and E2E regressions after adapting them to the 4.71 release surface.
  • Runs LatestCommitted customer workflows in Direct mode and retries reads until diagnostics show exactly one contacted region; unsupported Gateway cases are skipped.
  • Replaces the unavailable Confluent Avro serializer dependency with a repository-local test serializer.
  • Routes Maven, Spring, npm, pip, and NuGet dependencies through authenticated Azure Artifacts feeds.
  • Uses the current macOS test-proxy launch contract and validates dependency/version metadata without public registries.
  • Skips the 4.71-only code-quality lint job because its legacy sdk-build-tools dependency is unavailable in the isolated feed.
  • Removes CosmosNettyLeakDetectorFactory from the two customer-workflow TestNG suites because that listener does not exist in 4.71; all workflow tests, groups, and package selectors remain enabled.

Testing and engineering infrastructure changes

  • Restored and release-adapted PPCB unit, E2E, and customer-workflow coverage.
  • Restricted unsupported LATEST_COMMITTED Gateway cases to Direct mode and added diagnostic contacted-region retry validation.
  • Replaced the unavailable Confluent Avro test dependency with a repository-local serializer.
  • Routed Maven, npm, pip, and NuGet tooling through authenticated Azure Artifacts feeds.
  • Updated the macOS test-proxy launcher and NuGet source isolation.
  • Skipped the 4.71-only code-quality lint job because its legacy sdk-build-tools artifact is unavailable.
  • Removed the unavailable CosmosNettyLeakDetectorFactory listener from two TestNG suites without removing any tests.

Validation

  • azure-cosmos, azure-cosmos-test, and azure-cosmos-tests build in the local four-hotfix harness.
  • The Fix partitionLevelCircuitBreakerCfg missing from CosmosDiagnostics clientCfgs #49734 method-level diagnostics regression passes across all four hotfix worktrees.
  • The excluded-region customer workflow passes live in Direct mode; Gateway skips the unsupported case.
  • Kafka test compilation passes without the Confluent serializer artifact.
  • The pinned test-proxy package installs from Azure Artifacts and /Admin/IsAlive returns HTTP 200.
  • The unchanged FI case that receives HTTP 408 instead of the historical HTTP 201 expectation is accepted for this release and is not a hotfix blocker.
  • Fresh hosted requalification is in progress; no completed check currently reports failure.

Annie Liang (xinlian12) and others added 4 commits August 22, 2026 10:44
@xinlian12 @jeet1995
Fix PPCB failback with missing or stale addresses (Azure#50182)
7377cb8 
* Fix PPCB failback with missing or stale addresses

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeet1995
Fix PPCB diagnostics and failback observability. (Azure#50158)
f74871f 
* Improve PPCB failback diagnostics

* Validate PPCB state in diagnostics E2E tests

* Limit PPCB diagnostics assertion to data-plane requests

* Log PPCB failback backlog progress

* Add PPCB failback remaining meter

* Clarify PPCB failback meter name

* Reduce PPCB failback meter allocations

* Track PPCB pending recoveries by collection

* Correlate PPCB failback recovery diagnostics

* Harden PPCB failback recovery tests

* Fix PPCB all-region diagnostics assertion

* Optimize PPCB diagnostics snapshots

* Cache PPCB diagnostics snapshots

* Align PPCB failback flow with main

* Add benchmark fault injection support.

* Harden PPCB failback telemetry

Use a single injectable logger, prevent backlog telemetry failures from escaping the recovery flow, and align the pending failback metric name.

* Scope PR to PPCB diagnostics

Remove benchmark, fault-injection, metric, and recovery behavior changes. Retain immutable CosmosDiagnostics PPCB snapshots, lifecycle E2E assertions, and WARN logging for every failback failure.

* Reduce PPCB diagnostics overhead

Reuse immutable PPCB map references in response snapshots and shorten per-region diagnostic field names.

* Refactoring

* Simplify PPCB diagnostics snapshots

Represent holder state with one volatile immutable-map reference and align the compact timestamp serialization test.

* Ignore updates to empty PPCB diagnostics

Make updates to the shared uninitialized diagnostics sentinel a no-op and verify it remains null-serializing.

* Avoid copying PPCB diagnostics state

Retain the live PPCB diagnostics map reference to avoid per-publication map and wrapper allocations, accepting weak consistency.

* Log PPCB diagnostics lifecycle snapshots

Emit one full CosmosDiagnostics JSON payload for failed, post-failover, and post-failback E2E phases for PR evidence.

* Document PPCB diagnostics improvements

Add the unreleased changelog entry for per-region PPCB snapshots and failback WARN logging.

* Add PPCB failback outcome diagnostics

Track the latest background failback attempt time, outcome, and failure reason per partition-region; validate lifecycle state in E2E and focused recovery tests.

* Reset unavailable timestamp after PPCB failback

Use the available-state sentinel when recovery moves a region to HealthyTentative and cover it in the scheduled recovery test.

* Refine PPCB failback diagnostics

Keep failback attempt metadata partition-scoped, retain only the latest full failure message per region, and clear retained messages when recovery backlog drains.
@jeet1995
Prepare azure-cosmos 4.71.3-hotfix backport
15e4f9b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 16ce0941-555c-4190-8c4d-96c705086350
@jeet1995
Backport PPCB diagnostics client config fix (Azure#49734)
a6bf7a9
@azure-pipelines

azure-pipelines Bot commented Aug 28, 2026

Copy link
Copy Markdown
Azure Pipelines:
35 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@jeet1995

Abhijeet Mohanty (jeet1995) commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Aug 28, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jeet1995

Abhijeet Mohanty (jeet1995) commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - ci

@azure-pipelines

azure-pipelines Bot commented Aug 28, 2026

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

@jeet1995
Backport customer workflow tests from PR Azure#49568
7bde221 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 16ce0941-555c-4190-8c4d-96c705086350
@jeet1995
Use authenticated Maven mirror in pipelines
7cbaebb 
Route CI and live-test Maven dependency resolution through the authenticated Azure SDK feed so network-isolated agents do not access Maven Central directly.
@jeet1995

Abhijeet Mohanty (jeet1995) commented Aug 29, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Aug 29, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jeet1995
Authenticate Maven in remaining pipeline jobs
bd8653e 
Replace all remaining reporting-tools warmup calls with the authenticated Azure Artifacts Maven setup so version, native, validation, aggregate, and patch pipeline jobs work under network isolation.
@jeet1995

Abhijeet Mohanty (jeet1995) commented Aug 29, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Aug 29, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jeet1995
Remove obsolete reporting setup from Cosmos pipelines
93ecd40 
Rely on shared authenticated Maven setup in isolated test jobs instead of injecting the network-isolation-incompatible reporting-tools warmup from Cosmos test and Kafka pipelines.
@jeet1995

Abhijeet Mohanty (jeet1995) commented Aug 29, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Aug 29, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jeet1995
Run LatestCommitted workflow in Direct mode
07a367b 
Exclude the unsupported Gateway factory case while retaining Direct-mode LatestCommitted customer workflow coverage.
@jeet1995

Abhijeet Mohanty (jeet1995) commented Aug 29, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Aug 29, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jeet1995
Fix hotfix test and Kafka pipeline compatibility
1621e84 
Restrict unsupported customer workflow modes, initialize PPCB fixtures for the thin-client group, route Confluent dependencies through authenticated CFS, and authenticate Testcontainers image pulls through ACR.
@jeet1995
Route Spring Maven repository through CFS
b5f237d 
Backport PR Azure#49523's mirror correction so Spring Milestone cannot bypass the authenticated Azure Artifacts mirror on isolated agents.
@jeet1995
Retry reads until one region is contacted
13aa55a
@jeet1995
Remove Confluent serializer dependency from Kafka tests
bbca4c6
@jeet1995
Abhijeet Mohanty (jeet1995) deleted the release_azure-cosmos-4.71.3-hotfix branch August 31, 2026 13:16
@jeet1995

Abhijeet Mohanty (jeet1995) commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Aug 31, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jeet1995
Abhijeet Mohanty (jeet1995) marked this pull request as ready for review September 1, 2026 15:18
@azure-pipelines

azure-pipelines Bot commented Sep 1, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
32 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@jeet1995

Abhijeet Mohanty (jeet1995) commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Sep 1, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jeet1995

Abhijeet Mohanty (jeet1995) commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Sep 1, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joshfree Josh Free (joshfree) Awaiting requested review from joshfree joshfree is a code owner
@JonathanGiles Jonathan Giles (JonathanGiles) Awaiting requested review from JonathanGiles JonathanGiles is a code owner
@alzimmermsft Alan Zimmer (alzimmermsft) Awaiting requested review from alzimmermsft alzimmermsft is a code owner
@samvaity Sameeksha Vaity (samvaity) Awaiting requested review from samvaity samvaity is a code owner
@g2vinay Vinay Gera (g2vinay) Awaiting requested review from g2vinay g2vinay is a code owner
@rujche Rujun Chen (rujche) Awaiting requested review from rujche rujche is a code owner
@saragluna Xiaolu Dai (saragluna) Awaiting requested review from saragluna saragluna is a code owner
@moarychan Moary Chen (moarychan) Awaiting requested review from moarychan moarychan is a code owner
@hallipr Patrick Hallisey (hallipr) Awaiting requested review from hallipr hallipr is a code owner
@weshaggard Wes Haggard (weshaggard) Awaiting requested review from weshaggard weshaggard is a code owner
@benbp Ben Broderick Phillips (benbp) Awaiting requested review from benbp benbp is a code owner
@kirankumarkolli Kiran Kumar Kolli (kirankumarkolli) Awaiting requested review from kirankumarkolli kirankumarkolli is a code owner
@srnagar Srikanta (srnagar) Awaiting requested review from srnagar srnagar is a code owner
@anuchandy Anu Thomas (anuchandy) Awaiting requested review from anuchandy anuchandy is a code owner
@conniey Connie Yau (conniey) Awaiting requested review from conniey conniey is a code owner

Assignees

No one assigned

Labels

Cosmos

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jeet1995 @xinlian12