Skip to content

feat: add TcpStackAwareSocketChannel for z/OS TCP/IP stack recovery — GH#4776 - #4792

Open
balhar-jakub wants to merge 6 commits into
v3.x.xfrom
hermes/gh4776
Open

feat: add TcpStackAwareSocketChannel for z/OS TCP/IP stack recovery — GH#4776#4792
balhar-jakub wants to merge 6 commits into
v3.x.xfrom
hermes/gh4776

Conversation

@balhar-jakub

@balhar-jakub balhar-jakub commented Jul 9, 2026

Copy link
Copy Markdown
Member

Closes #4776

Problem

The z/OS TCP/IP stack can restart (EDC5122I / NetworkRecycledException) while Tomcat is reading from or writing to an accepted client socket. Tomcat must discard the dead connection safely.

Fix

  • Refactored isTcpStackRestarted and isRecycledClass from instance methods to package-private static methods on TomcatAcceptFixConfig
  • Added the opt-in apiml.tcpStackAwareSocketChannel.enabled configuration property, defaulting to false so non-z/OS deployments are unaffected
  • Created ExcludedSocketOps interface for Lombok @Delegate exclusion list
  • Created TcpStackAwareSocketChannel inner class:
    • Intercepts read(ByteBuffer), read(ByteBuffer[], int, int), write(ByteBuffer), write(ByteBuffer[], int, int)
    • Detects EDC5122I/NetworkRecycledException, safely closes the dead socket, and re-throws so Tomcat discards the connection
    • Delegates wrapper close and blocking-mode behavior safely to the accepted socket
  • Added unit coverage for read/write variants, socket closure, configuration disabled/enabled behavior, and z/OS restart detection

Configuration

This is a z/OS-only mitigation and is disabled by default. Enable it only for affected z/OS deployments after staging validation:

apiml:
  tcpStackAwareSocketChannel:
    enabled: true

Validation

  • ./gradlew :apiml-tomcat-common:clean :apiml-tomcat-common:build — BUILD SUCCESSFUL (77 tests)
  • ./gradlew clean && ./gradlew build — BUILD SUCCESSFUL

@balhar-jakub

Copy link
Copy Markdown
Member Author

QA + Security Review — PR #4792 (#4776)

Verdict: APPROVED

Build and Tests

  • ./gradlew :apiml-tomcat-common:clean build — BUILD SUCCESSFUL, 76/76 tests pass
  • 16 new tests added covering all read/write/safeClose/config scenarios

Pavel's Lens — All 8 Rules Checked

Rule Status Notes
1. Config Consistency OK apiml.tcpStackAwareSocketChannel.enabled has Spring default :true. Follows existing undocumented pattern (same as server.tomcat.retryRebindTimeoutSecs).
2. Deduplication OK isTcpStackRestarted/isRecycledClass properly extracted to static methods. Old instance method delegates. Minor: NETWORK_RECYCLED_EXCEPTION_CLASS constant on line 176 is now dead code — not used after refactor. Suggest removing in follow-up.
3. Null Safety OK isTcpStackRestarted checks getMessage() != null, getCause() != null, cause != t (cycle guard).
4. Test Parametrization OK Tests are varied enough (read/write, single/scatter, pass/fail, config on/off, integration) to not collapse cleanly into parameterized. 16 distinct test cases with good edge coverage.
5. Security Boundaries OK No auth/TLS/CORS/input validation changes. This is z/OS socket infrastructure. Config toggle safely disables. No secrets, no data exposure. safeClose is best-effort with catch-and-ignore.
6. z/OS Awareness EXCELLENT Purpose-built for z/OS TCP/IP stack recovery. Handles EDC5122I and com.ibm.net.NetworkRecycledException. Graceful degradation via config toggle.
7. Log Quality OK All log.debug with exception context. Messages identify operation type (read/scatter read/write/scatter write). safeClose silently swallows close errors — correct for best-effort.
8. TODO Tracking OK No TODO/FIXME/HACK comments found.

Minor Notes (non-blocking)

  1. NETWORK_RECYCLED_EXCEPTION_CLASS constant (line 176) is dead code after the static refactor — can be cleaned up in a follow-up.
  2. Instance isTcpStackRestarted wrapper (lines 277-279) delegates to static version — may still be referenced by pre-existing tests calling channel.isTcpStackRestarted(). Acceptable.

Security Assessment

  • No new auth/authz boundaries introduced
  • No input validation changes
  • No secrets or credentials in code
  • No data exposure via logs (debug level, exception context only)
  • MethodHandle usage for implCloseSelectableChannel/implConfigureBlocking follows existing pattern in FixedServerSocketChannel
  • @Delegate with ExcludedSocketOps correctly excludes final methods and manually overridden methods

APPROVED — ready for merge.

@balhar-jakub

Copy link
Copy Markdown
Member Author

DCO check failed — all 3 commits are missing Signed-off-by lines.

Fix: git rebase --signoff v3.x.x && git push --force-with-lease

@balhar-jakub
balhar-jakub marked this pull request as draft July 9, 2026 12:09
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
78.6% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

💡 Need a hand with PR review? Try Gitar by Sonar!

@balhar-jakub balhar-jakub moved this from New to In Progress in API Mediation Layer Backlog Management Jul 15, 2026
@balhar-jakub
balhar-jakub marked this pull request as ready for review August 20, 2026 08:18

@balhar-jakub balhar-jakub left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-Reviewer Review

Verdict: Approve with minor NITs.

Findings:

  • ℹ️ NIT — Typo in IMPL_CLOSE_SELECTABGLE_CHANNEL_HANLE (should be HANDLE) — pre-existing, but inherited.
  • ℹ️ NIT — Method counts in ExcludedSocketOps comment are slightly off (14/6 split is approximate).
  • ℹ️ NIT — close() is excluded but not overridden — defense-in-depth via implCloseSelectableChannel() works.
  • ℹ️ NIT — No toString() override.
  • ℹ️ NIT — safeClose swallows IOException silently; consider trace-level log.
  • ✅ Test coverage is excellent (every read/write variant tested, mockStatic for NetworkRecycledException).

Good:

  • Feature flag apiml.tcpStackAwareSocketChannel.enabled defaults to false — safe opt-in.
  • @Delegate approach correctly handles 20 excluded methods + delegates the rest.
  • isTcpStackRestarted now static (was instance) — cleaner since no instance state needed.
  • New tests cover both the existing isTcpStackRestarted and the new TcpStackAwareSocketChannel.

Optional follow-up:

  • Update apiml-tomcat-common CHANGELOG / docs to mention the new flag.
  • Consider a tenant-level integration test that toggles the flag and verifies behavior.

Required before merge: none. All findings are NIT-level polish.

balhar-jakub and others added 6 commits August 24, 2026 14:03
…StackAwareSocketChannel config (#4776)

Move isRecycledClass and isTcpStackRestarted from FixedServerSocketChannel
instance methods to package-private static methods on TomcatAcceptFixConfig.
Add apiml.tcpStackAwareSocketChannel.enabled config property (default true).
Update call sites in FixedServerSocketChannel.accept() to use static methods.

Signed-off-by: Jakub Balhar <jakub.balhar@broadcom.com>
…#4776)

Create ExcludedSocketOps interface (20 methods) and TcpStackAwareSocketChannel
inner class extending SocketChannel with @DeleGate. Intercepts read/write ops
to detect EDC5122I/NetworkRecycledException, closes dead socket, re-throws.
Wire into FixedServerSocketChannel.accept() guarded by config property
apiml.tcpStackAwareSocketChannel.enabled (default true).

Signed-off-by: Jakub Balhar <jakub.balhar@broadcom.com>
Add comprehensive tests for TcpStackAwareSocketChannel wrapper:
- Normal read/write passthrough
- EDC5122I detection on read/write/scatter/gather (close + rethrow)
- NetworkRecycledException detection via MockedStatic
- Non-EDC5122I IOException passthrough (no close)
- safeClose robustness (IOException on close ignored)
- configureBlocking/isBlocking delegation
- Integration test with real SocketChannel
- Config disabled returns raw SocketChannel

Update TcpStackRestartHandling tests to use static methods.
All 76 tests pass.

Signed-off-by: Jakub Balhar <jakub.balhar@broadcom.com>
Change default from true to false. The wrapper breaks Tomcat socket
acceptance on non-z/OS platforms (Linux CI runners), causing tests
to hang with Connection refused until BuildAndTest times out at 35
minutes. The feature only has value on z/OS where TCP/IP stack
restarts can occur — z/OS deployments must explicitly set
apiml.tcpStackAwareSocketChannel.enabled=true.

Signed-off-by: Jakub Balhar <jakub.balhar@broadcom.com>
Signed-off-by: Jakub Balhar <jakub.balhar@broadcom.com>
Signed-off-by: Jakub Balhar <jakub.balhar@broadcom.com>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

z/OS TCP/IP stack recovery only patches accept() socket, not existing client connections

1 participant