fix(network): close idle peer connections, back off on already connected, lower maxbackoff

[stevenvegt]

Problem

A node kept logging this every few seconds for the same four peers, and never got connected to them:

level=warning msg="Peer connection error" error="rpc error: code = Unknown desc = already connected"

Two things went wrong at once:

• The remote side (in our case an nginx in front of the node, but a half-open TCP connection does the same) kept a stream open for our node after our node had already dropped it. The remote therefore thought it was still connected to us and rejected every new connection. Nothing ever cleaned that up short of restarting the proxy.
• Our node treated that rejection as a clean disconnect and reconnected after 1 to 5 seconds, forever. The backoff never kicked in.

Changes

Idle timeout on connections (network.idletimeout, default 2m, 0 disables). Peers send gossip and diagnostics every 5 seconds on every stream, so a stream with no incoming message for two minutes is dead. The node now closes it and the regular reconnect logic takes over. Time spent handling a message (a large transaction list during sync, for example) does not count as idle. This works through reverse proxies, unlike gRPC keepalive pings, which proxies answer themselves.

Back off on "already connected". The rejection is now handled like an authentication failure: exponential backoff instead of a retry every few seconds. Fixing this exposed a race between the receive loop storing the peer's close status and the stream context cancelling the connection, which also affected the existing Unauthenticated path. The close status is now read only after the gRPC connection is closed and the receive loops have exited.

network.maxbackoff default lowered from 24h to 1h. The backoff is persisted across restarts and only reset when a peer's NutsComm address changes, so a peer that was unreachable for a few days could go unattempted for up to a day after it came back.

Tests

All new behavior was written test-first. New tests in connection_test.go and connection_manager_test.go:

• a client rejected with "already connected" calls Backoff() and not Reset()
• a silent stream is disconnected after the idle timeout
• a stream that keeps delivering messages stays connected
• a handler that blocks longer than the idle timeout does not trigger a disconnect
• a zero idle timeout disables the check
• TestDefaultConfig asserts the new maxbackoff and idletimeout defaults

go test -race ./network/... passes.

Operational notes

For deployments behind nginx, grpc_read_timeout and grpc_send_timeout at their 60s default (not raised to hours) plus so_keepalive on the listen directive let the proxy drop dead streams on its own as well.

[qltysh]

0 new issues

Tool	Category	Rule	Count

[qltysh]

Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (7)

Rating	File	% Diff	Uncovered Line #s
	network/cmd/cmd.go	100.0%
	network/network.go	100.0%
	network/config.go	100.0%
	network/transport/grpc/connection.go	95.5%	272-274
	network/transport/grpc/connection_list.go	100.0%
	network/transport/grpc/config.go	16.7%	112-116
	network/transport/grpc/connection_manager.go	100.0%
	Total	91.1%

🤖 Increase coverage with AI coding...

In the `fix/network-idle-timeout-and-already-connected-backoff` branch, add test coverage for this new code:

- `network/transport/grpc/config.go` -- Line 112-116
- `network/transport/grpc/connection.go` -- Line 272-274

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.