Skip to content

p2p: libp2p resource manager with configured limits - #4639

Open
pinebit wants to merge 3 commits into
mainfrom
pinebit/libp2p-resource-manager
Open

p2p: libp2p resource manager with configured limits#4639
pinebit wants to merge 3 commits into
mainfrom
pinebit/libp2p-resource-manager

Conversation

@pinebit

@pinebit pinebit commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Replace libp2p's NullResourceManager, which never rejects anything and leaves nodes open to stream, connection and memory exhaustion, with configured resource managers.

Validator nodes use autoscaled system limits with fixed per-peer limits: defaults for relays and unknown peers, elevated but bounded limits for authenticated cluster peers, raised protocol-scope defaults for charon protocols, and per-IP connection limits that allow cluster peers to share an IP (NAT or local clusters). This path is gated behind the new alpha feature flag libp2p_resource_manager and falls back to the null manager when disabled.

Relays derive limits from their existing connection config (p2p-max-connections, p2p-max-reservations) unconditionally: system connection, FD and stream limits track the configured capacity, transient limits match system limits to survive reconnect storms, and the circuit relay hop/stop protocol and relay service scopes are sized to full circuit load instead of go-libp2p's memory-scaled defaults, which would otherwise cap circuits far below the configured capacity.

Both managers replace go-libp2p's default per-IP connection rate limiter (burst of 16 connections per IP regardless of configured limits) with one sized to the per-IP connection limits, so reconnect bursts from peers behind a shared NAT IP are not throttled.

category: feature
ticket: none

Replace NullResourceManager (which never rejects anything, leaving nodes
open to stream, connection and memory exhaustion) with configured
resource managers. Validator nodes get autoscaled system limits with
fixed per-peer limits, elevated for authenticated cluster peers, gated
behind the alpha libp2p_resource_manager feature flag. Relays derive
limits from their connection config unconditionally, including circuit
relay protocol and service scopes and a per-IP connection rate limiter
sized to the per-IP connection limits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pinebit
pinebit requested a lite review from Copilot August 10, 2026 19:02
@pinebit pinebit changed the title p2p: replace null libp2p resource manager with configured limits p2p: libp2p resource manager with configured limits Aug 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces configured libp2p resource managers (instead of network.NullResourceManager) to enforce connection/stream/memory limits and reduce DoS/resource-exhaustion risk. It adds a feature flag for validator nodes while applying capacity-derived limits for relay nodes.

Changes:

  • Add validator and relay resource manager implementations with per-peer/per-protocol/per-subnet limits and a custom per-IP connection rate limiter.
  • Gate validator resource manager behind the new alpha feature flag libp2p_resource_manager, falling back to the null manager when disabled.
  • Update relay startup to use the configured resource manager and add internal tests validating key limit behaviors.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
p2p/resourcemanager.go Implements configured libp2p resource managers and limit/rate-limiter helpers.
p2p/resourcemanager_internal_test.go Adds tests asserting per-peer stream limits, relay conn limits, and burst behavior.
cmd/relay/p2p.go Switches relay node startup from null to configured resource manager.
app/featureset/featureset.go Adds libp2p_resource_manager feature flag definition and registration.
app/app.go Gates validator node resource manager usage behind the new feature flag.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread app/app.go Outdated
Comment thread cmd/relay/p2p.go
p2p.NewNode can fail before libp2p takes ownership of the resource
manager, leaking its background goroutines. Close it on the error path;
Close is idempotent so paths where libp2p already closed it are safe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pinebit
pinebit requested a review from KaloyanTanev August 10, 2026 19:13
Set explicit system and transient stream and connection limits instead of
inheriting go-libp2p's autoscaled defaults. Those derive from an eighth of
host memory and fall below the fixed per-peer limits on typical validator
hardware: 3072 system inbound streams and 63 transient inbound connections
on a 16GB host, against a 4096 stream cluster peer allowance and a 64
connection per-IP burst. The per-peer limits were therefore unreachable and
a single peer could exhaust the system stream budget. Memory and FD limits
stay host derived, since those track resources the host actually has.

This fixes TestNewResourceManagerPeerStreamLimits and
TestConnRateLimiterAllowsPerIPBurst, which passed only on hosts with enough
memory for the autoscaled ceilings to clear the fixed limits. The limit
config moves into clusterLimitConfig, mirroring relayLimitConfig, and
TestClusterLimitConfig asserts the limits no longer track host memory.

streamLimits takes separate inbound and outbound values so the protocol
scopes track each direction's system and peer limit rather than deriving
both from the inbound one. Built limits are unchanged by that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.06250% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.31%. Comparing base (aea5631) to head (59f9b82).

Files with missing lines Patch % Lines
app/app.go 0.00% 7 Missing ⚠️
p2p/resourcemanager.go 96.55% 2 Missing and 2 partials ⚠️
cmd/relay/p2p.go 40.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4639      +/-   ##
==========================================
+ Coverage   58.16%   58.31%   +0.15%     
==========================================
  Files         247      248       +1     
  Lines       34056    34180     +124     
==========================================
+ Hits        19807    19933     +126     
+ Misses      11779    11773       -6     
- Partials     2470     2474       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@KaloyanTanev

Copy link
Copy Markdown
Collaborator

I'd be cautious of including that, without gathering enough data on our side for amount of opened streams nodes have. That is not only for the happy case where we observe a properly functioning cluster running X amount of validators having Y amount of streams at most, but also in the unhappy case, when cluster connections drop, they reconnect. Also when QUIC is enabled.

Until we have that I'm afraid this might cause more harm than good.

@pinebit

pinebit commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Until we have that I'm afraid this might cause more harm than good.

Agreed on the data gap, and that's why the node path ships default off behind the alpha flag - the numbers below (agent verified) explain why the chosen limits have 10–100× headroom in both happy and reconnect cases, QUIC included, and rcmgr metrics during the alpha can confirm them empirically before promotion.

- One stream per message, closed immediately. Every send in p2p/sender.go opens a stream, writes, reads the response, and closes it via defer s.Close(), with a deadline set to the send timeout. So concurrent streams per peer = messages in flight, and no stream outlives its timeout. There is no long-lived stream accumulation to measure.
- Happy case scale: even a 20k-validator cluster runs a handful of concurrent duties per slot (attester, aggregator, proposer, sync), each producing a few in-flight consensus/parsigex messages per peer — low hundreds of concurrent streams per peer at the extreme, against a 4096 per-cluster-peer limit and 16384 system limit (deliberately fixed above go-libp2p's autoscaled ~3072-on-16GB, so limits don't silently depend on host memory).
- Unhappy case (drop + reconnect): streams die with their connection, so reconnect storms pressure the connection scopes, not stream scopes — and that's exactly what the PR sized for: 32 conns per cluster peer, 64 per IP (whole cluster behind one NAT), transient = 128 to absorb a full-cluster burst, and a custom per-IP rate limiter replacing go-libp2p's default burst-of-16, which would otherwise throttle a NAT'd cluster's reconnect regardless of limits.
- QUIC: the resource manager counts streams transport-agnostically — a native QUIC stream and a yamux-over-TCP stream cost the same. The differences cut in QUIC's favor (no per-connection FD). The one thing QUIC changes is dual-transport peers holding both a TCP and a QUIC connection, which 32 conns/peer covers comfortably.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants