New RPCConnectionManager: Single unified manager, no ringbuffer, OpenSSL-owned sockets

[Eddy Ashton (eddyashton)]

Summary

This PR replaces the old split RPC networking path with an OpenSSL-native connection layer. Previously, socket handling lived in host-side libuv code while TLS and protocol sessions were driven through enclave-side ringbuffer messages and memory BIOs. With CCF now running as a single process, that split is no longer useful, so RPC sockets, TLS, protocol session creation, and per-interface policy now live behind a single RPC connection manager.

TLS now terminates at the connection layer. Protocol sessions receive plaintext and write responses through a SessionWriter, so HTTP, HTTP/2, and custom protocols no longer own TLS state directly. This removes the RPC ringbuffer message path, the memory-BIO TLS session layer, and the old libuv RPC connection containers.

The new transport uses non-blocking sockets bound directly to OpenSSL, and splits the work in two. The existing host libuv loop owns the server's own state - accepting connections, uv_poll_t registration, the SSL_CTX, idle connection cleanup via uv_timer_t, and closing file descriptors - and performs no SSL operations itself. Every SSL operation for a connection instead runs on that connection's own OrderedTasks queue, keeping handshakes and bulk encryption off the loop thread. The loop only schedules a pass over a connection, driven either by file descriptor readiness or by a cross-thread request (a queued write, a close, or a certificate update) marshalled through uv_async_t. A connection is serviced by at most one pass at a time. Per-interface behavior such as certificates, session caps, metrics, HTTP parser settings, and custom protocol dispatch is centralized in RPCConnectionManager.

Session caps are applied when a connection is accepted, before any TLS state exists, rather than when its first request arrives. max_open_sessions_hard is documented as a bound on connections, and counting at first-request time missed any client that completed the TCP and TLS handshakes and then went silent while still holding a file descriptor and TLS state.

Node outbound requests are outside this transport and use libcurl.

UDP remains as a small datagram transport driven by uv_poll_t. The temporary QUIC/UDP echo behavior is stateless, so it consumes no session and no interface capacity, while custom UDP protocols are routed to per-peer sessions. Native QUIC is still future work and requires OpenSSL 3.5 or later.

Structural Breakdown

OpenSSLServer is the low-level inbound connection transport. It owns the listening and accepted socket file descriptors, SSL objects, uv_poll_t handles, read/write buffers, handshake state, graceful-close state, certificate reload requests, and idle-timeout sweeping.

OpenSSLSessionManager bridges transport connections to ccf::Session. It lazily creates sessions for inbound connections, forwards plaintext bytes into sessions, implements SessionWriter, and reports connection closure back to the owner.

RPCConnectionManager is the higher-level RPC owner. It replaces the old RPC session container and owns one transport bridge per TCP interface, plus UDP interface state. It applies per-interface admission and caps, certificates, parser settings, application protocol selection, session metrics, custom protocol routing, and UDP peer demultiplexing.

SessionWriter, Session, and PlaintextSession form the new session boundary. Sessions no longer encrypt or decrypt; they parse plaintext and emit plaintext responses to their writer. HTTP/1 and HTTP/2 sessions now use this boundary.

CustomProtocolSubsystemInterface now creates sessions from (ConnID, SessionWriter&) rather than a TLS context. This matches the new layering: custom protocols see plaintext and write through the transport-neutral writer.

DatagramServer is the UDP socket transport. It uses a uv_poll_t handle on the existing libuv loop. RPCConnectionManager echoes datagrams directly for the temporary QUIC behavior, and maps UDP peers to sessions for custom datagram protocols.

Startup wiring moved accordingly. The enclave creates and owns the RPC manager, binds RPC interfaces, resolves actual bound addresses including ephemeral ports, and reports those addresses back through the enclave entry point so the host can write the RPC addresses file.

The removed files are the old RPC transport stack: RPCSessions, TLSSession, host RPC connections, legacy UDP plumbing, the old QUIC session, and the TCP/UDP ringbuffer message types that were specific to the split RPC path. Ledger, consensus, and node-to-node uses of ringbuffer and libuv are not part of this change.

[Amaury Chamayou (achamayou)]

Had an agent look at why performance seems affected, particularly on the blocking commit benchmark. I found the Nagle regression right away, which seemed worth fixing. The rest is more debatable, some of the locking changes sound like easy wins, but need testing, but the big one is probably not having handshake handling delaying permanent regime requests for other sessions, and that requires non trivial re-design I think.

Executive summary

PR 8117 consistently regresses the Basic Blocking benchmark by roughly 26-28% against the recent main EWMA. The four observed branch results are 725.5, 792.4, 731.1, and 744.0 tx/s. The latest result is 744.0 tx/s, 26% below a main EWMA of approximately 1,010 tx/s.

The missing TCP_NODELAY setting was a real regression from the old TCP transport and has been restored in commit e2d94caa3. The old transport called uv_tcp_nodelay(..., 1) for every socket, while the new accept4() path initially left Nagle enabled. Runtime tracing confirms that accepted sockets now receive setsockopt(..., TCP_NODELAY, 1). However, the post-fix CI result improved only from 731.1 to 744.0 tx/s and remains 26% below main. Nagle was therefore not the principal cause.

The most likely remaining cause is the new architecture's concentration of TLS and socket work on one libuv loop, amplified by the benchmark's short, highly concurrent, latency-sensitive shape. PR 8117 moves SSL_read() and SSL_write() from per-session worker processing to the shared default libuv loop. Every response also crosses a mutexed queue and an async wake before that loop performs TLS encryption. With 128 clients each allowing exactly one request in flight, added response delay translates directly into lower throughput; pipelined Basic can hide the same latency.

Benchmark shape

pi_basic_blocking is configured in CMakeLists.txt as:

128 clients
100 blocking writes per client
primary target
max-writes-ahead = 0

The endpoint waits for consensus commitment before responding. Each client sends its next request only after receiving the previous response. Aggregate throughput is therefore approximately:

active clients / average request-response latency

At the main baseline, 128 / 1,010 is about 127 ms per request. At 744 tx/s it is about 172 ms per request, an added delay of roughly 45 ms in the fully-overlapped idealisation.

This benchmark also launches 128 client processes sequentially and exports throughput over the interval from the earliest send to the latest receive. Since each client sends only 100 requests, connection establishment and launch skew are a significant part of the measured interval. A slower 128-way TLS handshake ramp can reduce the reported throughput even if steady-state request handling is unchanged.

Evidence

CI A/B results

Run	Basic Blocking throughput	Relative to current main EWMA
PR run 1	725.5 tx/s	-28%
PR run 2	792.4 tx/s	-22%
PR run 3	731.1 tx/s	-28%
After TCP_NODELAY	744.0 tx/s	-26%

The regression is present in every PR run and survives the TCP_NODELAY fix. Other benchmarks do not move consistently with it: pipelined Basic was above baseline in the first three runs, while commit-latency microbenchmarks remained close to baseline. This points away from KV execution and consensus as the primary source.

Local measurements

The local 128-client benchmark is noisy because process startup is serialized and this development host is shared:

Variant	Full-run throughput	Client start spread	All-client overlap	All-active throughput
TCP_NODELAY, run 1	471 tx/s	about 16.5 s	3.7 s	1,039 tx/s
Nagle enabled	504 tx/s	14.9 s	none	invalid
TCP_NODELAY, run 2	622 tx/s	9.9 s	10.3 s	1,180 tx/s

The fixed full-run results vary too much to claim a local throughput win. The all-active fixed rates are nevertheless near or above the main CI baseline, while much of the full-run interval is launch ramp rather than steady load.

A stable 16-client variant with 1,000 requests per client produced 99% overlap and nearly identical results:

Variant	Throughput	p50 latency	p90 latency	p99 latency
TCP_NODELAY	159.50 tx/s	100.124 ms	102.784 ms	105.760 ms
Nagle enabled	159.34 tx/s	100.109 ms	102.830 ms	106.397 ms

At 16 clients the 100 ms signature cadence dominates, so this test does not exercise the high-concurrency bottleneck. It does show that Nagle is not a general per-request 25% cost.

CPU profiling was not available in the dev container: perf is not installed and /proc/sys/kernel/perf_event_paranoid is 4.

Hot-path comparison

Old path

The legacy design performed TLS work in each TLSSession, reached from the session's ordered worker task. Encrypted bytes then crossed the ringbuffer and were written by libuv. TCP sockets were configured with both TCP_NODELAY and keepalive.

New path

The new response path is:

HTTP response vector
  -> ThreadedSession::SendDataTask
  -> PlaintextSession::send_data_thread
  -> SessionWriter::write_outbound(span)
  -> OpenSSLServer::send
  -> pending_out under out_mutex
  -> uv_async_send
  -> one libuv loop drains pending_out
  -> copy into Conn::outbuf
  -> SSL_write on the loop thread
  -> socket send

All inbound handshakes, SSL_read() calls, SSL_write() calls, connection-map operations, and poll-interest updates for all RPC interfaces run on uv_default_loop(). This creates a single serialization point that did not exist when TLS processing happened in ordered worker tasks.

Likely contributors

1. Single-loop TLS encryption and handshake serialization

Confidence: high as an architectural bottleneck; medium as the full explanation for the measured 26%.

OpenSSLServer performs every handshake and TLS record operation on the shared libuv loop. Basic Blocking creates 128 TLS connections and waits synchronously for every small response. A burst of committed callbacks queues many responses, but one loop encrypts and writes them serially. Initial handshakes are also serialized on this loop, potentially widening the benchmark's client start ramp.

Recommended measurement:

• Run the benchmark with CPU sampling enabled on the CI benchmark host.
• Separate samples by the basic server process and inspect SSL_write, SSL_read, SSL_accept, drain_pending_out, uv__io_poll, and task-system frames.
• Record handshake completion timestamps and first-request timestamps for all 128 connections.
• Compare full-run throughput with all-clients-active throughput.

Potential improvement:

• Give RPC transport work dedicated loop threads, or shard connections across several loops.
• At minimum, separate expensive handshake processing from steady-state response encryption if OpenSSL object ownership can remain safe.
• This is a larger design change and should follow profiling rather than be attempted speculatively.

2. Lifecycle mutex on every socket event and response wake

Confidence: medium.

on_connection_poll() calls mark_loop_thread(), which locks lifecycle_mutex on every poll callback and rewrites the same thread ID. send() calls wake(), which takes the same mutex for every response. Under 128 active connections, the loop and worker threads repeatedly contend on a mutex whose steady-state information rarely changes.

Potential improvement:

• Set the loop thread ID once, rather than on every callback.
• Make the steady-state stop/initialisation flags atomic or otherwise arrange lock-free reads in wake().
• Keep the mutex for startup/shutdown transitions only.

This is relatively contained and should be measurable with mutex-contention profiling or a counter around failed/immediate lock acquisition.

3. Two avoidable response copies

Confidence: high that the copies exist; low-to-medium impact for tiny Basic responses.

ThreadedSession already owns the response as std::vector<uint8_t>, but SessionWriter::write_outbound accepts only a span. OpenSSLServer::send copies that span into a new OutItem::data, and drain_pending_out copies it again into Conn::outbuf before SSL_write.

Potential improvement:

• Add an ownership-taking writer API such as write_outbound(ConnID, std::vector<uint8_t>&&).
• Move the vector into OutItem.
• When Conn::outbuf is empty, move/swap OutItem::data into it rather than inserting.
• Preserve the span overload for callers that cannot transfer ownership.

This should help larger responses and high-throughput workloads even if it is not the main Basic Blocking regression.

4. Unconditional wake and queue synchronization

Confidence: medium-low, because libuv already coalesces async notifications.

Every send() locks out_mutex, appends to pending_out, then calls uv_async_send. Libuv coalesces pending async callbacks, but the application still executes the wake path and lifecycle lock for every response.

Potential improvement:

• Wake only when the queue transitions from empty to non-empty.
• Drain until no items remain, with a carefully designed pending flag to avoid missed wakeups.
• Reserve or use a queue structure that avoids repeated vector growth under bursts.

5. Repeated poll re-arming

Confidence: medium-low until syscall counts are collected.

finish_or_close() calls update_interest() after every successful response, and update_interest() calls uv_poll_start() even when the desired mask remains UV_READABLE. Cache the current event mask in Conn and re-arm only when UV_WRITABLE interest actually changes. Validate the benefit by counting epoll_ctl/poll-update syscalls before and after.

6. Session-map mutex on every inbound chunk

Confidence: low-to-medium.

OpenSSLSessionManager::on_data() takes sessions_mutex for every decrypted chunk, including every request on an established persistent connection. The loop is the sole inbound caller, while close operations may arrive from workers.

Potential improvement:

• Associate the session directly with connection state after first creation, or separate loop-owned lookup from cross-thread close coordination.
• Avoid changing this without race-focused tests; the current lock is simple and correct.

Benchmark improvements

The current benchmark metric conflates connection startup and steady-state request throughput. Both are useful, but they should be reported separately.

Recommended changes:

1. Add a start barrier so all submitters connect and wait before requests begin.
2. Export all_clients_active_average_throughput_tx/s to Bencher alongside the existing full-run throughput.
3. Increase requests per client so the steady window dominates client process and TLS setup.
4. Add explicit connection-ramp metrics: time from first to last handshake and first request.
5. Retain the existing metric under a name such as Basic Blocking including connection ramp if startup performance is intentional.

These changes would show whether PR 8117 regresses TLS connection establishment, steady blocking response latency, or both.

Recommended next actions

1. Keep TCP_NODELAY commit e2d94caa3; it restores an explicit legacy socket policy even though it does not recover the benchmark.
2. Re-run or extend CI with handshake timestamps and all-active throughput.
3. Profile the server process on the benchmark runner, focusing on the shared libuv TLS loop and lifecycle mutex.
4. Prototype the low-risk mutex cleanup and ownership-taking response path independently, benchmarking each change.
5. Consider loop sharding only if profiling confirms SSL_accept/SSL_write serialization dominates.

Validation performed

• openssl_server_test: 17 test cases passed, 368 assertions passed, 1 skipped.
• C++ format checks passed for src/host/tls/openssl_server.h.
• strace captured successful setsockopt(..., TCP_NODELAY, 1) calls on accepted sockets.
• Post-fix CI benchmark completed successfully at 744.0 Basic Blocking tx/s.
• Multiple local Basic Blocking runs were collected as described above.

The TCP_NODELAY fix was committed and pushed as e2d94caa3. This report is intentionally not committed.