fix(market-adapter): keep Kibana LP fetches alive under proxy connect… - #12
Open
Impulssi wants to merge 1 commit into
Open
fix(market-adapter): keep Kibana LP fetches alive under proxy connect…#12Impulssi wants to merge 1 commit into
Impulssi wants to merge 1 commit into
Conversation
Impulssi
force-pushed
the
fix/kibana-lp-fetch-stability
branch
from
August 26, 2026 06:03
f8aca53 to
f4d779c
Compare
…ion resets The Kibana console proxy at kibana.bitshares.dev now resets connections mid-transfer once a single search page streams enough data (observed with full _source payloads around ~8k documents, and even ~2k documents with _source: true). The LP fetcher queried with size 10000 and _source: true, so every fetch died at the availability probe — and worse, silently: - doKibanaRequest only listened for request-phase errors; a response stream aborted mid-body never emits 'end', so the surrounding promise stayed pending forever and fetch_lp_data hung with no error output. - There was no retry, so a single transient reset killed the whole fetch. ## Fix - kibana_client: handle 'aborted'/'error' on the response stream and reject the promise, so callers fail loudly instead of hanging forever. - kibana_candles: request only the _source branches the field map actually reads (plus the fixed timestamp/sequence fields) instead of full documents. Restricted projections keep each page small and fast (verified live: 2000-doc pages with restricted _source return in ~1.5s where full _source aborts). - kibana_candles: default kibanaPageSize 10000 -> 2000. - kibana_candles: retry transient page failures (aborted / connection reset / timeout) with linear backoff — search_after pagination is stateless on the server, so replaying a failed page is safe. - buildDirectionalDocumentQuery accepts an optional sourceFields argument; omitting it keeps the legacy _source: true behavior for other callers. ## Testing Notes - tests/test_kibana_candles.ts: new coverage for the source projection derivation, the _source wiring in the directional query (plus legacy fallback), and transient-page retry recovery. - npm run typecheck clean. - Verified live against kibana.bitshares.dev with fetch_lp_data.js (BTS/XBTSX.USDT pool 1.19.48, 2-month 1h fetch): the availability probe and both chunk fetches complete where they previously hung indefinitely.
Impulssi
force-pushed
the
fix/kibana-lp-fetch-stability
branch
from
August 26, 2026 06:34
f4d779c to
e1ccc05
Compare
froooze
added a commit
that referenced
this pull request
Aug 26, 2026
…ion resets The kibana.bitshares.dev console proxy resets connections mid-transfer when a single search page streams enough data (observed around ~8k documents with full _source, and even ~2k with _source: true). The LP fetcher queried size: 10000 with _source: true, so every fetch died mid-page and silently: doKibanaRequest only listened for request-phase errors, so a response stream aborted mid-body never settled the promise and fetch_lp_data hung forever with no error output. No retry existed either, so one transient reset killed the whole fetch. ## Response-stream settlement - Problem: aborted/error during the response body left the surrounding promise pending forever - Impact: availability probes and LP chunk fetches hung indefinitely without any log output - Solution: market_adapter/core/kibana_client.ts now handles 'aborted'/'error' on the response stream behind a settle guard, rejecting with an actionable message instead of hanging ## Restricted _source projection - Problem: full operation documents are heavy and trip the proxy's per-response limit - Impact: large pages were guaranteed to be killed on busy pools - Solution: market_adapter/core/kibana_candles.ts derives a minimal _source projection from the field map (sourceFieldsForFieldMap) plus fixed timestamp/sequence fields; buildDirectionalDocumentQuery keeps legacy _source: true behavior when no projection is passed ## Smaller pages and transient-failure retries - Problem: oversized pages plus zero retries made every fetch a single-shot failure - Impact: transient proxy resets aborted whole lookbacks - Solution: kibanaPageSize default lowered 10000 -> 2000, transient page failures (aborted/reset/socket/timeout) retried up to kibanaPageRetries total attempts (4 here; upstream PR used 3) with linear backoff; safe because search_after pagination is stateless server-side Adapted from upstream contribution bitshares#4 / #12, raised to 4 total attempts per page and adjusted for the newer request-phase settle guard already present in this tree. ## Risk Notes - Retry classification matches error strings; non-transient errors rethrow immediately unchanged - Projection branches capped at depth 3 return whole subtrees from ES, a superset of leaf reads, so no data loss ## Testing Notes - npm run typecheck - npm run build && node dist/tests/test_kibana_candles.js (new coverage: source projection derivation, _source wiring + legacy fallback, retry recovery after 3 failures within the 4-attempt budget)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ion resets
The Kibana console proxy at kibana.bitshares.dev now resets connections mid-transfer once a single search page streams enough data (observed with full _source payloads around ~8k documents, and even ~2k documents with _source: true). The LP fetcher queried with size 10000 and _source: true, so every fetch died at the availability probe — and worse, silently:
Fix
Testing Notes
Note: I'm new to this codebase and worked through this change with an AI
coding assistant (opencode). The bug was hit on my own live instance and
all measurements / reproduction steps are mine — but please flag anything
in the code that looks off and I'll dig in and follow up.