Skip to content

perf: reuse fused depth output buffer - #321

Open
Magnushst wants to merge 1 commit into
nkaz001:masterfrom
Magnushst:perf/reuse-fusion-output-buffer
Open

perf: reuse fused depth output buffer#321
Magnushst wants to merge 1 commit into
nkaz001:masterfrom
Magnushst:perf/reuse-fusion-output-buffer

Conversation

@Magnushst

Copy link
Copy Markdown

Summary

This removes the temporary Vec<Event> created for each accepted depth or snapshot event in the Python market-depth fusion path.

Append-style core methods now write directly into the persistent output vector. The existing vector-returning methods remain available as compatible wrappers.

Changes

  • Add update_bid_depth_into and update_ask_depth_into.
  • Reuse the binding's persistent fused-event vector.
  • Preserve add=false by restoring the previous output length after updating depth state.
  • Add deterministic differential and binding regression tests.

Correctness

A 20,000-event deterministic differential test compares emitted events and complete depth state after every update.

The sequence covers bid and ask updates, snapshots, deletions, stale timestamps, crossings and narrow and wide price ranges.

A binding test covers accepted output, stale events, add=false state updates and invalid sides.

All 260 benchmark pairs on each tested platform produced identical output counts and sequence checksums.

Performance

Representative one-million-event results:

Platform Baseline ns/event Candidate ns/event Improvement
Windows 11 58.100 32.124 44.69%
Debian under WSL2 41.911 30.116 28.37%

The platform results are independent and were not combined.

Median allocation calls fell from 980,071.5 to 31. Cumulative requested bytes fell by 65.15%.

All measured depth and snapshot workloads improved. The unchanged BBO and clear-event branches were not benchmarked.

Testing

Passed:

cargo test --locked -p hftbacktest --lib --no-default-features --features backtest
cargo test --locked --release -p hftbacktest --lib --no-default-features --features backtest
cargo test --locked -p py-hftbacktest process_event_reuses_output_without_changing_add_semantics
cargo test --locked --release -p py-hftbacktest process_event_reuses_output_without_changing_add_semantics
cargo clippy --locked -p py-hftbacktest --all-targets
maturin build --release --locked --manifest-path py-hftbacktest/Cargo.toml

The installed-wheel fusion smoke test also passed.

Workspace-wide checks were attempted. Remaining failures occur in unchanged examples, documentation tests and tests requiring absent data files. Detailed logs and raw benchmark samples are available on request.

Compatibility

  • Existing Rust method signatures remain available.
  • No Python API or data-format changes.
  • No new dependencies.
  • No unsafe code.
  • No change to event ordering or deterministic output.

Related issue

Closes #320.

Reviewer guide

The complete production and test change is contained in:

  • hftbacktest/src/depth/fuse.rs
  • py-hftbacktest/src/fuse.rs

The important invariants are the order of appended events, stale-timestamp handling and add=false state behaviour.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 19 complexity

Metric Results
Complexity 19

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid per-event allocation in Python market-depth fusion

1 participant