Add CryptoHFTData.com orderbook/trades support and full workflow notebook

[hmate9]

Summary

This PR adds CryptoHFTData as a historical order book and trade data provider for the Python data utilities, including conversion, documentation, a command-line example, and an end-to-end notebook.

Disclosure: I created CryptoHFTData.com and provide the historical market data used by this integration. The provider has broad spot and derivatives coverage and supports rate-limited keyless downloads through its free tier.

What’s included

• hftbacktest.data.utils.cryptohftdata
  • download(...)
  • convert(...)
  • download_and_convert(...)
• optional hftbacktest[cryptohftdata] dependency using cryptohftdata>=0.4.0
• keyless free-tier downloads, with optional explicit or CRYPTOHFTDATA_API_KEY credentials
• conversion of incremental depth, snapshots, and trades into event_dtype
• unit tests for conversion, validation, snapshots, ordering, pandas input, injected clients, keyless clients, and environment credentials
• Sphinx API reference and a detailed usage guide
• examples/cryptohftdata.py for command-line downloads and conversion
• executed examples/CryptoHFTData Workflow.ipynb covering download, conversion, snapshot reconstruction, replay, and statistics

Converter behavior

The converter maps CryptoHFTData’s normalized schema into HftBacktest’s event format:

• order book local_ts = received_time
• order book exch_ts = coalesce(transaction_time, event_time) * 1_000_000
• trade local_ts = received_time
• trade exch_ts = coalesce(trade_time, event_time) * 1_000_000
• is_buyer_maker=True becomes SELL_EVENT | TRADE_EVENT
• other trades become BUY_EVENT | TRADE_EVENT
• update rows become DEPTH_EVENT
• snapshot rows become side-specific DEPTH_CLEAR_EVENT and DEPTH_SNAPSHOT_EVENT sequences

The normal latency correction, stable event ordering, and event-order validation are applied before output.

Usage

pip install "hftbacktest[cryptohftdata]"

python examples/cryptohftdata.py \
  BTCUSDT binance_futures 2026-03-01 2026-03-02 \
  data/btcusdt_20260301_20260302.npz

An API key is optional. CRYPTOHFTDATA_API_KEY can be set for account access without putting credentials in source code.

Validation

• python -m pytest py-hftbacktest/tests/test_cryptohftdata.py -q — 9 passed
• ruff check on the provider, tests, and command-line example — passed
• Python compilation and command-line help smoke checks — passed
• Sphinx rendered the new CryptoHFTData tutorial and API reference successfully; the repository retains unrelated pre-existing documentation warnings for missing copied notebooks, optional Databento, and Pandoc
• live keyless test with CryptoHFTData SDK 0.4.0:
  • downloaded 239,047 KAVAUSDT Binance Futures order book rows and 15,699 trades for 2026-07-11
  • converted 258,093 events, including 8,089 snapshot events
  • reconstructed a 329-level terminal snapshot
  • replayed through HashMapMarketDepthBacktest to a valid 0.0445 / 0.0446 terminal book

The broader upstream Python test directory currently contains an existing test that expects an untracked tmp_20240501.npz fixture and a Numba function named test_run that pytest collects as a fixture-based test. Those failures are unrelated to this change; the complete CryptoHFTData test module passes.

Notes

• Date ranges are inclusive UTC days because the SDK downloads hourly partitions for each selected day.
• Some ranges begin with incremental depth updates. The guide explains how to include earlier warm-up data or reconstruct an end-of-range snapshot before replaying a later period.
• The executed BTCUSDT notebook intentionally does not commit credentials or generated NPZ files.

[AnMakc]

Does book data include snapshots per file (hourly) or per day?
In my test, there were no snapshots for binance BTC/USDT for 2 weeks of data, nor for spot nor perps.

[hmate9]

Does book data include snapshots per file (hourly) or per day? In my test, there were no snapshots for binance BTC/USDT for 2 weeks of data, nor for spot nor perps.

@AnMakc

deltas are most common, but after a disconnect it now also has snapshots:

>>> df = pd.read_parquet("BTCUSDT_orderbook_20260414_00.parquet")
>>> df.head()
         received_time     event_time  transaction_time   symbol event_type  first_update_id  ...  prev_final_update_id  last_update_id  side           price    quantity order_count
0  1776124800035760089  1776124799914               NaN  BTCUSDT     update     9.192883e+10  ...                   NaN             NaN   ask  74418.00000000  2.95160000         NaN
1  1776124800035760089  1776124799914               NaN  BTCUSDT     update     9.192883e+10  ...                   NaN             NaN   ask  74419.71000000  0.00000000         NaN
2  1776124800035760089  1776124799914               NaN  BTCUSDT     update     9.192883e+10  ...                   NaN             NaN   ask  74419.72000000  0.00008000         NaN
3  1776124800035760089  1776124799914               NaN  BTCUSDT     update     9.192883e+10  ...                   NaN             NaN   ask  74419.73000000  0.00007000         NaN
4  1776124800035760089  1776124799914               NaN  BTCUSDT     update     9.192883e+10  ...                   NaN             NaN   ask  74420.32000000  0.00000000         NaN

[5 rows x 13 columns]
>>> df.event_type.value_counts()
event_type
update      1102418
snapshot       5000
Name: count, dtype: int64
>>> 

[hmate9]

but even with just deltas, you can effectively create a snapshot by reading in deltas a few hours from before the exact timeframe youre interested in and "warming up" with that data

[hmate9]

@nkaz001 any feedback or plans to merge this?

[AnMakc]

@hmate9
I've been looking into data and there's still gaps for BTCUSTD in recent orderbook data, please see attached screenshot.
This is on Binance BTCUSDT perp futures orderbook deltas, gaps are detected by checking update ids.

I think it's expected to have one gap per day due to planned exchange engine restarts, but there're more frequent gaps, is it a known issue? Also, is there github/discord?

[hmate9]

@AnMakc thanks for flagging this. I rechecked the full range from your screenshot, plus the period since then, against the currently published CryptoHFTData files.

Method: for Binance USD-M futures orderbook updates, I collapsed the price-level rows back to event-level depth messages and checked the Binance continuity rule:

current.prev_final_update_id / pu == previous.final_update_id / u

Snapshot rows were treated as valid reset anchors.

Results as of 2026-06-20:

Period	Event-level messages scanned	Real pu gaps	Gap rate
2026-03-01 → 2026-05-20	142,539,453	353	2.48 / 1M events
2026-05-21 → latest 2026-06-20 hour	92,466,387	20	0.22 / 1M events

So yes, your complaint was valid for the earlier data, especially March. But the situation has improved a lot since then: roughly a 91% lower gap rate, and from 2026-05-28 through the latest available 2026-06-20 hour I found zero real pu chain gaps.

We have invested into making our collectors much more stable and more reliable. We still have rare disconnects from binance websockets but recover quickly.

[codacy-production]

Not up to standards ⛔

🔴 Issues 9 minor

Alerts:
⚠ 9 issues (≤ 0 issues of at least minor severity)

Results:
9 new issues

Category	Results
Documentation	9 minor

View in Codacy

🟢 Metrics 61 complexity · 4 duplication

Metric	Results
Complexity	61
Duplication	4

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}