diff --git a/STYLE_GUIDE.md b/STYLE_GUIDE.md index 622757f..76d8307 100644 --- a/STYLE_GUIDE.md +++ b/STYLE_GUIDE.md @@ -77,7 +77,7 @@ The Operate section covers topics related to running Sei infrastructure. This is **Contents include:** - **Node Operations**: Overview, Seictl setup, statesync, snapshot sync, node types, troubleshooting, API configuration, validators, oracle price feeder -- **Advanced Operations**: Configuration & monitoring, RocksDB backend, technical reference +- **Advanced Operations**: Configuration & monitoring, Giga SS Store migration, technical reference ## Style Guidelines diff --git a/assets/pebbledb-vs-rocksdb.png b/assets/pebbledb-vs-rocksdb.png deleted file mode 100644 index c6e7918..0000000 Binary files a/assets/pebbledb-vs-rocksdb.png and /dev/null differ diff --git a/docs.json b/docs.json index 2f4ef94..7e3bc63 100644 --- a/docs.json +++ b/docs.json @@ -339,7 +339,6 @@ "group": "Advanced Operations", "pages": [ "node/advanced-config-monitoring", - "node/rocksdb-backend", "node/giga-storage-migration", "node/technical-reference" ] @@ -899,6 +898,11 @@ "destination": "/node", "permanent": true }, + { + "source": "/node/rocksdb-backend", + "destination": "/node/node-operators#move-off-rocksdb", + "permanent": true + }, { "source": "/node/oracle-price-feeder", "destination": "/learn/oracles", diff --git a/node/giga-storage-migration.mdx b/node/giga-storage-migration.mdx index 755f614..a8abb1a 100644 --- a/node/giga-storage-migration.mdx +++ b/node/giga-storage-migration.mdx @@ -15,7 +15,7 @@ After migration the SS layer is repartitioned into two cooperating stores: | Layer | Cosmos backend | EVM backend | |---|---|---| | **SC** (State Commit, app hash) | `memiavl` | FlatKV | -| **SS** (State Store, historical queries) | single MVCC DB (PebbleDB or RocksDB) | dedicated EVM SS MVCC DB(s) under `data/evm_ss/` | +| **SS** (State Store, historical queries) | single PebbleDB MVCC database | dedicated EVM SS MVCC databases in the configured EVM SS directory | Only the **SS** layer changes for this migration. SC layer config is untouched and `memiavl` remains the authoritative source for the app hash, so this is @@ -43,9 +43,6 @@ local data directory and imports a fresh snapshot into the new layout. - EVM reads are served exclusively from a dedicated EVM SS database. - Non-EVM modules no longer pay write amplification for EVM state. -- A backend change (PebbleDB ↔ RocksDB) can be combined with the same state - sync, since `ss-backend` drives both the Cosmos SS MVCC DB and every EVM SS - sub-DB. ## What's different about EVM SS @@ -69,8 +66,7 @@ sc-enable = true [state-store] ss-enable = true -# DBBackend for the Cosmos SS MVCC DB and for every EVM SS sub-DB. -# Supported: pebbledb, rocksdb. Default pebbledb. +# Use PebbleDB for the Cosmos SS MVCC DB and every EVM SS sub-DB. ss-backend = "pebbledb" # Route EVM state to the dedicated EVM SS backend. @@ -80,14 +76,7 @@ ss-backend = "pebbledb" evm-ss-split = true ``` -If you want to switch SS backend in the same step: - -- **PebbleDB → RocksDB**: set `ss-backend = "rocksdb"`, build `seid` with - `-tags rocksdbBackend`, and install RocksDB per the - [RocksDB Backend Guide](/node/rocksdb-backend). `ss-backend` drives both the - Cosmos SS MVCC DB and every EVM SS sub-DB, so a single setting flips both. -- No data migration tool is needed across backends — the state sync populates - the new layout. +Keep `ss-backend = "pebbledb"` during this migration. RocksDB support for the state store will be removed. No target release has been published. If the node already uses RocksDB, follow [Move off RocksDB](/node/node-operators#move-off-rocksdb). ### Step 2: State sync into the new layout @@ -175,10 +164,9 @@ flipping `evm-ss-split` from `false` to `true` without state syncing. 1. **EVM SS directory missing or empty** (before the EVM SS is opened). When `evm-ss-split = true`, the composite state store refuses to proceed if - Cosmos SS already has committed history but the EVM SS directory - (`data/evm_ss/` by default) does not exist or is empty. Failing before - the sub-DBs are opened means a rejected config does not leave a confusing - empty `data/evm_ss/` behind. + Cosmos SS already has committed history but the configured EVM SS directory + does not exist or is empty. Failing before the sub-DBs are opened means a + rejected config does not leave a confusing empty directory behind. 2. **EVM SS DB empty post-open, pre-recovery.** Belt-and-suspenders for (1) when the directory exists but its DBs are empty. The WAL only covers the last `KeepRecent` blocks, so replay cannot rebuild a fresh EVM SS from @@ -190,18 +178,19 @@ flipping `evm-ss-split` from `false` to `true` without state syncing. If any check fires, the correct fix is either (a) complete the state sync described above, or (b) set `evm-ss-split = false` and restart. If -`data/evm_ss/` is stale from a failed attempt, remove it before state syncing. +the configured EVM SS directory is stale from a failed attempt, remove it +before state syncing. ## Rollback To roll back: 1. Set `evm-ss-split = false` in `app.toml`. -2. Restart the node. The EVM SS DB under `data/evm_ss/` is no longer opened - but stays on disk until manually removed. +2. Restart the node. The EVM SS DB is no longer opened but stays on disk until + you remove it. To fully reclaim the disk used by EVM SS, stop the node and delete -`data/evm_ss/` after reverting the setting. +the configured EVM SS directory after reverting the setting. Cleanly rolling back to `evm-ss-split = false` requires another state sync. Under `evm-ss-split = true`, EVM writes go only to the EVM SS DB, so Cosmos SS will not have those writes. Restarting with `evm-ss-split = false` stops opening the EVM SS DB, but EVM-state queries will miss anything written after the Giga state sync until you re-state-sync without the split. @@ -209,9 +198,12 @@ To fully reclaim the disk used by EVM SS, stop the node and delete ### Where do the data files live after migrating? -- Cosmos SS data lives under the same directory as before, typically - `data/pebbledb/` for the default `pebbledb` backend. -- EVM SS data lives under `data/evm_ss/`. +- Cosmos SS data uses `data/pebbledb/` in the legacy layout and + `data/state_store/cosmos/pebbledb/` in the current layout. +- EVM SS data uses `data/evm_ss/` in the legacy layout and + `data/state_store/evm/pebbledb/` in the current layout. +- A non-empty `ss-db-directory` or `evm-ss-db-directory` overrides the + corresponding default path. - SC data (`memiavl` + FlatKV) is untouched by this migration. ### Does Giga SS Store change the app hash or consensus? diff --git a/node/index.mdx b/node/index.mdx index d1f4bdd..c698b90 100644 --- a/node/index.mdx +++ b/node/index.mdx @@ -138,12 +138,10 @@ See the Network Versions table above for the current recommended version. ``` - Prebuilt binaries are attached from `v6.6.1` onward — for earlier versions, build - from source or use Docker. They are `linux/amd64` only, omit hardware Ledger - support, and don't include the optional - [RocksDB state-store backend](/node/rocksdb-backend). On other architectures, or - if you sign with a Ledger device, build from source or use Docker; a - RocksDB-enabled `seid` must be built from source. + Prebuilt binaries are attached from `v6.6.1` onward. For earlier versions, + build from source or use Docker. They are `linux/amd64` only and omit hardware + Ledger support. On other architectures, or if you sign with a Ledger device, + build from source or use Docker. @@ -416,8 +414,8 @@ seid query staking validator $(seid keys show --bech val -a) Optimize your node's performance with advanced settings and monitoring tools. - - Run with RocksDB instead of the default backend. + + Move EVM historical state into dedicated state-store databases on supported RPC nodes. Detailed technical specifications, API endpoints, and commands for node operators. diff --git a/node/node-operators.mdx b/node/node-operators.mdx index 386e9a5..eca9a8a 100644 --- a/node/node-operators.mdx +++ b/node/node-operators.mdx @@ -2,7 +2,7 @@ title: 'Sei Node Operations Guide' sidebarTitle: 'Overview' description: 'Detailed guide for running and maintaining Sei nodes. Learn about configuration management, database maintenance, service management, and update procedures.' -keywords: ['sei node', 'blockchain node', 'node configuration', 'seidb', 'node maintenance', 'system operations'] +keywords: ['sei node', 'blockchain node', 'node configuration', 'seidb', 'node maintenance', 'system operations', 'rocksdb'] --- This guide covers the detailed operational aspects of running a Sei node, @@ -25,6 +25,15 @@ $HOME/.sei/config/ └── priv_validator_key.json # Validator private signing key (if running as a validator) ``` +The state-store databases live outside the config directory: + +- Cosmos SS uses `$HOME/.sei/data/{backend}` in the legacy layout and + `$HOME/.sei/data/state_store/cosmos/{backend}` in the current layout. +- EVM SS uses `$HOME/.sei/data/evm_ss` in the legacy layout and + `$HOME/.sei/data/state_store/evm/{backend}` in the current layout. +- Non-empty `ss-db-directory` and `evm-ss-db-directory` settings override + those locations. + The snippets below are opinionated tuning recommendations layered on top of the defaults. For the *full* unmodified `app.toml`, `config.toml`, and `client.toml` shipped by the latest tagged `seid` release, jump to @@ -89,6 +98,8 @@ sc-enable = true [state-store] # Historical SS layer for queries. Required for any node serving RPC. ss-enable = true +# State-store backend. Use PebbleDB; RocksDB support will be removed. +ss-backend = "pebbledb" # 0 = keep everything; 100,000 is roughly 28 hours of pacific-1 history. ss-keep-recent = 100000 @@ -103,6 +114,8 @@ The full unmodified `app.toml`, `config.toml`, and `client.toml` produced by `seid init` against the latest tagged `seid` release. Use these as the canonical reference for every available knob and its default value. +The generated `app.toml` below mirrors the latest tagged release and may still list RocksDB as a state-store option. Do not use RocksDB for new or resynced nodes. RocksDB support for the SeiDB state store will be removed. No target release has been published. + @@ -1366,9 +1379,8 @@ legacy single-database IAVL store with separate hot- and historical-data tiers: modes is set to a non-default value. 2. **State Store (SS)** — versioned raw key/value pairs used for historical queries. Required for any node that serves RPC. The default backend is - **PebbleDB**; **RocksDB** is available for iteration-heavy workloads such - as archive nodes or RPC nodes that run a lot of `debug_trace*` (see the - [RocksDB Backend Guide](/node/rocksdb-backend) for build instructions). + **PebbleDB**. Do not use RocksDB for new nodes. If you already use RocksDB, + follow [Move off RocksDB](#move-off-rocksdb) before support is removed. The legacy IAVL backend is still selectable via `sc-enable = false` but is deprecated and slated for removal — new deployments and existing nodes @@ -1420,7 +1432,7 @@ snapshot-keep-recent = 2 # serving RPC must keep it on. ss-enable = true -# pebbledb (default) or rocksdb (faster iteration, archive-friendly). +# Use PebbleDB. RocksDB support will be removed. ss-backend = "pebbledb" ss-async-write-buffer = 100 @@ -1448,6 +1460,59 @@ Setting small (more frequent) pruning intervals may collide with snapshot creation. Too-large (less frequent) intervals mean pruning takes longer overall, which can cause missed blocks and excessive resync time. +#### PebbleDB version encoding + +Fresh PebbleDB state stores use descending-version MVCC encoding. Sei includes +the version in each key and sorts newer versions first. This lets +latest-version reads reach the newest visible value without scanning older +versions. The store records this format with the `s/_mvcc_descending` sentinel +key, which `seid` detects automatically when it opens the database. + +PebbleDB state stores created by older builds use ascending-version encoding. +`seid` detects these stores and opens them in compatibility mode without an +error. They remain on the slower ascending read path. Rebuild the state store +through [state sync](/node/statesync) with a current `seid` release to adopt +descending encoding. A filesystem snapshot preserves the source store's +encoding, so confirm its encoding with the snapshot provider. + +### Move off RocksDB + +RocksDB support for the SeiDB state store will be removed. No target release has been published. Check the [Sei release notes](https://github.com/sei-protocol/sei-chain/releases) before every upgrade. + +RocksDB and PebbleDB use different on-disk formats. Changing +`ss-backend = "rocksdb"` to `ss-backend = "pebbledb"` against the same data +does not convert the store. Rebuild the state store instead. + +For a non-archive node: + +1. Stop `seid` and back up the validator key, validator state, node key, + configuration, and genesis file that you need to preserve. +2. Check `ss-db-directory` and `evm-ss-db-directory` in `app.toml`. An empty + value uses a path under `$HOME/.sei/data`. If either setting points + elsewhere, move the old RocksDB data out of the active path and keep it + with the backup. Clearing `$HOME/.sei/data` does not clear a custom + directory. +3. Set `ss-backend = "pebbledb"`, then rebuild through + [state sync](/node/statesync) or a provider-confirmed + [PebbleDB snapshot](/node/snapshot). +4. Confirm the startup log reports `"SeiDB SS is enabled"` with the PebbleDB + backend. Test the RPC methods your node serves before you delete the old + RocksDB backup. + +Do not use state sync or a pruned snapshot to migrate an archive node. Both start from a recent height and discard the earlier state-store versions that an archive node must retain. + +There is no documented in-place migration for a RocksDB archive node. Build a +separate replacement from a trusted full-history PebbleDB source. Keep the +RocksDB node on a compatible `seid` release until the replacement has caught +up and you have tested historical queries at old heights. If you cannot obtain +a full-history PebbleDB source, do not wipe the existing archive data. Contact +the [Sei Tech Chat](https://t.me/+KZdhZ1eE-G01NmZk) before you upgrade. + +PebbleDB can be slower for iteration-heavy historical queries, including +`debug_trace*` requests. Descending-version encoding improves recent reads, +but it does not remove the cost of long-history iteration. Benchmark the +replacement under your trace workload before you cut over. + ### Giga Storage and Giga Executor These are two **separate** opt-in features that ship in newer `seid` diff --git a/node/rocksdb-backend.mdx b/node/rocksdb-backend.mdx deleted file mode 100644 index 05098a4..0000000 --- a/node/rocksdb-backend.mdx +++ /dev/null @@ -1,122 +0,0 @@ ---- -title: 'RocksDB Backend Setup & Performance' -sidebarTitle: 'RocksDB Backend' -description: 'Build and enable the RocksDB backend for seid to significantly improve iteration-heavy operations and traceBlock latency, especially on long-history RPC and archive nodes.' -keywords: ['rocksdb', 'sei node', 'state store', 'performance', 'traceBlock', 'archive node', 'rpc'] ---- -## Overview - -As Sei nodes accumulate more history and state data, iteration-heavy operations—like those performed during `traceBlock` calls—can become significantly slower. This is especially true for archive or RPC nodes where the state store contains millions of versions. - -To address this, Sei supports **RocksDB** as an alternative backend to PebbleDB. RocksDB provides native **multi-version concurrency control (MVCC)** and **column family support**, which leads to substantial iteration performance improvements as history grows. - -## Why RocksDB? - -PebbleDB lacks native MVCC support, meaning Sei must manually encode versions into keys. This retrofit approach causes iteration time to grow linearly with the amount of stored history. As a result: - -- Archive nodes with large state stores experience **slower debug trace latency**. -- Each key lookup may require scanning multiple key versions. -- Iterations (e.g., over Oracle or EVM module keys) become increasingly expensive over time. - -By contrast, **RocksDB** supports native user-defined timestamps and optimized column families for versioned data access. This means: - -- Iterations over keys at a single version are **much faster**. -- Historical data growth has **minimal effect on iteration cost**. -- The performance advantage **amplifies with larger node history**. - -In Sei’s benchmarks, RocksDB achieved up to **10–30× faster traceBlock iteration times** compared to PebbleDB, with even greater benefits observed on archive nodes. - - - -### PebbleDB descending-version encoding - -Recent PebbleDB builds partly narrow this gap for latest-version reads. Because PebbleDB has no native MVCC, Sei encodes the version into each key. Freshly created PebbleDB state stores now use a **descending-version MVCC encoding**, which sorts newer versions before older ones for the same logical key. This lets latest-version reads land directly on the newest visible version instead of scanning through older versions, improving read performance on the fast path. Fresh stores are marked on disk with a sentinel key (`s/_mvcc_descending`) so the mode is detected automatically on open. - -Legacy PebbleDB stores written by earlier builds use the older **ascending-version encoding**. These are detected automatically on open and read using the legacy ascending path—no error is raised—but they stay unmarked and cannot benefit from the descending fast path unless the store is recreated or migrated. This mirrors the migration constraint on RocksDB: archive nodes that cannot recreate their state store will continue running on the slower legacy path. Note that even with descending encoding, PebbleDB still lacks native MVCC and column-family support, so RocksDB remains the recommended backend for iteration-heavy archive and long-history RPC workloads. - -## Example: TraceBlock Latency Comparison - -The following chart compares iteration (trace time) performance between **PebbleDB** and **RocksDB** over a 3 million block history: - -![PebbleDB vs RocksDB Trace Times](/assets/pebbledb-vs-rocksdb.png) -*Trace Times: Pebble vs Rocks (3M history): RocksDB shows a significantly flatter latency curve as state grows, while PebbleDB’s iteration times increase sharply for older blocks.* - -## Setup Instructions - -RocksDB only needs to be built once. After that, you can install `seid` with RocksDB support directly. - -### Prerequisites - -Ensure your system includes the following packages: - -```bash -sudo apt-get update -sudo apt-get install -y build-essential pkg-config cmake git zlib1g-dev \ - libbz2-dev libsnappy-dev liblz4-dev libzstd-dev libjemalloc-dev \ - libgflags-dev liburing-dev -``` - -### Build & Install - -Refer to the Sei `Makefile` [here](https://github.com/sei-protocol/sei-chain/blob/main/Makefile#L116) for the official targets. - -```bash -# Step 1: Build RocksDB (one-time setup) -make build-rocksdb - -# Step 2: Install seid with RocksDB backend -make install-rocksdb -``` - -Once installed, your `seid` binary will be built with RocksDB backend support: - -```bash -seid version -# should include "rocksdbBackend" build tag -``` - -### Configuration - -After installation, update your configuration file to enable the RocksDB backend: - -```bash -~/.sei/config/app.toml -``` - -Set - -```toml -ss-backend = "rocksdb" -``` - -### Node Setup Notes - -- RPC Nodes — must perform a state sync when spinning up a new node configured with RocksDB. -- Archive Nodes — currently, RocksDB is not supported for existing data unless syncing from genesis. A migration route from PebbleDB to RocksDB is being developed and will be shared soon. - -## Summary - -| Feature | PebbleDB | RocksDB | -| ----------------------------- | ------------------ | ------------------------------------- | -| MVCC Support | No | ✅ Native via user-defined timestamps | -| Column Families | No | ✅ Yes | -| Iteration Speed (Large State) | Slows with history | ✅ Up to 30× faster | -| Installation | Default | One-time build (`make build-rocksdb`) | - -## TL;DR - -- RocksDB backend drastically improves trace iteration and historical query performance. -- Install once, then run: - -```bash -make install-rocksdb -``` - -- Update ~/.sei/config/app.toml to use RocksDB: - -```toml -ss-backend = "rocksdb" -``` - -- RPC nodes require state sync; archive nodes must currently sync from genesis -- Expect **10–30× faster** trace latencies, especially on archive nodes or long-history setups. diff --git a/node/snapshot.mdx b/node/snapshot.mdx index 1b7fa61..2fcb826 100644 --- a/node/snapshot.mdx +++ b/node/snapshot.mdx @@ -12,6 +12,10 @@ Follow this guide to join an existing network through **snapshot sync**. To quic Snapshot sync allows a new node to join a network by downloading a recent, compressed copy of the entire application state and extracting it directly into the data directory. This reduces the initial sync time from days to minutes. +RocksDB support for the SeiDB state store will be removed. No target release has been published. For new or resynced nodes, use a snapshot only after its provider confirms that the state store uses PebbleDB. Keep `ss-backend = "pebbledb"` in `app.toml`. See [Move off RocksDB](/node/node-operators#move-off-rocksdb) if you are replacing an existing RocksDB node. + +Do not restore a pruned snapshot onto an archive node. It does not contain the earlier state-store versions that the archive node must retain. Use only a provider-confirmed full-history PebbleDB archive snapshot, or follow the archive-node guidance in [Move off RocksDB](/node/node-operators#move-off-rocksdb). + ## Snapshot Providers You can select from various providers for downloading snapshots: @@ -23,6 +27,12 @@ You can select from various providers for downloading snapshots: