-
Notifications
You must be signed in to change notification settings - Fork 6
docs: prepare state store guidance for RocksDB removal #62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,16 +2,16 @@ | |
| title: 'Sei Node Operations Guide' | ||
| sidebarTitle: 'Overview' | ||
| description: 'Detailed guide for running and maintaining Sei nodes. Learn about configuration management, database maintenance, service management, and update procedures.' | ||
| keywords: ['sei node', 'blockchain node', 'node configuration', 'seidb', 'node maintenance', 'system operations'] | ||
| keywords: ['sei node', 'blockchain node', 'node configuration', 'seidb', 'node maintenance', 'system operations', 'rocksdb'] | ||
| --- | ||
|
|
||
| This guide covers the detailed operational aspects of running a Sei node, | ||
| including configuration management, maintenance procedures, and best practices | ||
| for stable and performant operations. | ||
|
|
||
| ## Configuration Management | ||
|
|
||
| ### Directory Structure | ||
|
|
||
| The Sei node configuration is stored in `$HOME/.sei/config/`: | ||
|
|
||
|
|
@@ -25,15 +25,24 @@ | |
| └── priv_validator_key.json # Validator private signing key (if running as a validator) | ||
| ``` | ||
|
|
||
| The state-store databases live outside the config directory: | ||
|
|
||
| - Cosmos SS uses `$HOME/.sei/data/{backend}` in the legacy layout and | ||
| `$HOME/.sei/data/state_store/cosmos/{backend}` in the current layout. | ||
| - EVM SS uses `$HOME/.sei/data/evm_ss` in the legacy layout and | ||
| `$HOME/.sei/data/state_store/evm/{backend}` in the current layout. | ||
| - Non-empty `ss-db-directory` and `evm-ss-db-directory` settings override | ||
| those locations. | ||
|
|
||
| The snippets below are opinionated tuning recommendations layered on top of | ||
| the defaults. For the *full* unmodified `app.toml`, `config.toml`, and | ||
| `client.toml` shipped by the latest tagged `seid` release, jump to | ||
| [Default Configurations](#default-configurations) at the bottom of this | ||
| section. | ||
|
|
||
| ### Essential Configuration Parameters | ||
|
|
||
| #### Network Settings (config.toml) | ||
|
|
||
| ```toml | ||
| [p2p] | ||
|
|
@@ -59,7 +68,7 @@ | |
| timeout-broadcast-tx-commit = "10s" | ||
| ``` | ||
|
|
||
| #### Application Settings (app.toml) | ||
|
|
||
| ```toml | ||
| # Minimum gas prices to prevent spam transactions | ||
|
|
@@ -89,6 +98,8 @@ | |
| [state-store] | ||
| # Historical SS layer for queries. Required for any node serving RPC. | ||
| ss-enable = true | ||
| # State-store backend. Use PebbleDB; RocksDB support will be removed. | ||
| ss-backend = "pebbledb" | ||
|
alexander-sei marked this conversation as resolved.
|
||
| # 0 = keep everything; 100,000 is roughly 28 hours of pacific-1 history. | ||
| ss-keep-recent = 100000 | ||
|
|
||
|
|
@@ -97,12 +108,14 @@ | |
| rs-backend = "pebbledb" | ||
| ``` | ||
|
|
||
| ### Default Configurations | ||
|
|
||
| The full unmodified `app.toml`, `config.toml`, and `client.toml` produced by | ||
| `seid init` against the latest tagged `seid` release. Use these as the | ||
| canonical reference for every available knob and its default value. | ||
|
|
||
| <Warning>The generated `app.toml` below mirrors the latest tagged release and may still list RocksDB as a state-store option. Do not use RocksDB for new or resynced nodes. RocksDB support for the SeiDB state store will be removed. No target release has been published.</Warning> | ||
|
|
||
| <Tabs> | ||
| <Tab title="app.toml"> | ||
|
|
||
|
|
@@ -1349,7 +1362,7 @@ | |
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ## Database Management | ||
|
|
||
| ### Architecture | ||
|
|
||
|
|
@@ -1366,15 +1379,14 @@ | |
| modes is set to a non-default value. | ||
| 2. **State Store (SS)** — versioned raw key/value pairs used for historical | ||
| queries. Required for any node that serves RPC. The default backend is | ||
| **PebbleDB**; **RocksDB** is available for iteration-heavy workloads such | ||
| as archive nodes or RPC nodes that run a lot of `debug_trace*` (see the | ||
| [RocksDB Backend Guide](/node/rocksdb-backend) for build instructions). | ||
| **PebbleDB**. Do not use RocksDB for new nodes. If you already use RocksDB, | ||
| follow [Move off RocksDB](#move-off-rocksdb) before support is removed. | ||
|
|
||
| The legacy IAVL backend is still selectable via `sc-enable = false` but is | ||
| deprecated and slated for removal — new deployments and existing nodes | ||
| should run on SeiDB. | ||
|
|
||
| ### SeiDB Configuration | ||
|
|
||
| The full set of knobs is in the auto-generated | ||
| [Default Configurations](#default-configurations) above. The block below | ||
|
|
@@ -1420,7 +1432,7 @@ | |
| # serving RPC must keep it on. | ||
| ss-enable = true | ||
|
|
||
| # pebbledb (default) or rocksdb (faster iteration, archive-friendly). | ||
| # Use PebbleDB. RocksDB support will be removed. | ||
| ss-backend = "pebbledb" | ||
|
|
||
| ss-async-write-buffer = 100 | ||
|
|
@@ -1448,7 +1460,60 @@ | |
| snapshot creation. Too-large (less frequent) intervals mean pruning takes | ||
| longer overall, which can cause missed blocks and excessive resync time. | ||
|
|
||
| #### PebbleDB version encoding | ||
|
Check warning on line 1463 in node/node-operators.mdx
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] Inserting this Either move the new subsection below that paragraph, or lift the pruning paragraph up to directly follow the TOML block. |
||
|
|
||
| Fresh PebbleDB state stores use descending-version MVCC encoding. Sei includes | ||
| the version in each key and sorts newer versions first. This lets | ||
| latest-version reads reach the newest visible value without scanning older | ||
| versions. The store records this format with the `s/_mvcc_descending` sentinel | ||
| key, which `seid` detects automatically when it opens the database. | ||
|
|
||
| PebbleDB state stores created by older builds use ascending-version encoding. | ||
| `seid` detects these stores and opens them in compatibility mode without an | ||
| error. They remain on the slower ascending read path. Rebuild the state store | ||
| through [state sync](/node/statesync) with a current `seid` release to adopt | ||
| descending encoding. A filesystem snapshot preserves the source store's | ||
| encoding, so confirm its encoding with the snapshot provider. | ||
|
|
||
| ### Move off RocksDB | ||
|
|
||
| <Warning>RocksDB support for the SeiDB state store will be removed. No target release has been published. Check the [Sei release notes](https://github.com/sei-protocol/sei-chain/releases) before every upgrade.</Warning> | ||
|
|
||
| RocksDB and PebbleDB use different on-disk formats. Changing | ||
| `ss-backend = "rocksdb"` to `ss-backend = "pebbledb"` against the same data | ||
| does not convert the store. Rebuild the state store instead. | ||
|
|
||
| For a non-archive node: | ||
|
|
||
| 1. Stop `seid` and back up the validator key, validator state, node key, | ||
| configuration, and genesis file that you need to preserve. | ||
| 2. Check `ss-db-directory` and `evm-ss-db-directory` in `app.toml`. An empty | ||
| value uses a path under `$HOME/.sei/data`. If either setting points | ||
| elsewhere, move the old RocksDB data out of the active path and keep it | ||
| with the backup. Clearing `$HOME/.sei/data` does not clear a custom | ||
| directory. | ||
| 3. Set `ss-backend = "pebbledb"`, then rebuild through | ||
| [state sync](/node/statesync) or a provider-confirmed | ||
| [PebbleDB snapshot](/node/snapshot). | ||
| 4. Confirm the startup log reports `"SeiDB SS is enabled"` with the PebbleDB | ||
| backend. Test the RPC methods your node serves before you delete the old | ||
| RocksDB backup. | ||
|
|
||
| <Danger>Do not use state sync or a pruned snapshot to migrate an archive node. Both start from a recent height and discard the earlier state-store versions that an archive node must retain.</Danger> | ||
|
|
||
| There is no documented in-place migration for a RocksDB archive node. Build a | ||
| separate replacement from a trusted full-history PebbleDB source. Keep the | ||
| RocksDB node on a compatible `seid` release until the replacement has caught | ||
| up and you have tested historical queries at old heights. If you cannot obtain | ||
| a full-history PebbleDB source, do not wipe the existing archive data. Contact | ||
| the [Sei Tech Chat](https://t.me/+KZdhZ1eE-G01NmZk) before you upgrade. | ||
|
|
||
| PebbleDB can be slower for iteration-heavy historical queries, including | ||
| `debug_trace*` requests. Descending-version encoding improves recent reads, | ||
| but it does not remove the cost of long-history iteration. Benchmark the | ||
| replacement under your trace workload before you cut over. | ||
|
|
||
| ### Giga Storage and Giga Executor | ||
|
|
||
| These are two **separate** opt-in features that ship in newer `seid` | ||
| releases. Both default to off; only enable them deliberately and after | ||
|
|
@@ -1492,7 +1557,7 @@ | |
| occ_enabled = false | ||
| ``` | ||
|
|
||
| ### Database Maintenance | ||
|
|
||
| The database is typically stable and can be left alone, although some attention | ||
| may be required: | ||
|
|
@@ -1519,9 +1584,9 @@ | |
|
|
||
| <Warning>The wipe command above deletes the entire local database (everything except `priv_validator_state.json`) and the `wasm` folder. It does not compact data in place — after running it, the node must be re-synced from a [snapshot](/node/snapshot) or via [state sync](/node/statesync) before it can serve traffic again.</Warning> | ||
|
|
||
| ## Service Management | ||
|
|
||
| ### Systemd Commands | ||
|
|
||
| ```bash | ||
| # Check service status | ||
|
|
@@ -1540,7 +1605,7 @@ | |
| journalctl -fu seid -o cat | ||
| ``` | ||
|
|
||
| ### Log Management | ||
|
|
||
| Prevent logs from consuming excessive disk space by enabling rotation: | ||
|
|
||
|
|
@@ -1561,7 +1626,7 @@ | |
| EOF | ||
| ``` | ||
|
|
||
| ## Update Procedures | ||
|
|
||
| <Info> | ||
| Upgrade with the same method you originally installed with: `make install` | ||
|
|
@@ -1571,7 +1636,7 @@ | |
| `which -a seid` lists every copy. | ||
| </Info> | ||
|
|
||
| ### Minor Updates | ||
|
|
||
| For minor updates that are non-consensus-breaking: | ||
|
|
||
|
|
@@ -1613,7 +1678,7 @@ | |
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ### Major Updates | ||
|
|
||
| For major upgrades that introduce state-breaking changes: | ||
|
|
||
|
|
@@ -1657,7 +1722,7 @@ | |
| halt-height so the swap takes seconds at upgrade time. | ||
| </Tip> | ||
|
|
||
| ## Performance Optimization | ||
|
|
||
| Performance optimizations can yield different results depending on your system's | ||
| hardware, workload, and network conditions. Before implementing any changes, | ||
|
|
@@ -1665,7 +1730,7 @@ | |
| your specific configuration and requirements. Always back up important data | ||
| before making modifications. | ||
|
|
||
| ### Memory Management (sysctl tuning) | ||
|
|
||
| Optimizing memory management settings can help improve performance and | ||
| stability, particularly for high-load nodes. These settings control swap usage | ||
|
|
@@ -1679,7 +1744,7 @@ | |
| vm.dirty_writeback_centisecs = 100 # Frequency (in hundredths of a second) at which the system writes "dirty" pages to disk | ||
| ``` | ||
|
|
||
| ### Network Stack Optimization | ||
|
|
||
| Tuning the network stack can enhance packet processing efficiency and | ||
| throughput, particularly for nodes handling a large number of peers and high | ||
|
|
@@ -1693,7 +1758,7 @@ | |
| net.core.wmem_max = 16777216 # send buffer size for network sockets | ||
| ``` | ||
|
|
||
| ### Storage Optimization | ||
|
|
||
| Optimizing storage settings can significantly reduce write latency and improve | ||
| database performance, especially for nodes using NVMe SSDs. | ||
|
|
@@ -1703,9 +1768,9 @@ | |
| blockdev --setra 4096 /dev/nvme0n1 # readahead value to optimize sequential reads | ||
| ``` | ||
|
|
||
| ## Backup and Recovery | ||
|
|
||
| ### Regular Backups | ||
|
|
||
| Automate backups to avoid data loss: | ||
|
|
||
|
|
@@ -1724,7 +1789,7 @@ | |
| systemctl start seid | ||
| ``` | ||
|
|
||
| ### Recovery Procedure | ||
|
|
||
| Restoring from backup in case of corruption or accidental deletion: | ||
|
|
||
|
|
@@ -1742,7 +1807,7 @@ | |
| systemctl start seid | ||
| ``` | ||
|
|
||
| ## Security Considerations | ||
|
|
||
| - Use firewalls and rate-limiting to prevent attacks | ||
| - Keep your system and node software updated | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[blocker] This calls
$HOME/.sei/data/evm_ssthe legacy layout and$HOME/.sei/data/state_store/evm/{backend}the current one, but the auto-generated defaultapp.tomlembedded ~290 lines below in this same file says the opposite (line 320):That block is regenerated from the latest tagged
seidrelease viaseid init(.github/workflows/sync-default-configs.yml), so it is the authoritative reference on this page. Nothing else in the repo mentions adata/state_store/...layout.One of the two is wrong, and readers hit both on a single page. This is not cosmetic: the new
verify_snapshot_backendfunction innode/snapshot.mdxderives its scanned paths from this claim, so if the layout mapping is wrong the safety check scans the wrong directories.Please reconcile against the actual
seidrelease — either correct the prose, or note the release in which thestate_store/...layout landed so the stale generated comment is explained. The same claim is duplicated atnode/giga-storage-migration.mdx:201-205and needs the same fix.