diff --git a/evm/differences-with-ethereum.mdx b/evm/differences-with-ethereum.mdx index d3500f8..5afff91 100644 --- a/evm/differences-with-ethereum.mdx +++ b/evm/differences-with-ethereum.mdx @@ -137,6 +137,49 @@ updated by both native Cosmos bank send transactions and EVM send transactions. As a result, if certain offchain applications only parse EVM transactions, they may find certain state changes unattributable to any EVM transaction. + + +## EVM Transaction Envelope Restrictions + +On Sei, EVM transactions are carried inside a Cosmos transaction envelope, but that envelope must remain empty of Cosmos-specific fields. This introduces two Sei-specific divergences from a plain Ethereum transaction that tooling constructing raw transactions should be aware of: + +### No Cosmos wrapper fields on EVM transactions + +An EVM transaction must not populate any of the Cosmos wrapper fields. If any of the following are set, the transaction is rejected: + +- `memo` +- `timeout_height` +- extension options (`extension_options` / `non_critical_extension_options`) +- `signer_infos` +- fee amount, fee `payer`, and fee `granter` +- top-level `signatures` + +The EVM transaction's own signature (`v`, `r`, `s`) lives inside the EVM payload itself, so none of these Cosmos-level fields are needed. This check is applied uniformly to all EVM transactions. + + + +### Whole-block rejection on transaction decode failure + +During proposal processing, Sei decodes every transaction in a proposed block. If any transaction fails to decode — or panics during decode — the **entire block proposal is rejected** rather than the offending transaction being silently skipped (treated as nil) while the rest of the block proceeds. + +This also affects block gas accounting: a transaction that could not be decoded no longer contributes zero gas and gets skipped; instead its presence causes the block proposal to be rejected outright. + + + +This is a consensus-level behavior change. Tooling that produces malformed or non-canonical transactions can no longer rely on such transactions being individually dropped from an otherwise-valid block — a single undecodable transaction now invalidates the whole proposal. Ensure your signing and broadcast tooling only submits transactions that decode cleanly. + + + +### Rejection of bloated (non-canonical) transaction bodies + +The transaction decoder now rejects "bloated" transaction bodies — those whose raw protobuf wire encoding is larger than the canonical re-marshal of the decoded body. Non-canonical encodings (for example, padded fields or an oversized `Any.Value`) that were previously silently canonicalized and accepted are now rejected on decode with a decode error. + + + +These are consensus-level validation changes. Transactions that were previously accepted may now be rejected if they carry Cosmos wrapper fields or use a non-canonical protobuf encoding. Ensure your transaction-signing tooling produces a minimal, canonical envelope with no Cosmos wrapper fields populated. + + + ## Finality Sei has instant finality — a transaction is final as soon as its block is diff --git a/evm/evm-parity/state-proofs.mdx b/evm/evm-parity/state-proofs.mdx index 8fa8462..c8684ee 100644 --- a/evm/evm-parity/state-proofs.mdx +++ b/evm/evm-parity/state-proofs.mdx @@ -13,6 +13,16 @@ Ethereum stores state in a Merkle Patricia Trie (MPT) and `eth_getProof` returns The RPC method exists and responds correctly, but the proof data structure is not compatible with Ethereum MPT proof verifiers. +### Store backends + +`eth_getProof` resolves the underlying account store by unwrapping known KVStore wrappers until it reaches a proof-capable queryable store. This means proofs are served across a broader range of node configurations, not just a classic IAVL store. Supported roots include: + +- Classic IAVL stores +- store/v2 memiavl commitment stores +- Any other proof-capable (queryable) store reached through `tracekv`, Giga cache, or prefix store wrappers + +If none of these can be reached, the call returns a `cannot find a proof-capable queryable KV store` error. In all cases the returned proof data is IAVL-format, not Ethereum MPT. + ## What This Affects Most applications do not call `eth_getProof` directly. It is primarily used by: diff --git a/evm/evm-parity/transaction-types.mdx b/evm/evm-parity/transaction-types.mdx index badc49d..dd7ead2 100644 --- a/evm/evm-parity/transaction-types.mdx +++ b/evm/evm-parity/transaction-types.mdx @@ -23,6 +23,18 @@ Type 4 (EIP-7702) SetCode transactions must include a non-empty authorization li Each authorization entry must also carry a valid (non-nil) chain ID. If you are constructing SetCode transactions directly, ensure at least one authorization is present before submitting. + +### Access List and Auth List Entry Validation + +EVM transactions now undergo stricter semantic validation during `ValidateBasic`. Malformed transactions that were previously accepted may now be rejected. + +**Access list entries (type 1 and type 2):** Each access list tuple is validated for well-formed hex encoding. Every address must be a canonical hex address of the correct length, and every storage key must be a canonical hex hash of the correct length. Entries with wrong-length or non-hex values are rejected. + +**Auth list entries (type 4):** In addition to the non-empty auth list and non-nil chain ID requirements above, each authorization entry is validated for a canonical hex address and well-formed signature values. + +**Signature values:** The `v`, `r`, and `s` signature values (for both the transaction itself and, for type 4, each auth list entry) must not exceed their maximum byte length and must not contain a leading zero byte. Non-canonical signature encodings are rejected. + + ## Not Supported | Type | EIP | Name | Notes | diff --git a/evm/installing-seid-cli.mdx b/evm/installing-seid-cli.mdx index d4d5dce..0a090fc 100644 --- a/evm/installing-seid-cli.mdx +++ b/evm/installing-seid-cli.mdx @@ -97,7 +97,6 @@ Available Commands: add-wasm-genesis-message Wasm genesis subcommands blocktest run EF blocktest collect-gentxs Collect genesis txs and output a genesis.json file - compact Compact the application DB fully (only if it is a levelDB) config Create or query an application CLI configuration file debug Tool for helping with debugging your application ethreplay replay EVM transactions @@ -106,9 +105,7 @@ Available Commands: help Help about any command init Initialize private validator, p2p, genesis, and application configuration files keys Manage your application's keys - latest_version Prints the latest version of the app DB migrate Migrate genesis to a specified target version - prune Prune app history states by keeping the recent heights and deleting old heights query Querying subcommands rollback rollback cosmos-sdk and tendermint state by one height start Run the full node diff --git a/evm/precompiles/distribution.mdx b/evm/precompiles/distribution.mdx index 877bd1f..666a9bb 100644 --- a/evm/precompiles/distribution.mdx +++ b/evm/precompiles/distribution.mdx @@ -164,6 +164,12 @@ bool success = DISTR_CONTRACT.setWithdrawAddress(0x742d35Cc6634C0532925a3b8D4C9d require(success, "Failed to set withdraw address"); ``` + + **Recipient Validation**: The withdrawal address must be able to receive external funds. If the target address cannot receive external funds — for example, an unassociated EVM-cast address — the call is rejected with `ErrInvalidRecipient` and fails. Ensure the withdrawal address is associated (or otherwise permitted to receive funds) before setting it. + + Additionally, if a previously set withdrawal address later becomes invalid (blocked or no longer able to receive external funds), reward withdrawals automatically fall back to sending rewards to the delegator's own address instead. + + ### withdrawDelegationRewards Withdraws accumulated rewards from a specific validator. diff --git a/evm/precompiles/oracle.mdx b/evm/precompiles/oracle.mdx index 6ee1132..778c47b 100644 --- a/evm/precompiles/oracle.mdx +++ b/evm/precompiles/oracle.mdx @@ -6,4 +6,4 @@ keywords: ['oracle precompile', 'ethers.js', 'price feeds', 'exchange rates', 't --- **Address:** `0x0000000000000000000000000000000000001008` -**Deprecation Notice:** The native Sei Oracle is deprecated and will be shut off soon. We strongly recommend migrating to one of the third-party oracle providers, such as [Chainlink](/evm/oracles/chainlink), [Pyth](/evm/oracles/pyth-network), [Redstone](/evm/oracles/redstone), or [API3](/evm/oracles/api3). +**Retired as of v6.6:** The native Sei Oracle precompile has been retired. On-chain oracle data queries are disabled — calling `getExchangeRates` or `getOracleTwaps` now reverts with the error `oracle precompile is retired; oracle data queries are disabled`. You must migrate to one of the third-party oracle providers, such as [Chainlink](/evm/oracles/chainlink), [Pyth](/evm/oracles/pyth-network), [Redstone](/evm/oracles/redstone), or [API3](/evm/oracles/api3). diff --git a/evm/reference.mdx b/evm/reference.mdx index aef1579..1e66aaa 100644 --- a/evm/reference.mdx +++ b/evm/reference.mdx @@ -59,6 +59,50 @@ All endpoints follow the standard JSON-RPC format: For additional public and commercial endpoints, see [RPC providers](/learn/rpc-providers) and [Chains & endpoints](/learn/dev-chains). + +### Tendermint `/status` endpoint + +Alongside the EVM JSON-RPC surface, Sei nodes expose the underlying Tendermint/CometBFT RPC, including the `/status` endpoint. Its `SyncInfo` object reports a `last_committed_block_height` field: the height of the last block finalized by consensus. + +```bash +curl -s $SEI_TENDERMINT_RPC/status | jq '.result.sync_info.last_committed_block_height' +``` + +The field is JSON-serialized as a string (`last_committed_block_height`), matching the other height fields in `SyncInfo`. + +| Field | JSON | Description | +| :- | :- | :- | +| `last_committed_block_height` | string | Height of the last block finalized by consensus. | + +**Consensus-engine behavior:** + +- **Under CometBFT** — commit and app-apply happen in a single step, so `last_committed_block_height` is guaranteed to equal `latest_block_height`. +- **Under Autobahn** — the value is derived from the latest `CommitQC`. Because consensus finalizes a block before the app executes it, the invariant is `last_committed_block_height >= latest_block_height`; the two can briefly differ while the app catches up. + + +Under Autobahn (`AutobahnConfigFile` set) the CometBFT block store is not populated, so `/status` derives `latest_block_height` and `latest_app_hash` from the app layer (`ABCIInfo`) instead of the block store. Several other `SyncInfo` fields (block hash/time, earliest-block metadata, `catching_up`, and peer-height fields) remain unpopulated in this mode. + + + + +### Tendermint block and validator endpoints under Autobahn + +Under Autobahn (`AutobahnConfigFile` set) the CometBFT `BlockStore` and `StateStore` are not populated. Rather than returning empty or failing responses, the `/block`, `/block_by_hash`, `/block_results`, and `/validators` Tendermint RPC endpoints route through the GigaRouter's in-memory state (the finalized global blocks retained by the Autobahn data layer). This keeps downstream consumers — including evmrpc, block explorers, and monitoring tooling — working without individually branching on the consensus engine. + +Responses are served only for heights still inside Autobahn's retained window (heights pruned per `RetainHeight` are no longer available). Requests are validated against the current ABCI head, so out-of-range heights return the same `ErrHeightNotAvailable`/`ErrHeightExceedsChainHead`-class errors as the CometBFT path. + +| Endpoint | Autobahn behavior | +| :- | :- | +| `/block` | Returns the fully-populated translated block at the requested height: `BlockID.Hash` (the Autobahn header hash), the header (`ChainID`, `Height`, `Time`), and `Data.Txs`. Other header fields (`AppHash`, `ProposerAddress`, `LastCommit`, …) stay at zero values. | +| `/block_by_hash` | Resolves the block by its Autobahn header hash via an in-memory hash index. An unknown or wrong-size hash returns `{Block: nil}` with no error, matching CometBFT semantics for a missing block. | +| `/block_results` | Returns a valid-but-empty `ResultBlockResults` at the requested height. `ConsensusParamUpdates.Block.MaxGas` is populated from the producer config; `TxsResults` (per-tx `ExecTxResult` details) is intentionally empty because `FinalizeBlock` responses are not persisted under Autobahn. Populating these is a separate follow-up. | +| `/validators` | Returns the genesis committee at any retained height — the committee is fixed at genesis under Autobahn (no validator-updates path). `block_height` matches the requested height, and pagination (`page`/`per_page`) behaves the same as the CometBFT path. | + + +Because `FinalizeBlock` responses are not stored on disk under Autobahn, `/block_results` cannot surface per-transaction execution results (`TxsResults` is empty). For per-transaction EVM data, use the EVM JSON-RPC methods (`eth_getTransactionReceipt`, `eth_getBlockReceipts`) instead. + + + ### Filter and subscription limits @@ -81,6 +125,8 @@ Every method below is also browsable interactively in the explorer above; this s **Sei-specific behavior:** Decodes to an ethtypes.Transaction, wraps it in a Cosmos MsgEVMTransaction, and broadcasts via CometBFT (async BroadcastTx by default; slow mode uses BroadcastTxCommit). Non-zero CheckTx codes surface as ABCI errors, not geth mempool errors. Legacy (non-1559) txs must set gasPrice at or above the governance minimum (currently 50 gwei on mainnet); blob (EIP-4844) txs are not enabled. Supports per-sender EvmProxy forwarding. +**Under Autobahn (`AutobahnConfigFile` set):** transaction broadcast routes through Autobahn's producer-backed mempool instead of CometBFT's `TxMempool`. This mempool admits EVM transactions strictly in sequential per-sender nonce order: a transaction whose nonce does not match the sender's next expected nonce is rejected with a bad-nonce error, so senders must submit nonces contiguously. Synchronous broadcast (`broadcast_tx` / `BroadcastTx`) blocks while the mempool is full and only returns once capacity is available, whereas the async path may drop the transaction if the mempool is full. The `unsafe_flush_mempool` Tendermint RPC endpoint is not supported under Autobahn and returns `unsafe_flush_mempool is not supported with autobahn mempool`. + **Parameters:** | # | Name | Type | Description | @@ -195,7 +241,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns the EVM transaction matching the given hash, or null if not found. -**Sei-specific behavior:** Sees EVM transactions only; if the hash resolves to a non-EVM Cosmos tx it errors. Pending lookups read from the CometBFT mempool, not a geth txpool. Use the legacy sei_getTransactionByHash to also surface Cosmos txs with synthetic representations. +**Sei-specific behavior:** Sees EVM transactions only; if the hash resolves to a non-EVM Cosmos tx it errors. Pending lookups resolve the transaction directly from the CometBFT mempool via an EVM-hash index (rather than a geth txpool), so a pending EVM tx is found by its hash without scanning through unconfirmed-transaction pages. Use the legacy sei_getTransactionByHash to also surface Cosmos txs with synthetic representations. **Parameters:** @@ -220,7 +266,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns the receipt of a transaction by hash, or null if not found. -**Sei-specific behavior:** Receipt is reconstructed from keeper.GetReceipt + CometBFT block data rather than from a native MPT receipt trie; status/logs are standard Ethereum format. +**Sei-specific behavior:** Receipt is reconstructed from keeper.GetReceipt + CometBFT block data rather than from a native MPT receipt trie; status/logs are standard Ethereum format. An EVM transaction that bumps the sender's nonce but then fails during state transition — for example an EIP-7623 floor-data-gas shortfall (post-Pectra), which fails inside go-ethereum's `Execute()` before any opcode runs — now returns a `status=0` failed-tx receipt (with `gasUsed` equal to the tx's `gasLimit` and a populated `VmError` reason) under the Giga executor, rather than returning null indefinitely and hanging clients that poll for it. If a receipt exists but its block height is above the safe-latest watermark (for example when Tendermint status momentarily lags the receipt store), the method returns JSON null rather than an error — the Ethereum JSON-RPC 'not yet mined' signal — so clients simply poll again, matching `eth_getBlockByNumber` behavior. **Parameters:** @@ -485,7 +531,7 @@ Every method below is also browsable interactively in the explorer above; this s **Limited.** Returns a Merkle proof for an account and the requested storage slots. -**Sei-specific behavior:** Sei stores state in an IAVL tree, not an Ethereum Merkle-Patricia trie. The result is a Sei-specific ProofResult{address, hexValues, storageProof} where storageProof entries are CometBFT/IAVL crypto.ProofOps, NOT eth-style MPT proof nodes. There is no accountProof, balance, codeHash, nonce, or storageHash field (Sei has no per-account state root); standard eth_getProof verifiers will not work. +**Sei-specific behavior:** Sei stores state in an IAVL-style tree, not an Ethereum Merkle-Patricia trie. The handler unwraps the EVM store through any intervening wrappers (cache, tracing, Giga cache, prefix stores) until it reaches an underlying proof-capable queryable store — classic IAVL, a store/v2 memiavl commitment, or any other proof-capable root — so proofs work across these backends rather than only classic IAVL. If no proof-capable queryable store can be reached it returns `cannot find a proof-capable queryable KV store`. The result is a Sei-specific ProofResult{address, hexValues, storageProof} where storageProof entries are CometBFT/IAVL crypto.ProofOps, NOT eth-style MPT proof nodes. There is no accountProof, balance, codeHash, nonce, or storageHash field (Sei has no per-account state root); standard eth_getProof verifiers will not work. **Parameters:** @@ -554,7 +600,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns block information by block hash, with full transactions when fullTx is true. -**Sei-specific behavior:** Block hashes are CometBFT block hashes (computed from the Tendermint header), so they differ from Ethereum block hashes and are not interchangeable across chains. Under the eth namespace synthetic txs and bank transfers are excluded. The genesis hash returns a synthetic genesis block; unknown/zero hash returns null. Uncles array is always empty. +**Sei-specific behavior:** Block hashes are CometBFT block hashes (computed from the Tendermint header), so they differ from Ethereum block hashes and are not interchangeable across chains. Under the eth namespace synthetic txs and bank transfers are excluded. The genesis block hash (`0xF9D3845DF25B43B1C6926F3CEDA6845C17F5624E12212FD8847D0BA01DA1AB9E`) is recognized and returns the encoded genesis block directly, keeping hash-based lookups consistent with `eth_getBlockByNumber("0x0")`; other unknown/zero hashes return null. Uncles array is always empty. **Parameters:** @@ -581,7 +627,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns the number of EVM transactions in a block by number, as a hex quantity. -**Sei-specific behavior:** Counts EVM transactions only (via getEvmTxCount); synthetic/bank-transfer txs are excluded. Genesis returns 0x0; non-existent/future blocks return null. +**Sei-specific behavior:** Counts the same transactions that appear in `eth_getBlockByNumber`'s transaction list: EVM transactions are counted only when a receipt exists for them, and `MsgExecuteContract` (wasm execute-contract) and `MsgSend` (bank send) messages are also included. Genesis returns 0x0; non-existent/future blocks return null. Because the receipt store can be configured with a smaller `KeepRecent` than the block/state stores, this method now verifies that the requested block's receipts have not been pruned before counting: if they have, it returns an error of the form `requested height N receipts have been pruned; earliest available is M` rather than a count. This means the call can fail for older blocks even when the block data itself is still available. **Parameters:** @@ -631,7 +677,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns all EVM transaction receipts for a given block. -**Sei-specific behavior:** Under the eth namespace synthetic/shell receipts are excluded (includeShellReceipts=false). Genesis returns an empty array; zero hash returns null. transactionIndex is recomputed sequentially over the compacted receipt list. +**Sei-specific behavior:** Under the eth namespace synthetic/shell receipts are excluded (includeShellReceipts=false). Genesis returns an empty array. An empty (zero) or non-existent/unknown block hash returns `result: null` rather than an error, matching the Ethereum RPC spec. transactionIndex is recomputed sequentially over the compacted receipt list. **Parameters:** @@ -699,7 +745,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns base fees, gas-used ratios, and reward percentile data over a range of blocks. -**Sei-specific behavior:** Base fees and rewards reflect Sei's x/evm fee market (GetNextBaseFee), not an Ethereum EIP-1559 mempool, and Sei does not burn the base fee. Watermark-aware: pruned/historical blocks may not be available as far back as on Ethereum archive nodes. +**Sei-specific behavior:** Base fees and rewards reflect Sei's x/evm fee market (GetNextBaseFee), not an Ethereum EIP-1559 mempool, and Sei does not burn the base fee. Watermark-aware: pruned/historical blocks may not be available as far back as on Ethereum archive nodes. Matching go-ethereum/execution-apis semantics, `baseFeePerGas` contains one more element than `gasUsedRatio` (`len(baseFeePerGas) == len(gasUsedRatio) + 1`): the trailing element is the projected base fee for the child of the newest block in the range. Each block's base fee uses header base fee semantics (the same value reported in the block header — `GetNextBaseFee` at the parent's committed height) with a `DefaultMinFeePerGas` fallback for early blocks. Note that when some heights in the range have pruned/partial base-fee data, `baseFeePerGas` may still be shorter than `gasUsedRatio + 1` for those rows. **Parameters:** @@ -786,6 +832,8 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Polls a filter and returns new logs (log filters) or block hashes (block filters) since the last poll. +**Sei-specific behavior:** For log filters, returns an empty array (`[]`) rather than `null` when no logs match or a bounded filter's block range has been fully consumed, in line with the Ethereum JSON-RPC spec. + **Parameters:** | # | Name | Type | Description | @@ -1089,7 +1137,7 @@ The `filter` object applies only to `logs` subscriptions. For `newHeads`, pass t **Supported.** Replays a transaction by hash and returns an execution trace using the configured tracer. -**Sei-specific behavior:** HTTP-only (the debug namespace is not registered on the WebSocket server). Supports geth tracers (callTracer, prestateTracer, flatCallTracer, struct/opcode logger); callTracer/prestateTracer/flatCallTracer results are pre-baked/cached via TraceBaker. Requires trace-enabled/archive state for the target height. +**Sei-specific behavior:** HTTP-only (the debug namespace is not registered on the WebSocket server). Supports geth tracers (callTracer, prestateTracer, flatCallTracer, struct/opcode logger); callTracer/prestateTracer/flatCallTracer results are pre-baked/cached via TraceBaker. Requires trace-enabled/archive state for the target height. Subject to the max block lookback guard (`max_trace_lookback_blocks`): a request whose target block is older than the configured lookback is rejected with an error of the form `block number X is beyond max lookback of Y`. This guard now applies consistently across all `debug_trace*` endpoints, and such attempts increment the `evmrpc_historical_debug_trace_attempts_total` metric. **Parameters:** @@ -1231,6 +1279,45 @@ The `filter` object applies only to `logs` subscriptions. For `newHeads`, pass t } ``` + + +#### `debug_traceTransactionProfile` + +**Limited.** Sei extension that replays a transaction by hash and returns its execution trace alongside a detailed timing and store-access profile. + +**Sei-specific behavior:** Sei-specific extension (not part of upstream go-ethereum's debug namespace). HTTP-only and subject to historical-debug-trace availability guards. In addition to the standard trace result, it returns a `profile` object breaking down where time was spent — total wall time, historical DB lookup time, and per-phase timings (transaction lookup, block load, historical tx replay, block-context build, tx prepare, execution, and trace-result assembly) — plus a per-module `store` access trace (reads, iterators with the keys they surfaced, and per-operation `stats` roll-ups). Per-tx caps bound the trace size: at most 16 iterators and 64 keys per iterator are retained per module, with overflow flagged via `truncated`. To run this method across a whole block range and generate aggregate reports, use the `seidb trace-profile-report` command. + +**Parameters:** + +| # | Name | Type | Description | +| :- | :- | :- | :- | +| 1 | `hash` | DATA, 32 bytes | Transaction hash to trace and profile. | +| 2 | `config` | object | Optional tracer config (tracer name, tracerConfig, timeout, reexec, disableStorage/Stack/Memory). | + +The response `result` contains a `trace` field (the standard tracer output) and a `profile` object shaped as follows: + +- `totalNanos` — total wall-clock nanoseconds for the profiled trace. +- `historicalDbLookupNanos` — nanoseconds spent in historical store lookups (get/has/iterator/iteratorNext). +- `otherNanos` — remaining time not attributed to historical lookups or execution. +- `phases` — per-phase timings: `lookupTransactionNanos`, `loadBlockNanos`, `replayHistoricalTxsNanos`, `buildBlockContextNanos`, `prepareTxNanos`, `executionNanos`, `traceResultNanos`. +- `store` — per-module store access trace: `modules` (each with `reads`, `has`, `iterators`, and per-op `stats`) and top-level `stats`. + +**Example request:** + +```json +{ + "jsonrpc": "2.0", + "id": 1, + "method": "debug_traceTransactionProfile", + "params": [ + "0x5c504ed432cb51138bcf09aa5e8a410dd4a1e204ef84bfed1be16dfba1b22060", + { + "timeout": "60s" + } + ] +} +``` + ## Sei Custom Endpoints @@ -1240,7 +1327,11 @@ Sei extends the standard Ethereum JSON-RPC API with custom endpoints that enhanc **Deprecation Notice:** All `sei_*` and `sei2_*` JSON-RPC methods are deprecated and scheduled for removal. Do not build new integrations on these endpoints. Use standard `eth_*` and `debug_*` methods instead. -Access is controlled by the `enabled_legacy_sei_apis` setting under `[evm]` in `app.toml`. Only methods explicitly listed in this allowlist are available. Disabled methods return a standard JSON-RPC error (code `-32601`, data `"legacy_sei_deprecated"`). Allowed methods pass through unchanged, with an optional `Sei-Legacy-RPC-Deprecation` HTTP response header signaling deprecation. +Access is controlled by the `enabled_legacy_sei_apis` setting under `[evm]` in `app.toml`. Only methods explicitly listed in this allowlist are available. Disabled methods return a standard JSON-RPC error (code `-32601`, data `"legacy_sei_deprecated"`). Allowed single-object calls pass through unchanged, with an optional `Sei-Legacy-RPC-Deprecation` HTTP response header signaling deprecation. + +**Batch requests:** JSON-RPC batches (a top-level JSON array) are handled by the gate rather than passed through wholesale. Only the allowed methods in the batch are forwarded to the inner handler as a filtered subset, and the inner responses are then merged back by matching each response's JSON-RPC `id`. Disallowed methods yield the usual `-32601` legacy deprecation error in their slot, and any batch element that is not a JSON object returns a JSON-RPC `-32600` `"Invalid Request"` error in its slot — a malformed element no longer causes the whole batch body to bypass the gate and reach the inner handler. + +Per JSON-RPC 2.0, **notifications** — requests that fully omit the `id` member — do not produce a response entry, including within a batch. A request that includes `"id": null` is **not** a notification: `null` is a valid id, so the request receives a response like any other id. Only requests with no `id` member at all are treated as notifications and omitted from the batch response. As a result, the merged response array is **not** 1:1 with the request batch when notifications are present — it is ordered like the batch minus the notification slots. If a batch would produce no response objects at all (for example, an all-notification batch, or an empty result), the gateway returns an **empty HTTP body** with HTTP 200 rather than an empty JSON array (`[]`). ### Legacy API Configuration @@ -1270,7 +1361,9 @@ To enable additional legacy methods, add them to this array. All other `sei_*` a | `sei_getTransactionErrorByHash` | Get error message for a failed transaction | | `sei_getVMError` | Get VM error details for a transaction | | `sei_getBlockByHash` | Get block by hash (includes synthetic txs) | +| `sei_getBlockByHashExcludeTraceFail` | Get block by hash, excluding untraceable txs — synthetic txs **and** ante-deferred stub transactions (correct-nonce ante failures such as insufficient funds/fee, whose stub receipts carry `EffectiveGasPrice == 0` and `GasUsed == 0`). Note this differs from the regular `eth_getBlockByHash`, which still includes these ante-failure stubs. | | `sei_getBlockByNumber` | Get block by number (includes synthetic txs) | +| `sei_getBlockByNumberExcludeTraceFail` | Get block by number, excluding untraceable txs — synthetic txs **and** ante-deferred stub transactions (correct-nonce ante failures such as insufficient funds/fee, whose stub receipts carry `EffectiveGasPrice == 0` and `GasUsed == 0`). Note this differs from the regular `eth_getBlockByNumber`, which still includes these ante-failure stubs. | | `sei_getBlockReceipts` | Get block receipts (includes synthetic txs) | | `sei_getBlockTransactionCountByHash` | Get tx count by block hash (includes synthetic txs) | | `sei_getBlockTransactionCountByNumber` | Get tx count by block number (includes synthetic txs) | @@ -1279,6 +1372,7 @@ To enable additional legacy methods, add them to this array. All other `sei_*` a | `sei_getTransactionByHash` | Get transaction by hash (includes synthetic txs) | | `sei_getTransactionCount` | Get account transaction count | | `sei_getTransactionReceipt` | Get transaction receipt (includes synthetic txs) | +| `sei_getTransactionReceiptExcludeTraceFail` | Get transaction receipt, excluding only txs that never executed. Ante-rejected txs (rejected before reaching the VM, identified by `EffectiveGasPrice=0`) and chain-generated synthetic txs (`ShellEVMTxType`) are filtered out; reverted and out-of-gas transactions are now included, because they ran in the VM and produce real traces. (Previously any receipt with `status=0` was excluded, which over-filtered reverts and OOG failures.) The `*ExcludeTraceFail` trace variants (e.g. `debug_traceBlockByHash`/`debug_traceBlockByNumber` exclude-trace-fail forms) apply the same filtering. | | `sei_getFilterLogs` | Get filter logs (includes synthetic logs) | | `sei_getLogs` | Get logs (includes synthetic logs) | | `sei_getFilterChanges` | Get filter changes (includes synthetic events) | @@ -1413,3 +1507,32 @@ Sei-Legacy-RPC-Deprecation: All sei_* and sei2_* JSON-RPC methods are deprecated ``` Clients can use this header to detect legacy API usage and plan migration. + + + +## Tendermint RPC status response + +The underlying Tendermint (CometBFT) RPC also exposes a `/status` endpoint whose response includes a `validator_info` object. This is separate from the EVM JSON-RPC surface but is often consumed by tooling such as CosmJS. + +The `validator_info` object always includes both a `pub_key` and an `address` field. For nodes that are not validators, `pub_key` is returned as a zero (empty) public key and `address` is derived from it, rather than the fields being omitted. This guarantees a stable response shape so that clients like CosmJS can parse the `/status` response without special-casing non-validator nodes. + +```json +{ + "jsonrpc": "2.0", + "id": 1, + "result": { + "validator_info": { + "address": "...", + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "..." + }, + "voting_power": "0" + } + } +} +``` + + +On validator nodes, `pub_key` contains the node's actual validator public key and `voting_power` reflects its voting power. On non-validator nodes, `pub_key` carries a zero public key and `voting_power` is `0`. + diff --git a/evm/tracing/index.mdx b/evm/tracing/index.mdx index 3473127..dfd4248 100644 --- a/evm/tracing/index.mdx +++ b/evm/tracing/index.mdx @@ -56,6 +56,7 @@ Debug tracing is your primary tool for understanding EVM transaction execution o | `debug_traceBlockByNumber` | Trace entire block | Block-level analysis | | `debug_traceCall` | Simulate and trace | Testing before execution | | `debug_traceStateAccess` | State access patterns | Performance optimization | +| `debug_traceTransactionProfile` | Trace plus timing/store-access profiling | Latency breakdown and DB-access analysis | ## Transaction Analysis Example @@ -408,6 +409,52 @@ debug_traceStateAccess(hash) - **`opcodeTracer`**: Opcode-level execution - **Custom JS**: Custom analysis logic + + +## Pre-Baked Trace Cache + +RPC nodes can optionally pre-compute and cache `debug_trace*` results in the background so that trace requests are served from a local on-disk cache instead of re-executing the block live on every call. This is an opt-in feature configured through new `[evm]` fields in `app.toml` and is recommended for RPC nodes only. + +When enabled, a background worker re-executes each committed block with the configured tracers and stores the results in a Pebble database at `/data/trace_db`. The following methods serve from this cache on hit, and otherwise fall through to live re-execution: + +- `debug_traceTransaction` +- `debug_traceBlockByNumber` and `debug_traceBlockByHash` + +### `*ExcludeTraceFail` filtering semantics + +The `*ExcludeTraceFail` endpoints (for example `sei_getTransactionReceiptExcludeTraceFail` and the `ExcludeTraceFail` block-trace variants) filter out transactions whose trace would be empty or meaningless — specifically, transactions that were included in a block but never actually executed in the VM. Two classes are excluded: + +- **Ante-deferred stub transactions**: txs that passed the nonce check but failed a later ante step (for example insufficient funds or insufficient fee) and so never reached the VM. The chain writes a stub receipt for these, identified by `EffectiveGasPrice == 0 && GasUsed == 0` (both fields unset because the tx never executed). +- **Chain-generated synthetic transactions**: txs with the internal `ShellEVMTxType`, which have no real EVM execution. + +Everything that ran in the VM is **included**, even if it failed. Reverted and out-of-gas (OOG) transactions have `Status == 0` but `EffectiveGasPrice > 0` and `GasUsed > 0`, and produce a real trace (the `REVERT` or OOG condition simply appears inside that trace), so they are returned rather than filtered out. + +This filter applies to both the block endpoints (`sei_getBlockByNumberExcludeTraceFail` and `sei_getBlockByHashExcludeTraceFail`) and the block-trace endpoints (`sei_traceBlockByNumberExcludeTraceFail` and `sei_traceBlockByHashExcludeTraceFail`): both now drop these ante-deferred stub transactions. The regular `eth_getBlockBy*` endpoints continue to **include** them, so a tx that appears in a normal block response may be absent from the `*ExcludeTraceFail` variant. The trace-side filter also catches stubs whose tracer embeds the error in the returned JSON while leaving the trace's `Error` field empty — it checks the underlying receipt shape rather than relying on `Error` alone. + +### When the cache is used + +A request is only served from cache when trace baking is enabled **and** the request uses a *bakeable* tracer configuration: + +- The tracer is one of `callTracer`, `prestateTracer`, or `flatCallTracer`. +- No custom `tracerConfig` is supplied. A per-call `tracerConfig` (for example `{"withLog": true}`) is not part of the cache key, so any custom tracer config makes the request un-bakeable and it falls through to live re-execution. + +Requests that use the struct logger (no tracer), a JavaScript tracer, or any other named tracer are always executed live. + +### Configuration + +Trace baking is controlled by these `[evm]` fields in `app.toml`: + +| Field | Default | Description | +| ----- | ------- | ----------- | +| `trace_bake_enabled` | `false` | Enables the background trace-baking worker that caches results at `/data/trace_db`. RPC nodes only. | +| `trace_bake_workers` | `1` | Number of re-execution worker goroutines. | +| `trace_bake_queue_size` | `4096` | Bounds the in-flight height queue. Heights are dropped when the queue is full so consensus never blocks; dropped blocks fall through to live re-execution. | +| `trace_bake_tracers` | `["callTracer"]` | Which tracers to bake per block. Eligible values: `callTracer`, `prestateTracer`, `flatCallTracer`. | +| `trace_bake_window_blocks` | `0` | Rolling prune window: blocks older than `(latest - window)` are pruned. `0` disables pruning, so the cache grows forever. | + +Enabling trace baking adds a persistent on-disk store at `/data/trace_db` and increases disk usage. The store's write-ahead log is flushed when the node shuts down cleanly. + + ## Next Steps 1. **[JavaScript Tracers](/evm/tracing/javascript-tracers)** - Custom analysis scripts diff --git a/evm/transactions.mdx b/evm/transactions.mdx index d9eacbe..7368ac9 100644 --- a/evm/transactions.mdx +++ b/evm/transactions.mdx @@ -285,6 +285,43 @@ EVM transactions in Sei follow the Ethereum transaction format with standard pro | Out of Gas | Gas limit too low for the operation | Use `eth_estimateGas` to set appropriate limit | | Contract Execution Failed | Contract function reverted | Test with `eth_call` before sending transaction | + +## Transaction Validation + +Before an EVM transaction is accepted, Sei performs strict semantic validation of the transaction's fields. Transactions that are malformed at the encoding level are rejected during basic validation, even if they were accepted by older node versions. If you construct and sign transactions manually, make sure they are well-formed to avoid rejection. + +The following checks are enforced: + +| Field | Requirement | +| --- | --- | +| Signature values (`v`, `r`, `s`) | Each value must not exceed its maximum byte length (32 bytes for `r`/`s`) and must not contain a leading zero byte. | +| `accessList` addresses | Each access-list address must be a valid, correctly-sized hex-encoded address. | +| `accessList` storage keys | Each storage key must be a valid, correctly-sized hex-encoded 32-byte hash. | +| Authorization list (EIP-7702 `setCode` txs) | Each entry must include a non-nil chain ID, a valid hex address, and well-formed `v`/`r`/`s` signature values. | + + + Transactions with malformed signature values, access-list entries, or authorization-list entries that previously passed validation may now be rejected. Ensure your signing tooling produces canonical, correctly-encoded values. + + +### Cosmos wrapper fields + +EVM transactions must not carry any Cosmos SDK wrapper fields. A transaction is rejected if it populates a memo, timeout height, extension options, signer infos, fee amount/payer/granter, or Cosmos-level signatures. This validation is applied uniformly across the transaction pipeline. + +Additionally, the transaction decoder now rejects "bloated" transaction bodies whose raw wire encoding is larger than the canonical re-marshaled form (for example, padded fields or oversized `Any.Value` entries). Non-canonical encodings that previously decoded successfully will now fail with a decode error, so tooling should always emit canonical protobuf encodings. + + +### Receipts for nonce-bumping failed transactions + +Some EVM transactions pass basic validation and bump the sender's nonce, yet still fail during state transition — for example, a transaction whose gas limit clears the intrinsic-gas check but falls short of the EIP-7623 floor-data-gas requirement (which can occur in normal operation after Pectra). Because these transactions bump the nonce, they are considered to have happened and therefore produce a receipt. + +For such failures, Sei writes a `status=0` (failed) receipt with `gasUsed` equal to the transaction's gas limit and a `VmError` describing the state-transition reason. `eth_getTransactionReceipt` returns this failed-tx receipt instead of `null`. + + + Previously, a nonce-bumping transaction that failed during state transition could return `null` from `eth_getTransactionReceipt` indefinitely, causing clients that poll for a receipt to hang. Clients should now expect a `status=0` receipt for these transactions and treat it as a normal failed-transaction result. + + + + ## Additional Resources diff --git a/node/advanced-config-monitoring.mdx b/node/advanced-config-monitoring.mdx index 4fc13d9..2c81b72 100644 --- a/node/advanced-config-monitoring.mdx +++ b/node/advanced-config-monitoring.mdx @@ -504,6 +504,194 @@ scrape_configs: - targets: ['localhost:9100'] ``` + + +## EVM RPC OpenTelemetry Metrics + +The EVM RPC layer emits OpenTelemetry metrics through the process-wide `MeterProvider` (for example, a Prometheus exporter). These are emitted in parallel with the legacy `sei_*` metrics so you can migrate dashboards incrementally. + +### Available Metrics + +| Metric | Type | Description | +| --- | --- | --- | +| `evmrpc_request_latency_seconds` | Histogram | EVM RPC request latency in seconds. | +| `evmrpc_websocket_connects_total` | Counter | Number of new websocket connections. | +| `evmrpc_redirected_requests_total` | Counter | Number of EVM RPC requests forwarded to another validator. Labeled by `endpoint` and `connection`. | +| `evmrpc_historical_debug_trace_attempts_total` | Counter | Number of `debug_trace*` requests targeting historical blocks beyond the configured max block lookback. Labeled by `endpoint` and `connection`. | + +The `evmrpc_request_latency_seconds` histogram carries the following labels: + +| Label | Description | +| --- | --- | +| `endpoint` | The RPC method being served (for example, `eth_getBalance`). | +| `connection` | The connection type serving the request (for example, `http` or `websocket`). | +| `success` | Boolean indicating whether the request succeeded. | +| `error_class` | A low-cardinality classification of the failure. An empty string denotes success. Possible values include `panic`, `execution_reverted`, `evm_not_supported`, `sei_legacy_disabled`, `association_missing`, `jsonrpc_error`, and `unknown`. | +| `jsonrpc_code` | A low-cardinality bucket for the JSON-RPC error code: `spec` (predefined range `-32700..-32600`), `server` (server-defined range `-32099..-32000`), or `other`. An empty string denotes no code (success or an untyped error). | + +### Migrating from Legacy Metrics + +The following legacy `sei_*` metrics remain available today but are deprecated and scheduled for removal once dashboards migrate to the `evmrpc_*` OpenTelemetry metrics: + +| Legacy metric | Replacement | +| --- | --- | +| `sei_rpc_request_latency_ms` | `evmrpc_request_latency_seconds` | +| `sei_websocket_connect` | `evmrpc_websocket_connects_total` | +| `sei_rpc_request` | `evmrpc_request_latency_seconds` (use the histogram count with the `success` and `error_class` labels) | + +Update your Prometheus and Grafana dashboards to consume the `evmrpc_*` metrics before the legacy metrics are removed. Note that latency changed units from milliseconds (`sei_rpc_request_latency_ms`) to seconds (`evmrpc_request_latency_seconds`), so adjust any thresholds and panel formatting accordingly. + + + +## FlatKV OpenTelemetry Metrics + +The FlatKV state store emits OpenTelemetry metrics through the process-wide `MeterProvider` (for example, a Prometheus exporter). These metrics let node operators observe commit throughput, catchup progress, snapshotting, rollbacks, and snapshot imports. + +### Available Metrics + +| Metric | Type | Description | +| --- | --- | --- | +| `flatkv_open_latency` | Histogram | Time taken to open the FlatKV store (seconds). | +| `flatkv_apply_changesets_latency` | Histogram | Time taken to apply changesets to FlatKV (seconds). | +| `flatkv_commit_latency` | Histogram | Time taken to commit FlatKV changes (seconds). | +| `flatkv_commit_batch_latency` | Histogram | Time taken to commit a FlatKV data DB batch (seconds). | +| `flatkv_batch_read_old_values_latency` | Histogram | Time taken to batch read old FlatKV values (seconds). | +| `flatkv_num_kv_pairs` | Counter | Number of key-value pairs applied to FlatKV. | +| `flatkv_pending_writes` | Gauge | Current number of pending FlatKV writes. | +| `flatkv_current_version` | Gauge | Current committed FlatKV version. | +| `flatkv_catchup_latency` | Histogram | Time taken to replay FlatKV WAL entries (seconds). | +| `flatkv_catchup_replay_num_blocks` | Counter | Number of FlatKV WAL entries replayed during catchup. | +| `flatkv_snapshot_write_latency` | Histogram | Time taken to write a FlatKV snapshot (seconds). | +| `flatkv_snapshot_prune_latency` | Histogram | Time taken to prune FlatKV snapshots (seconds). | +| `flatkv_snapshot_prune_attempts` | Counter | Total number of FlatKV snapshot prune attempts. | +| `flatkv_current_snapshot_height` | Gauge | Current FlatKV snapshot height. | +| `flatkv_rollback_latency` | Histogram | Time taken to rollback FlatKV state (seconds). | +| `flatkv_import_latency` | Histogram | Time taken to import FlatKV snapshot data (seconds). | +| `flatkv_import_kv_pairs` | Counter | Number of key-value pairs imported into FlatKV. | +| `flatkv_import_worker_flush_latency` | Histogram | Time taken to flush a FlatKV import worker batch (seconds). | +| `flatkv_flush_latency` | Histogram | Time taken to flush a FlatKV data DB (seconds). | + +### Labels + +FlatKV metrics carry the following labels where applicable: + +| Label | Description | +| --- | --- | +| `db` | The data DB the measurement applies to (for example, `accountDB`, `storageDB`, `codeDB`, or `legacyDB`). Present on per-DB metrics such as `flatkv_commit_batch_latency`, `flatkv_flush_latency`, `flatkv_num_kv_pairs`, `flatkv_pending_writes`, `flatkv_import_kv_pairs`, and `flatkv_import_worker_flush_latency`. | +| `success` | Boolean indicating whether the operation succeeded. Present on latency and attempt metrics that can fail. | +| `read_only` | Boolean present on `flatkv_open_latency` indicating whether the store was opened read-only. | + +### Enabling Pebble Internal Metrics + +Pebble's internal (per-DB) metrics are governed by a single FlatKV-level knob, `EnablePebbleMetrics`. When set, this value is propagated to every data DB (account, code, storage, legacy, and metadata) during initialization and overrides any per-DB `EnableMetrics` settings, so configure Pebble metrics through `EnablePebbleMetrics` rather than the individual per-DB knobs. + + + +## LittDB OpenTelemetry Metrics + +LittDB now emits its metrics through the process-wide OpenTelemetry `MeterProvider` instead of a private Prometheus client. When `MetricsEnabled` is set, LittDB configures a Prometheus exporter on the global provider and serves `/metrics` on `MetricsPort` (default `9101`). The previous `MetricsNamespace` and `MetricsRegistry` config fields have been removed; all metric names now use a fixed `litt_` prefix. + +### Available Metrics + +| Metric | Type | Unit | Description | +| --- | --- | --- | --- | +| `litt_table_size_bytes` | Gauge | bytes | The size of individual tables in the database. | +| `litt_table_key_count` | Gauge | count | The number of keys in individual tables in the database. | +| `litt_open_iterator_count` | Gauge | count | The number of currently-open iterators for individual tables in the database. A persistently nonzero value indicates a leaked iterator, which suspends garbage collection for the table. | +| `litt_bytes_read` | Counter | bytes | The number of bytes read from disk since startup. | +| `litt_keys_read` | Counter | count | The number of keys read from disk since startup. | +| `litt_cache_hits` | Counter | count | The number of cache hits since startup. | +| `litt_cache_misses` | Counter | count | The number of cache misses since startup. | +| `litt_read_latency_seconds` | Histogram | seconds | Read latency of the database, including both cache hits and cache misses. | +| `litt_cache_miss_latency_seconds` | Histogram | seconds | Read latency measured only when a cache miss occurs. | +| `litt_bytes_written` | Counter | bytes | The number of bytes written to disk since startup (values only, not metadata). | +| `litt_keys_written` | Counter | count | The number of keys written to disk since startup. | +| `litt_write_latency_seconds` | Histogram | seconds | Write latency of the database. | +| `litt_flush_count` | Counter | count | The number of times a flush operation has been performed. | +| `litt_flush_latency_seconds` | Histogram | seconds | Latency of a flush operation. | +| `litt_segment_flush_latency_seconds` | Histogram | seconds | Segment flush latency; a subset of the time spent during a flush operation. | +| `litt_keymap_flush_latency_seconds` | Histogram | seconds | Keymap flush latency; a subset of the time spent during a flush operation. | +| `litt_garbage_collection_latency_seconds` | Histogram | seconds | Latency of garbage collection operations. | +| `litt_chunk_cache_key_count` | Gauge | count | The number of keys in the chunk cache. | +| `litt_chunk_cache_weight_bytes` | Gauge | bytes | The weight of the chunk cache in bytes. | +| `litt_chunk_cache_keys_added` | Counter | count | The number of keys added to the chunk cache. | +| `litt_chunk_cache_weight_added_bytes` | Counter | bytes | The weight of the entries added to the chunk cache. | +| `litt_chunk_cache_eviction_latency_seconds` | Histogram | seconds | Eviction latency of the chunk cache. | + +### Attributes + +| Attribute | Description | +| --- | --- | +| `table` | The table the observation applies to. Present on per-table metrics such as `litt_bytes_read`, `litt_read_latency_seconds`, `litt_table_size_bytes`, and the flush/GC latency histograms. | +| `cache` | The cache instance the observation applies to (`chunk_read` or `chunk_write`). Present on the `litt_chunk_cache_*` metrics, which distinguish read and write caches by this attribute rather than by separate metric names. | + +### Migrating from Legacy Metrics + +Metric names, units, and shape changed with the OpenTelemetry migration, so existing Prometheus and Grafana dashboards must be updated: + +- Latency metrics moved from millisecond summaries (for example `{namespace}_read_latency_ms`) to second histograms (`litt_read_latency_seconds`). Adjust thresholds and panel formatting from milliseconds to seconds accordingly. +- Counters and gauges gained a fixed `litt_` prefix and explicit units, for example `bytes_read` became `litt_bytes_read` and the cache weight gauge became `litt_chunk_cache_weight_bytes`. +- The per-cache series that were previously separate metric names (for example `chunk_read_cache_*` and `chunk_write_cache_*`) are now the shared `litt_chunk_cache_*` metrics distinguished by the `cache` attribute. +- The `MetricsNamespace` and `MetricsRegistry` config fields no longer exist. Metric names are fixed, and metrics are always backed by the global OTel provider; supply the scrape port via `MetricsPort`. + + + +## IBC OpenTelemetry Metrics + +The IBC modules emit OpenTelemetry metrics through the process-wide `MeterProvider` (for example, a Prometheus exporter). These are emitted in parallel with the legacy `ibc_*` and `tx_msg_*` telemetry counters so you can migrate dashboards incrementally. The metrics are grouped by meter, one per IBC module. + +### Transfer Metrics (`ibc_transfer_keeper` meter) + +| Metric | Type | Description | +| --- | --- | --- | +| `ibc_transfer_tx_msg` | Gauge | Last amount of tokens transferred via IBC per denom class. | +| `ibc_transfer_packet_receive` | Gauge | Last amount of tokens received in an IBC packet per denom class. | +| `ibc_transfer_send` | Counter | Total number of IBC transfers sent. | +| `ibc_transfer_receive` | Counter | Total number of IBC transfers received. | + +The transfer gauges carry a `denom_class` attribute. `ibc_transfer_send` carries `destination_port`, `destination_channel`, and a boolean `source` attribute indicating whether the sending chain is the token source. `ibc_transfer_receive` carries `source_port`, `source_channel`, and the boolean `source` attribute. + +### Core Client Metrics (`ibc_core_client_keeper` meter) + +| Metric | Type | Description | +| --- | --- | --- | +| `ibc_client_create` | Counter | Total number of IBC client creates. | +| `ibc_client_update` | Counter | Total number of IBC client updates. | +| `ibc_client_upgrade` | Counter | Total number of IBC client upgrades. | +| `ibc_client_misbehaviour` | Counter | Total number of IBC client misbehaviour events. | + +These metrics carry a `client_type` attribute. `ibc_client_update`, `ibc_client_upgrade`, and `ibc_client_misbehaviour` also carry a `client_id` attribute. `ibc_client_update` additionally carries an `update_type` attribute (`msg` or `proposal`). + +### Connection Metrics (`ibc_connection` meter) + +| Metric | Type | Description | +| --- | --- | --- | +| `ibc_connection_open_init` | Counter | Total number of IBC connection open-init handshakes. | +| `ibc_connection_open_try` | Counter | Total number of IBC connection open-try handshakes. | +| `ibc_connection_open_ack` | Counter | Total number of IBC connection open-ack handshakes. | +| `ibc_connection_open_confirm` | Counter | Total number of IBC connection open-confirm handshakes. | + +### Channel Metrics (`ibc_channel` meter) + +| Metric | Type | Description | +| --- | --- | --- | +| `ibc_channel_open_init` | Counter | Total number of IBC channel open-init handshakes. | +| `ibc_channel_open_try` | Counter | Total number of IBC channel open-try handshakes. | +| `ibc_channel_open_ack` | Counter | Total number of IBC channel open-ack handshakes. | +| `ibc_channel_open_confirm` | Counter | Total number of IBC channel open-confirm handshakes. | +| `ibc_channel_close_init` | Counter | Total number of IBC channel close-init handshakes. | +| `ibc_channel_close_confirm` | Counter | Total number of IBC channel close-confirm handshakes. | + +### Core Packet Metrics (`ibc_core` meter) + +| Metric | Type | Description | +| --- | --- | --- | +| `ibc_core_tx_msg_recv_packet` | Counter | Total number of IBC recv packet messages. | +| `ibc_core_timeout_packet` | Counter | Total number of IBC timeout packets. | +| `ibc_core_tx_msg_acknowledge_packet` | Counter | Total number of IBC acknowledge packet messages. | + +These packet metrics carry `source_port`, `source_channel`, `destination_port`, and `destination_channel` attributes. `ibc_core_timeout_packet` additionally carries a `timeout_type` attribute (`height` or `channel-closed`). + ## Performance Testing diff --git a/node/giga-storage-migration.mdx b/node/giga-storage-migration.mdx index 755f614..ca2bf3a 100644 --- a/node/giga-storage-migration.mdx +++ b/node/giga-storage-migration.mdx @@ -209,9 +209,117 @@ To fully reclaim the disk used by EVM SS, stop the node and delete ### Where do the data files live after migrating? -- Cosmos SS data lives under the same directory as before, typically - `data/pebbledb/` for the default `pebbledb` backend. -- EVM SS data lives under `data/evm_ss/`. +As of Sei v6.6, new nodes use a structured subdirectory layout. Existing nodes +with legacy flat paths automatically keep using them (the legacy path takes +precedence when present). + + + +## FlatKV EVM SC migration flow + +Everything above concerns the **SS** (State Store) layer. The **SC** (State +Commit) layer has its own, separate migration path that moves the hot `evm/` +data out of `memiavl` and into FlatKV in place, without a state sync. It is +driven entirely by `app.toml`'s `sc-write-mode` and is coordinated across a +quorum by stopping the nodes, editing config, and restarting. + +Unlike the SS split, the SC-side migration **does change how `evm/` data +contributes to the app hash** (memiavl IAVL root before the migration; FlatKV +lattice hash after). Because of that, every validator in a quorum must flip at +the same coordinated stop — a node flipped while its peers are still on the old +mode will produce a different AppHash on the very next block and consensus will +halt. The safe sequence is always: stop everyone, rewrite `app.toml` +everywhere, restart everyone. + +This SC-side FlatKV EVM migration flow is exercised by the cluster/devnet integration harness. Do not run it against testnet/mainnet nodes unless the release notes for your version explicitly call it out as supported. + +### Write modes + +The migration is a transition from the `memiavl_only` write mode (v0, where +memiavl is the sole SC backend and FlatKV is not allocated) to `migrate_evm` +(the in-flight mode that drains `evm/` keys from memiavl into FlatKV). Once the +migration completes, operators flip `sc-write-mode` again to `evm_migrated` so +subsequent restarts don't spin up the migration manager. + + + +Beyond `evm_migrated`, `flatkv_only` is the fully-supported **terminal** +steady-state write mode: FlatKV is the sole SC backend and `memiavl` is not +allocated at all. In this mode every module's SC state is served from FlatKV, +and state-sync snapshot export/restore plus app-hash parity work correctly, so +a node can boot directly into the post-migration shape without ever running the +migration manager. Use `flatkv_only` when you want a node to come up already in +the post-migration steady state rather than draining keys out of memiavl at +runtime. + +A correctness bug in the WAL replay path — where empty (zero-length) +values written with no delete flag were dropped on replay (catchup, read-only +clone, snapshot export, and state-sync restore), diverging the FlatKV state and +the consensus AppHash from the live chain — is fixed. Empty-value writes are now +preserved across a WAL round-trip and state-sync, which is what makes +`flatkv_only` state-sync reliable. + +While in a migration mode: + +- Caller reads of not-yet-migrated keys fall back to FlatKV for brand-new keys + written after the migration started, and to memiavl otherwise. +- Iteration is forwarded to memiavl (with a completeness caveat: keys already + migrated out of memiavl are skipped). This is only safe for best-effort + callers. +- The migration boundary advances at most once per block. + +### Operator-facing knobs + +**`sc-keys-to-migrate-per-block`** (`app.toml`, `[state-commit]` section) +controls how many EVM keys the in-flight migration drains from memiavl into +FlatKV per block. It defaults to `1024`, which is appropriate for production +drains. Lowering it spreads the migration across more blocks. It must be `> 0`, +and it is ignored entirely when `sc-write-mode` is not a migration mode. + +```toml copy +[state-commit] +sc-write-mode = "migrate_evm" +sc-keys-to-migrate-per-block = 1024 +``` + +**`GIGA_MIGRATE_FROM_MEMIAVL`** is a cluster/docker environment variable used by +the local devnet setup. When set to `true` it boots every node in +`memiavl_only` mode — the v0 starting point for the FlatKV EVM migrate flow. It +is mutually exclusive with `GIGA_STORAGE`; if both are set, +`GIGA_MIGRATE_FROM_MEMIAVL` takes precedence. + +```bash copy +GIGA_MIGRATE_FROM_MEMIAVL=true make docker-cluster-start +``` + +### Checking migration status + +The `seidb migrate-evm-status` subcommand reports the on-disk FlatKV EVM +migrate state of a FlatKV directory as JSON. It clones the latest snapshot and +WAL into a temp dir before reading, so it can be run against a live node's data +directory without contending for the FlatKV writer lock. + +```bash copy +seidb migrate-evm-status --db-dir [--height ] +``` + +`--db-dir` (short `-d`) points at the FlatKV data directory; `--height` selects +a target version (`0`, the default, selects the latest available version). The +emitted JSON includes `migrate_evm_complete` (true once the migration finishes), +`migration_version`, `version_at`, and whether an in-flight boundary is still +present. Poll it until `migrate_evm_complete` reports `true` on every validator +before flipping `sc-write-mode` to `evm_migrated`. + +When the migration finishes, each node also emits a `migration complete` summary +log line and a set of `seidb_migration_*` OpenTelemetry counters covering keys +and bytes migrated. + +- Cosmos SS data lives under `data/state_store/cosmos/{backend}` on new nodes + (e.g. `data/state_store/cosmos/pebbledb` for the default `pebbledb` backend). + Existing nodes with legacy `data/pebbledb/` continue using it. +- EVM SS data lives under `data/state_store/evm/{backend}` on new nodes + (e.g. `data/state_store/evm/pebbledb`). Existing nodes with legacy + `data/evm_ss/` continue using it. - SC data (`memiavl` + FlatKV) is untouched by this migration. ### Does Giga SS Store change the app hash or consensus? @@ -245,3 +353,18 @@ block this scenario at startup. No, same as SeiDB. SS stores raw KVs and does not reconstruct IAVL-style proofs. + + + +### Does enabling Giga Storage change the receipt backend? + +In the `localnode` and `rpcnode` configuration scripts, setting +`GIGA_STORAGE=true` defaults `RECEIPT_BACKEND` to `pebble` unless you set +`RECEIPT_BACKEND` explicitly. To use a different value while running with Giga +Storage, provide an explicit `RECEIPT_BACKEND` env var, which takes precedence +over the default. + +`pebbledb` (aka `pebble`) is now the only supported receipt-store backend. The +former `parquet` option has been removed: setting `RECEIPT_BACKEND=parquet` (or +`rs-backend = "parquet"` in `app.toml`) is rejected with an error +(`unsupported receipt-store backend; supported: pebbledb`). diff --git a/node/index.mdx b/node/index.mdx index dd84036..344f767 100644 --- a/node/index.mdx +++ b/node/index.mdx @@ -171,6 +171,18 @@ See the Network Versions table above for the current recommended version. ``` Available architectures: linux/amd64 and linux/arm64. + + + + + Some builds are published as testing-only image variants and must **never** be used in production. These are compiled with special Go build tags: + + - `mock_balances-*` — mocks account balances. + - `mock_block_validation-*` — bypasses `AppHash` and `DataHash` block validation. + - `mock_chain_validation-*` — swallows most halting consensus validation failures (every swallow-eligible check except `ErrLastCommitVerify`), computing each check authentically but continuing instead of halting. Combines `mock_balances` and `mock_chain_validation`. + + Production images are built with full validation enabled. The `mock_block_validation` and `mock_chain_validation` variants are intended solely for testing environments where relaxing validation checks is required; running either on a production network is unsafe. + @@ -295,11 +307,17 @@ pending-size = 5000 max-pending-txs-bytes = 1073741824 +# Deprecated: these fields no longer have any effect and are ignored. pending-ttl-duration = "3s" +# Deprecated: these fields no longer have any effect and are ignored. pending-ttl-num-blocks = 5 ``` + + `pending-ttl-duration` and `pending-ttl-num-blocks` are deprecated and now have no effect. They are ignored regardless of the value you set. + + diff --git a/node/node-operators.mdx b/node/node-operators.mdx index 386e9a5..53566ee 100644 --- a/node/node-operators.mdx +++ b/node/node-operators.mdx @@ -83,7 +83,9 @@ enable = true max-open-connections = 1000 [state-commit] -# SeiDB state-commit (memiavl + FlatKV). Recommended on every node. +# SeiDB state-commit (memiavl + FlatKV) is mandatory. The legacy IAVL backend +# has been fully removed; if sc-enable is false the node panics at startup with +# "SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated". sc-enable = true [state-store] @@ -93,10 +95,20 @@ ss-enable = true ss-keep-recent = 100000 [receipt-store] -# Storage backend for EVM transaction receipts (pebbledb or parquet). +# Storage backend for EVM transaction receipts. pebbledb (aka pebble) is the +# only supported backend. rs-backend = "pebbledb" ``` + +`pebbledb` (aka `pebble`) is the only supported receipt-store backend. Setting +`rs-backend = "parquet"` (or `RECEIPT_BACKEND=parquet`) is rejected with the +error `unsupported receipt-store backend; supported: pebbledb`. Enabling Giga +Storage (`GIGA_STORAGE=true`) in the localnode and rpcnode configuration scripts +defaults the receipt backend to `pebble` when `RECEIPT_BACKEND` is not set. The +former `receipt-store.tx-index-backend` config field has been removed. + + ### Default Configurations The full unmodified `app.toml`, `config.toml`, and `client.toml` produced by @@ -126,6 +138,8 @@ minimum-gas-prices = "0.01usei" # MinRetainBlocks defines the minimum block height offset from the current block # for pruning Tendermint blocks. Set to 0 to disable pruning. This only affects # Tendermint block pruning, not application state (see "pruning-*" configs). +# This value also drives receipt store retention: the receipt store's KeepRecent +# is derived from min-retain-blocks (0 means keep everything / no pruning). min-retain-blocks = 100000 # ConcurrencyWorkers defines how many workers to run for concurrent transaction execution. @@ -230,19 +244,6 @@ sc-snapshot-prefetch-threshold = 0.8 # Maximum snapshot write rate in MB/s (global across all trees). 0 = unlimited. Default 100. sc-snapshot-write-rate-mbps = 100 -# WriteMode defines the write routing mode for EVM data in the SC layer. -# Valid values: memiavl_only, migrate_evm, evm_migrated, migrate_all_but_bank, -# all_migrated_but_bank, migrate_bank, flatkv_only, test_only_dual_write -sc-write-mode = "memiavl_only" - -# KeysToMigratePerBlock controls how many EVM keys the in-flight migration -# (sc-write-mode = migrate_evm / migrate_bank / migrate_all_but_bank) drains -# from memiavl into flatkv per block. Default 1024 is appropriate for -# production drains; lower it (e.g. 256) to spread the migration across more -# blocks for test runs that need to observe the resume / hybrid-read path. -# Must be > 0; ignored entirely when not in a migration mode. -sc-keys-to-migrate-per-block = 1024 - ############################################################################### ### FlatKV (EVM) Configuration ### ############################################################################### @@ -304,7 +305,7 @@ ss-prune-interval = 600 ss-import-num-workers = 1 # EVMDBDirectory defines the directory for the optional EVM state-store DB(s). -# If unset, defaults to /data/evm_ss when EVM SS is enabled. +# If unset, defaults to /data/state_store/evm/{backend} when EVM SS is enabled. evm-ss-db-directory = "" # EVMSplit controls whether EVM data is routed to a dedicated SS backend. @@ -812,6 +813,12 @@ genesis-file = "config/genesis.json" # Path to the JSON file containing the private key to use for node authentication in the p2p protocol node-key-file = "config/node_key.json" +# NOTE: Out-of-process ABCI support has been removed. The full node now runs +# only with Tendermint in-process, so the legacy `proxy-app` and `abci` base +# config fields no longer appear in the generated template and have no effect. +# When upgrading, remove any `proxy-app` and `abci` lines from your existing +# config.toml. + ####################################################################### ### Advanced Configuration Options ### ####################################################################### @@ -963,6 +970,18 @@ upnp = false # Maximum number of connections (inbound and outbound). max-connections = 100 +# Maximum number of outbound connections to regular (non-persistent) peers. +# Inbound and outbound connections are now managed in separate pools, so this +# field caps the outbound pool while the inbound pool is derived as +# max-connections - max-outbound-connections. Persistent and unconditional +# peers bypass both limits. +# +# This field is optional: when it is left unset, max-outbound-connections +# defaults to 20, or to half of max-connections (rounded up) when +# max-connections is below 40. When it is set explicitly, the effective value +# is min(max-connections, max-outbound-connections). +# max-outbound-connections = 20 + # Rate limits the number of incoming connection attempts per IP address. max-incoming-connection-attempts = 100 @@ -1040,20 +1059,32 @@ max-tx-bytes = 1048576 # XXX: Unused due to https://github.com/tendermint/tendermint/issues/5796 max-batch-bytes = 0 -# ttl-duration, if non-zero, defines the maximum amount of time a transaction -# can exist for in the mempool. +# ttl-duration defines the maximum amount of time a transaction can exist in +# the mempool. A zero value now disables time-based TTL purging entirely +# (the field is treated as an optional/disabled setting when unset or zero). # # Note, if ttl-num-blocks is also defined, a transaction will be removed if it -# has existed in the mempool at least ttl-num-blocks number of blocks or if it's +# has existed in the mempool at least ttl-num-blocks number of blocks or if its # insertion time into the mempool is beyond ttl-duration. ttl-duration = "5s" -# ttl-num-blocks, if non-zero, defines the maximum number of blocks a transaction -# can exist for in the mempool. +# ttl-num-blocks defines the maximum number of blocks a transaction can exist +# for in the mempool. A zero value now disables block-based TTL purging entirely +# (the field is treated as an optional/disabled setting when unset or zero). # # Note, if ttl-duration is also defined, a transaction will be removed if it # has existed in the mempool at least ttl-num-blocks number of blocks or if -# it's insertion time into the mempool is beyond ttl-duration. +# its insertion time into the mempool is beyond ttl-duration. +# +# Expiration behavior: expired PENDING transactions are always pruned. Expired +# READY transactions are pruned only when keep-invalid-txs-in-cache-related +# removal is enabled via the mempool's RemoveExpiredTxsFromQueue setting. +# +# NOTE: The Tendermint mempool was substantially rewritten. Transaction +# admission, ordering, eviction, and gossip are now driven by a unified +# transaction store with inclusion-ordering, soft/hard size limits, and +# cache-based deduplication. After upgrading, node operators may observe +# different mempool ordering, eviction, and gossip behavior than before. ttl-num-blocks = 10 tx-notify-threshold = 0 @@ -1066,8 +1097,12 @@ pending-size = 5000 max-pending-txs-bytes = 1073741824 +# Deprecated: pending-ttl-duration is no longer used and this field has no +# effect. Pending transactions are no longer purged based on a time-based TTL. pending-ttl-duration = "0s" +# Deprecated: pending-ttl-num-blocks is no longer used and this field has no +# effect. Pending transactions are no longer purged based on a block-count TTL. pending-ttl-num-blocks = 0 # Defines the percentage of transactions with the lowest priority hint @@ -1181,9 +1216,22 @@ peer-query-maj23-sleep-duration = "2s" # For additional information, see ADR-74: # https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-074-timeout-params.md +# If false, all of the Unsafe*TimeoutOverride fields below are ignored. +# Defaults to false. +# +# As of v6.6.0, the timeout overrides below only take effect when this field is +# set to true, OR when the onchain timeout params still equal the legacy +# pacific-1 "badParams" values (a backward-compatibility carve-out during +# migration). When unsafe-overrides-enabled is false and the onchain timeout +# params differ from those legacy values, the overrides are ignored and the +# onchain (or default) timeout params are used instead. +unsafe-overrides-enabled = false + # This field provides an unsafe override of the Propose timeout consensus parameter. # This field configures how long the consensus engine will wait for a proposal block before prevoting nil. -# If this field is set to a value greater than 0, it will take effect. +# The override is applied only when unsafe-overrides-enabled = true (or the onchain +# timeout params still equal the legacy badParams values) and this field is set to a +# value greater than 0. unsafe-propose-timeout-override = "0s" # This field provides an unsafe override of the ProposeDelta timeout consensus parameter. diff --git a/node/technical-reference.mdx b/node/technical-reference.mdx index 186396a..4399a13 100644 --- a/node/technical-reference.mdx +++ b/node/technical-reference.mdx @@ -37,6 +37,155 @@ seid tendermint show-validator seid query node info ``` + + +### seidb Tooling Commands + +The `seidb` binary provides low-level tooling for inspecting and maintaining a node's on-disk state. + +#### Reporting FlatKV EVM Migration Status + +The `migrate-evm-status` subcommand reads the on-disk FlatKV EVM migration state from a FlatKV data directory and prints a JSON summary. It is primarily intended for integration and operator tooling that polls each validator to determine whether the FlatKV EVM migration has completed, without needing a custom RPC handler or having to grep through node logs. + +```bash +# Report FlatKV EVM migration status at the latest available version +seidb migrate-evm-status --db-dir + +# The --db-dir flag may be abbreviated as -d +seidb migrate-evm-status -d /root/.sei/data/state_commit/flatkv + +# Report status at a specific FlatKV version (0 selects the latest) +seidb migrate-evm-status --db-dir --height +``` + +The command opens FlatKV read-only — it hardlink-clones the latest snapshot and copies the WAL into a temporary directory before opening — so it can be run safely against a directory that a live node is still writing to. + +The emitted JSON contains the following fields: + +- `version_at` — the FlatKV version that was read. +- `migration_version` — the on-disk migration version (`0` means the FlatKV EVM migration has not yet completed). +- `migrate_evm_complete` — `true` once the migration version has reached the FlatKV EVM (v1) target. +- `boundary_present` — `true` while the migration is in flight (the in-progress resume cursor is still present). +- `boundary_hex` — hex-encoded migration boundary cursor, included only when a boundary is present. +- `version_raw_hex` — hex-encoded raw migration-version bytes, included only when a migration version is present. + + + +#### Comparing EVM State Across Backends + +The `evm-logical-digest` subcommand computes a backend-independent digest of the EVM logical state (the account, code, and storage buckets) so that a memIAVL node and a FlatKV node can be compared at the same chain height. Because a freshly migrated FlatKV node stamps a per-key `blockHeight` into each value that differs from the memIAVL leaf versions, a raw byte-for-byte digest would diverge even when the underlying EVM state is identical. This command strips the serialization-version and `blockHeight` header on both sides and digests only the height-independent logical payload (storage word, bytecode, or balance+nonce+codehash), producing a comparable `FINAL_DIGEST` per backend. + +```bash +# FlatKV digest at a height (WAL-replays to it). Prints per-bucket +# bucket_digest values and one FINAL_DIGEST line for backend comparison. +seidb evm-logical-digest --backend flatkv \ + --db-dir /root/.sei/data/state_commit/flatkv --height 213200000 + +# memIAVL digest at the same height (0 = current symlink), using the default +# semantic normalization. memiavl resolves snapshot-/evm or current/evm +# and does not replay WAL in this tool. +seidb evm-logical-digest --backend memiavl \ + --db-dir /root/.sei/data/state_commit/memiavl --height 213200000 + +# Translator-based memIAVL digest, which feeds each leaf through the current +# migration mapping (flatkv.ImportTranslator). +seidb evm-logical-digest --backend memiavl \ + --db-dir /root/.sei/data/state_commit/memiavl --height 213200000 \ + --memiavl-normalization translator +``` + +Two backends match when the FlatKV `FINAL_DIGEST` equals the memIAVL `FINAL_DIGEST`. FlatKV also writes an internal migration-version marker row that a memiavl-only node never owns, so the command omits that row from the final comparison automatically. + +The command accepts the following flags: + +- `--backend` — backend to read: `flatkv` or `memiavl`. +- `--db-dir` (`-d`) — for FlatKV, the FlatKV data directory; for memIAVL, the memIAVL root directory containing `current/` and `snapshot-*`. +- `--height` — target version. FlatKV WAL-replays to it; memIAVL resolves `snapshot-/evm` (`0` selects the `current` symlink). +- `--memiavl-normalization` — memIAVL normalization mode: `semantic`/`independent` (raw EVM key/value decoder, the default `semantic`) or `translator` (current migration mapping). +- `--inspect-bucket` — inspect one normalized bucket (`account`, `code`, `storage`, or `legacy`) instead of printing the global digest. +- `--key-offset` — inspect mode: byte offset into the physical key before applying `--key-prefix` or sharding. +- `--key-prefix` — inspect mode: hex prefix, relative to `--key-offset`, used to filter physical keys. +- `--shard-next-bytes` — inspect mode: group matching keys by this many bytes after `--key-prefix`. +- `--list` — inspect mode: list matching key/logical-value pairs instead of shard `bucket_digest` values. +- `--list-limit` — inspect mode: maximum pairs to print with `--list` (default `1000`; a value `<= 0` means unlimited). +- `--details` — inspect list mode: include backend-specific version metadata. +- `--find-hash` — optional 32-byte hex per-entry hash to hunt for. When two `bucket_digest` values differ by exactly one entry, their XOR is that entry's hash; this prints every matching entry so a single diverging row can be located. + + + +### Autobahn (GigaRouter) Config Generation + +When running with the Autobahn (GigaRouter) networking layer, you can generate the Autobahn JSON config from a set of node directories. Each directory must contain `validator_pubkey.txt`, `node_pubkey.txt`, `autobahn_address.txt`, and `evmrpc_url.txt`. + +```bash +# Generate an autobahn JSON config from one or more node directories +seid tendermint gen-autobahn-config [node-dirs...] --output + +# The --output flag may be abbreviated as -o +seid tendermint gen-autobahn-config ./node0 ./node1 ./node2 -o autobahn.json + +# Choose where autobahn consensus and data WALs are persisted (default: data/autobahn) +seid tendermint gen-autobahn-config ./node0 ./node1 --output autobahn.json --persistent-state-dir data/autobahn + +# Pass an empty value to disable persistence and run in-memory only +seid tendermint gen-autobahn-config ./node0 ./node1 --output autobahn.json --persistent-state-dir= +``` + +The `--persistent-state-dir` flag controls where autobahn persists its consensus and data write-ahead logs (WALs) across restarts. It defaults to `data/autobahn`, so persistence is enabled by default without any operator action; the consensus and data layers write to distinct subdirectories under this shared on-disk root. A relative path is resolved against the node's `--home` directory at config load time, while absolute paths are used as-is. Passing an empty value (`--persistent-state-dir=`) disables persistence entirely, running both the consensus and data layers in-memory only. When set, the flag populates the `PersistentStateDir` field in the generated config. + +The command reads the following files from each node directory: + +- `validator_pubkey.txt` — the validator public key in `validator:` format. +- `node_pubkey.txt` — the p2p node public key in `node:ed25519:public:` format. +- `autobahn_address.txt` — the network address (`host:port`) the node advertises to peers. + +The `validator_pubkey.txt` and `node_pubkey.txt` files are written automatically alongside `priv_validator_key.json` and `node_key.json` whenever those keys are saved, so they are typically already present in each node's config directory. + +The generated `autobahn.json` file describes the validator set along with gas and transaction limits, block interval, view timeout, and dial interval. To have a node consume it, reference the file from `config.toml` using the `autobahn-config-file` key. + + + +#### Giga Mode Behavior and Per-Block Limits + +When a node is started in Giga mode — that is, when `autobahn-config-file` is set in `config.toml` — the block production and networking behavior differs significantly from standard Tendermint consensus: + +- **The CometBFT `TxMempool` is not used.** Under Giga the standard mempool (and its gossip reactor) is disabled entirely. Transactions instead route through the Autobahn producer-backed mempool. +- **Consensus reactor, state sync, and block sync are disabled.** In Giga mode these reactors are skipped entirely, and both state sync and block sync are forced off regardless of other configuration. +- **Transactions are admitted through the producer mempool.** The RPC broadcast endpoints call the producer's `InsertTx`/`TryInsertTx` rather than the CometBFT mempool's `CheckTx`. `BroadcastTx` uses `InsertTx`, which blocks while the mempool is full; the async path uses `TryInsertTx`, which returns a `mempool is full` error instead of blocking. +- **Sequential EVM nonce ordering is enforced.** For EVM transactions, the producer mempool admits transactions strictly in nonce order per sender. A transaction whose nonce does not match the next expected nonce is rejected with a `bad nonce` error. Because admission is sequential, the mempool can track pending nonces (`EvmNextPendingNonce`) as callers submit them. + +Each Autobahn block payload is bounded by the following limits, enforced by the producer as it fills a block: + +- **Maximum transactions per block:** the lower of the configured `max_txs_per_block` and the built-in maximum. +- **Maximum total transaction bytes per block:** a fixed per-block byte cap; a single transaction larger than this cap is rejected with a `transaction too large` error. +- **Wanted gas per block (`MaxGasWantedPerBlock`):** derived from the genesis `MaxGasWanted` block param. A transaction whose `GasWanted` exceeds this per-block limit is rejected as too large. +- **Estimated gas per block (`MaxGasEstimatedPerBlock`):** derived from the genesis `MaxGas` block param. A transaction whose (normalized) estimated gas exceeds this per-block limit is rejected as too large. + +When filling a block the producer seals the current block and starts a new one as soon as adding the next transaction would exceed any of the transaction-count, byte, wanted-gas, or estimated-gas limits. + + + +#### Autobahn Committee and Network Message Limits + +Beyond the per-block payload limits, Giga mode enforces structural limits on the validator committee and on incoming consensus network messages: + +- **Maximum validators per committee:** the Autobahn committee is capped at a hard limit of 100 validators (`MaxValidators`). Committee creation rejects any validator set exceeding this limit — building a committee from more than 100 validators fails with a `too many validators` error rather than being silently truncated. +- **Bounded consensus network messages.** Autobahn consensus protobuf messages carry declared size and count constraints that are checked against the raw wire bytes before the message is decoded. Payloads that violate these constraints are rejected during decoding, before any allocation, which protects nodes from oversized or malformed inputs that could otherwise decode into much larger in-memory structures. + +The enforced message constraints include: + +- **Per-field maximum sizes** on fixed-width fields such as hashes, signatures, and public keys. +- **Maximum repeated-field counts** on validator-related lists — signature and quorum-certificate lists are capped at 100 entries (matching the 100-validator committee cap). +- **Transaction payload caps:** a block payload may carry at most 2000 transactions, with a combined transaction byte budget of roughly 2 MB across all transactions in the payload. + +Any message whose fields exceed these limits is rejected at decode time, so an oversized network payload never reaches the consensus logic. + + + + Because Giga replaces the CometBFT mempool, the `unsafe_flush_mempool` RPC endpoint is not supported under Giga and returns `unsafe_flush_mempool is not supported with autobahn mempool`. + + + ### Key Management Proper key management is crucial for security. These commands help you manage @@ -111,6 +260,12 @@ snapshot-keep-recent = 2 size = 5000 max-txs-bytes = 1073741824 cache-size = 10000 +# ttl-duration and ttl-num-blocks now treat zero (or unset) as "TTL purging disabled". +# A non-zero value defines the time / number of blocks after which a transaction +# is removed from the mempool; leaving them unset or set to zero disables TTL-based +# purging entirely. +ttl-duration = "0s" +ttl-num-blocks = 0 # State store configuration [state-store] @@ -135,9 +290,8 @@ external-address = "" seeds = "" persistent_peers = "" upnp = false -max_num_inbound_peers = 40 -max_num_outbound_peers = 10 -allowed_pools = "" +max-connections = 100 +max-outbound-connections = 20 max_packet_msg_payload_size = 10240 handshake_timeout = "20s" dial_timeout = "3s" @@ -149,6 +303,9 @@ cors_allowed_origins = [] cors_allowed_methods = ["HEAD", "GET", "POST"] cors_allowed_headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time"] max_open_connections = 900 +# timeout_broadcast_tx_commit is now enforced by the BroadcastTxCommit RPC: when set +# greater than 0 it is applied as a context timeout on the request, so a +# BroadcastTxCommit call will be cancelled if it does not complete within this duration. timeout_broadcast_tx_commit = "10s" # Consensus Configuration @@ -166,6 +323,20 @@ double_sign_check_height = 0 + + + + The `[consensus]` section may still parse a `stateless-leader-election` field, but it is **deprecated and ignored**. Stateless (seed-based) leader election is now always enabled regardless of the value set, so this field no longer has any effect. It is retained only for config-parsing compatibility and can be safely omitted. + + + + Out-of-process ABCI support has been removed. The full node now runs only with Tendermint in-process; external stand-alone ABCI processes (socket or gRPC) are no longer supported. As a result: + + - The `seid start` flags `--address` and `--transport` are **deprecated and ignored**. + - The Tendermint node flags `--proxy-app` and `--abci` are **deprecated and ignored**. + - The `proxy-app` and `abci` fields in `config.toml` have been removed and no longer have any effect. Node operators upgrading should delete these lines from their `config.toml` if present. + + ## Network Parameters Understanding network parameters helps you operate your node effectively. @@ -187,6 +358,11 @@ Slashing Parameters: - slash_fraction_downtime: 0% (no stake slash; jail only) - slash_fraction_double_sign: 0% (no stake slash; double-signing still triggers permanent tombstoning) + +Oracle Parameters: + - min_valid_per_window: 0% (default changed from 5% now that the Oracle + Price Feeder is retired; distinct from the + slashing module's min_signed_per_window above) ``` @@ -206,17 +382,25 @@ $HOME/.sei/ │ ├── config.toml # Tendermint configuration │ ├── genesis.json # Chain genesis file │ ├── node_key.json # Node identity key -│ └── priv_validator_key.json # Validator signing key +│ ├── node_pubkey.txt # Node public key in autobahn format ("node:ed25519:public:"), written when the node key is saved +│ ├── priv_validator_key.json # Validator signing key +│ └── validator_pubkey.txt # Validator public key in autobahn format ("validator:"), written when the validator key is saved ├── data/ │ ├── application.db # Application state -│ ├── blockstore.db # Block data -│ ├── cs.wal/ # Consensus write-ahead logs -│ ├── evidence.db # Evidence of misbehavior -│ ├── state.db # Tendermint state -│ └── tx_index.db # Transaction index +│ └── tendermint/ # Tendermint consensus DBs (new subdirectory layout) +│ ├── blockstore.db # Block data +│ ├── cs.wal/ # Consensus write-ahead logs +│ ├── evidence.db # Evidence of misbehavior +│ ├── peerstore.db # Peer store +│ ├── state.db # Tendermint state +│ └── tx_index.db # Transaction index └── keyring-file/ # Local key storage ``` + + New nodes place the Tendermint consensus databases (blockstore, state, tx_index, evidence, peerstore, and cs.wal) under `data/tendermint/`. Existing nodes that already have these databases in the legacy flat layout directly under `data/` (e.g. `data/blockstore.db`, `data/cs.wal/`) continue using those legacy paths automatically — the legacy location takes precedence when present, so no migration is required. + + This reference guide provides essential technical information for operating Sei nodes and validators. For API documentation and other detailed specifications, please refer to the respective sections in our documentation set. diff --git a/node/troubleshooting.mdx b/node/troubleshooting.mdx index 2d129dd..da4f6ea 100644 --- a/node/troubleshooting.mdx +++ b/node/troubleshooting.mdx @@ -90,18 +90,98 @@ top -p $(pgrep seid) If you encounter an AppHash mismatch, you'll need to capture the state for comparison with a known good version: ```bash -# For SeiDB (most non-archive nodes): +# For SeiDB (all supported nodes): git clone https://github.com/sei-protocol/sei-db.git cd sei-db/tools make install systemctl stop seid seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump systemctl restart seid +``` + + +On Giga Storage nodes, EVM state lives in a FlatKV store rather than in the memIAVL trees, so an AppHash comparison there requires dumping the FlatKV state as well. Use the `dump-flatkv` command to iterate and dump the physical `(key, value)` pairs into per-bucket files (formatted to match `dump-iavl` so the same diff tooling works on both): + +```bash +# For FlatKV (Giga Storage nodes hold EVM state in FlatKV): +systemctl stop seid +seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump +systemctl restart seid +``` + +The `dump-flatkv` command accepts the following flags: + +- `--db-dir` / `-d`: FlatKV database directory. +- `--output-dir` / `-o`: output directory; one file is written per bucket. +- `--height`: FlatKV target version; `0` (the default) selects the latest available version. +- `--bucket` / `-b`: restrict the dump to a single bucket (`account`, `code`, `storage`, or `legacy`). Defaults to all buckets. + +For example, to dump only the `storage` bucket at a specific version: + +```bash +seidb dump-flatkv --db-dir $HOME/.sei/data/flatkv --output-dir /home/ubuntu/flatkv-dump --height 12345678 --bucket storage +``` + + + +### Comparing EVM State Between memIAVL and FlatKV + +When debugging an AppHash mismatch that involves EVM state, a byte-for-byte physical dump can diverge between backends even when the underlying state is identical. This is because every FlatKV value embeds a per-key block-height stamp (the height the key was last written or migrated), which differs from the memIAVL leaf versions on a freshly migrated node. + +The `evm-logical-digest` command works around this by computing a backend-independent digest of the EVM *logical* state — it strips the serialization-version and block-height header on both sides and digests only the logical payload (account balance/nonce/code hash, bytecode, and storage words). This lets a memIAVL node and a FlatKV node be compared at the same chain height: -# For Legacy IAVL DB: -seid debug dump-iavl +```bash +# FlatKV digest at a height (WAL-replays to it): +seidb evm-logical-digest --backend flatkv \ + --db-dir $HOME/.sei/data/state_commit/flatkv --height 213200000 + +# memIAVL digest at the same height (0 = current symlink): +seidb evm-logical-digest --backend memiavl \ + --db-dir $HOME/.sei/data/state_commit/memiavl --height 213200000 ``` +Each run prints per-bucket `bucket_digest` values and a single `FINAL_DIGEST` line covering the `account`, `code`, `storage`, and `legacy` buckets. Compare the `FINAL_DIGEST` lines from both backends at the same height — they should match. Because FlatKV can contain a FlatKV-only migration-version marker that a memIAVL-only node never owns, that row is automatically omitted from the FlatKV final result so the comparison is apples-to-apples. + +The `evm-logical-digest` command accepts the following flags: + +- `--backend`: backend to read (`flatkv` or `memiavl`). +- `--db-dir` / `-d`: for FlatKV, the FlatKV data directory; for memIAVL, the memIAVL root directory (containing `current/` and `snapshot-*`). +- `--height`: target version. FlatKV WAL-replays to it; memIAVL resolves `snapshot-/evm` (`0` selects the `current` symlink). +- `--memiavl-normalization`: memIAVL normalization mode — `semantic`/`independent` (raw EVM key/value decoder, the default) or `translator` (routes each leaf through the current migration mapping). + +For targeted debugging, the command also supports inspecting a single normalized bucket instead of printing the global digest: + +- `--inspect-bucket`: inspect one bucket (`account`, `code`, `storage`, or `legacy`) instead of the global digest. +- `--key-offset`: byte offset into the physical key before applying `--key-prefix` and sharding. +- `--key-prefix`: hex prefix, relative to `--key-offset`, used to filter physical keys. +- `--shard-next-bytes`: group matching keys by this many bytes after `--key-prefix`. +- `--list`: list matching key/logical-value pairs instead of shard digests. +- `--list-limit`: maximum pairs to print with `--list` (default `1000`; `<=0` means unlimited). +- `--details`: include backend-specific version metadata in list mode. +- `--find-hash`: a 32-byte hex per-entry hash to hunt for. When two `bucket_digest` values differ by exactly one entry, their XOR is that entry's hash; passing it prints every matching entry as a `FOUND-HASH` line. + +For example, to list the first 50 `account` rows with version metadata, or to shard the `storage` bucket under a key prefix by the next 2 bytes: + +```bash +seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \ + --inspect-bucket account --list --list-limit 50 --details + +seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \ + --inspect-bucket storage --key-prefix 03 --shard-next-bytes 2 +``` + + + +**The legacy IAVL backend has been fully removed and SeiDB State Commit (SC) is now mandatory.** SC must be enabled via `sc-enable = true` in the `[state-commit]` section of `app.toml`. If SC is not enabled, the node no longer falls back to IAVL — it panics at startup with: + +```text +SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated +``` + +The `seid debug dump-iavl` command has also been removed along with the IAVL backend. To inspect state, use the `seidb dump-iavl` tool shown above. + + + Always include the app hash, commit hash, and block height from your logs when reporting issues. ### Identifying AppHash Errors