Repository navigation
Probe Edge Blockbook servers for inSync and fail over when they fall behind - #462
Conversation
f735269 to
a6e6b04
Compare
j0ntz
left a comment
There was a problem hiding this comment.
Commit subjects: 6 of 7 exceed the 50-character limit (commit-subject-50). Suggested replacements:
| Commit | Len | Suggestion |
|---|---|---|
1623f37 |
59 | Drop Blockbook servers that report inSync false |
cea9e9e |
63 | Quarantine dropped servers from the refill |
27fe15a |
64 | Re-query addresses after a stale server drop |
23da251 |
66 | Keep an out-of-sync server if it is the last |
c2eb9a6 |
58 | Guard block height handler on closed sockets |
613cb32 |
51 | Test keepalive re-probe, de-dupe, and recovery |
On the Trezor 403s: in the app, Socket.ts connects through setupBrowser, which sends the WebView's own user agent. Only the Node fallback (nodejsWS.ts) sends NodeJS-WS-agent, so the code supports the PR's read that the 403s come from the Node/CLI path rather than production.
j0ntz
left a comment
There was a problem hiding this comment.
Correction to the Trezor note in my review: a live handshake against wss://ltc1 through wss://ltc5.trezor.io/websocket returns 101 for both a WebView user agent and NodeJS-WS-agent, so neither app path triggers the 403. All five return 403 only when the User-Agent header is missing or is curl's default, and GET https://ltc1.trezor.io/api/ with curl's user agent also returns 403. The reported 403s fit a probe tool that sends no browser-like user agent, which still supports the PR's read that production is unaffected.
a6e6b04 to
83d32f1
Compare
|
Rechecked the handshake against |
|
Approved after bugbot |
83d32f1 to
bbebbbe
Compare
4c42d54 to
bbebbbe
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5608508. Configure here.
The engine scored servers on response time alone, so an Edge Blockbook whose indexer had fallen behind its backend kept answering pings quickly and kept its top score. Nothing ever moved the wallet off it. Blockbook only reports inSync on its REST GET /api/ response; the WebSocket getInfo message does not carry it. Each socket URI now looks up its HTTPS twin in engineInfo.serverConfigs (the same list broadcastTx uses) and probes it once the socket has connected, then again on every keepalive. Probes closer than ten seconds apart are skipped, since the first keepalive can land right next to the connect probe. A server whose twin says inSync false is disconnected through the normal close path, which scores it down and schedules a refill. Third-party sockets have no twin and are never probed. A probe that fails, times out, or returns something unexpected is recorded as unknown and never treated as unhealthy, so a REST outage cannot take a working socket down with it.
Dropping an out-of-sync server only closes its socket. Its score is still near the top, since it answered every ping, so the very next refill would reconnect to it and the probe would drop it again, a few seconds later, forever. PluginState now keeps an in-memory quarantine keyed by URI with a five-minute expiry. getLocalServers withholds quarantined URIs, and the refill loop skips any that were already picked before the quarantine began. It lives on PluginState rather than on the engine so every wallet of the same currency shares one finding instead of each discovering it on its own. dumpData lists the live entries for support logs.
Closing a stale server's socket moved the address subscriptions to the surviving connection, but nothing asked that connection for history. The subscribe loop skips addresses already marked processing, and a Blockbook only pushes transactions it sees after the subscription, so blocks the stale server had not indexed were never fetched. The balance stayed short until an unrelated event happened to touch the address. ServerStates now emits SERVER_DROPPED from its deliberate-drop path. The processor answers by clearing the processing flag on every subscribed address and queueing a UTXO refresh for each. The next task pass adds them back to the transactions cache, and each query starts from the address's stored lastQueriedBlockHeight. An address is queued with the height of whichever server subscribed it, but any server can answer the query, so the recorded height is capped at the answering server's own height. A lagging server can no longer move lastQueriedBlockHeight past blocks it never indexed, and the range the dropped server missed is fetched again. Re-queued addresses are counted a second time by the progress ratio, so it is clamped at 1 rather than reporting more than fully synced. The lookahead is now extended before progress is counted, so addresses it adds are in the total before an address's flag can bring the ratio to 1.
A stale server still serves balances and history, just behind the chain. Dropping it with nothing else connected would leave the wallet with no data at all until a refill succeeds. The drop now happens only while some other connection is open and not known to be out of sync. A peer whose probe has not answered yet does not count, since it may be about to fail the same way. The check runs on every probe result and once each new connection has finished connecting, so a stale server that was kept for lack of a peer goes as soon as one turns up, whether that peer is probed or not. A third-party server that cannot be probed counts as usable. The re-check runs from the connect promise rather than the socket's open event: that event fires before the connection is marked connected, so a re-check there could never see the new peer.
BLOCK_HEIGHT_CHANGED indexed the connection cache without checking the entry still existed, while the NEW_ADDRESS_TRANSACTION handler next to it already does. A height notification that lands after the entry is removed would throw inside the emitter.
The probe tests so far only exercised the state a server starts in. The paths that matter in production, a server that falls out of sync while connected and one that comes back, run off the 50 second keepalive and had no coverage. The socket's keepalive and wake-up intervals and the probe de-dupe window are now injectable through ServerStates, Blockbook, and the socket config, defaulting to the production values. Tests run the cycle at a few hundred milliseconds with a probe whose answers can change mid-test, and cover: re-probing on each keepalive, dropping a server that falls out of sync, keeping the last leg when every server does, honouring the de-dupe window, and taking a recovered server back after its quarantine.
e8a7f15 to
725d488
Compare
Every engine probed its servers on its own keepalive, de-duplicated only within that engine. A user with several wallets of one currency sent one GET /api/ per wallet per keepalive to the same Edge Blockbook, though every answer is the same. PluginState now keeps the latest probe per server URI, next to the quarantine it already shares. A probe asked for less than ten seconds after the last one started gets that one's result, whether it is still in flight or has answered, so one request serves every wallet. Each engine still applies the answer to its own connection and runs its own drop check.
725d488 to
9c38daa
Compare

CHANGELOG
Does this branch warrant an entry to the CHANGELOG?
Dependencies
none
Description
Asana task
Asana: https://app.asana.com/1/9976422036640/project/1213843652804305/task/1218875483565488
Last weekend the LTC Edge Blockbook fell behind its backend. It kept answering pings, so it kept its top score, and wallets stayed pinned to it: a completed resync showed a short balance that corrected itself an hour later. The engine scored servers on round-trip time only and never read Blockbook's
inSync.The Trezor 403s in the same ticket are not a production problem. Cloudflare rejects tool user agents (curl, okhttp, CFNetwork); the plugin's WebView user agent passes. Verified in the simulator against the plugin's own logs.
inSyncis only on Blockbook's RESTGET /api/, not on the WebSocketgetInfo, and Trezor's REST is closed to everyone, so only Edge's own servers can be probed. Each commit stands alone:blockbookentries ofengineInfo.serverConfigsand probes it after connect and on every keepalive (de-duplicated within 10 s).inSync: falsedisconnects the server through the normal close path and schedules a refill. No twin means never probed; a failed, timed-out, or unparseable probe isunknown, never unhealthy.getLocalServers, skipped by refill, and listed indumpData. Refill keeps retrying while a quarantine is in force so the server returns when it expires.SERVER_DROPPEDengine event. The processor clears the processing flag on every subscribed address and queues a UTXO refresh, so the surviving connection fetches the range the stale server missed, starting from each address's storedlastQueriedBlockHeight. Progress is now per-address completion flags rather than a counter, so re-queued work cannot push the ratio past 1 or report 100% early.BLOCK_HEIGHT_CHANGEDhandler, matching its sibling.GET /api/on its own keepalive. Each wallet still applies the answer to its own connection.Also fixed on the way:
reconnect()overwrote its pending timer instead of clearing it, so a timer could outlivestop()and switch a stopped engine back on.Design calls: no hysteresis beyond Blockbook's own 5 s grace, and no block-height comparison between servers (reorgs).
Testing. Unit: 1340 passing, suite exits on its own. Integration, in the iOS simulator with a local proxy in front of ltc-wusa1 whose
/api/reports a flippableinSync, with two LTC wallets and ltc1.trezor.io as the peer:health: healthy -> unhealthy, dropped, quarantined 300000ms, 171 and 122 addresses re-queued, reconnect scheduledFollow-up, not in this PR: after a drop the re-query fires ~460 requests in one second at the surviving Trezor socket, which closes it; it converges after a few reconnects. Per-server request pacing (the socket's 50 in-flight cap is global) would fix that and the same burst on a fresh wallet whose first pick is a third-party server.
Note
Medium Risk
Changes core multi-server sync, connection selection, and balance correctness paths; behavior is heavily unit-tested but affects all UTXO Blockbook wallets when servers fall behind.
Overview
UTXO wallets now detect lagging Edge Blockbook indexers and fail over instead of staying on a fast but stale connection.
On connect and each socket keepalive, probeable servers get a shared REST
GET /api/check forinSync(Edge hosts only; failed probes count as unknown, not unhealthy). Servers reporting behind are quarantined for five minutes (currency-wide), disconnected when another healthy or unknown peer exists, and only the last stale connection is kept if nothing else is usable. Dropping a server emitsSERVER_DROPPED, which re-queues address tx/UTXO sync from storedlastQueriedBlockHeight, caps that cursor to the answering server’s height, and tracks sync progress with per-address flags so re-queries cannot report 100% early. Probe deduplication across wallets limits one REST call per server per ~10s.reconnect()now clears its pending timer sostop()cannot resurrect the engine.Reviewed by Cursor Bugbot for commit 725d488. Bugbot is set up for automated code reviews on this repo. Configure here.