Skip to content

Probe Edge Blockbook servers for inSync and fail over when they fall behind - #462

Merged
peachbits merged 7 commits into
masterfrom
matthew/server-health-probe
Oct 8, 2026
Merged

peachbits merged 7 commits into
masterfrom
matthew/server-health-probe

Conversation

@peachbits

@peachbits peachbits commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

  • Yes
  • No

Dependencies

none

Description

Asana task

Asana: https://app.asana.com/1/9976422036640/project/1213843652804305/task/1218875483565488

Last weekend the LTC Edge Blockbook fell behind its backend. It kept answering pings, so it kept its top score, and wallets stayed pinned to it: a completed resync showed a short balance that corrected itself an hour later. The engine scored servers on round-trip time only and never read Blockbook's inSync.

The Trezor 403s in the same ticket are not a production problem. Cloudflare rejects tool user agents (curl, okhttp, CFNetwork); the plugin's WebView user agent passes. Verified in the simulator against the plugin's own logs.

inSync is only on Blockbook's REST GET /api/, not on the WebSocket getInfo, and Trezor's REST is closed to everyone, so only Edge's own servers can be probed. Each commit stands alone:

  1. Probe. A socket URI finds its HTTPS twin in the blockbook entries of engineInfo.serverConfigs and probes it after connect and on every keepalive (de-duplicated within 10 s). inSync: false disconnects the server through the normal close path and schedules a refill. No twin means never probed; a failed, timed-out, or unparseable probe is unknown, never unhealthy.
  2. Quarantine. Five minutes in memory on PluginState, shared by every wallet of the currency, withheld from getLocalServers, skipped by refill, and listed in dumpData. Refill keeps retrying while a quarantine is in force so the server returns when it expires.
  3. Re-query on drop. A SERVER_DROPPED engine event. The processor clears the processing flag on every subscribed address and queues a UTXO refresh, so the surviving connection fetches the range the stale server missed, starting from each address's stored lastQueriedBlockHeight. Progress is now per-address completion flags rather than a counter, so re-queued work cannot push the ratio past 1 or report 100% early.
  4. Last leg. A stale server is only dropped while another connection is open and not known unhealthy. Re-evaluated on every probe result and once each new connection finishes connecting.
  5. Block height guard. Null check in the BLOCK_HEIGHT_CHANGED handler, matching its sibling.
  6. Tests. Keepalive, wake-up, and probe de-dupe intervals are injectable (production defaults unchanged). Over-time tests cover re-probing, dropping a server that falls out of sync, keeping the last leg, the de-dupe window, and taking a recovered server back.
  7. Shared probe. The de-dupe lives on PluginState, keyed by server URI, so every wallet of a currency shares one probe per server per 10 s instead of each sending its own GET /api/ on its own keepalive. Each wallet still applies the answer to its own connection.

Also fixed on the way: reconnect() overwrote its pending timer instead of clearing it, so a timer could outlive stop() and switch a stopped engine back on.

Design calls: no hysteresis beyond Blockbook's own 5 s grace, and no block-height comparison between servers (reorgs).

Testing. Unit: 1340 passing, suite exits on its own. Integration, in the iOS simulator with a local proxy in front of ltc-wusa1 whose /api/ reports a flippable inSync, with two LTC wallets and ltc1.trezor.io as the peer:

Time Observed
18:03:27 Both wallets connect; proxy probed healthy
18:03:45 Flag flipped to false
18:04:22 Next keepalive: health: healthy -> unhealthy, dropped, quarantined 300000ms, 171 and 122 addresses re-queued, reconnect scheduled
18:09:26 Quarantine expired, proxy reconnected and probed healthy

Follow-up, not in this PR: after a drop the re-query fires ~460 requests in one second at the surviving Trezor socket, which closes it; it converges after a few reconnects. Per-server request pacing (the socket's 50 in-flight cap is global) would fix that and the same burst on a fresh wallet whose first pick is a third-party server.


Note

Medium Risk
Changes core multi-server sync, connection selection, and balance correctness paths; behavior is heavily unit-tested but affects all UTXO Blockbook wallets when servers fall behind.

Overview
UTXO wallets now detect lagging Edge Blockbook indexers and fail over instead of staying on a fast but stale connection.

On connect and each socket keepalive, probeable servers get a shared REST GET /api/ check for inSync (Edge hosts only; failed probes count as unknown, not unhealthy). Servers reporting behind are quarantined for five minutes (currency-wide), disconnected when another healthy or unknown peer exists, and only the last stale connection is kept if nothing else is usable. Dropping a server emits SERVER_DROPPED, which re-queues address tx/UTXO sync from stored lastQueriedBlockHeight, caps that cursor to the answering server’s height, and tracks sync progress with per-address flags so re-queries cannot report 100% early. Probe deduplication across wallets limits one REST call per server per ~10s. reconnect() now clears its pending timer so stop() cannot resurrect the engine.

Reviewed by Cursor Bugbot for commit 725d488. Bugbot is set up for automated code reviews on this repo. Configure here.

@peachbits
peachbits force-pushed the matthew/server-health-probe branch from f735269 to a6e6b04 Compare October 2, 2026 18:51

@j0ntz j0ntz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit subjects: 6 of 7 exceed the 50-character limit (commit-subject-50). Suggested replacements:

Commit Len Suggestion
1623f37 59 Drop Blockbook servers that report inSync false
cea9e9e 63 Quarantine dropped servers from the refill
27fe15a 64 Re-query addresses after a stale server drop
23da251 66 Keep an out-of-sync server if it is the last
c2eb9a6 58 Guard block height handler on closed sockets
613cb32 51 Test keepalive re-probe, de-dupe, and recovery

On the Trezor 403s: in the app, Socket.ts connects through setupBrowser, which sends the WebView's own user agent. Only the Node fallback (nodejsWS.ts) sends NodeJS-WS-agent, so the code supports the PR's read that the 403s come from the Node/CLI path rather than production.

Comment thread src/common/plugin/PluginState.ts Resolved

@j0ntz j0ntz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to the Trezor note in my review: a live handshake against wss://ltc1 through wss://ltc5.trezor.io/websocket returns 101 for both a WebView user agent and NodeJS-WS-agent, so neither app path triggers the 403. All five return 403 only when the User-Agent header is missing or is curl's default, and GET https://ltc1.trezor.io/api/ with curl's user agent also returns 403. The reported 403s fit a probe tool that sends no browser-like user agent, which still supports the PR's read that production is unaffected.

@peachbits
peachbits force-pushed the matthew/server-health-probe branch from a6e6b04 to 83d32f1 Compare October 5, 2026 22:12
@peachbits

Copy link
Copy Markdown
Contributor Author

Rechecked the handshake against wss://ltc1.trezor.io/websocket: okhttp/4.12.0, a CFNetwork user agent, and curl's default get 403, while NodeJS-WS-agent and an iOS WebView user agent get 101. That matches the PR description's list (curl, okhttp, CFNetwork), so it stays as written.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@j0ntz

j0ntz commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Approved after bugbot

@peachbits
peachbits force-pushed the matthew/server-health-probe branch from 83d32f1 to bbebbbe Compare October 7, 2026 22:19

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@peachbits
peachbits force-pushed the matthew/server-health-probe branch from 4c42d54 to bbebbbe Compare October 8, 2026 17:27

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5608508. Configure here.

The engine scored servers on response time alone, so an Edge Blockbook
whose indexer had fallen behind its backend kept answering pings quickly
and kept its top score. Nothing ever moved the wallet off it.

Blockbook only reports inSync on its REST GET /api/ response; the
WebSocket getInfo message does not carry it. Each socket URI now looks up
its HTTPS twin in engineInfo.serverConfigs (the same list broadcastTx
uses) and probes it once the socket has connected, then again on every
keepalive. Probes closer than ten seconds apart are skipped, since the
first keepalive can land right next to the connect probe. A server whose
twin says inSync false is disconnected through the normal close path,
which scores it down and schedules a refill.

Third-party sockets have no twin and are never probed. A probe that
fails, times out, or returns something unexpected is recorded as unknown
and never treated as unhealthy, so a REST outage cannot take a working
socket down with it.
Dropping an out-of-sync server only closes its socket. Its score is
still near the top, since it answered every ping, so the very next
refill would reconnect to it and the probe would drop it again, a few
seconds later, forever.

PluginState now keeps an in-memory quarantine keyed by URI with a
five-minute expiry. getLocalServers withholds quarantined URIs, and the
refill loop skips any that were already picked before the quarantine
began. It lives on PluginState rather than on the engine so every wallet
of the same currency shares one finding instead of each discovering it
on its own. dumpData lists the live entries for support logs.
Closing a stale server's socket moved the address subscriptions to the
surviving connection, but nothing asked that connection for history.
The subscribe loop skips addresses already marked processing, and a
Blockbook only pushes transactions it sees after the subscription, so
blocks the stale server had not indexed were never fetched. The balance
stayed short until an unrelated event happened to touch the address.

ServerStates now emits SERVER_DROPPED from its deliberate-drop path. The
processor answers by clearing the processing flag on every subscribed
address and queueing a UTXO refresh for each. The next task pass adds
them back to the transactions cache, and each query starts from the
address's stored lastQueriedBlockHeight.

An address is queued with the height of whichever server subscribed it,
but any server can answer the query, so the recorded height is capped at
the answering server's own height. A lagging server can no longer move
lastQueriedBlockHeight past blocks it never indexed, and the range the
dropped server missed is fetched again.

Re-queued addresses are counted a second time by the progress ratio, so
it is clamped at 1 rather than reporting more than fully synced. The
lookahead is now extended before progress is counted, so addresses it
adds are in the total before an address's flag can bring the ratio to 1.
A stale server still serves balances and history, just behind the
chain. Dropping it with nothing else connected would leave the wallet
with no data at all until a refill succeeds.

The drop now happens only while some other connection is open and not
known to be out of sync. A peer whose probe has not answered yet does
not count, since it may be about to fail the same way. The check runs
on every probe result and once each new connection has finished
connecting, so a stale server that was kept for lack of a peer goes as
soon as one turns up, whether that peer is probed or not. A third-party
server that cannot be probed counts as usable.

The re-check runs from the connect promise rather than the socket's
open event: that event fires before the connection is marked
connected, so a re-check there could never see the new peer.
BLOCK_HEIGHT_CHANGED indexed the connection cache without checking the
entry still existed, while the NEW_ADDRESS_TRANSACTION handler next to
it already does. A height notification that lands after the entry is
removed would throw inside the emitter.
The probe tests so far only exercised the state a server starts in. The
paths that matter in production, a server that falls out of sync while
connected and one that comes back, run off the 50 second keepalive and
had no coverage.

The socket's keepalive and wake-up intervals and the probe de-dupe
window are now injectable through ServerStates, Blockbook, and the
socket config, defaulting to the production values. Tests run the cycle
at a few hundred milliseconds with a probe whose answers can change
mid-test, and cover: re-probing on each keepalive, dropping a server
that falls out of sync, keeping the last leg when every server does,
honouring the de-dupe window, and taking a recovered server back after
its quarantine.
@peachbits
peachbits force-pushed the matthew/server-health-probe branch from e8a7f15 to 725d488 Compare October 8, 2026 19:26
Every engine probed its servers on its own keepalive, de-duplicated only
within that engine. A user with several wallets of one currency sent one
GET /api/ per wallet per keepalive to the same Edge Blockbook, though
every answer is the same.

PluginState now keeps the latest probe per server URI, next to the
quarantine it already shares. A probe asked for less than ten seconds
after the last one started gets that one's result, whether it is still
in flight or has answered, so one request serves every wallet. Each
engine still applies the answer to its own connection and runs its own
drop check.
@peachbits
peachbits force-pushed the matthew/server-health-probe branch from 725d488 to 9c38daa Compare October 8, 2026 22:40
@peachbits
peachbits merged commit e0efe33 into master Oct 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants