Harden HF16 finality, block production, vote relay and admin recovery

.github/workflows/docker-pr-build.yml

@@ -28,7 +28,9 @@ jobs:
       - uses: docker/setup-buildx-action@v4
         id: setup-buildx
 
+      # Fork PRs build without registry secrets and never publish images.
       - uses: docker/login-action@v4
+        if: github.event.pull_request.head.repo.full_name == github.repository
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
@@ -84,7 +86,7 @@ jobs:
         with:
           context: .
           file: ./share/vizd/docker/Dockerfile-production
-          push: true
+          push: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
           tags: vizblockchain/vizd:${{ steps.tag.outputs.value }}
           cache-from: type=gha,scope=pr-${{ github.base_ref }}
           cache-to:   type=gha,scope=pr-${{ github.base_ref }},mode=max
@@ -97,7 +99,7 @@ jobs:
           context: .
           file: ./share/vizd/docker/Dockerfile-testnet
           build-args: TESTNET_VALIDATORS=21
-          push: true
+          push: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
           tags: vizblockchain/vizd:${{ steps.tag.outputs.value }}-testnet21
           cache-from: type=gha,scope=pr-testnet21
           cache-to:   type=gha,scope=pr-testnet21,mode=max

.github/workflows/tests.yml

@@ -23,6 +23,15 @@ jobs:
       - name: P2P corruption recovery regression (offline)
         run: python3 tests/p2p/test_corruption_recovery.py -v
 
+      - name: HF16 offline regression checks
+        run: |
+          set -euo pipefail
+          python3 tests/p2p/test_vote_ingress.py
+          python3 tests/validator/test_signing_cache.py
+          python3 tests/chain/test_snapshot_restore.py
+          python3 tests/consensus_sim/production_safety_offline.py
+          node --test tests/block_log/archive_merge.test.cjs
+
       - name: Install build deps
         run: |
           set -euo pipefail
@@ -125,7 +134,7 @@ jobs:
           set -euo pipefail
           cd build
           if [ "${{ matrix.validators }}" = "2" ]; then
-            tests/consensus_sim/consensus_sim_tests --run_test=hf16_boundary_suite --log_level=test_suite
+            tests/consensus_sim/consensus_sim_tests --run_test=hf16_boundary_suite,hf16_production_safety_suite,validator_committed_state_suite --log_level=test_suite
           else
             ctest -R '^consensus_sim$' --output-on-failure
           fi

docs/consensus/hf16.md

@@ -10,7 +10,7 @@ After the master image is built, check its revision/version, startup, synchroniz
 
 - The runtime slot changes from **3 seconds to 1 second** at HF16. Use the node's current `get_config` value and block timestamps; the compile-time legacy `CHAIN_BLOCK_INTERVAL` is not the post-fork slot.
 - A QC collects signatures from scheduled validators. The implementation uses `max(floor(2N/3)+1, floor(0.75N))`: **15 of 21**, not the old 14-confirmation post-validation description.
-- A producer handover carries a parent QC. Timeout certificates (TCs) allow progress after skipped rounds. LIB uses the **2-chain** rule: certified parent and child on the local main branch, adjacent heights and consecutive rounds. One isolated QC is insufficient.
+- A producer handover carries a parent QC. Timeout certificates (TCs) allow progress after skipped rounds. LIB uses the **2-chain** rule: certified parent and child on the local main branch, adjacent heights and consecutive rounds. The signed child header must carry that parent's QC. A child using the late-QC grandparent fallback does not finalize its parent merely because the local vote pool later collects QCs for both. One isolated QC is insufficient.
 - Votes are limited to one block per validator per round; the highest voted round is persisted before broadcasting. Repeating the same vote is allowed. Conflicting votes can be proved by header evidence and produce virtual operation `validator_equivocation` (ID **106**).
 - Header extension tags are append-only: **3 QC**, **4 TC**, **5 equivocation proof**. Transaction extensions and existing operation IDs are unchanged.
 - Block-denominated governance values whose wall-time meaning must survive the slot change are scaled at activation. Do not mechanically replace every 3-second timer: independent retry/shutdown timers remain separate.

docs/development/testing.md

@@ -4,6 +4,10 @@ VIZ Ledger uses Boost.Test for unit tests and provides utility programs for inte
 
 ---
 
+## Pull-request Docker builds
+
+Fork pull requests build images without registry login or publication. Internal pull requests retain Docker Hub login and image publication. The separate master/release workflows are unchanged.
+
 ## Unit Tests
 
 The unit test binary is built via CMake as part of the `libraries/chain` target.

docs/p2p/messages.md

@@ -6,6 +6,12 @@ DLT P2P uses a binary protocol over raw TCP. Each message is framed with a 4-byt
 
 ---
 
+## Vote relay bounds
+
+Block and timeout vote relay caches retain only chain-accepted hashes. Each cache has at most 8192 entries and a 120-second lifetime. Expiry cleanup examines at most 64 oldest entries per lookup.
+
+Accepted local block votes have a separate bounded retry cache, so remote relay churn cannot suppress their rebroadcast. Malformed names/heights use the existing strike policy; a generic delegate refusal is not proof of peer misconduct.
+
 ## Message Type Summary
 
 | Type ID | Name | Direction | Purpose |

docs/plugins/node-admin.md

@@ -29,6 +29,10 @@ request arrives **from 127.0.0.1**, so checking the source address on the public
 protects nothing. `node_admin` therefore opens its own HTTP listener, bound to loopback by
 default, that is never proxied.
 
+This listener is HTTP-only, including MCP. WebSocket upgrade requests are rejected
+with HTTP 403 and the connection is closed, even when a bearer token is supplied;
+there is no admin WebSocket API on this port.
+
 An endpoint on a non-loopback address is refused unless `admin-allow-remote = true` is set
 explicitly. A bearer token on an open port is a different threat model, not a convenience.
 

docs/plugins/snapshot.md

@@ -242,6 +242,61 @@ P2P sync fills the gap to the live chain head.
 
 ---
 
+## Admin-Requested In-Process Restore
+
+`admin.snapshot.restore` schedules a restore on the snapshot worker thread. Its
+`status: started` response means the job was accepted, not that import or resume
+succeeded. `admin.snapshot.status` exposes whether the job is still running;
+completion and failure details are logged by the worker.
+
+The chain restore method applies these failure boundaries:
+
+- With a started P2P plugin, a synchronous pause exception aborts before database
+  close/import and leaves the pre-restore syncing flag unchanged. Missing or
+  unstarted P2P is skipped, as before.
+- After pause, block production is gated through `is_syncing()`. If database
+  close throws, the method does not import or resume: close may have partially
+  changed state. It keeps the gate set and requests application shutdown.
+- An import exception, zero imported head, or unreadable head returns failure
+  without resuming P2P or clearing the production gate. Import handling may
+  already have requested shutdown; the restore method does not assume that it
+  has, and does not automatically recover from this state.
+- An immediate peer-reset/resume exception returns failure and retains the
+  production gate. P2P flags may already have changed before the exception; this
+  is not a guarantee that P2P remains paused after a failed resume.
+- Only after a nonzero head and no immediate resume error does the method clear
+  the syncing gate and report success. The existing post-pause P2P catchup gate
+  still applies.
+
+These are local error-handling boundaries, **not whole-node quiescence**. The
+recovery mutex only excludes cooperating users; pausing P2P and setting the
+production gate do not establish that all API calls, transactions or already
+in-flight validator/database work have drained. P2P reset and the resume queue
+are asynchronous, so later errors in those posted tasks are not observed by
+this method. Plugin lifecycle changes during restore are not synchronized here.
+After a post-close failure, inspect the logs and use the normal stopped-node
+recovery procedure rather than assuming the old database is intact.
+
+### Offline regression
+
+Run from the repository root with a C++17 compiler available:
+
+```bash
+python3 tests/chain/test_snapshot_restore.py
+```
+
+The test extracts the production `restore_from_snapshot()` method and compiles
+it against controlled storage, P2P, filesystem and application dependencies. It
+covers preflight, absent/unstarted P2P, pause exceptions with both initial syncing
+values, partially mutating close failures, failed/empty import, unreadable head,
+peer-reset/resume exceptions (including partially changed resume flags), and
+successful restore ordering. It starts no node and performs no real DB restore;
+it does not prove thread quiescence or test the queued P2P task bodies.
+`CXX` selects the compiler; `RESTORE_SOURCE` can select an alternate `plugin.cpp`
+for baseline regression checks.
+
+---
+
 ## Automatic Runtime Recovery: `--auto-recover-from-snapshot`
 
 Enabled by default. When shared memory corruption is detected during block processing or block generation, the node:

docs/plugins/validator.md

@@ -157,6 +157,40 @@ Production resumes automatically on the next tick.
 ## `on_block_applied()` — Signal Handler
 
 Connected to `database::applied_block`. Runs for every incoming block.
+This deferred callback does not refresh the signing-key snapshot.
+
+### Committed signing-key snapshot
+
+Block headers must use the key committed at the current head, not a pending
+`validator_update_operation` key. The plugin seeds configured validators' keys at
+startup, then replaces the complete snapshot on `database::committed_chain_state`.
+That signal runs synchronously under the database write lock after successful
+`_push_block()` and memory checking, inside the callback that temporarily removes
+pending transactions. It fires before those transactions are restored. Signal
+callbacks must not yield or acquire database locks.
+
+Reading all configured validators at this boundary also captures rotations
+reverted by a fork switch, including cases where the adopted blocks contain no
+rotation at all. Applying only incoming block operations cannot undo such a
+rotation; reading the database in deferred `applied_block` instead sees pending
+state. The snapshot stores both head height and ID, and removes validators absent
+from the committed index. Visible null keys still disable production; a committed
+null key is not replaced by a pending re-enable.
+
+Offline regression (requires Python 3 and a C++14 compiler):
+
+```sh
+python3 tests/validator/test_signing_cache.py
+# Negative control against the pre-fix source; expected to exit 1:
+python3 tests/validator/test_signing_cache.py --source-ref d4fd4bcd
+```
+
+The test extracts and compiles the real plugin helpers with a controlled database
+fixture, and guards the source hook placement and connection lifecycle. It does
+not replace a native chain integration run. The companion scenario
+`tests/validator/test_committed_chain_state.cpp` is included in the testnet
+`consensus_sim_tests` target and verifies that real `push_block()` exposes committed keys in
+the synchronous signal and restored pending keys in deferred `applied_block`.
 
 ### Missed slot detection
 
@@ -205,7 +239,7 @@ Included in P2P FORWARD stagnation logs when the node is stuck with no peer ahea
 5. **Slaves must sync before producing in emergency mode** — producing on stale head = minority fork.
 6. **Participation < 33% stops production** — network partition guard (overridable).
 7. **21 consecutive own-validator blocks → rollback to LIB** — minority fork recovery.
-8. **All database reads are fresh** — no state caching; emergency mode can activate/deactivate any block.
+8. **Committed signing keys are refreshed at each successful push boundary** — other production state is read fresh; emergency mode can activate/deactivate any block.
 
 ---
 

docs/storage/block-log-tools.md

@@ -75,7 +75,7 @@ node block-archive.cjs merge /var/lib/vizd/block-archive --from=83700000 --out=/
 
 Order matters: the merged log must reach at least the snapshot block minus one. If it ends earlier, snapshot import sees a gap and resets the log (`plugins/snapshot/plugin.cpp`, "gap detected"). If it ends after the snapshot, the node replays the extra blocks locally before P2P sync.
 
-It refuses gaps in the range (`GAP a-b, merge stopped`) and never overwrites existing files.
+It refuses gaps in the range (`GAP a-b, merge stopped`) and never overwrites existing files. Overlapping ranges emit each height once, including nested ranges. Without `--to`, the upper bound is the highest archived height, not the last filename's end.
 
 ### Filters