Incident response runbook and incident communication for testnet

Part of the testnet readiness umbrella (#166).

Why

Item 7 of the pre-testnet requirements research on discussion No. 132, raised again in the Sep 18, 2026 Testnet-Ready session: list in advance what can go wrong during and after launch, and plan the response, so we protect uptime and reputation.

Nothing in the repo covers what happens during an incident. The "public status page" in Developer/issues/validator-operations.md is a different thing: it exists to show validators the network isn't empty before they join.

Scope

A written, rehearsed runbook plus a way to tell external participants what is going on mid-incident. The Security Alliance web3 incident response template is the reference starting point.

Incident catalogue. DIN-specific failure modes, each with severity, detection signal and first response. Starting list:
- Smart contract bug or exploit in platform contracts (DinCoordinator, DinToken, DinValidatorStake, DinModelRegistry, treasury/fee/emission contracts) or task contracts
- A GI stuck in a phase with no way forward. This has already happened: Model_0 GI 2 has sat in DINaggregatorsRegistrationStarted since July (see discussion No. 150)
- Wrong or unexpected slashing
- Key compromise: DIN-Representative, proxy admin, model owner or validator
- Malicious or broken service code in a model manifest running on validator nodes
- Poisoned model updates getting through aggregation
- IPFS provider or pinning outage, or RPC outage
- Faucet or test-token drain
Emergency levers. What we can actually do on-chain today. None of the contracts in foundry/src/ has a pause function, so the only lever is a proxy upgrade via UpgradePlatform.s.sol. Decide whether testnet needs a pause/guardian mechanism, or record that we accept not having one.
Roles and contact. Who is on call, who can sign which transactions, and how the team reaches each other fast. The on-call/backup rota itself lives in the internal incident log (item 7).
Participant communication. Where validators, clients and model owners hear about an incident (e.g. the validator Signal group, a status page) and message templates for "investigating / mitigated / resolved".
Post-incident. A short post-mortem template, and how fixes feed back into the DevNet release cycle.
Rehearsal. At least one tabletop run-through with the team before the first external cohort joins.
Internal incident log and maintainer rota. Use discussion No. 170 as the model: it already does this for red or missing develop CI runs. It names backup maintainers, sets an escalation threshold (about 6 hours unaddressed, with more commits on top), opens one thread per incident with the commit hash, run URL and cause, and closes each thread with the fix and a green run, so the discussion doubles as the log. Testnet incidents need the same thing, with one difference: it must be internal, not public.
- Why not public GitHub: an exploit, key compromise, faucet drain or wrong slashing must not be disclosed before it is mitigated. Discussion No. 170 is fine in public because a red CI run on develop exposes nothing an attacker can use.
- What it needs: one record per incident (severity, what was detected and when, affected contracts/models/addresses or tx hashes, who is handling it, timeline, resolution, link to the post-mortem). It also needs a named primary and backup maintainer with an escalation rule: who steps in, after how long.
- Platform (to decide):
  - (a) a private repo in the org, e.g. InfiniteZeroFoundation/incidents, with one issue per incident. Same tooling as today, with access control, labels for severity, and a searchable history.
  - (b) a private Slack channel for real-time coordination, paired with (a) or a doc as the durable log.
  - (c) a private shared doc or tracker (Notion, Google Doc) with a fixed per-incident template.
  - (d) Monitor-driven incident response. OpenZeppelin Monitor (self-hosted, open source) watches DIN contracts for catalogue signals (stuck GI phase, unexpected slashing, faucet drain, proxy admin changes, upgrades) and raises incidents in PagerDuty (free tier, up to 5 users). PagerDuty holds the on-call schedule (primary and backup), the escalation policy and the per-incident record and timeline. Post-mortems are linked from each incident. Contract vulnerabilities are fixed through GitHub private security advisories. SEAL 911 is listed as the external escalation for live exploits. Revisit Hypernative if DIN moves to mainnet with real value at risk.
  - For contract vulnerabilities specifically, also consider GitHub's private security advisories on this repo, which keep the fix and disclosure private until it ships.
- Public side: once an incident is mitigated, only a sanitized summary goes out through the participant channel (item 4). The internal record stays internal.

Acceptance criteria

Runbook doc merged in the repo (location to be decided: Documentation/public/ for the participant-facing part, Documentation/technical/ for the internal procedures)
Incident catalogue covers at least the list above, each with a first response
Pause/guardian decision recorded
Participant communication channel set up and linked from onboarding docs
Internal incident log platform chosen and set up (access limited to the team), with primary and backup maintainers named and the escalation rule written down
One tabletop rehearsal done, with findings recorded

Timing

Long-running. It must be done before the first external cohort joins testnet, and should be revised after each DevNet release.

@umeradl

First draft of the incident-response runbook, for review. @umeradl could you check the "Emergency levers" table and the "Gaps found while drafting" section for accuracy? Names in "Roles and contacts" still need filling in. Once we agree the content, I'll add it to the repo.

InfiniteZero Testnet Incident-Response Runbook

Draft for review, 25 October 2026.

Purpose and scope

This runbook tells the InfiniteZero team what to do, in what order, when something goes wrong on the public testnet. It is a draft for issue #167, to be refined by the core team before launch.

An incident is anything that puts participants' funds, data, trust or ability to take part at risk, for example:

a smart-contract bug or exploit (such as security: DinEmission.fundGI lets anyone use up a model's emission schedule on far-future GIs #238, the emission drain);
a leaked or misused admin key;
honest validators being slashed or jailed wrongly;
training rounds stalling, or the network being unusable;
a broken dincli release, or an infrastructure outage (RPC, IPFS);
scams or impersonation targeting our community.

Testnet tokens have no real-money value, but treat every incident as rehearsal for mainnet: the same contracts, keys and habits carry over.

Roles and contacts

Every incident has one named incident lead who makes the calls; everyone else supports them. Fill in names and backups before launch.

Role	Responsibility	Primary	Backup	How to reach
Incident lead	Declares the incident, sets severity, decides actions, closes it	TBD	TBD	TBD
Engineering lead	Diagnoses, prepares and executes on-chain or code fixes	TBD	TBD	TBD
Key holder(s)	Signs admin transactions (platform owner, ProxyAdmins, DIN-Representative)	TBD	TBD	TBD
Comms lead	Posts updates to the community and answers questions	TBD	TBD	TBD

One person can hold more than one role during testnet, but the incident lead and the key holder should be different people where possible, so no single person acts alone on an admin key.

Open question: what response time can the team realistically commit to outside working hours? Set this before writing the severity targets below into anything public.

Severity levels

Set the severity in the first 15 minutes; raise or lower it as you learn more. When unsure, pick the higher level.

Level	Meaning	Examples	First response	Updates to community
SEV1 Critical	Active loss or misuse of funds or control; the network can't be trusted	Admin key leaked; exploit being used now (e.g. #238 drain in progress); contracts slashing honest validators at scale	Within 1 hour, all roles	Within 1 hour, then every 4 hours
SEV2 Major	Serious harm likely, or a core function broken, but contained	A training round stuck (#222); a bad `dincli` release; RPC or IPFS outage blocking participation; a vulnerability reported privately	Within 4 hours	Within 4 hours, then daily
SEV3 Minor	Limited impact, workaround exists	A single validator wrongly jailed; a docs error; a scam post in our channels	Within 1 working day	When resolved

Response times are proposals; adjust them to what the team can commit to (see Roles and contacts).

The first 30 minutes

Whoever spots a problem starts here, even if they are not the incident lead.

Raise the alarm. Post in the private team channel: what you saw, where (transaction link, contract, model) and when. Don't discuss suspected exploits in public channels yet.
Name an incident lead. The first available person from the Roles table takes it until handed over.
Set a severity using the table above.
Start an incident log. A shared note with timestamps: what happened, what was decided, what was done, by whom. It becomes the post-mortem.
Contain before you fix. Use the levers in Emergency levers to stop further harm first (for example, removing a misbehaving slasher contract or freezing minting). Two people check every admin transaction before it's signed.
Preserve evidence. Record transaction hashes, block numbers, addresses and screenshots before anything changes.
Tell the community (SEV1 and SEV2) using the first template in Communication, even if all you can say is that you're investigating.
Pick the playbook below that fits, and follow it.

Playbooks

Each playbook follows the same order: contain, fix, communicate. The lever names refer to the Emergency levers table.

1. Admin key leaked or misused (SEV1)

The deployer key owns every platform contract and all seven ProxyAdmins, so it can change settings and upgrade contract code.

Contain: if you still control the key, immediately transfer ownership of every platform contract and every ProxyAdmin to a fresh, secure address (hardware wallet or Safe). Two people verify each transaction.
If the attacker is faster: on testnet, redeploy a fresh platform from develop, publish the new addresses, and update dincli/config/din_info.json and the docs. Announce the old addresses as compromised.
Never call retireFaucet in a panic: it is one-way and also locks the mint-cap setting.
Communicate: SEV1 template; tell participants which addresses to stop using.

2. Smart-contract bug being exploited (SEV1)

No contract has a global pause, so containment means switching off the specific path being abused.

Freeze new minting if tokens are being created: set the mint cap to the current total minted. A cap of 0 means unlimited, so never set it to 0 to stop minting.
Stop a misbehaving task contract from slashing: remove it as a slasher via DinCoordinator.removeSlasherContract.
Stop a model from receiving new emission: disableModel (note: its live task contracts keep running until the Three dormant/no-op mechanisms awaiting a product decision: disableModel kill-switch, rejectModel fee handling, S6 slashing (M-2, L-4, BL-31) #224 kill-switch is built).
Fix: platform contracts can be upgraded through their ProxyAdmin after the fix is reviewed and tested; task contracts are not upgradeable, so the model owner redeploys them.
Communicate: SEV1 template; publish a post-mortem once fixed.

3. Honest validators slashed or jailed (SEV1 at scale, SEV3 for one)

Contain: remove the responsible task contract as a slasher so no further slashing happens.
Note: there is no admin "unjail". A wrongly jailed validator must wait out the jail period (7 days by default) and then call reactivate.
Make good: agree any compensation (test tokens) case by case and record it in the incident log.
Fix the root cause before re-adding the slasher.

4. Training round stuck (SEV2)

Until #222 is built, a stuck round cannot be recovered on-chain, and its rewards and validators' registration slots stay locked.

Ask the model owner to stop starting new rounds on that model.
Tell affected validators what is locked and what happens next.
Once No recovery path when a T1/T2 batch never reaches reveal quorum — GI stuck forever (BL-30) #222 ships, abort the round under the agreed policy (anyone may abort after the timeout; rewards return to whoever funded them).

5. Broken `dincli` release (SEV2)

Tell participants which version to stay on or roll back to.
Fix on develop, test, publish a new release, then announce it.

6. Infrastructure outage: RPC, IPFS, OP Sepolia (SEV2)

Check whether the problem is ours or the provider's (OP Sepolia status, RPC provider, IPFS gateway).
Point participants to a backup RPC endpoint or IPFS provider (dincli supports several IPFS providers and an opt-in gateway fallback).
No on-chain action is usually needed; post status updates until resolved.

7. A participant's key is leaked (SEV3, SEV2 if many)

Advise them to move remaining funds and unstake (the unbonding period applies).
Consider freezing the address with blacklistValidator: nobody can then stake or unstake with it, so an attacker can't withdraw the stake. It's reversible with unblacklistValidator once the owner has a safe new key.
Remind everyone to use encrypted keystores, never plaintext keys.

8. Scams and impersonation (SEV3, SEV2 if spreading)

Another project already uses the name and ticker "DIN", so expect confusion and fake airdrops.

Remove scam posts and ban the accounts in our channels.
Post the scam-warning template: we sell no tokens, run no airdrop, and never ask for keys.
Pin the official links (GitHub organisation, Telegram, Signal).

Emergency levers

These are the on-chain actions the team can take today on develop's contracts. There is no global pause, so every response uses one of these targeted levers. All except reactivate need the platform owner key (the deployer, today); task-contract settings need the model owner's key.

Lever	Contract and function	Effect	Reversible?
Freeze new minting	`DinCoordinator.setMintCap` (set to current total minted)	Stops both the ETH faucet and emission minting. A cap of 0 means unlimited.	Yes, raise the cap
Retire the faucet	`DinCoordinator.retireFaucet`	Permanently stops ETH-to-DIN minting and locks the mint-cap setting	No, one-way
Stop a contract slashing	`DinCoordinator.removeSlasherContract`	That task contract can no longer slash or jail anyone	Yes, re-add it
Disable a model	`DINModelRegistry.disableModel`	Blocks registration and manifest updates and new emission funding; live task contracts keep running (#224)	Yes, `enableModel`
Freeze an address	`DinValidatorStake.blacklistValidator`	The address can't stake or unstake	Yes, `unblacklistValidator`
Change penalty settings	`DinValidatorStake.setS5GlobalParams`, `setS5RecidivismParams`, `setMinStake`, `setUnbondingPeriod`	Adjusts penalties and stake rules; `setS5GlobalParams(0, 0)` switches the cross-model escalation off	Yes
Change emission	`DinEmission.setEmissionParams`	Adjusts the per-round subsidy schedule	Yes
Change the reward split	`DINTaskAuditor.setRewardSplit` (model owner)	Changes who receives each round's pool	Yes
Upgrade a platform contract	The contract's ProxyAdmin	Replaces the contract's code after a reviewed fix	Yes, by another upgrade
Move control to a safe key	`transferOwnership` on each contract and ProxyAdmin	Hands control to a new owner address	Only by the new owner
Leave jail	`DinValidatorStake.reactivate` (the validator)	Ends a jail once its period has expired	Not applicable

Practise each lever on the DevNet 3.0 rehearsal before launch, so nobody uses one for the first time during an incident.

Communication

The comms lead posts in the Telegram community (main channel) and a pinned GitHub Discussion, with Signal as a mirror. Say what you know, what you don't, and when the next update comes. Never post exploit details until the issue is fixed.

1. Initial notice (SEV1/SEV2)

⚠️ We're investigating an issue affecting [what: e.g. staking / training rounds / dincli] on the InfiniteZero testnet, first seen at [time, UTC].

What to do for now: [e.g. don't start new rounds / don't stake / no action needed].

Testnet tokens have no real-money value. Next update by [time, UTC].

2. Update

Update on [issue], [time, UTC]: [what we've learned and done]. [What participants should do]. Next update by [time, UTC].

3. Resolved

✅ Resolved: [issue]. Cause: [one line]. What we changed: [one line]. What you need to do: [nothing / action]. A full write-up will follow at [link]. Thank you for your patience.

4. Scam warning

🚫 Scam warning: InfiniteZero has no token for sale and no airdrop running, and we will never ask for your private key or seed phrase. Official links only: github.com/InfiniteZeroFoundation and this Telegram group. Please report suspicious messages to the admins.

After an incident

Within 5 working days of a SEV1 or SEV2, the incident lead publishes a short, blameless post-mortem as a GitHub Discussion, and opens an issue for every follow-up action. Blameless means it describes what happened and why the system allowed it, not who to blame.

# Post-mortem: [title], [date]

Severity: SEV[n] · Duration: [start] to [end] (UTC) · Incident lead: [name]

## Summary
[Two or three sentences: what happened and the impact.]

## Impact
[Who was affected, and how: participants, funds, rounds, downtime.]

## Timeline (UTC)
- [time]: [event]

## Root cause
[Why it happened, and why existing checks didn't catch it.]

## What went well / what didn't
- ...

## Follow-up actions
- [ ] [action] (#issue, owner)

Review this runbook after every SEV1 or SEV2, and update it with anything the incident taught you.

Before launch

Tick these off before the public testnet opens; together they make this runbook usable on day one.

Gaps found while drafting

Writing this runbook against develop (commit 55c4b52) surfaced limits in what the team can do during an incident. Each is a decision for the team; raise the ones you agree with as issues.

Gap	Why it matters in an incident	Already tracked?
No global pause on any contract	An exploit can only be stopped path by path, which is slower and easier to get wrong	No
No admin way to release a wrongly jailed validator	Honest validators stay jailed for the full period (7 days by default) even after the team confirms a mistake	No
Disabling a model doesn't stop its live training rounds	The main brake on a misbehaving model is partial	Yes, #224 (decision recorded)
Stuck rounds can't be recovered	Rewards and validators' registration slots stay locked	Yes, #222 (policy decided)
A mint cap of 0 means unlimited; retiring the faucet also locks the cap	Easy to make a mistake under pressure; both are noted in the levers table	No (behaviour, not a bug)
One deployer key controls every contract and all seven ProxyAdmins	A single leaked key gives full control	Partly, in the DevNet 3.0 issue (key custody)
Task contracts aren't upgradeable	A bug in them is fixed only by each model owner redeploying	No (by design)

Navigation Menu

Repository navigation

Incident response runbook and incident communication for testnet #167

Why

Scope

Acceptance criteria

Timing

InfiniteZero Testnet Incident-Response Runbook

Purpose and scope

Roles and contacts

Severity levels

The first 30 minutes

Playbooks

1. Admin key leaked or misused (SEV1)

2. Smart-contract bug being exploited (SEV1)

3. Honest validators slashed or jailed (SEV1 at scale, SEV3 for one)

4. Training round stuck (SEV2)

5. Broken `dincli` release (SEV2)

6. Infrastructure outage: RPC, IPFS, OP Sepolia (SEV2)

7. A participant's key is leaked (SEV3, SEV2 if many)

8. Scams and impersonation (SEV3, SEV2 if spreading)

Emergency levers

Communication

After an incident

Before launch

Gaps found while drafting

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Incident response runbook and incident communication for testnet #167

Description

Why

Scope

Acceptance criteria

Timing

Activity

abrahamnash commented on Oct 26, 2026

InfiniteZero Testnet Incident-Response Runbook

Purpose and scope

Roles and contacts

Severity levels

The first 30 minutes

Playbooks

1. Admin key leaked or misused (SEV1)

2. Smart-contract bug being exploited (SEV1)

3. Honest validators slashed or jailed (SEV1 at scale, SEV3 for one)

4. Training round stuck (SEV2)

5. Broken dincli release (SEV2)

6. Infrastructure outage: RPC, IPFS, OP Sepolia (SEV2)

7. A participant's key is leaked (SEV3, SEV2 if many)

8. Scams and impersonation (SEV3, SEV2 if spreading)

Emergency levers

Communication

After an incident

Before launch

Gaps found while drafting

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

5. Broken `dincli` release (SEV2)