Skip to content

task_210726_6.md

Latest commit

 

History

History
199 lines (132 loc) · 16.7 KB

task_210726_6.md

File metadata and controls

199 lines (132 loc) · 16.7 KB
Raw

Task: Scoring Validation & Auditor Evaluation Mechanism + Reward Engine Bootstrap (Issues #39/#40, partial #41/#38)

ID: task_210726_6 Assigned to: Similoluwa Abidoye (@Abidoyesimze) Created: 2026-07-21 Status: Open Repo: https://github.com/InfiniteZeroFoundation/DevNet (base branch: develop) Roadmap ref: P3-SCR, WP 2.2, P3-4.1 (S3 threshold only), P3-5.1/5.2 (partial reward engine) Dates: Jul 21–27, 2026 (5 working days; deadline Monday, July 27)

Context

Issues #38 (slashing), #39 (scoring), #40 (auditing), and #41 (rewards) are all open and unassigned. Robbert is concurrently running task_210726_5 this same week (issues #37/#42/#43): a new DinTreasury, a new DinFeeRouter, DinToken.burn(), DinCoordinator mint-cap/treasury-routing, and governable MIN_STAKE plus real jailValidator()/reactivate() in DinValidatorStake.

Several sub-items of #38 and #41 depend on contracts Robbert hasn't merged yet. Rather than block this task on his PR landing, it's scoped to the parts of #39/#40/#41/#38 that have zero dependency on task_210726_5, plus the parts of #38 that are safe to do regardless of merge order.

In scope this week:

  • #39 (scoring) — full scope
  • #40 (auditing) — full scope
  • #41 (rewards) — the self-contained per-GI reward engine (deposit, role split, settlement, claim)
  • #38 (slashing) — taxonomy spec, invariant tests against existing slash(), dispute-resolution scaffold only (stretch — attempt only if Parts 1-3 finish early)

Explicitly out of scope this week:

  • #41: the decaying protocol-emission top-up — needs DinEmission, itself blocked on the still-open P3-5.2 simulation (out of scope for everyone right now, not just you)
  • #38: DAO-settable partial-slash fraction of MIN_STAKE (needs Robbert's governable storage), the 50/50 burn/treasury slashed-stake destination (needs DinTreasury/DinFeeRouter), jail/blacklist tier wiring (needs Robbert's jailValidator())
  • Exact reward role-split percentages — 60/20/15/5 client/auditor/aggregator/treasury is a proposal pending Umer's P3-5.2 simulation at 10–50 validators/100–500 clients. Build the mechanism as DAO-settable with the proposal as the initial default — don't treat the numbers as final.
  • Your existing PRs #22/#30 are untouched by this task — that's a separate merge-queue conversation, not part of this scope.

If anything here conflicts with what you find in the live code, flag it — this spec was written against develop as of 2026-07-21 (commit c003ff1).

Setup

git clone https://github.com/<your-fork>/DevNet
cd DevNet
git checkout develop && git pull
git checkout -b feature/scoring-auditing-rewards

cd foundry && forge build && forge test

Unlike your security-review trial task (which pinned to a fixed historical commit because the platform contracts were mid-flight under PR #13), this task builds new functionality into DINTaskAuditor.sol/DINTaskCoordinator.sol — files Robbert's concurrent task doesn't touch at all. Building against develop's moving tip is the right call here since there's no file-overlap ambiguity. Still: re-pull and re-run forge build && forge test right before opening your PR, in case his PR has landed by then.

Read before writing anything:

  1. Developer/design/MECHANISM_DESIGN.md §5 (Rewards), §6 (Scoring & Auditing)
  2. Developer/issues/scoring-mechanism.md, auditor-evaluation-mechanism.md, client-reward-mechanism.md, validator-reward-mechanism.md
  3. Developer/design/whitepaper-summary.md §4.4, §8 (gap items 3, 5, 6, 11, 14)
  4. Developer/design/scoring-auditing-references.md — prior-work mapping (BlockFlow, 2CP, Shapley-value data valuation, IPFS) for the median-scoring, commit-reveal, and encrypted-test-data mechanisms below; includes a runnable permutation-averaging reference implementation for §1a
  5. foundry/src/DINTaskAuditor.sol and foundry/src/DINTaskCoordinator.sol in full — the two contracts you'll modify
  6. Documentation/public/workflows/model-workflow.md — the GI lifecycle

Reference papers (public links)

  • V. Mugunthan, R. Rahman, L. Kagal, "BlockFlow: An Accountable and Privacy-Preserving Solution for Federated Learning" — arXiv:2007.03856 — source of the median-scoring mechanism and S3's 0.5-deviation threshold; its 5-stage round protocol (evaluation-commit → evaluation-reveal) is the closest precedent for §2a
  • H. Cai, D. Rueckert, J. Passerat-Palmbach, "2CP: Decentralized Protocols to Transparently Evaluate Contributivity in Blockchain Federated Learning Environments" — arXiv:2011.07516 — step-by-step marginal-gain contributivity formula behind §1b's marginal-gain gate
  • R. Jia et al., "Towards Efficient Data Valuation Based on the Shapley Value" — arXiv:1902.10275 — permutation-averaged Monte Carlo Shapley estimation, relevant to §1a's "permutation averaging"
  • R. Jia et al., "Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms" — arXiv:1908.08619 — closed-form KNN-Shapley; same lineage as the already-rejected TKNN-Shapley idea (Developer/rejected-ideas/tknn-shapley.md) — do not resurface for per-datapoint valuation, only its model/update-level form applies here
  • J. Benet, "IPFS - Content Addressed, Versioned, P2P File System" — arXiv:1407.3561 — content-addressing basis for §2b's encrypted test-data distribution
  • LabeliaLabs/distributed-learning-contributivity (GitHub) — reference implementation of 9 contributivity-scoring methods including a permutation-MC Shapley approximation (mplc/contributivity.py:truncated_MC), directly usable for §1a's spec-to-code mapping

1. Scoring: P3-SCR validation + on-chain median (Issue #39)

1a. Locate and map spec to code — do this first, don't assume

MECHANISM_DESIGN §6 and issue #39 describe "sequential fold-in" and "permutation averaging" as already shipped (WP 2.2 ✅). We searched cache_model_0/services/scoring.py and aggregator.py and found neither term anywhere. Your first deliverable: a short mapping of which function(s) actually implement fold-in duplicate discounting and permutation averaging today — or a clear statement that they don't exist yet and need to be built from spec. Post this before writing tests against the wrong target.

1b. Empirical validation (pytest)

Once the mapping is confirmed:

  • Poison detection: label-flip, sign-flip, scaling, garbage-update injection
  • Cosine-to-consensus / norm-bound / marginal-gain gates at multiple ε values
  • Duplicate discounting via fold-in (or documented absence)
  • Sequential fold-in and permutation averaging correctness (or documented absence)

1c. On-chain median at finalizeEvaluation

finalizeEvaluation (line 552) already computes and stores finalAvgScore (line 599) per submission, gated separately by minEligibilityQuorum and minScoreQuorum (two distinct parameters, lines 39-40 — confirm these aren't accidentally conflated, since both are counted via similar vote-tallying paths). Confirm this is correct and usable as the canonical per-model score for the S3 slashing reference and reward basis — don't rebuild it from scratch, verify and fix if needed.

1d. Threshold framing

Ship the S3 deviation threshold in shadow mode: computed/emitted, not gating anything yet. Settable parameter, placeholder default — don't hard-code a specific number.

2. Auditing: commit-then-reveal, encrypted test data, resampling (Issue #40)

Three design decisions, all resolved in principle (Abraham, 2026-07-21) — your job is implementation:

2a. Commit-then-reveal for auditor score submission

Auditors currently call setAuditScorenEligibility (line 515) in plaintext in one transaction, letting a later auditor copy an earlier one's score. Implement two-phase submission: commitAuditScore(gi, batchId, modelIndex, commitHash) where commitHash = keccak256(abi.encodePacked(score, vote, salt)), then revealAuditScore(gi, batchId, modelIndex, score, vote, salt) validating against the stored hash before running the existing scoring/eligibility logic. Add a reveal-window mechanic — check DINShared.sol's GIstates enum before adding a new state, since it's shared across both task contracts. Auditors who commit but never reveal must not silently corrupt quorum counting.

2b. Encrypted audit test-data distribution

Current flow (assignAuditTestDataset, line 421) assigns a single plaintext-shared testDataCID. Whitepaper §5.2.3b specifies per-validator key encryption with on-chain key mapping. Add mapping(uint256 => mapping(uint256 => mapping(address => bytes))) public encryptedTestDataKey (GI → batch → auditor → their encrypted key), populated when assignAuditTestDataset runs. No dincli/Python decryption change required — on-chain plumbing only.

2c. Test-set resampling strategy

Reserved pool (~40%), resampled half per round, full pool in the final round. Primarily an off-chain sampling-policy design (where the test dataset is assembled before its CID is pinned) — document the algorithm, wire it into an existing dataset-assembly script if one exists under dincli//cache_model_0/, otherwise write it as a design doc under Developer/design/ and flag implementation as follow-up.

2d. Aggregator auditing (S4 disputes) — spec only here

Referenced for completeness; built once in Part 4, don't duplicate.

3. Rewards: per-GI pool, role split, settlement, claim (Issue #41, core scope only)

What's broken today

DINTaskAuditor.totalDepositedRewards (line 16) and RewardDeposited (line 102) are declared but never written or emitted anywhere in the codebase. DINTaskCoordinator.endGI() (line 783) does nothing but flip GIstate to GIended — there is no settlement, no claimable-balance crediting, no claim function. This is a green field, not a partial implementation.

What to build

  1. Per-GI pool, not a running total. Replace totalDepositedRewards with mapping(uint256 => uint256) public giRewardPool (keyed by GI number) — a single running total can't support the "funded per-GI, settled per-GI" model MECHANISM_DESIGN §5 describes.
  2. DIN-denominated deposit. Add IERC20 public dinToken + setDinToken(address) (onlyOwner, deploy-time — mirror Robbert's DINModelRegistry.setDinToken pattern for consistency). Add depositRewards(uint256 gi, uint256 amount) external pulling DIN via safeTransferFrom, crediting giRewardPool[gi], emitting RewardDeposited.
  3. Precondition on _startGI. Wire DINTaskCoordinator._startGI (line 159) to check the auditor contract's giRewardPool(_GI) > 0 (or a DAO-settable minimum) before allowing the GI to start — revert with a clear custom error if unfunded. Follow the existing cross-contract reference pattern between coordinator and auditor.
  4. DAO-settable role split, not hard-coded. struct RewardSplit { uint16 clientBps; uint16 auditorBps; uint16 aggregatorBps; uint16 treasuryBps; } — mirror Robbert's bps-based DinSplit/EthSplit pattern for codebase consistency. Default to 6000/2000/1500/500 (the 60/20/15/5 proposal). Setter validates the sum equals 10000. These numbers are explicitly not final.
  5. Settlement at endGI. Once GIstate == AggregatorsSlashed, compute and credit mapping(address => uint256) public claimable:
    • Clients: proportional to finalAvgScore among accepted (approved == true) submissions, scaled by clientBps
    • Auditors: equal share per completed assignment (correctness enforced by slashing, not reward weighting), scaled by auditorBps
    • Aggregators: per finalized T1/T2 batch, scaled by aggregatorBps
    • Treasury share: accrue in a local treasuryAccrued variable with a // TODO(task_210726_5): forward to DinTreasury once merged comment and a settable treasury address for later wiring — DinTreasury doesn't exist on develop yet, don't block on it.
    • Split computation must happen fully on-chain so payouts are independently verifiable.
  6. Pull-payment claim only. claimRewards() external nonReentrant — transfers claimable[msg.sender], zeroes first (checks-effects-interactions), reverts on zero balance. No push-transfer loops in endGI — this is exactly the class of unbounded-loop gas-DoS risk your own PR #22 findings warned about elsewhere in these contracts.
  7. Anti-gaming — reuse Parts 1-2, don't reimplement: zero reward below passScore (already gated via approved), fold-in duplicate discounting naturally reduces a duplicate's finalAvgScore contribution (if it exists per 1a), median-bounded score inflation bounds the client-reward basis.

Explicitly not in scope

Protocol-emission top-up, actual DinTreasury forwarding (stub only), final role-split percentages, aggregator reward scaling with batch size (note as a TODO if time runs short).

4. Slashing: taxonomy spec + invariants + dispute scaffold (Issue #38, stretch only)

Attempt only if Parts 1-3 finish with time to spare. If the budget runs tight, cut this entirely and say so in the PR — most of #38's actionable work is blocked on task_210726_5 anyway.

4a. Conditions taxonomy — spec only, no contract change

Write up S1–S6 (missed audit vote, missed T1/T2 submission, score deviation from median (S3, reuse Part 1's on-chain median), invalid aggregation via recomputation (S4), recidivism (S5), registration without capacity (S6)) as a design doc, mapping each to where it would hook into existing contracts.

4b. Invariant/fuzz tests against slash() as it exists today

DinValidatorStake.slash() (line 165) exists and is untouched by Robbert's task. Write Foundry invariant tests proving, against current behavior: staked supply never increases after a slash; slashed amount's actual destination today (confirm: does it strand as pure accounting with no transfer/burn at all, as the design docs claim — verify with a test, don't take it on faith); no negative stake states are reachable.

4c. Dispute-resolution scaffold — stub the payout

Dispute bond + window + fresh-subgroup re-evaluation mechanism (likely in DINTaskCoordinator, alongside T1/T2 finalize logic). Stub the actual bounty/bond-return payout behind a DinTreasury-shaped interface with the same TODO pattern as Part 3 — don't implement real fund movement against a contract that doesn't exist yet.

Explicitly not in scope

DAO-settable partial-slash fraction, burn/treasury destination routing, jail/blacklist tier wiring, dispute bond size/window numeric values (implement as settable parameters with placeholder defaults).

Deliverables

Scoring (#39)

  • Spec-to-code mapping for fold-in/permutation averaging posted before writing tests
  • pytest coverage for poison detection, cosine/norm/marginal-gain gates, fold-in/permutation (or documented absence)
  • On-chain canonical score at finalizeEvaluation confirmed or fixed
  • S3 deviation threshold shipped in shadow mode

Auditing (#40)

  • Commit-then-reveal implemented with reveal-window handling
  • Per-auditor encrypted test-data key mapping added on-chain
  • Test-set resampling policy documented (implemented if a hook point exists)

Rewards (#41, core scope)

  • Per-GI reward pool (giRewardPool) replacing the dead counter
  • depositRewards, wired as a precondition of _startGI
  • DAO-settable role split (bps-based), defaulted to 60/20/15/5
  • On-chain settlement at endGI crediting claimable
  • Pull-payment claimRewards() — no push-transfer loops
  • Treasury forwarding stubbed with a clear TODO

Slashing (#38, stretch only)

  • S1–S6 taxonomy spec written
  • Invariant/fuzz tests against current slash() behavior
  • Dispute-resolution scaffold with stubbed treasury payout

Submission

PR into develop, referencing #39, #40, #41 (and #38 if attempted). Describe what you built per part, what you deferred and why, and the spec-to-code mapping from §1a. If Robbert's PR lands on develop before yours, re-pull/re-test and note whether anything needs adjusting now that DinTreasury/DinFeeRouter/governable MIN_STAKE/jailing actually exist.

Notes

  • No dincli/Python changes required beyond §1's pytest work.
  • Flag any PR touching DinValidatorStake.sol, DINTaskAuditor.sol, or DINTaskCoordinator.sol for Umer's review given cryptoeconomic sensitivity.
  • If §1a finds fold-in/permutation genuinely doesn't exist, don't silently build a full implementation from scratch inside this budget — write the design doc and a minimal reference implementation, then flag scope in the PR.
  • Every "stub"/TODO in Parts 3-4 is intentional — reviewers shouldn't ask you to un-stub before merge.