Start DIN-Indexer #24

Activity

1. robertocarlous commented on Jul 10, 2026

   

   robertocarlous

   on Jul 10, 2026

   Last edited by robertocarlous

   Collaborator

   More actions

   Hi Umer, Please i need some clarification and answer to the following Questions

   1. Missing event fields — propose additions or use call handlers?

   ModelRegistrationRequested only emits requestId and requester. The isOpenSource flag and fee amount are not in the payload to filter pending requests by type in the indexer I'd either need to add those fields to the event, or use a contract call inside the mapping handler. Same for ManifestUpdateRequested which emits requestId and modelId but not the requester address. My preference is to propose minimal event additions (one or two extra fields) rather than call handlers, since call handlers make the subgraph less portable. Is that acceptable, or do you want zero contract changes and call handlers instead?

   2. RewardClaimed, GIStarted, GIEnded in the ROADMAP entity schema

   The ROADMAP (P4-IDX1 description) lists these as target events for the entity schema, but none of them exist in the 4 platform contracts. They look like task-level contract events (DINTaskCoordinator). Are these planned additions to the platform contracts, or were they listed by mistake and the schema should only cover what the platform contracts actually emit today?

   3. Directory name

   The issue says pick one: indexer/ or subgraph/. I lean toward subgraph/ if we go with The Graph (names the tool clearly), indexer/ if we keep it tool-agnostic. Which do you prefer?

   4. Local Graph node vs Hardhat chainId

   The Hardhat local network is configured with chainId: 1337. The Graph node's networks.toml needs to match that chain. Should the local dev setup target hardhat network (chainId 1337) or the localhost fork of Optimism Sepolia (chainId 11155420 as configured in hardhat.config.ts)?

   5. Confirming the only in-scope P4-IDX2 replacement target

   The lms.py (~line 62) and aggregation.py (~lines 76, 95, 136) loops are against DINTaskAuditor / DINTaskCoordinator — task-level contracts, which are explicitly out of scope. Confirming: the only loop I replace in P4-IDX2 is the dindao.py pending-requests enumeration (lines 443–467), correct?
2. umeradl commented on Jul 10, 2026

   

   umeradl

   on Jul 10, 2026

   MemberAuthor

   More actions

   Hi Robert, answers to each below.

   1. Missing event fields — event additions confirmed, no call handlers.

   Yes — propose minimal event additions. That's the direction Developer/issues/indexer.md already commits to ("adding missing events for important transitions / improving indexed event fields" are listed as the good minimal contract changes), and the issue's event-coverage audit checkbox says exactly this: "propose minimal contract-side event additions rather than storage changes." Call handlers are also a practical problem beyond portability — they don't work on all Graph node backends and would slow down indexing.

   Concretely, for the two events you named:

   • ModelRegistrationRequested(uint256 indexed requestId, address indexed requester) → add bool isOpenSource and uint256 fee
   • ManifestUpdateRequested(uint256 indexed requestId, uint256 indexed modelId) → add address indexed requester

   Since these land as part of the event-coverage audit (P4-IDX1 checkbox), write them up as proposals in the audit doc first rather than pushing contract changes directly — then land all event additions together as one small reviewed PR once the audit is complete. Note the platform contracts are behind proxies now (PR #13), but since they're only deployed locally (anvil/hardhat) at this point, redeploying/upgrading with the new event signatures is low-friction — no live-chain migration concern.

   2. RewardClaimed/GIStarted/GIEnded — scope the schema to what the platform contracts actually emit today.

   You're right — none of those three events exist in the platform contracts (they're not even task-level events under those names: DINTaskAuditor has RewardDeposited, not RewardClaimed, and neither task contract emits GI start/end events). Treat the ROADMAP wording as placeholder text from before any event audit was done.

   The priority is: work on the platform contracts first and get a running indexer/subgraph end-to-end — schema, mappings, local Graph node, the dindao.py call-site integration. Only after that's running do we start on task-level contracts. Task-level indexing is also a different kind of effort: the events follow the federated-learning lifecycle (Global Iterations, LMS, auditor scoring, two-tier aggregation, slashing), so understanding the FL workflow matters a lot there — Documentation/public/workflows/model-workflow.md is the place to build that understanding when the time comes. For now, anything you notice about missing GI/reward events goes in the P4-IDX3 handoff notes, not this schema.

   One heads-up for the schema: SlasherContractAdded/SlasherContractRemoved are emitted by both DinCoordinator and DinValidatorStake with identical signatures — keep the source contract distinguishable in your entities so the two streams don't get conflated.

   3. Directory name: subgraph/.

   Go with subgraph/ — we're committing to The Graph per P4-IDX1, so name the directory after the tool. Alongside the code, document how to run it locally (setup steps, docker-compose if needed — this is already a P4-IDX2 checkbox) and how dincli integrates with it. Integration order is: dincli now (the dindao.py call site), then the SDK and the daemon (dind) later, in that order, as each becomes available — write the run/integration docs so those follow-on consumers can plug in without rework (your daemon event schema from P4-IDX1 already feeds dind's P4-7.1).

   4. Local Graph node — target chainId 1337.

   Point the Graph node at the plain local chain, chainId 1337. Both local options use it: foundry/anvil.sh runs anvil with --chain-id 1337, and the hardhat network in hardhat.config.ts is also chainId: 1337. The 11155420 you saw is only the sepolia_op_devnet network (the live chain). No forked chain needed for indexer dev — deploy the platform contracts fresh to the local node and index that. Use anvil (./foundry/anvil.sh) as the canonical local chain, since that's what the documented devnet workflow uses (dincli system configure-network --network local).

   5. Confirmed — dindao.py is the only in-scope replacement.

   Correct. The only loop you replace in P4-IDX2 is the pending-requests enumeration in dincli/cli/dindao.py (list-pending-requests, both the model-request and manifest-request loops). You read it right: the lms.py and aggregation.py loops run against DINTaskCoordinator/DINTaskAuditor — task-level contracts, out of scope here. List them in your write-up as follow-up candidate call sites (per the P4-IDX2 checkbox) and leave the code untouched — we'll sort those loops out once we start working on the task-level contracts.

   Timeline. No hard deadline on this issue — the ROADMAP dates were drawn up for a one-developer setup and don't apply now that we're working as a team. The indexer/graph is needed for DevNet 3.0, so there's runway. And nothing is blocking anything: the contracts and ABIs are there, so go straight through design into implementation and tests. If contract changes land later (e.g. the event additions from your audit), just update the mappings then — that's cheap compared to waiting.
3. umeradl

   added

   Checked

   and removed

   Checked

   on Jul 10, 2026
4. robertocarlous commented on Jul 10, 2026

   

   robertocarlous

   on Jul 10, 2026

   Collaborator

   More actions

   Thank You Umer, i will look into this!
5. umeradl commented on Jul 13, 2026

   

   umeradl

   on Jul 13, 2026

   MemberAuthor

   More actions

   @robertocarlous One correction and a note on what to work on next.

   Correction — there is no Phase 5. The "deferred to P5+" wording in this issue body and my earlier comment was a mistake: Developer/ROADMAP.md has no P5 — the roadmap currently runs through P3 (cryptoeconomic layer) and P4 (daemon + indexer). Please disregard the P5+ references.

   Next step: until your next weekly task is assigned, continue working on the subgraph — extend it to the task-level contracts (DINTaskCoordinator / DINTaskAuditor) via dynamic data sources, the follow-up previously labelled out of scope here. Since task-level events follow the federated-learning lifecycle (Global Iterations, LMS, auditor scoring, two-tier aggregation, slashing), read Documentation/public/workflows/model-workflow.md first to build that context before designing the task-level entities. The lms.py and aggregation.py RPC-loop call sites from your PR #29 handoff notes are the natural integration targets.
6. robertocarlous commented on Jul 13, 2026

   

   robertocarlous

   on Jul 13, 2026

   Collaborator

   More actions

   Thanks for the correction noted. I'll treat the task-level dynamic data sources as the immediate continuation of this work, not a future milestone.

   Read through Documentation/Model-workflow.md. The GI lifecycle is clear: aggregator/auditor registration → LMS submission by clients → auditor evaluation/scoring → T1 and T2 aggregation → slashing → GI end. Each phase maps to a set of events on DINTaskCoordinator and DINTaskAuditor. Will audit those contract events next, then design the task-level entities and dynamic data source config before writing any handlers.

   The lms.py (~line 62) and aggregation.py (~lines 76, 95, 136) RPC loops from the handoff notes are the planned integration targets once the schema is in place.
7. robertocarlous commented on Jul 13, 2026

   

   robertocarlous

   on Jul 13, 2026

   Collaborator

   More actions

   Hi @umeradl Pls can you take look at this

   Task-level contract event audit — findings and a design question before proceeding

   Read through both contracts. Here is what exists and what is missing before I design the entities.

   Existing events

   DINTaskCoordinator emits 4 events: DINValidatorRegistered (aggregator registration), Tier1BatchAuto, Tier2BatchAuto (batch creation), and AggregatorSlashed.

   DINTaskAuditor emits 8 usable events: DINAuditorRegistered, AuditorsBatchAuto, AuditorsBatchesCreated, AuditScoreSubmitted, EligibilityVoted, EligibilityFinalized, PassScoreUpdated, AuditorSlashed. One declared event — RewardDeposited — is never emitted (no depositRewards function exists); flagging as dead code.

   Critical gaps

   Three functions that are direct targets for replacing RPC loops emit no event:

   • submitLocalModel() on DINTaskAuditor — no event. lms.py ~line 62 iterates getClientModels() over RPC and cannot be replaced without a LocalModelSubmitted event.
   • submitT1Aggregation() and submitT2Aggregation() on DINTaskCoordinator — no events. aggregation.py ~lines 76, 95, 136 iterates T1/T2 batches over RPC and cannot be replaced without submission events.

   Additionally, the entire 23-state GIstates machine is silent — none of the 14 phase-transition functions (startGI, startLMsubmissions, closeLMsubmissions, startT1Aggregation, endGI, etc.) emit an event.

   Proposed minimal additions (same approach as the platform-contract audit)

   Contract | Addition | Signature -- | -- | -- DINTaskAuditor | LocalModelSubmitted | (uint indexed gi, uint indexed modelIndex, address indexed client, bytes32 modelCID) DINTaskCoordinator | T1AggregationSubmitted | (uint indexed gi, uint indexed batchId, address indexed aggregator, bytes32 cid) DINTaskCoordinator | T2AggregationSubmitted | (uint indexed gi, uint indexed batchId, address indexed aggregator, bytes32 cid)

   Design question before proceeding

   For the GI state machine: should I propose a single catch-all GIStateChanged(uint indexed gi, uint8 indexed newState) emitted by all 14 silent transition functions — or would you prefer individual named events per phase (GIStarted, LMSOpened, LMSClosed, T1Started, GIEnded, etc.)? The catch-all is fewer contract changes; individual events are more readable in the subgraph and easier to filter in dind. Flagging this before designing the entities so I pick the right schema shape.
8. umeradl commented on Jul 13, 2026

   

   umeradl

   on Jul 13, 2026

   MemberAuthor

   More actions

   @robertocarlous Good audit — I checked everything against develop and it's all accurate: the event inventory, RewardDeposited declared but never emitted (no depositRewards function exists), no events on submitLocalModel / submitT1Aggregation / submitT2Aggregation, and the fully silent GI state machine. Answers below.

   1. GI state machine — go with the single catch-all GIStateChanged.

   event GIStateChanged(uint indexed GI, uint8 indexed newState);

   Reasons:

   • The state machine is bigger than the 14 owner-callable transition functions: GIstate is written at 23 sites in DINTaskCoordinator (constructor, the pre-GI setup steps AwaitingDINTaskAuditorToBeSet → GenesisModelCreated, and writes inside the batch-creation and aggregation-close flows). Individual named events would mean ~20 declarations and 23 emit sites to keep in sync — a lot of ABI surface for one enum.
   • The clean implementation is an internal setter: replace every GIstate = GIstates.X; with _setGIstate(GIstates.X) that assigns and emits. One declaration, one emit site, and it's impossible to miss a transition when states are added later.
   • Filterability in dind isn't actually lost: with newState indexed, any consumer can topic-filter for a single state — e.g. subscribe to GIStateChanged with newState == 16 (T1AggregationStarted) to queue aggregation jobs. That's exactly the P4-7.1 wiring. The named T1AggregationStarted event in the ROADMAP is placeholder wording from before this audit, same as the GIStarted/GIEnded names we already corrected.
   • Readability belongs in the subgraph, not the Solidity ABI: define a GraphQL enum in schema.graphql mirroring GIstates (DINShared.sol is the source of truth — dincli already mirrors it in its GI state helpers) and map uint8 → enum in one handler. Queries read just as well as named events would, with one handler instead of fourteen near-identical ones.
   • It's also the Developer/issues/indexer.md default: minimal contract changes for event coverage.

   One simplification for you: the entire state machine lives in DINTaskCoordinator — DINTaskAuditor only reads state cross-contract — so this is a single-contract change.

   2. Three proposed submission events — confirmed, signatures are right.

   Verified against storage: modelIndex / client / modelCID match the LMSubmission struct, and bytes32 is correct for the T1/T2 CIDs. Two small notes:

   • Match the existing param style (uint indexed GI, capitalized) so the new events stay consistent with the current ABI.
   • T2AggregationSubmitted.batchId is always 0 today (TC_OnlyOneTier2Batch enforces it) — keep it anyway for symmetry, just noting so it doesn't surprise you in the data.

   One more candidate to run through the same audit filter: closeT1Aggregation / closeT2Aggregation select the majority CID per batch, and that winner lives only in storage. With submission events alone, the indexer would have to re-implement the vote-majority logic to know the finalized CID — and the T2 winner is the new global model, the single most important value in a GI. Consider proposing T1BatchFinalized(uint indexed GI, uint indexed batchId, bytes32 winningCID) and T2Finalized(uint indexed GI, bytes32 globalModelCID) alongside. Your call after reading the close functions.

   3. RewardDeposited dead code — agreed, flag it.

   Document it in the audit write-up as dead code with a removal proposal, but don't remove it yet — the P3 cryptoeconomics work (fee routing / reward distribution, P3-5.1–5.3) may introduce a real deposit flow, and that work should decide whether the event gets an emitter or gets deleted.

   Process — same as the platform-contract audit: write these up as proposals in the audit doc first, then land all task-contract event additions as one small reviewed PR. Put the audit doc under Documentation/technical/audits/ (the foundry security review already lives there — follow that convention). One difference from the platform contracts: task contracts aren't proxied — they're deployed fresh per model by the model owner — so the new events only apply to newly deployed task contracts; already-deployed tasks on sepolia_op_devnet won't emit them. That's fine for dynamic data sources going forward, just state it explicitly in the write-up.
9. umeradl

   assigned

   umeradl

   and

   robertocarlous

   and unassigned

   umeradl

   on Jul 15, 2026
10. umermjd11 commented on Sep 8, 2026

    

    umermjd11

    on Sep 8, 2026

    Collaborator

    More actions

    Status check (2026-09-08)

    Comprehensive implementation in PR #29 — open, not draft, appears to close this issue on merge:

    • Covers P4-IDX1 (event-coverage audit, entity schema, subgraph manifest, daemon event schema), P4-IDX2 (AssemblyScript mapping handlers, docker-compose local Graph node stack, 14 example queries), P4-IDX3 (both RPC enumeration loops in dincli/cli/dindao.py replaced with GraphQL + silent RPC fallback, 6 unit tests, handoff notes)
    • Goes beyond this issue's stated scope: also indexes the task-level contracts (DINTaskCoordinator/DINTaskAuditor) via dynamic data sources, which this issue explicitly listed as out of scope ("deferred to P5+") — not a problem, just wider than asked
    • 5 handlers are commented out pending contract event additions proposed in the audit — flagged clearly in the PR as the one real gap, not hidden

    Not yet merged: dincli/cli/dindao.py:443 on current develop still has the raw for idx in range(totalModelRequests) RPC loop this PR is meant to replace.

    Leaving this open until #29 merges — but of the four P4-kickoff issues (#20/#21/#23/#24), this looks closest to being done outright.
11. umermjd11

    added

    Long run

    on Sep 10, 2026
12. umeradl

    added a parent issue on Sep 25, 2026

    • Testnet readiness: tracking what must be in place before testnet opens #166
13. umeradl

    removed a parent issue on Sep 25, 2026

    • Testnet readiness: tracking what must be in place before testnet opens #166