diff --git a/Documentation/public/roles/aggregators.md b/Documentation/public/roles/aggregators.md index 72786b2e..2858a923 100644 --- a/Documentation/public/roles/aggregators.md +++ b/Documentation/public/roles/aggregators.md @@ -103,7 +103,7 @@ dincli aggregator reveal-t1 [--gi ] [--batch ] | `--gi ` | No | Global Iteration index. Defaults to the current GI if omitted | | `--batch ` | No | Reveal only this batch. Defaults to every batch you have a local commit for | -> ⚠️ Aggregation is commit-then-reveal: your CID only counts once revealed. **A commit that is never revealed is slashed like a missed submission.** Reveal from the same machine you committed from, since the salt lives in the local commit cache (`/aggregations/commits/`). +> ⚠️ Aggregation is commit-then-reveal: your CID only counts once revealed. **A commit that is never revealed is slashed like a missed submission.** Reveal from the same machine you committed from, since the salt lives in the local commit cache (`/aggregations/commits//`). --- diff --git a/Documentation/technical/testing/dincli-testing-guide.md b/Documentation/technical/testing/dincli-testing-guide.md index 1a7bd546..14be0cf2 100644 --- a/Documentation/technical/testing/dincli-testing-guide.md +++ b/Documentation/technical/testing/dincli-testing-guide.md @@ -37,9 +37,9 @@ python -m pytest tests/dincli/test_01_platform.py \ ``` That's it. The conftest will: -- Compile contracts: `npx hardhat compile` always (the task-contract deploy and - `dump-abi` tests still use Hardhat artifacts), plus `forge build` for the - default `foundry` toolchain +- Compile contracts: `forge build` always (the platform deploy, the + task-contract deploys and the `dump-abi` tests all use `foundry/out`), plus + `npx hardhat compile` only for the `hardhat` platform toolchain - Kill any existing chain node and start a fresh one (clean EVM state): Anvil for `foundry`, a Hardhat node for `hardhat` - Start the IPFS daemon if it is not already running @@ -165,9 +165,9 @@ that runs `pytest tests/dincli/ -v -x -m integration --tb=short` with heavy to run on every commit or push. Maintainers run it on demand, for example before merging a change to the GI flow. - **Not a required check:** it lives outside `ci.yml`, so `CI OK` never - depends on it, and it will not become required. It is expected to fail after - `test_01` until task_061026_21 moves the suite to the foundry task contracts - and demo wallets. + depends on it, and it will not become required. Locally, an isolated run + (`DIN_TEST_ISOLATED=1`) walks one full GI from `startGI` to `GIended`, plus + reward claim/withdraw and slot release: 186 tests in about 30 minutes. - **Setup:** Foundry (`FOUNDRY_VERSION`), Node 20, Python 3.12 with CPU torch, Kubo `v0.43.1` (sha256-verified) and the runner's Docker. `forge build` and `npx hardhat compile` run first, in the runner's normal environment, so the @@ -206,7 +206,8 @@ other than `.env.example` in the repo root or `foundry/`, Foundry and Kubo ```bash (cd foundry && npm ci && forge build) -(cd hardhat && npm ci && npx hardhat compile) +# only for PLATFORM_DEPLOY_TOOLCHAIN=hardhat: +# (cd hardhat && npm ci && npx hardhat compile) python -m pip install --index-url https://download.pytorch.org/whl/cpu torch python -m pip install -e ".[test]" docker build -f dincli/docker/worker/Dockerfile -t din-worker:dev . @@ -224,10 +225,10 @@ at the end of the session. The logs stay in `DIN_TEST_RESULTS_DIR`. ### Managed services (`managed_services` fixture) Before any test runs, the fixture does the following: -1. **Solidity compilation** — runs `npx hardhat compile` in `hardhat/` (the - task-contract deploys and `dump-abi` tests use Hardhat artifacts), and - `forge build` in `foundry/` when the toolchain is `foundry` (the platform is - deployed from `foundry/out/` via `DeployPlatform.s.sol`). In isolated mode +1. **Solidity compilation** — runs `forge build` in `foundry/` (the task + contracts and `dump-abi` tests use `foundry/out/`, and so does the default + platform deploy via `DeployPlatform.s.sol`), and `npx hardhat compile` in + `hardhat/` only when `PLATFORM_DEPLOY_TOOLCHAIN=hardhat`. In isolated mode the `DIN_TEST_SVM_DIR` compilers, if set, are copied into scratch first. 2. **Fresh chain node** — kills any running node and launches a clean one on port `8545`: `foundry/anvil.sh` (70 accounts, chain ID 1337) for `foundry`, @@ -240,6 +241,27 @@ Before any test runs, the fixture does the following: on the host to execute containerized client training, evaluation, and aggregation in Phase 4. +### Seeded default artifacts (`seeded_default_artifacts` fixture, isolated mode) + +`model-owner task cache-default-artifacts` and `add-default-test-data` fetch +the CIDs in din_info's `default_*` entries. The isolated Kubo node is offline +and empty, and the published `default_manifest` has no provider on the public +network, so isolated runs seed their own copy (`tests/dincli/seed.py`): + +- the reference model's services, per-role requirements and test dataset + from `cache_model_0/`, the task-contract ABIs from `dincli/abis/` (the + `cache_model_0/abis/` copies predate the richer GI states), and a manifest + built from `cache_model_0/manifest.json` with those CIDs; +- the `cid_services/` files for `system dataset distribute-mnist`, which + reproduce their pinned CIDs. + +Files are added with Kubo's default layout, the one dincli verifies +downloads against. The fixture then points this run's din_info `default_*` +entries (including `default_test_dataset`) at the new CIDs. It runs after +`din_info_backup`, so `din_info.json` is restored unpatched at teardown. +Legacy (non-isolated) runs still fetch the published defaults through the +host's IPFS node. + ### Fixture ordering and `autouse` Only two fixtures in the conftest are `autouse=True`: `managed_services` and @@ -265,10 +287,11 @@ The full session setup chain, in execution order: ``` din_tmp (create/wipe temp dirs) - └─ managed_services (autouse: compile, fresh Hardhat node, IPFS) + └─ managed_services (autouse: forge build, fresh chain node, IPFS) bootstrap (autouse) depends on: ├─ managed_services (already up) ├─ din_info_backup (snapshot din_info.json for restore at teardown) + ├─ seeded_default_artifacts (isolated mode: seed IPFS, patch din_info defaults) └─ run ─ din_env, workdir then: configure-demo + configure-network --network local ``` @@ -306,7 +329,7 @@ If you see torch import errors in Phase 4, confirm those paths are correct. ### Chain accounts -The GI harness uses accounts 0–22 and 50–58 (59 accounts total). +The GI harness uses accounts 0–22 and 50–58 (32 accounts; indices up to 58). `foundry/anvil.sh` already starts Anvil with `--accounts 70`. For the `hardhat` toolchain, Hardhat's default is 20 accounts, so `hardhat.config.ts` must set `accounts.count` to at least 60. If you see an "account index out of @@ -347,24 +370,30 @@ graph TD | Phase | File | Description | |-------|------|--------------| -| 1 | `test_01_platform.py` | Deploy 4 platform contracts as DIN-Representative (account 0) | -| 2 | `test_02_task_contracts.py` | Deploy task contracts + authorise slashers (accounts 0 and 1) | +| 1 | `test_01_platform.py` | Deploy the 7 platform contracts as DIN-Representative (account 0) and import them | +| 2 | `test_02_task_contracts.py` | Deploy task contracts from `foundry/out` (default `--model-id`, DinToken set) + authorise slashers (accounts 0 and 1) | | 3 | `test_03_registration.py` | Genesis model creation, IPFS upload, registry request + approval | -| 4 | `test_04_gi.py` | Full Global Iteration: aggregator/auditor/client/eval/aggregation/slash/end | +| 4 | `test_04_gi.py` | One complete GI, state asserted after every phase: funding, registration + keys, LMS, seeds + test data, evaluation and aggregation with reveals, slashing, end, claims, release slots | The harness is the blocking prerequisite for contract upgrade redeployments (P3 WP 6.1). Pass all phases before pushing contract changes to testnet. ### Phase 1: Platform Deployment (`test_01_platform.py`) - **Role**: `Account 0` (DIN-Representative). -- **Deploys**: `DinCoordinator` (and `DinToken` internally), `DinValidatorStake`, - `DINModelRegistry`. -- **Outputs**: official ABIs and platform contract addresses, stored in the - shared session `state`. +- **Deploys**: the seven platform contracts behind transparent proxies + (`DinCoordinator`, `DinToken`, `DinValidatorStake`, `DINModelRegistry`, + `DinTreasury`, `DinFeeRouter`, `DinEmission`) with `DeployPlatform.s.sol` + (or the hardhat script), then `system import-deployments`. Checks each + per-proxy `proxyAdmin*` key and its `proxy_admin_*` din_info entry. +- **Outputs**: platform contract addresses in din_info and the shared session + `state`. The `dump-abi` tests write to the run's scratch dir (`--output`), + so no tracked file changes. ### Phase 2: Task & Slasher Setup (`test_02_task_contracts.py`) - **Role**: `Account 1` (Model Owner). -- **Deploys**: `DINTaskCoordinator`, `DINTaskAuditor`. +- **Deploys**: `DINTaskCoordinator`, `DINTaskAuditor` from `foundry/out`, + leaving `--model-id` to its default (`totalModels()` = 0) and checking that + each deploy set the contract's DinToken. - **Authorizations**: `Account 0` authorizes the new task contracts as legitimate slashers; the model owner then registers the slashers back onto the task contracts. @@ -380,19 +409,31 @@ The harness is the blocking prerequisite for contract upgrade redeployments 6. `Account 0` inspects and approves the registration request. ### Phase 4: Global Iteration (GI) Lifecycle (`test_04_gi.py`) -The core federated training round: -1. **Window opening** — opens registration for aggregators and auditors. +One complete GI (GI 1 of model 0). After every phase the test asserts the GI +state by name (`task gi show-state`): +1. **Funding** — the model owner buys DIN and deposits 50 DIN into GI 1's + reward pool, then starts the GI (`GIstarted`). 2. **Staking & joining** — 12 aggregators (accounts `11–22`) and 9 auditors - (accounts `50–58`) buy DIN tokens, stake, and register. + (accounts `50–58`) buy DIN tokens, stake, and register; each auditor + registers an encryption key. An aggregator trying to register as an + auditor is refused (dual-role guard). 3. **Training (LMS)** — a distributed MNIST shard dataset is generated and - assigned to 9 clients (accounts `2` and `4–10`); clients run local - training inside containerized workers (Docker) and upload model weights. -4. **Evaluation** — auditors receive partitioned batches of trained local - models, evaluate accuracy against test data, and submit scores on-chain. -5. **Aggregation** — aggregators pull approved weights and perform Tier 1 - (sub-batch averaging) and Tier 2 (global consensus model) aggregation. -6. **Slashing & cleanup** — under-performing validators are slashed and the - Global Iteration is ended. + assigned to 9 clients (accounts `2–10`, i.e. `range(2, 11)`); clients + train inside containerized workers (Docker), then submit with `submit-lm`. +4. **Seed, batches, test data** — an auditor locks the auditor seed + (`AuditSeedLocked`), the owner creates the batches + (`AuditorsBatchesCreated`) and assigns encrypted test data + (`AuditTestDataAssigned`). +5. **Evaluation** — auditors evaluate and commit, the owner opens the reveal + window, auditors reveal, the owner closes (`LMSevaluationClosed`). +6. **Aggregation** — an aggregator locks the T1/T2 seed (`AggSeedLocked`), + the owner creates the batches; T1 then T2 each go commit → reveal → + close; the owner records the T2 score. +7. **Slashing & end** — auditors then aggregators are slashed and the GI + ends (`GIended`, ordinal 28). +8. **Rewards** — one client, one auditor and one aggregator claim and + withdraw their GI 1 reward (each DIN balance must rise); the owner + releases GI 1's registration slots. --- diff --git a/dincli/cli/aggregator.py b/dincli/cli/aggregator.py index fcd32a27..432e9aed 100644 --- a/dincli/cli/aggregator.py +++ b/dincli/cli/aggregator.py @@ -44,20 +44,31 @@ # commands rather than recomputed -- recomputing would also require # re-running the aggregation worker a second time. Mirrors auditor.py's # commit-store pattern for the auditor-side commit-reveal flow. -def _agg_commit_store_path(model_base_dir: Path, tier: int, gi: int, batch_id: int) -> Path: +# +# The store is per wallet (commits/
/...): several aggregators in +# the same batch can run from one machine (one cache dir), and a shared +# file would let the last committer overwrite the others' salts, failing +# their reveals. _load_agg_commit still reads the old shared path, so a +# commit made before this change can be revealed after upgrading. +def _agg_commit_store_path(model_base_dir: Path, tier: int, gi: int, batch_id: int, address: Optional[str]) -> Path: tier_name = "t1" if tier == TIER1 else "t2" - return model_base_dir / "aggregations" / "commits" / f"{tier_name}_gi_{gi}_batch_{batch_id}.json" + commits_dir = model_base_dir / "aggregations" / "commits" + if address is not None: + commits_dir = commits_dir / address + return commits_dir / f"{tier_name}_gi_{gi}_batch_{batch_id}.json" -def _save_agg_commit(model_base_dir: Path, tier: int, gi: int, batch_id: int, cid_bytes32: bytes, salt: bytes) -> None: - path = _agg_commit_store_path(model_base_dir, tier, gi, batch_id) +def _save_agg_commit(model_base_dir: Path, tier: int, gi: int, batch_id: int, cid_bytes32: bytes, salt: bytes, address: str) -> None: + path = _agg_commit_store_path(model_base_dir, tier, gi, batch_id, address) path.parent.mkdir(parents=True, exist_ok=True) with open(path, "w") as f: json.dump({"cid": cid_bytes32.hex(), "salt": salt.hex()}, f) -def _load_agg_commit(model_base_dir: Path, tier: int, gi: int, batch_id: int) -> Optional[dict]: - path = _agg_commit_store_path(model_base_dir, tier, gi, batch_id) +def _load_agg_commit(model_base_dir: Path, tier: int, gi: int, batch_id: int, address: str) -> Optional[dict]: + path = _agg_commit_store_path(model_base_dir, tier, gi, batch_id, address) + if not path.exists(): + path = _agg_commit_store_path(model_base_dir, tier, gi, batch_id, None) # pre-per-wallet layout if not path.exists(): return None with open(path) as f: @@ -132,12 +143,12 @@ def register( if stake < MIN_STAKE: console.print(f"[bold red]✗ Aggregator does not have enough stake.[/bold red]") - return + raise typer.Exit(1) else: console.print(f"[bold green]✓ Aggregator has enough stake.[/bold green]") try: - build_and_send_tx( + receipt = build_and_send_tx( ctx, taskCoordinator_contract.functions.registerDINaggregator(curr_GI), "Registering aggregator", @@ -147,9 +158,12 @@ def register( ) except Exception as e: console.print(f"[bold red]✗ Could not register aggregator. {e}[/bold red]") - - - + raise typer.Exit(1) + # A refused registration (e.g. TA_DualRoleNotAllowed) must not exit 0. + if receipt is None: + raise typer.Exit(1) + + @app.command("lock-seed", help="Lock the T1/T2 batch-assignment seed for the current GI (permissionless)") def lock_seed( ctx: typer.Context, @@ -425,7 +439,7 @@ def aggregate_t1( # mine, so saving only on a receipt could strand a real commit # without its preimage. A failed attempt leaves a stale cache # that the next retry replaces (the batch isn't committed yet). - _save_agg_commit(model_base_dir, TIER1, curr_GI, bid, aggregated_cid_bytes32, salt) + _save_agg_commit(model_base_dir, TIER1, curr_GI, bid, aggregated_cid_bytes32, salt, account.address) build_and_send_tx( ctx, taskCoordinator_contract.functions.commitT1Aggregation(curr_GI, bid, commit_hash), @@ -466,7 +480,7 @@ def reveal_t1( if batch_id is not None and bid != batch_id: continue - commit = _load_agg_commit(model_base_dir, TIER1, curr_GI, bid) + commit = _load_agg_commit(model_base_dir, TIER1, curr_GI, bid, account.address) if commit is None: continue @@ -638,7 +652,7 @@ def aggregate_t2( commit_hash = _agg_commit_hash(aggregated_cid_bytes32, salt, account.address, curr_GI, TIER2, i) # Cache before sending -- see aggregate-t1. - _save_agg_commit(model_base_dir, TIER2, curr_GI, i, aggregated_cid_bytes32, salt) + _save_agg_commit(model_base_dir, TIER2, curr_GI, i, aggregated_cid_bytes32, salt, account.address) build_and_send_tx( ctx, taskCoordinator_contract.functions.commitT2Aggregation(curr_GI, i, commit_hash), @@ -672,7 +686,7 @@ def reveal_t2( model_base_dir = ctx.obj.get_model_base_dir(model_id) - commit = _load_agg_commit(model_base_dir, TIER2, curr_GI, 0) + commit = _load_agg_commit(model_base_dir, TIER2, curr_GI, 0, account.address) if commit is None: console.print("[yellow]No local T2 commit found to reveal.[/yellow]") return diff --git a/dincli/cli/auditor.py b/dincli/cli/auditor.py index 6ef7fa95..3603bd14 100644 --- a/dincli/cli/auditor.py +++ b/dincli/cli/auditor.py @@ -59,19 +59,31 @@ def _decrypt_aes_gcm(ciphertext: bytes, K: bytes) -> bytes: # time must be reproduced exactly at reveal time, so it's cached to disk # between the two commands rather than recomputed (recomputing would also # require re-running the scoring worker a second time). -def _commit_store_path(model_base_dir: Path, gi: int, batch_id: int, model_index: int) -> Path: - return model_base_dir / "audits" / "commits" / f"gi_{gi}_batch_{batch_id}_lm_{model_index}.json" - - -def _save_commit(model_base_dir: Path, gi: int, batch_id: int, model_index: int, score: int, vote: bool, salt: bytes) -> None: - path = _commit_store_path(model_base_dir, gi, batch_id, model_index) +# +# The store is per wallet (commits/
/...): an audit batch has +# several auditors, who may run from one machine (one cache dir), and a +# shared file would let the last committer overwrite the others' salts, +# failing their reveals with TA_RevealHashMismatch. _load_commit still +# reads the old shared path, so a commit made before this change can be +# revealed after upgrading. +def _commit_store_path(model_base_dir: Path, gi: int, batch_id: int, model_index: int, address: Optional[str]) -> Path: + commits_dir = model_base_dir / "audits" / "commits" + if address is not None: + commits_dir = commits_dir / address + return commits_dir / f"gi_{gi}_batch_{batch_id}_lm_{model_index}.json" + + +def _save_commit(model_base_dir: Path, gi: int, batch_id: int, model_index: int, score: int, vote: bool, salt: bytes, address: str) -> None: + path = _commit_store_path(model_base_dir, gi, batch_id, model_index, address) path.parent.mkdir(parents=True, exist_ok=True) with open(path, "w") as f: json.dump({"score": score, "vote": vote, "salt": salt.hex()}, f) -def _load_commit(model_base_dir: Path, gi: int, batch_id: int, model_index: int) -> Optional[dict]: - path = _commit_store_path(model_base_dir, gi, batch_id, model_index) +def _load_commit(model_base_dir: Path, gi: int, batch_id: int, model_index: int, address: str) -> Optional[dict]: + path = _commit_store_path(model_base_dir, gi, batch_id, model_index, address) + if not path.exists(): + path = _commit_store_path(model_base_dir, gi, batch_id, model_index, None) # pre-per-wallet layout if not path.exists(): return None with open(path) as f: @@ -156,12 +168,12 @@ def register( if stake < MIN_STAKE: console.print(f"[bold red]✗ Auditor does not have enough stake.[/bold red]") - return + raise typer.Exit(1) else: console.print(f"[bold green]✓ Auditor has enough stake.[/bold green]") try: - build_and_send_tx( + receipt = build_and_send_tx( ctx, taskAuditor_contract.functions.registerDINAuditor(curr_GI), "Registering auditor", @@ -171,7 +183,10 @@ def register( ) except Exception as e: console.print(f"[bold red]✗ Could not register auditor. {e}[/bold red]") - raise typer.Exit() + raise typer.Exit(1) + # A refused registration (e.g. TA_DualRoleNotAllowed) must not exit 0. + if receipt is None: + raise typer.Exit(1) @app.command("register-encryption-key", help="Generate this wallet's X25519 key and register its public key on DinValidatorStake") def register_encryption_key(ctx: typer.Context): @@ -568,7 +583,7 @@ def evaluate_lms( console.print(f"[bold red]✗ Batch {batch_id}: owner signature on rawCID invalid — skipping.[/bold red]") continue - raw_cid = get_cid_from_bytes32("0x" + raw_cid_bytes.hex()) + raw_cid = get_cid_from_bytes32(raw_cid_bytes.hex()) pt_path = model_base_dir / "dataset" / "auditor" / "TestDatasets" / f"auditorDataset_{curr_GI}_{batch_id}.pt" if not pt_path.exists(): @@ -650,7 +665,7 @@ def evaluate_lms( # a real commit without its preimage. A failed attempt leaves # a stale cache that the next retry replaces (the LM isn't # committed yet, so the hasCommittedLM guard lets it through). - _save_commit(model_base_dir, curr_GI, batch_id, model_index, score_int, vote_bool, salt) + _save_commit(model_base_dir, curr_GI, batch_id, model_index, score_int, vote_bool, salt, account.address) build_and_send_tx( ctx, task_auditor_contract.functions.commitAuditScore(curr_GI, batch_id, model_index, commit_hash), @@ -708,7 +723,7 @@ def reveal_lms( if lmi is not None and lmi != model_index: continue - commit = _load_commit(model_base_dir, curr_GI, batch_id, model_index) + commit = _load_commit(model_base_dir, curr_GI, batch_id, model_index, account.address) if commit is None: console.print(f"[yellow]No local commit found for LM {model_index} from batch {batch_id} -- skipping.[/yellow]") continue diff --git a/dincli/cli/modelownerd/model.py b/dincli/cli/modelownerd/model.py index acf86e70..ee46fe02 100644 --- a/dincli/cli/modelownerd/model.py +++ b/dincli/cli/modelownerd/model.py @@ -6,7 +6,7 @@ import json from web3 import Web3 -from dincli.cli.utils import (build_and_send_tx, get_env_key, +from dincli.cli.utils import (build_and_send_tx, get_env_key, load_din_info, get_manifest, get_manifest_key, GIstateToStr, @@ -137,6 +137,10 @@ def create_genesis_model( return #delete-this-for-production +# The reference MNIST test dataset used by `add-default-test-data --default-test-data`. +DEFAULT_TEST_DATASET_CID = "bafybeigjtcu2nzsffoy5pjmui25bnc43yduzn6aopi4wnrbtxfleqmw46y" + + @model_app.command("add-default-test-data") def add_default_test_data(ctx: typer.Context, task_coordinator_address: str = typer.Option(None, "--taskCoordinator", help="Task coordinator address"), model_id: int = typer.Option(None, "--model-id", help="Model ID"), default_test_data: bool = typer.Option(False, "--default-test-data", help="use default test data")): effective_network, w3, account, console = ctx.obj.get_en_w3_account_console() @@ -154,7 +158,10 @@ def add_default_test_data(ctx: typer.Context, task_coordinator_address: str task_dir = ctx.obj.get_task_dir(task_coordinator_address) test_dataset_path = task_dir.joinpath("dataset","test","test_dataset.pt") if default_test_data: - default_test_dataset_ipfs_hash = "bafybeigjtcu2nzsffoy5pjmui25bnc43yduzn6aopi4wnrbtxfleqmw46y" + # din_info's per-network `default_test_dataset` overrides the published default. + default_test_dataset_ipfs_hash = load_din_info().get(effective_network, {}).get( + "default_test_dataset", DEFAULT_TEST_DATASET_CID + ) console.print(f"[bold yellow] Using default test dataset with IPFS CID {default_test_dataset_ipfs_hash} [/bold yellow]") retrieve_from_ipfs(default_test_dataset_ipfs_hash, test_dataset_path) diff --git a/dincli/services/cid_utils.py b/dincli/services/cid_utils.py index 5a2a9263..86da0c50 100644 --- a/dincli/services/cid_utils.py +++ b/dincli/services/cid_utils.py @@ -27,7 +27,8 @@ def get_cid_from_bytes32(byte32_hex: str, version: int = 1, encoding: str = "bas Converts a raw 32-byte SHA2-256 digest (hex) back into a CID. Args: - byte32_hex: 64-character hex string (raw digest, NO '1220' prefix). + byte32_hex: 64-character hex string (raw digest, NO '1220' prefix; an + optional '0x' prefix is stripped). version: 0 or 1. encoding: 'base32' (recommended for v1) or 'base58btc'. @@ -36,8 +37,13 @@ def get_cid_from_bytes32(byte32_hex: str, version: int = 1, encoding: str = "bas """ from multibase import encode # local import keeps top-level deps minimal + digest_hex = str(byte32_hex) + # Accept a 0x-prefixed digest too: "1220" + "0x..." isn't valid hex. + if digest_hex[:2].lower() == "0x": + digest_hex = digest_hex[2:] + # Reconstruct the full multihash: 12 = sha2-256, 20 = 32-byte length - multihash_hex = "1220" + str(byte32_hex) + multihash_hex = "1220" + digest_hex multihash_bytes = bytes.fromhex(multihash_hex) # Build CIDv1 (dag-pb is standard for IPFS file data) diff --git a/tests/dincli/NOTES.md b/tests/dincli/NOTES.md index e6e4fae0..c9b87f6a 100644 --- a/tests/dincli/NOTES.md +++ b/tests/dincli/NOTES.md @@ -10,27 +10,26 @@ logic callable by `dind` without going through the CLI layer. ## Platform contracts (Phase 1 → `sdk/platform.py`) -PR 13 made the platform contracts transparent proxies. INTERIM: the hardhat -script (OZ upgrades plugin) deploys + wires token → coordinator → -`setCoordinator` → stake → `updateValidatorStakeContract` → registry, and -dincli imports the resulting `deployments/.json` via -`system import-deployments`. This is scaffolding only — the committed decision -record (`Documentation/technical/upgradable-contracts/proxy-deployment-architecture.md`) -chose native web3.py proxy deploys in `dinrep deploy` (backlog: -`Developer/issues/dincli-native-proxy-deployment.md`); once implemented, -Phase 1 returns to per-contract dincli deploy tests. - -Foundry migration: foundry/ has the same contracts but no deploy script yet. -When a forge deploy script lands, `deploy_platform` wraps `forge script ---broadcast` instead of `npx hardhat run`; `import_deployments` stays the -handoff (forge script should write the same deployments JSON shape, or the -importer grows a broadcast/run-latest.json parser). +Phase 1 deploys the seven platform contracts behind transparent proxies with +`forge script foundry/script/DeployPlatform.s.sol --broadcast` (the default +`PLATFORM_DEPLOY_TOOLCHAIN=foundry`; `hardhat` runs +`hardhat/scripts/deploy-platform.ts` instead), and dincli imports the +resulting `foundry/deployments/.json` via `system import-deployments`. +The test checks each per-proxy `proxyAdmin*` key and its `proxy_admin_*` +din_info entry. Native web3.py proxy deploys in `dinrep deploy` remain the +recorded long-term decision +(`Documentation/technical/upgradable-contracts/proxy-deployment-architecture.md`; +backlog `Developer/issues/dincli-native-proxy-deployment.md`); once +implemented, Phase 1 returns to per-contract dincli deploy tests. + +Task contracts (Phase 2 onward) are deployed from `foundry/out` too. +`dump-abi` tests write to the run's scratch dir (`--output`), never `--official`. | Test | SDK function | |------|-------------| | `test_deploy_platform_via_script` | `deploy_platform(network) → addresses dict` (wraps the deploy script) | | `test_import_deployments_into_din_info` | `import_deployments(network, file) → dict` (body of `system import-deployments`) | -| `test_dump_abi_*` | `dump_abi(artifact_path, official=True)` | +| `test_dump_abi_*` | `dump_abi(artifact_path, output_dir)` | --- @@ -38,8 +37,8 @@ importer grows a broadcast/run-latest.json parser). | Test | SDK function | |------|-------------| -| `test_deploy_task_coordinator` | `deploy_task_coordinator(network, account, artifact_path) → address` | -| `test_deploy_task_auditor` | `deploy_task_auditor(network, account, artifact_path) → address` | +| `test_deploy_task_coordinator` | `deploy_task_coordinator(network, account, artifact_path, model_id=None) → address` (also sets DinToken) | +| `test_deploy_task_auditor` | `deploy_task_auditor(network, account, artifact_path, model_id=None) → address` (also sets DinToken) | | `test_din_rep_authorizes_*` | `dinrep_add_slasher(network, account, task_coordinator=None, task_auditor=None)` | | `test_model_owner_registers_*` | `model_owner_add_slasher(network, account, task_coordinator=None, task_auditor=None)` | @@ -62,8 +61,42 @@ importer grows a broadcast/run-latest.json parser). ## GI lifecycle (Phase 4 → `sdk/gi.py`) -| Test | SDK function | -|------|-------------| +One complete GI, with the GI state asserted after every phase (`_assert_state`). + +| Test | SDK function | GI state after | +|------|-------------|----------------| +| `test_fund_gi_reward_pool` | `deposit_rewards(network, account, model_id, amount, gi=None)` | — | +| `test_gi_start` | `start_gi(network, account, model_id)` | `GIstarted` | +| `test_gi_aggregators_open/close` | `open/close_aggregator_registration(network, account, model_id)` | | +| `test_aggregator_register` | `register_aggregator(network, account, model_id)` | | +| `test_gi_auditors_open/close` | `open/close_auditor_registration(network, account, model_id)` | `DINauditorsRegistrationClosed` | +| `test_auditor_register` | `register_auditor(network, account, model_id)` | | +| `test_auditor_register_encryption_key` | `register_encryption_key(network, account)` | | +| `test_aggregator_cannot_register_as_auditor` | (negative: `register_auditor` raises) | | +| `test_lms_open/close` | `open_lms(network, account, model_id)` / `close_lms(...)` | `LMSstarted` / `LMSclosed` | +| `test_client_train_and_submit` | `train_lm(...)` + `submit_lm(network, account, model_id)` | | +| `test_auditor_lock_seed` | `lock_audit_seed(network, account, model_id)` | `AuditSeedLocked` | +| `test_create_auditor_batches` | `create_auditor_batches(network, account, model_id)` | `AuditorsBatchesCreated` | +| `test_create_testdataset_and_submit` | `assign_audit_test_data(network, account, model_id)` | `AuditTestDataAssigned` | +| `test_lms_evaluation_start` | `start_lms_evaluation(network, account, model_id)` | `LMSevaluationStarted` | +| `test_auditor_evaluate` | `evaluate_lms_batch(network, account, model_id)` (commit) | | +| `test_lms_evaluation_start_reveal` | `start_lms_evaluation_reveal(network, account, model_id)` | `LMSevaluationRevealStarted` | +| `test_auditor_reveal` | `reveal_audit_scores(network, account, model_id)` | | +| `test_lms_evaluation_close` | `close_lms_evaluation(network, account, model_id)` | `LMSevaluationClosed` | +| `test_aggregator_lock_seed` | `lock_agg_seed(network, account, model_id)` | `AggSeedLocked` | +| `test_create_t1nt2_batches` | `create_aggregation_batches(network, account, model_id)` | `T1nT2Bcreated` | +| `test_t1_aggregation_start` / `_start_reveal` / `_close` | `start/start_reveal/close_t1_aggregation(network, account, model_id)` | `T1AggregationStarted` / `…RevealStarted` / `T1AggregationDone` | +| `test_aggregator_aggregate_t1` / `test_aggregator_reveal_t1` | `aggregate_t1(...)` (commit) / `reveal_t1(...)` | | +| `test_t2_aggregation_start` / `_start_reveal` / `_close` | `start/start_reveal/close_t2_aggregation(network, account, model_id)` | `T2AggregationStarted` / `…RevealStarted` / `T2AggregationDone` | +| `test_aggregator_aggregate_t2` / `test_aggregator_reveal_t2` | `aggregate_t2(...)` (commit) / `reveal_t2(...)` | | +| `test_t2_set_score` | `set_tier2_score(network, account, model_id)` | | +| `test_slash_auditors` | `slash_auditors(network, account, model_id)` | `AuditorsSlashed` | +| `test_slash_aggregators` | `slash_aggregators(network, account, model_id)` | `AggregatorsSlashed` | +| `test_gi_end` / `test_gi_final_state_is_ended` | `end_gi(network, account, model_id)` | `GIended` (28) | +| `test_reward_claim_and_withdraw` | `claim_reward(network, account, model_id, gi)` + `withdraw_rewards(...)` | | +| `test_gi_release_slots` | `release_gi_slots(network, account, model_id, gi=None)` | | + +------|-------------| | `test_gi_start` | `start_gi(network, account, model_id)` | | `test_gi_aggregators_open/close` | `open/close_aggregator_registration(network, account, model_id)` | | `test_aggregator_register` | `register_aggregator(network, account, model_id)` | diff --git a/tests/dincli/conftest.py b/tests/dincli/conftest.py index b0a298bb..a35be1fe 100644 --- a/tests/dincli/conftest.py +++ b/tests/dincli/conftest.py @@ -4,10 +4,10 @@ tests/dincli/constants.py). The conftest automatically handles all prerequisites: - 1. Compiles Solidity contracts via npx hardhat compile (task-level - contract deploys and dump-abi tests always need hardhat's ABIs/bytecode, - regardless of which toolchain deploys the platform contracts); also runs - `forge build` when PLATFORM_DEPLOY_TOOLCHAIN is "foundry" + 1. Builds Solidity contracts via forge build (task-level contract deploys + and dump-abi tests read foundry/out/ artifacts, regardless of which + toolchain deploys the platform contracts); also runs npx hardhat compile + when PLATFORM_DEPLOY_TOOLCHAIN is "hardhat" 2. Kills any running chain node and starts a fresh one (clean EVM state) — Hardhat node for the "hardhat" toolchain, Anvil for "foundry" (both listen on HARDHAT_RPC / chain-id 1337, so the rest of the harness is @@ -58,6 +58,7 @@ ) from tests.dincli.demo import bootstrap_demo, prepare_demo_accounts +from tests.dincli.seed import apply_din_info_overrides, seed_default_artifacts from tests.dincli.services import start_service, rpc_ready, ipfs_ready from tests.dincli.diagnostics import Diagnostics, write_cleanup_summary from tests.dincli.reports import collect_worker_reports @@ -181,8 +182,8 @@ def _seed_solc(din_tmp: Path) -> None: def _build_foundry_contracts(results_dir: Path) -> None: """Run forge build. Aborts the session on failure. - Needed only when PLATFORM_DEPLOY_TOOLCHAIN is "foundry" — DeployPlatform.s.sol - reads compiled artifacts under foundry/out/, separate from hardhat/artifacts/. + Always needed: task-contract deploys and dump-abi tests read foundry/out/ + (ARTIFACT_BASE), and DeployPlatform.s.sol does too for the foundry toolchain. """ print("\n[setup] Building Foundry contracts...") log_path = results_dir / "forge_build.log" @@ -411,13 +412,14 @@ def managed_services(din_tmp): chain_proc = None ipfs_proc = None - # 1. Compile contracts — task-level contract deploys and dump-abi tests - # always need hardhat's ABIs/bytecode, regardless of deploy toolchain. - _compile_contracts(results_dir) - if PLATFORM_DEPLOY_TOOLCHAIN == "foundry": - if ISOLATED_MODE: - _seed_solc(din_tmp) - _build_foundry_contracts(results_dir) + # 1. Build contracts — task-level contract deploys and dump-abi tests read + # foundry/out/ regardless of deploy toolchain; hardhat compiles only when + # it deploys the platform. + if ISOLATED_MODE: + _seed_solc(din_tmp) + _build_foundry_contracts(results_dir) + if PLATFORM_DEPLOY_TOOLCHAIN == "hardhat": + _compile_contracts(results_dir) if ISOLATED_MODE: env = _isolated_env(din_tmp) @@ -507,6 +509,9 @@ def _isolated_env(din_tmp): "IPFS_API_URL_ADD": "http://127.0.0.1:5001/api/v0/add", "IPFS_API_URL_RETRIEVE": "http://127.0.0.1:5001/api/v0", "NO_COLOR": "1", "TERM": "dumb", + # Rich wraps at 80 columns without a TTY; the suite matches whole + # output lines (addresses, GI state), so keep them on one line. + "COLUMNS": "250", } @@ -531,6 +536,7 @@ def din_env(din_tmp): env["PYTHONPATH"] = str(DEVNET_ROOT) + (":" + existing if existing else "") env["NO_COLOR"] = "1" env["TERM"] = "dumb" + env["COLUMNS"] = "250" # see _isolated_env return env @@ -655,8 +661,39 @@ def _run( # --------------------------------------------------------------------------- +@pytest.fixture(scope="session") +def seeded_default_artifacts(managed_services, din_info_backup, din_tmp): + """Isolated mode: add the reference model's default artifacts to the offline + Kubo node and point this run's din_info `default_*` entries at them (see + tests/dincli/seed.py). Runs after din_info_backup, so the tracked + din_info.json is restored unpatched at the end of the session. + + Legacy (non-isolated) runs fetch the published defaults through the + host's IPFS node instead, and get None. + """ + if not ISOLATED_MODE: + yield None + return + env = _isolated_env(din_tmp) + + def add(path: Path) -> str: + result = _get_diagnostics().run( + # Kubo's default layout, the one dincli verifies downloads against + # (`ipfs add -n -Q`, dincli/services/ipfs.py); --cid-version 1 + # switches to a different chunking for multi-chunk files. + [IPFS_BIN, "add", "-Q", "--pin", str(path)], + cwd=din_tmp, env=env, timeout=60, label="ipfs_seed", + ) + result.check_returncode() + return result.stdout.strip() + + overrides = seed_default_artifacts(add, din_tmp) + apply_din_info_overrides(DIN_INFO_PATH, "local", overrides) + yield overrides + + @pytest.fixture(scope="session", autouse=True) -def bootstrap(managed_services, din_info_backup, run): +def bootstrap(managed_services, din_info_backup, seeded_default_artifacts, run): """ Configure demo mode and local network, and connect the named demo role wallets (account 0 = dinrep / DIN-Representative, account 1 = modelowner). diff --git a/tests/dincli/constants.py b/tests/dincli/constants.py index 8c71a559..e7d685f5 100644 --- a/tests/dincli/constants.py +++ b/tests/dincli/constants.py @@ -55,7 +55,9 @@ def _load_dotenv_values(path: Path) -> dict: def _env(name: str) -> "str | None": """Real environment first, then the repo-root .env file.""" return os.environ.get(name) or _DOTENV.get(name) -ARTIFACT_BASE = DEVNET_ROOT / "hardhat" / "artifacts" / "contracts" +# Contract artifacts come from `forge build` (same X.sol/X.json layout as hardhat; +# get_contract_instance reads bytecode.object). Hardhat task contracts are off the GI path. +ARTIFACT_BASE = DEVNET_ROOT / "foundry" / "out" # din_info.json lives inside the *imported* dincli package (files("dincli")). # PYTHONPATH points the tests at the live devnet checkout, so deploys write there. DIN_INFO_PATH = DEVNET_ROOT / "dincli" / "config" / "din_info.json" @@ -109,6 +111,8 @@ def _venv_site_packages(python_bin: str) -> str: IPFS_BIN = _env("IPFS_BIN") or shutil.which("ipfs") or "/usr/local/bin/ipfs" DIN_TEMP = Path(_env("DIN_TEST_TMPDIR") or str(Path.home() / "tempdir" / "dincli")) +# dump-abi tests write here (--output), never to the tracked dincli/abis/ (--official). +DUMP_ABI_DIR = DIN_TEMP / "abis" RESULTS_DIR = Path(_env("DIN_TEST_RESULTS_DIR") or str(DIN_TEMP / "results")) if ISOLATED_MODE: if not os.environ.get("DIN_TEST_TMPDIR") or not os.environ.get("DIN_TEST_RESULTS_DIR"): diff --git a/tests/dincli/seed.py b/tests/dincli/seed.py new file mode 100644 index 00000000..8d9cd8f0 --- /dev/null +++ b/tests/dincli/seed.py @@ -0,0 +1,106 @@ +"""Seed the isolated IPFS node with the reference model's default artifacts. + +`model-owner task cache-default-artifacts` and `add-default-test-data` fetch +the CIDs in din_info.json's `default_*` entries. Isolated runs use a fresh, +offline Kubo node, which can't fetch those CIDs, and since Nov 2026 no +provider on the public IPFS network serves `default_manifest` at all. This +module adds the repo's reference MNIST model (`cache_model_0/`, +`cid_services/`, the bundled `dincli/abis/`) to the isolated node instead, +and returns the din_info overrides pointing at the CIDs it got. + +A CID that a repo file already reproduces is kept, so the content is +unchanged. Otherwise the repo's version of that artifact is added and its +new CID is used, both in din_info and in the default manifest: + +- services: `cache_model_0/services/` +- task-contract ABIs: `dincli/abis/` (current foundry ABIs; the copies + in `cache_model_0/abis/` predate the richer GI states) +- per-role requirements: `cache_model_0/requirements//requirements.txt` +- default test dataset: `cache_model_0/dataset/test/test_dataset.pt` +""" + +import json +from pathlib import Path +from typing import Callable + +from dincli.services.cid_utils import get_bytes32_from_cid +from tests.dincli.constants import DEVNET_ROOT + +MODEL_DIR = DEVNET_ROOT / "cache_model_0" +CID_SERVICES_DIR = DEVNET_ROOT / "cid_services" +BUNDLED_ABIS_DIR = DEVNET_ROOT / "dincli" / "abis" +CID_SERVICES_JSON = DEVNET_ROOT / "dincli" / "config" / "cid_services.json" + +AddFn = Callable[[Path], str] + + +def _service_file(manifest_path: str) -> Path: + return MODEL_DIR / manifest_path.lstrip("/") + + +def _abi_file(manifest_path: str) -> Path: + return BUNDLED_ABIS_DIR / Path(manifest_path).name + + +def build_seeded_manifest(add: AddFn) -> dict: + """The reference manifest with every artifact CID replaced by one `add` returned.""" + manifest = json.loads((MODEL_DIR / "manifest.json").read_text(encoding="utf-8")) + + for entry in manifest.values(): + if isinstance(entry, dict) and entry.get("path") and entry.get("ipfs"): + entry["ipfs"] = add(_service_file(entry["path"])) + + for contract in manifest.get("task_contracts", {}).values(): + artifact = contract.get("artifact") or {} + if artifact.get("path") and artifact.get("ipfs"): + artifact["ipfs"] = add(_abi_file(artifact["path"])) + + for role in list(manifest.get("requirements.txt", {})): + manifest["requirements.txt"][role] = add(MODEL_DIR / "requirements" / role / "requirements.txt") + + return manifest + + +def seed_default_artifacts(add: AddFn, scratch: Path) -> dict: + """Add the reference artifacts via `add` and return the din_info `default_*` overrides. + + `scratch` is a writable directory for the generated manifest file. + """ + manifest = build_seeded_manifest(add) + manifest_path = scratch / "seeded_default_manifest.json" + manifest_path.write_text(json.dumps(manifest, indent=3), encoding="utf-8") + + services = { + name: add(MODEL_DIR / "services" / name) + for name in ("model.py", "modelowner.py", "client.py", "auditor.py", "aggregator.py", "scoring.py") + } + abis = {name: add(BUNDLED_ABIS_DIR / name) for name in ("DINTaskCoordinator.json", "DINTaskAuditor.json")} + requirements = { + role: add(MODEL_DIR / "requirements" / role / "requirements.txt") + for role in ("clients", "auditors", "aggregators", "modelowner") + } + + # `system dataset distribute-mnist` fetches its service by the CIDs in + # cid_services.json; the repo copies reproduce them, so adding them is enough. + for entry in json.loads(CID_SERVICES_JSON.read_text(encoding="utf-8"))["services"].values(): + service = CID_SERVICES_DIR / entry["path"].lstrip("/").removeprefix("services/") + requirements_file = CID_SERVICES_DIR / "requirements" / Path(entry["path"]).with_suffix(".txt").name + for path, expected in ((service, entry["ipfs"]), (requirements_file, entry["requirements"])): + got = add(path) + # Compare digests, as dincli does: the node may return a CIDv0 for a v1 CID. + if get_bytes32_from_cid(got) != get_bytes32_from_cid(expected): + raise RuntimeError(f"{path} hashes to {got}, but cid_services.json expects {expected}") + + return { + "default_manifest": add(manifest_path), + "default_services": services, + "default_abis": abis, + "default_requirements": requirements, + "default_test_dataset": add(MODEL_DIR / "dataset" / "test" / "test_dataset.pt"), + } + + +def apply_din_info_overrides(din_info_path: Path, network: str, overrides: dict) -> None: + din_info = json.loads(din_info_path.read_text(encoding="utf-8")) + din_info.setdefault(network, {}).update(overrides) + din_info_path.write_text(json.dumps(din_info, indent=4) + "\n", encoding="utf-8") diff --git a/tests/dincli/test_01_platform.py b/tests/dincli/test_01_platform.py index 5df8afa2..361d0973 100644 --- a/tests/dincli/test_01_platform.py +++ b/tests/dincli/test_01_platform.py @@ -40,6 +40,7 @@ from tests.dincli.constants import ( ARTIFACT_BASE, DIN_INFO_PATH, + DUMP_ABI_DIR, FOUNDRY_DIR, FORGE_BIN, HARDHAT_DIR, @@ -65,13 +66,23 @@ "dinModelRegistry": ("registry", "registry_address"), } +# Per-proxy ProxyAdmin keys (OZ v5 deploys one ProxyAdmin per proxy) that both +# deploy scripts write → the din_info keys import-deployments maps them to. +# Foundry also writes proxyAdminTreasury/FeeRouter/Emission; hardhat doesn't. +PROXY_ADMIN_KEYS = { + "proxyAdminToken": "proxy_admin_token", + "proxyAdminCoordinator": "proxy_admin_coordinator", + "proxyAdminStake": "proxy_admin_stake", + "proxyAdminRegistry": "proxy_admin_registry", +} + def _assert_artifacts_exist(): missing = [name for name, path in ARTIFACTS.items() if not path.exists()] if missing: pytest.skip( - f"Hardhat artifacts missing for: {missing}. " - "Run: cd hardhat && npx hardhat compile" + f"Foundry artifacts missing for: {missing}. " + "Run: cd foundry && forge build" ) @@ -132,9 +143,8 @@ def test_deploy_platform_via_script(state): assert result.returncode == 0, f"{label} failed:\n{result.stderr}" deployments = json.loads(DEPLOYMENTS_FILE.read_text()) - for key in (*DEPLOYMENT_KEYS, "proxyAdmin"): + for key in (*DEPLOYMENT_KEYS, *PROXY_ADMIN_KEYS): assert deployments.get(key, "").startswith("0x"), f"missing {key} in {DEPLOYMENTS_FILE}" - state["proxy_admin_address"] = deployments["proxyAdmin"] def test_import_deployments_into_din_info(run, state): @@ -161,13 +171,16 @@ def test_import_deployments_into_din_info(run, state): f"din_info['{info_key}'] not updated from deployments['{dep_key}']" ) state[state_key] = din_info[info_key] - assert din_info.get("proxy_admin") == deployments["proxyAdmin"] + for dep_key, info_key in PROXY_ADMIN_KEYS.items(): + assert din_info.get(info_key) == deployments[dep_key], ( + f"din_info['{info_key}'] not updated from deployments['{dep_key}']" + ) def test_dump_abi_coordinator(run): - """Dump coordinator ABI into bundled abis/.""" + """Dump coordinator ABI into scratch (never the tracked dincli/abis/).""" result = run([ - "system", "dump-abi", "--official", + "system", "dump-abi", "--output", str(DUMP_ABI_DIR), "--artifact", str(ARTIFACTS["coordinator"]), ]) assert result.returncode == 0 @@ -175,7 +188,7 @@ def test_dump_abi_coordinator(run): def test_dump_abi_token(run): result = run([ - "system", "dump-abi", "--official", + "system", "dump-abi", "--output", str(DUMP_ABI_DIR), "--artifact", str(ARTIFACTS["token"]), ]) assert result.returncode == 0 @@ -183,7 +196,7 @@ def test_dump_abi_token(run): def test_dump_abi_validator_stake(run): result = run([ - "system", "dump-abi", "--official", + "system", "dump-abi", "--output", str(DUMP_ABI_DIR), "--artifact", str(ARTIFACTS["validator_stake"]), ]) assert result.returncode == 0 @@ -191,7 +204,7 @@ def test_dump_abi_validator_stake(run): def test_dump_abi_registry(run): result = run([ - "system", "dump-abi", "--official", + "system", "dump-abi", "--output", str(DUMP_ABI_DIR), "--artifact", str(ARTIFACTS["registry"]), ]) assert result.returncode == 0 diff --git a/tests/dincli/test_02_task_contracts.py b/tests/dincli/test_02_task_contracts.py index e0fc1990..1294e5e2 100644 --- a/tests/dincli/test_02_task_contracts.py +++ b/tests/dincli/test_02_task_contracts.py @@ -13,10 +13,12 @@ """ import json +import re + import pytest from pathlib import Path -from tests.dincli.constants import ARTIFACT_BASE, DIN_INFO_PATH +from tests.dincli.constants import ARTIFACT_BASE, DIN_INFO_PATH, DUMP_ABI_DIR ARTIFACTS = { "task_coordinator": ARTIFACT_BASE / "DINTaskCoordinator.sol/DINTaskCoordinator.json", @@ -28,7 +30,7 @@ def ensure_artifacts(): missing = [name for name, path in ARTIFACTS.items() if not path.exists()] if missing: - pytest.skip(f"Hardhat artifacts missing for: {missing}.") + pytest.skip(f"Foundry artifacts missing for: {missing} (run forge build).") # --------------------------------------------------------------------------- @@ -38,7 +40,7 @@ def ensure_artifacts(): def test_connect_wallet_model_owner(run, state): """Switch to account 1 — model owner.""" - result = run(["system", "connect-wallet", "modelowner"]) + result = run(["system", "connect-demo-wallet", "modelowner"]) assert result.returncode == 0 @@ -49,9 +51,19 @@ def test_eth_balance_model_owner(run): assert "ETH" in result.stdout or "eth" in result.stdout.lower() +def _deployed_address(stdout: str, contract: str) -> str: + """The address in the CLI's "✓ contract deployed at: 0x…" line.""" + match = re.search(rf"{contract} contract deployed at:\s*(0x[a-fA-F0-9]{{40}})", stdout) + assert match, f"No deployed {contract} address in output:\n{stdout}" + return match.group(1) + + def test_deploy_task_coordinator(run, state): """Deploy DINTaskCoordinator as model owner. + --model-id is left to its default (DINModelRegistry.totalModels() = 0), + and the deploy sets DinToken on the new coordinator. + SDK candidate: deploy_task_coordinator(network, account, artifact_path) """ result = run([ @@ -59,21 +71,14 @@ def test_deploy_task_coordinator(run, state): "--artifact", str(ARTIFACTS["task_coordinator"]), ], timeout=180) assert result.returncode == 0 - - # Address is written to .env in cwd — parse it from stdout as fallback - # The CLI prints the deployed address; extract it for state. - for line in result.stdout.splitlines(): - if "0x" in line and len(line.strip()) >= 42: - import re - match = re.search(r"0x[a-fA-F0-9]{40}", line) - if match: - state["task_coordinator_address"] = match.group(0) - break + assert "totalModels() = 0" in result.stdout + assert "DinToken set on DINTaskCoordinator" in result.stdout + state["task_coordinator_address"] = _deployed_address(result.stdout, "DINTaskCoordinator") def test_dump_abi_task_coordinator(run): result = run([ - "system", "dump-abi", "--official", + "system", "dump-abi", "--output", str(DUMP_ABI_DIR), "--artifact", str(ARTIFACTS["task_coordinator"]), ]) assert result.returncode == 0 @@ -82,6 +87,10 @@ def test_dump_abi_task_coordinator(run): def test_deploy_task_auditor(run, state): """Deploy DINTaskAuditor as model owner. + --model-id defaults to the coordinator's modelId(); the deploy sets + DinToken on the new auditor (rewards deposit needs it) before linking + it to the coordinator. + SDK candidate: deploy_task_auditor(network, account, artifact_path) """ result = run([ @@ -89,19 +98,13 @@ def test_deploy_task_auditor(run, state): "--artifact", str(ARTIFACTS["task_auditor"]), ], timeout=180) assert result.returncode == 0 - - for line in result.stdout.splitlines(): - if "0x" in line: - import re - match = re.search(r"0x[a-fA-F0-9]{40}", line) - if match: - state["task_auditor_address"] = match.group(0) - break + assert "DinToken set on DINTaskAuditor" in result.stdout + state["task_auditor_address"] = _deployed_address(result.stdout, "DINTaskAuditor") def test_dump_abi_task_auditor(run): result = run([ - "system", "dump-abi", "--official", + "system", "dump-abi", "--output", str(DUMP_ABI_DIR), "--artifact", str(ARTIFACTS["task_auditor"]), ]) assert result.returncode == 0 @@ -112,7 +115,7 @@ def test_din_rep_authorizes_task_coordinator_as_slasher(run, state): SDK candidate: dinrep_add_slasher(network, account, task_coordinator=addr) """ - run(["system", "connect-wallet", "dinrep"]) + run(["system", "connect-demo-wallet", "dinrep"]) result = run(["dinrep", "add-slasher", "--taskCoordinator"]) assert result.returncode == 0 @@ -128,7 +131,7 @@ def test_model_owner_registers_task_coordinator_slasher(run): SDK candidate: model_owner_add_slasher(network, account, task_coordinator=addr) """ - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "add-slasher", "--taskCoordinator"]) assert result.returncode == 0 diff --git a/tests/dincli/test_03_registration.py b/tests/dincli/test_03_registration.py index 64470683..f417b55c 100644 --- a/tests/dincli/test_03_registration.py +++ b/tests/dincli/test_03_registration.py @@ -41,7 +41,7 @@ def test_model_owner_create_task_dir(run): SDK candidate: create_task_dir(network, model_owner_account) """ - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "task", "create-task-dir"]) assert result.returncode == 0 @@ -137,7 +137,7 @@ def test_model_owner_submit_registration_request(run): def test_din_rep_lists_pending_requests(run): """DIN-Representative can see the pending registration request.""" - run(["system", "connect-wallet", "dinrep"]) + run(["system", "connect-demo-wallet", "dinrep"]) result = run(["dinrep", "registry", "list-pending-requests"]) assert result.returncode == 0 @@ -160,7 +160,7 @@ def test_din_rep_approves_registration_request(run, state): def test_model_owner_checks_request_status(run): """Model owner confirms their request was approved.""" - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["task", "model-owner", "my-requests", "--ip"]) assert result.returncode == 0 @@ -206,13 +206,13 @@ def test_model_owner_update_manifest_request(run, din_tmp): def test_din_rep_lists_manifest_requests(run): - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["dinrep", "registry", "list-pending-requests"]) assert result.returncode == 0 def test_din_rep_explores_manifest_request(run): - run(["system", "connect-wallet", "dinrep"]) + run(["system", "connect-demo-wallet", "dinrep"]) result = run(["dinrep", "registry", "explore-request", "0", "-t", "manifest"]) assert result.returncode == 0 diff --git a/tests/dincli/test_04_gi.py b/tests/dincli/test_04_gi.py index c7b1243d..7cc5f22f 100644 --- a/tests/dincli/test_04_gi.py +++ b/tests/dincli/test_04_gi.py @@ -1,61 +1,90 @@ """ -Phase 4 — Complete Global Iteration (GI). - -Drives a full GI lifecycle for model 0: - GI start → aggregator registration → auditor registration → - LMS open → client training + submission → LMS close → - auditor batches → LMS evaluation → aggregation T1+T2 → - slash → GI end - -Expected final state: GIended (index 23). - -Account map: +Phase 4 — one complete Global Iteration (GI) on the foundry task contracts. + +Walks GI 1 of model 0 end to end (task_061026_21 Part D, Decision 3) and +asserts the GI state by name after every phase: + + 1. fund GI 1 (rewards deposit) → gi start GIstarted + 2. aggregator / auditor registration (+ auditor encryption keys; + an aggregator registering as an auditor is refused) + 3. LMS → auditor lock-seed AuditSeedLocked + → auditor-batches create AuditorsBatchesCreated + → create-testdataset --submit AuditTestDataAssigned + 4. lms-evaluation start → evaluate --submit → start-reveal → reveal → close + 5. aggregator lock-seed AggSeedLocked + → create-t1nt2-batches T1nT2Bcreated + 6. T1 start → aggregate-t1 --submit → start-reveal → reveal-t1 → close; + the same for T2; then T2 set-score + 7. slash auditors → slash aggregators → gi end GIended + 8. rewards claim + withdraw for one client, one auditor and one + aggregator (DIN balance rises) → gi release-slots + +Account map (anvil, 70 accounts): 0 = DIN-Representative 1 = model owner - 2–10 = clients (9 accounts; account 3 is skipped — matches task description) + 2–10 = clients (9 accounts) 11–22 = aggregators (12 accounts) 50–58 = auditors (9 accounts) Confirmation-prompt map (input_text reference): - model-owner gi start : 1 × y (test dataset for GM evaluation) - client train-lms : 3 × y (dataset placed? confirm containerised training × 2) - model-owner slash auditors : 1 × y (only if tier2_score == 0; pass extra y to be safe) - model-owner slash aggregators: 1 × y (same guard) - -SDK candidates: - start_gi(network, account, model_id) - register_aggregator(network, account, model_id) - register_auditor(network, account, model_id) - open_lms / close_lms(network, account, model_id) - train_and_submit_lm(network, account, model_id) - create_auditor_batches(network, account, model_id) - start_lms_evaluation / close_lms_evaluation(network, account, model_id) - evaluate_lms(network, account, model_id) - start_t1_aggregation / close_t1_aggregation(network, account, model_id) - aggregate_t1(network, account, model_id) - start_t2_aggregation / close_t2_aggregation(network, account, model_id) - aggregate_t2(network, account, model_id) - slash_auditors / slash_aggregators(network, account, model_id) - end_gi(network, account, model_id) + model-owner gi start : 1 × y (test dataset for GM evaluation) + client train-lms : 3 × y (dataset placed? confirm containerised training × 2) + model-owner slash auditors : 1 × y (only if tier2_score == 0; pass extra y to be safe) + model-owner slash aggregators: 1 × y (same guard) """ +import json +import re + import pytest -from tests.dincli.constants import TORCHENV_PYTHON, TORCHENV_SITE_PACKAGES +from eth_account import Account +from web3 import Web3 + +from tests.dincli.constants import DIN_INFO_PATH, HARDHAT_RPC, TORCHENV_PYTHON, TORCHENV_SITE_PACKAGES +from tests.dincli.demo import DEMO_MNEMONIC MODEL_ID = "0" +GI = "1" AGGREGATOR_ACCOUNTS = list(range(11, 23)) # 11..22 inclusive AUDITOR_ACCOUNTS = list(range(50, 59)) # 50..58 inclusive -CLIENT_ACCOUNTS = list(range(2, 11)) # 2, 4..10 (account 3 is skipped per task) +CLIENT_ACCOUNTS = list(range(2, 11)) # 2..10 inclusive + +REWARD_DEPOSIT_DIN = "50" +MODEL_OWNER_DIN_BUY_ETH = "0.001" + +_BALANCE_OF_ABI = [{ + "inputs": [{"name": "account", "type": "address"}], + "name": "balanceOf", + "outputs": [{"name": "", "type": "uint256"}], + "stateMutability": "view", + "type": "function", +}] def _gi_state(run) -> str: - result = run(["task", "gi", "show-state", MODEL_ID], check=False) - for line in result.stdout.splitlines(): - line = line.strip() - if line and not line.startswith("["): - return line - return result.stdout.strip() + """The GI state name printed by `task gi show-state` ("Global iteration state: ").""" + result = run(["task", "gi", "show-state", MODEL_ID]) + match = re.search(r"Global iteration state:\s*(\w+)", result.stdout) + assert match, f"No GI state in show-state output:\n{result.stdout}" + return match.group(1) + + +def _assert_state(run, expected: str) -> None: + actual = _gi_state(run) + assert actual == expected, f"Expected GI state {expected}, got {actual}" + + +def _address(acc: int) -> str: + Account.enable_unaudited_hdwallet_features() + return Account.from_mnemonic(DEMO_MNEMONIC, account_path=f"m/44'/60'/0'/0/{acc}").address + + +def _din_balance(acc: int) -> int: + token = json.loads(DIN_INFO_PATH.read_text(encoding="utf-8"))["local"]["token"] + w3 = Web3(Web3.HTTPProvider(HARDHAT_RPC)) + contract = w3.eth.contract(address=Web3.to_checksum_address(token), abi=_BALANCE_OF_ABI) + return contract.functions.balanceOf(_address(acc)).call() # --------------------------------------------------------------------------- @@ -65,7 +94,7 @@ def _gi_state(run) -> str: def test_show_initial_gi_state(run): """Confirm model 0 GI state before starting.""" - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["task", "gi", "show-state", MODEL_ID]) assert result.returncode == 0 @@ -79,6 +108,25 @@ def test_add_default_test_data_for_gi(run): assert result.returncode == 0 +def _assert_no_failures(result): + """A command that skips its work (✗ lines, failed txs) can still exit 0.""" + assert "✗" not in result.stdout, result.stdout + assert "failed" not in result.stdout.lower(), result.stdout + + +def test_fund_gi_reward_pool(run): + """Step 1: the model owner buys DIN and deposits it into GI 1's reward pool. + + `gi start` refuses an unfunded GI. The deposit needs `setDinToken` on + DINTaskAuditor, which `model-owner deploy task-auditor` sends. + """ + run(["system", "connect-demo-wallet", "modelowner"]) + run(["dintoken", "buy", MODEL_OWNER_DIN_BUY_ETH]) + result = run(["model-owner", "rewards", "deposit", MODEL_ID, "--amount", REWARD_DEPOSIT_DIN]) + assert result.returncode == 0 + assert f"GI {GI} reward pool" in result.stdout + + def test_gi_start(run): """Model owner starts GI for model 0. @@ -93,6 +141,7 @@ def test_gi_start(run): input_text="y\n", ) assert result.returncode == 0 + _assert_state(run, "GIstarted") def test_gi_aggregators_open(run): @@ -125,7 +174,7 @@ def test_aggregator_register(run, acc): def test_show_registered_aggregators(run): - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "gi", "show-registered-aggregators", MODEL_ID]) assert result.returncode == 0 @@ -162,8 +211,24 @@ def test_auditor_register(run, acc): assert result.returncode == 0 +@pytest.mark.parametrize("acc", AUDITOR_ACCOUNTS) +def test_auditor_register_encryption_key(run, acc): + """Each auditor registers an X25519 key. create-testdataset --submit refuses keyless auditors.""" + result = run(["--demokey", str(acc), "auditor", "register-encryption-key"]) + assert result.returncode == 0 + + +def test_aggregator_cannot_register_as_auditor(run): + """task_061026_20 Part B (#180): an aggregator of this GI can't also be an auditor.""" + result = run( + ["--demokey", str(AGGREGATOR_ACCOUNTS[0]), "auditor", "register", MODEL_ID], + check=False, + ) + assert result.returncode != 0, f"Dual-role registration was not refused:\n{result.stdout}" + + def test_show_registered_auditors(run): - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "gi", "show-registered-auditors", MODEL_ID]) assert result.returncode == 0 @@ -171,6 +236,7 @@ def test_show_registered_auditors(run): def test_gi_auditors_close(run): result = run(["model-owner", "gi", "reg", "auditors-close", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "DINauditorsRegistrationClosed") # --------------------------------------------------------------------------- @@ -185,6 +251,7 @@ def test_lms_open(run): """ result = run(["model-owner", "lms", "open", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "LMSstarted") def test_distribute_mnist(run): @@ -230,9 +297,10 @@ def test_lms_close(run): SDK candidate: close_lms(network, account, model_id) """ - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "lms", "close", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "LMSclosed") # --------------------------------------------------------------------------- @@ -240,6 +308,14 @@ def test_lms_close(run): # --------------------------------------------------------------------------- +def test_auditor_lock_seed(run): + """An auditor (permissionless) locks the auditor batch-assignment seed once its block is mined.""" + result = run(["--demokey", str(AUDITOR_ACCOUNTS[0]), "auditor", "lock-seed", MODEL_ID], timeout=180) + assert result.returncode == 0 + run(["system", "connect-demo-wallet", "modelowner"]) + _assert_state(run, "AuditSeedLocked") + + def test_create_auditor_batches(run): """Assign submitted LMs to auditor batches. @@ -247,6 +323,7 @@ def test_create_auditor_batches(run): """ result = run(["model-owner", "auditor-batches", "create", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "AuditorsBatchesCreated") def test_show_auditor_batches(run): @@ -260,6 +337,7 @@ def test_create_testdataset_and_submit(run): "model-owner", "auditor-batches", "create-testdataset", MODEL_ID, "--submit", ], python=TORCHENV_PYTHON, timeout=300) assert result.returncode == 0 + _assert_state(run, "AuditTestDataAssigned") def test_lms_evaluation_start(run): @@ -269,6 +347,7 @@ def test_lms_evaluation_start(run): """ result = run(["model-owner", "lms-evaluation", "start", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "LMSevaluationStarted") def test_show_lms_evaluation(run): @@ -297,21 +376,41 @@ def test_auditor_evaluate(run, acc): timeout=300, ) assert result.returncode == 0 + _assert_no_failures(result) def test_show_lms_evaluation_models_after(run): - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "lms-evaluation", "show", MODEL_ID, "--models"]) assert result.returncode == 0 +def test_lms_evaluation_start_reveal(run): + """Close the commit window and open the reveal window.""" + result = run(["model-owner", "lms-evaluation", "start-reveal", MODEL_ID]) + assert result.returncode == 0 + _assert_state(run, "LMSevaluationRevealStarted") + + +@pytest.mark.parametrize("acc", AUDITOR_ACCOUNTS) +def test_auditor_reveal(run, acc): + """Each auditor reveals the scores it committed (from its own per-wallet commit cache).""" + result = run(["--demokey", str(acc), "auditor", "lms-evaluation", "reveal", MODEL_ID]) + assert result.returncode == 0 + _assert_no_failures(result) + # Every LM in this auditor's batch must have a commit to reveal. + assert "No local commit found for LM" not in result.stdout, result.stdout + + def test_lms_evaluation_close(run): """Close LMS evaluation window. SDK candidate: close_lms_evaluation(network, account, model_id) """ + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "lms-evaluation", "close", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "LMSevaluationClosed") # --------------------------------------------------------------------------- @@ -319,6 +418,14 @@ def test_lms_evaluation_close(run): # --------------------------------------------------------------------------- +def test_aggregator_lock_seed(run): + """An aggregator (permissionless) locks the T1/T2 batch-assignment seed once its block is mined.""" + result = run(["--demokey", str(AGGREGATOR_ACCOUNTS[0]), "aggregator", "lock-seed", MODEL_ID], timeout=180) + assert result.returncode == 0 + run(["system", "connect-demo-wallet", "modelowner"]) + _assert_state(run, "AggSeedLocked") + + def test_create_t1nt2_batches(run): """Create T1 and T2 aggregation batches. @@ -326,6 +433,7 @@ def test_create_t1nt2_batches(run): """ result = run(["model-owner", "aggregation", "create-t1nt2-batches", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "T1nT2Bcreated") def test_show_t1_batches_before_agg(run): @@ -345,6 +453,7 @@ def test_t1_aggregation_start(run): """ result = run(["model-owner", "aggregation", "T1", "start", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "T1AggregationStarted") @pytest.mark.parametrize("acc", AGGREGATOR_ACCOUNTS) @@ -363,21 +472,38 @@ def test_aggregator_aggregate_t1(run, acc): timeout=300, ) assert result.returncode == 0 + _assert_no_failures(result) def test_show_t1_batches_after_agg(run): - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "aggregation", "show-t1-batches", MODEL_ID, "--detailed"]) assert result.returncode == 0 +def test_t1_aggregation_start_reveal(run): + result = run(["model-owner", "aggregation", "T1", "start-reveal", MODEL_ID]) + assert result.returncode == 0 + _assert_state(run, "T1AggregationRevealStarted") + + +@pytest.mark.parametrize("acc", AGGREGATOR_ACCOUNTS) +def test_aggregator_reveal_t1(run, acc): + """Each aggregator reveals its T1 commits (a no-op for one outside every T1 batch).""" + result = run(["--demokey", str(acc), "aggregator", "reveal-t1", MODEL_ID]) + assert result.returncode == 0 + _assert_no_failures(result) + + def test_t1_aggregation_close(run): """Close T1 aggregation. SDK candidate: close_t1_aggregation(network, account, model_id) """ + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "aggregation", "T1", "close", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "T1AggregationDone") # --------------------------------------------------------------------------- @@ -388,11 +514,7 @@ def test_t1_aggregation_close(run): def test_t2_aggregation_start(run): result = run(["model-owner", "aggregation", "T2", "start", MODEL_ID]) assert result.returncode == 0 - - -def test_show_gi_state_t2(run): - result = run(["task", "gi", "show-state", MODEL_ID]) - assert result.returncode == 0 + _assert_state(run, "T2AggregationStarted") def test_show_t2_batches(run): @@ -416,21 +538,49 @@ def test_aggregator_aggregate_t2(run, acc): timeout=300, ) assert result.returncode == 0 + _assert_no_failures(result) def test_show_t2_batches_after_agg(run): - run(["system", "connect-wallet", "modelowner"]) + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "aggregation", "show-t2-batches", MODEL_ID, "--detailed"]) assert result.returncode == 0 +def test_t2_aggregation_start_reveal(run): + result = run(["model-owner", "aggregation", "T2", "start-reveal", MODEL_ID]) + assert result.returncode == 0 + _assert_state(run, "T2AggregationRevealStarted") + + +@pytest.mark.parametrize("acc", AGGREGATOR_ACCOUNTS) +def test_aggregator_reveal_t2(run, acc): + """Each aggregator reveals its T2 commit (a no-op for one outside the T2 batch).""" + result = run(["--demokey", str(acc), "aggregator", "reveal-t2", MODEL_ID]) + assert result.returncode == 0 + _assert_no_failures(result) + + def test_t2_aggregation_close(run): """Close T2 aggregation. SDK candidate: close_t2_aggregation(network, account, model_id) """ + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["model-owner", "aggregation", "T2", "close", MODEL_ID]) assert result.returncode == 0 + _assert_state(run, "T2AggregationDone") + + +def test_t2_set_score(run): + """Score the final GM on the owner's test data and record it on-chain.""" + result = run( + ["model-owner", "aggregation", "T2", "set-score", MODEL_ID], + python=TORCHENV_PYTHON, + timeout=300, + ) + assert result.returncode == 0 + assert "Tier 2 score set" in result.stdout # --------------------------------------------------------------------------- @@ -451,6 +601,7 @@ def test_slash_auditors(run): input_text="y\n", ) assert result.returncode == 0 + _assert_state(run, "AuditorsSlashed") def test_slash_aggregators(run): @@ -463,6 +614,7 @@ def test_slash_aggregators(run): input_text="y\n", ) assert result.returncode == 0 + _assert_state(run, "AggregatorsSlashed") def test_gi_end(run): @@ -475,10 +627,47 @@ def test_gi_end(run): def test_gi_final_state_is_ended(run): - """Assert final GI state is GIended (index 23) — the key regression gate.""" - run(["system", "connect-wallet", "modelowner"]) + """The GI ends in GIended (ordinal 28) — the key regression gate.""" + run(["system", "connect-demo-wallet", "modelowner"]) result = run(["task", "gi", "show-state", MODEL_ID]) + assert "Global iteration state: GIended" in result.stdout, result.stdout + assert "Global iteration numerical state: 28" in result.stdout, result.stdout + + +# --------------------------------------------------------------------------- +# Rewards + release slots +# --------------------------------------------------------------------------- + + +def _claim_and_withdraw(run, acc) -> bool: + """Claim and withdraw GI 1's reward for `acc`; True if its DIN balance rose.""" + before = _din_balance(acc) + claim = run(["--demokey", str(acc), "rewards", "claim", MODEL_ID, "--gi", GI], check=False) + if claim.returncode != 0: + return False + withdraw = run(["--demokey", str(acc), "rewards", "withdraw", MODEL_ID], check=False) + return withdraw.returncode == 0 and _din_balance(acc) > before + + +@pytest.mark.parametrize("role,accounts", [ + ("client", CLIENT_ACCOUNTS), + ("auditor", AUDITOR_ACCOUNTS), + ("aggregator", AGGREGATOR_ACCOUNTS), +]) +def test_reward_claim_and_withdraw(run, role, accounts): + """One participant of each role claims and withdraws GI 1's reward, and its DIN balance rises. + + Tries the role's accounts in order: a participant whose work was rejected + or slashed, or that sat outside every batch, has no share. + """ + for acc in accounts: + if _claim_and_withdraw(run, acc): + return + pytest.fail(f"No {role} in {accounts} could claim and withdraw a GI {GI} reward") + + +def test_gi_release_slots(run): + """Release the ended GI's validator registration slots (once per GI).""" + run(["system", "connect-demo-wallet", "modelowner"]) + result = run(["model-owner", "gi", "release-slots", MODEL_ID, "--gi", GI]) assert result.returncode == 0 - assert "GIended" in result.stdout or "23" in result.stdout, ( - f"Expected GIended state, got:\n{result.stdout}" - ) diff --git a/tests/test_add_default_test_data_cid.py b/tests/test_add_default_test_data_cid.py new file mode 100644 index 00000000..55169960 --- /dev/null +++ b/tests/test_add_default_test_data_cid.py @@ -0,0 +1,40 @@ +"""`model-owner model add-default-test-data --default-test-data` takes its CID +from din_info's `default_test_dataset` when set, else the published default.""" +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from dincli.cli.modelownerd import model + + +class _Console: + def print(self, *a, **k): + pass + + +@pytest.fixture +def fetched(monkeypatch, tmp_path): + calls = [] + monkeypatch.setattr(model, "resolve_task_coordinator_address", lambda net, addr, console: "0xC") + monkeypatch.setattr(model, "retrieve_from_ipfs", lambda cid, path: calls.append((cid, Path(path)))) + return calls + + +def _ctx(tmp_path): + return SimpleNamespace(obj=SimpleNamespace( + get_en_w3_account_console=lambda: ("local", None, None, _Console()), + get_model_base_dir=lambda model_id: tmp_path / f"model_{model_id}", + )) + + +def test_uses_din_info_override(monkeypatch, tmp_path, fetched): + monkeypatch.setattr(model, "load_din_info", lambda: {"local": {"default_test_dataset": "bafyOverride"}}) + model.add_default_test_data(_ctx(tmp_path), task_coordinator_address=None, model_id=0, default_test_data=True) + assert fetched == [("bafyOverride", tmp_path / "model_0" / "dataset" / "test" / "test_dataset.pt")] + + +def test_falls_back_to_published_default(monkeypatch, tmp_path, fetched): + monkeypatch.setattr(model, "load_din_info", lambda: {"local": {}}) + model.add_default_test_data(_ctx(tmp_path), task_coordinator_address=None, model_id=0, default_test_data=True) + assert fetched[0][0] == model.DEFAULT_TEST_DATASET_CID diff --git a/tests/test_aggregator_commit_retry.py b/tests/test_aggregator_commit_retry.py index 7d30e156..b7871500 100644 --- a/tests/test_aggregator_commit_retry.py +++ b/tests/test_aggregator_commit_retry.py @@ -103,7 +103,7 @@ def worker_stubs(tmp_path): def _read_cache(model_base_dir, tier, batch_id=0): - with open(_agg_commit_store_path(model_base_dir, tier, GI, batch_id)) as f: + with open(_agg_commit_store_path(model_base_dir, tier, GI, batch_id, ACCOUNT)) as f: return json.load(f) @@ -115,7 +115,7 @@ def test_already_committed_batch_is_skipped_and_cache_untouched(tmp_path, tier, state = {"committed": True, "commits": []} contract = _make_contract(state) ctx = _make_ctx(contract, tmp_path) - _save_agg_commit(tmp_path, tier, GI, 0, OLD_CID, OLD_SALT) + _save_agg_commit(tmp_path, tier, GI, 0, OLD_CID, OLD_SALT, ACCOUNT) before = _read_cache(tmp_path, tier) with patch.object(aggregator, "build_and_send_tx") as send, \ @@ -140,7 +140,7 @@ def test_cache_written_before_send_and_matches_sent_commit(tmp_path, worker_stub state = {"committed": False, "commits": []} contract = _make_contract(state) ctx = _make_ctx(contract, tmp_path) - cache_path = _agg_commit_store_path(tmp_path, tier, GI, 0) + cache_path = _agg_commit_store_path(tmp_path, tier, GI, 0, ACCOUNT) seen_at_send = {} def _send(ctx_, fn, *args, **kwargs): diff --git a/tests/test_auditor_commit_retry.py b/tests/test_auditor_commit_retry.py index c2cd4aad..32a808e2 100644 --- a/tests/test_auditor_commit_retry.py +++ b/tests/test_auditor_commit_retry.py @@ -127,7 +127,7 @@ def _run(ctx): def _read_cache(model_base_dir, model_index): - with open(_commit_store_path(model_base_dir, GI, BATCH, model_index)) as f: + with open(_commit_store_path(model_base_dir, GI, BATCH, model_index, ACCOUNT)) as f: return json.load(f) @@ -136,7 +136,7 @@ def test_already_committed_lm_is_skipped_and_cache_untouched(tmp_path, worker_st leave LM 0's cache and on-chain commit alone and only commit LM 1.""" state = {"committed": {0}, "commits": []} ctx = _make_ctx(_make_auditor_contract(state), tmp_path) - _save_commit(tmp_path, GI, BATCH, 0, 55, False, OLD_SALT) + _save_commit(tmp_path, GI, BATCH, 0, 55, False, OLD_SALT, ACCOUNT) before = _read_cache(tmp_path, 0) with patch.object(auditor, "build_and_send_tx") as send: @@ -172,7 +172,7 @@ def test_cache_written_before_send_and_matches_sent_commit(tmp_path, worker_stub def _send(ctx_, fn, *args, **kwargs): model_index = state["commits"][-1][2] - seen_at_send.append(_commit_store_path(tmp_path, GI, BATCH, model_index).exists()) + seen_at_send.append(_commit_store_path(tmp_path, GI, BATCH, model_index, ACCOUNT).exists()) return None with patch.object(auditor, "build_and_send_tx", side_effect=_send): diff --git a/tests/test_cid_utils_0x_prefix.py b/tests/test_cid_utils_0x_prefix.py new file mode 100644 index 00000000..bc27dab8 --- /dev/null +++ b/tests/test_cid_utils_0x_prefix.py @@ -0,0 +1,25 @@ +"""get_cid_from_bytes32 must accept a 0x-prefixed digest. + +`auditor lms-evaluation evaluate` once built "1220" + "0x", which +fromhex() rejects at position 5, so every auditor failed to decrypt its test +data and nobody committed a score. +""" + +import pytest + +from dincli.services.cid_utils import get_bytes32_from_cid, get_cid_from_bytes32 + +# A raw test-data CID the model owner uploaded in a full-GI run. +RAW_CID = "bafybeieoufpd24dlkgvf73tkk4bmuf2snfcs5skw34yashifrwvtoc3qsm" + + +@pytest.mark.parametrize("prefix", ["", "0x", "0X"]) +def test_digest_round_trips_with_or_without_0x(prefix): + digest = get_bytes32_from_cid(RAW_CID) + assert get_cid_from_bytes32(prefix + digest) == RAW_CID + + +def test_raw_bytes_hex_as_evaluate_passes_it(): + raw_cid_bytes = bytes.fromhex(get_bytes32_from_cid(RAW_CID)) + assert get_cid_from_bytes32(raw_cid_bytes.hex()) == RAW_CID + assert get_cid_from_bytes32("0x" + raw_cid_bytes.hex()) == RAW_CID diff --git a/tests/test_commit_store_per_wallet.py b/tests/test_commit_store_per_wallet.py new file mode 100644 index 00000000..8c20522f --- /dev/null +++ b/tests/test_commit_store_per_wallet.py @@ -0,0 +1,53 @@ +"""The auditor and aggregator commit stores are per wallet. + +Several auditors share an audit batch, and several aggregators share a T1/T2 +batch. When they run from one machine (one dincli cache dir; e.g. `--demokey` +in the tests/dincli harness), a shared commit file let the last committer +overwrite the others' salts, so their reveals failed the commit hash. Each +wallet now gets commits/
/..., and the loaders still read the old +shared path so a commit made before upgrading can be revealed. +""" +import json + +from dincli.cli.aggregator import TIER1, TIER2, _agg_commit_store_path, _load_agg_commit, _save_agg_commit +from dincli.cli.auditor import _commit_store_path, _load_commit, _save_commit + +A = "0x" + "aa" * 20 +B = "0x" + "bb" * 20 +GI, BATCH, LMI = 1, 0, 3 + + +def test_two_auditors_same_batch_keep_their_own_salts(tmp_path): + _save_commit(tmp_path, GI, BATCH, LMI, 70, True, b"\x01" * 32, A) + _save_commit(tmp_path, GI, BATCH, LMI, 40, False, b"\x02" * 32, B) + + a, b = _load_commit(tmp_path, GI, BATCH, LMI, A), _load_commit(tmp_path, GI, BATCH, LMI, B) + assert (a["score"], a["vote"], a["salt"]) == (70, True, b"\x01" * 32) + assert (b["score"], b["vote"], b["salt"]) == (40, False, b"\x02" * 32) + assert _commit_store_path(tmp_path, GI, BATCH, LMI, A).parent.name == A + + +def test_auditor_reads_legacy_shared_commit(tmp_path): + legacy = _commit_store_path(tmp_path, GI, BATCH, LMI, None) + legacy.parent.mkdir(parents=True) + legacy.write_text(json.dumps({"score": 9, "vote": True, "salt": ("03" * 32)})) + + assert _load_commit(tmp_path, GI, BATCH, LMI, A)["score"] == 9 + assert _load_commit(tmp_path, GI + 1, BATCH, LMI, A) is None + + +def test_two_aggregators_same_batch_keep_their_own_salts(tmp_path): + for tier in (TIER1, TIER2): + _save_agg_commit(tmp_path, tier, GI, BATCH, b"\xaa" * 32, b"\x01" * 32, A) + _save_agg_commit(tmp_path, tier, GI, BATCH, b"\xaa" * 32, b"\x02" * 32, B) + assert _load_agg_commit(tmp_path, tier, GI, BATCH, A)["salt"] == b"\x01" * 32 + assert _load_agg_commit(tmp_path, tier, GI, BATCH, B)["salt"] == b"\x02" * 32 + + +def test_aggregator_reads_legacy_shared_commit(tmp_path): + legacy = _agg_commit_store_path(tmp_path, TIER1, GI, BATCH, None) + legacy.parent.mkdir(parents=True) + legacy.write_text(json.dumps({"cid": "ab" * 32, "salt": "04" * 32})) + + assert _load_agg_commit(tmp_path, TIER1, GI, BATCH, A)["salt"] == b"\x04" * 32 + assert _load_agg_commit(tmp_path, TIER2, GI, BATCH, A) is None diff --git a/tests/test_integration_harness.py b/tests/test_integration_harness.py index c95fcfa8..2572f437 100644 --- a/tests/test_integration_harness.py +++ b/tests/test_integration_harness.py @@ -212,7 +212,7 @@ def run(args): if fail_bootstrap: raise RuntimeError("bootstrap command failed") - fixture = harness.bootstrap.__wrapped__(None, None, run) + fixture = harness.bootstrap.__wrapped__(None, None, None, run) if fail_bootstrap: with pytest.raises(RuntimeError, match="bootstrap command failed"): next(fixture) @@ -237,7 +237,7 @@ def run(args): if fail_bootstrap: raise RuntimeError("bootstrap command failed") - fixture = harness.bootstrap.__wrapped__(None, None, run) + fixture = harness.bootstrap.__wrapped__(None, None, None, run) if fail_bootstrap: with pytest.raises(RuntimeError, match="bootstrap command failed"): next(fixture) diff --git a/tests/test_integration_seed.py b/tests/test_integration_seed.py new file mode 100644 index 00000000..623b5091 --- /dev/null +++ b/tests/test_integration_seed.py @@ -0,0 +1,53 @@ +"""tests/dincli/seed.py: seeding the isolated IPFS node with the reference +model's default artifacts (no IPFS node needed: `add` is faked).""" +import json + +from tests.dincli import seed + + +def _fake_add(log): + def add(path): + assert path.is_file(), f"seed references a missing file: {path}" + log.append(path) + if path.is_relative_to(seed.DEVNET_ROOT): + return "cid:" + path.relative_to(seed.DEVNET_ROOT).as_posix() + return "cid:generated/" + path.name + return add + + +def test_manifest_artifacts_point_at_repo_files(): + added = [] + manifest = seed.build_seeded_manifest(_fake_add(added)) + + assert manifest["getGenesisModelIpfs"]["ipfs"] == "cid:cache_model_0/services/modelowner.py" + assert manifest["Score_model_by_auditor"]["ipfs"] == "cid:cache_model_0/services/auditor.py" + assert manifest["task_contracts"]["DINTaskCoordinator_Contract"]["artifact"]["ipfs"] == "cid:dincli/abis/DINTaskCoordinator.json" + assert manifest["requirements.txt"]["auditors"] == "cid:cache_model_0/requirements/auditors/requirements.txt" + # source / compilation entries carry no CID and stay untouched + assert manifest["task_contracts"]["DINTaskAuditor_Contract"]["source"]["ipfs"] is None + + +def test_seed_returns_every_default_key(tmp_path, monkeypatch): + # cid_services entries are checked against their expected CIDs; echo them back. + expected = { + (seed.CID_SERVICES_DIR / e["path"].lstrip("/").removeprefix("services/")): e["ipfs"] + for e in json.loads(seed.CID_SERVICES_JSON.read_text())["services"].values() + } + expected.update({ + (seed.CID_SERVICES_DIR / "requirements" / (e["path"].rsplit("/", 1)[1].rsplit(".", 1)[0] + ".txt")): e["requirements"] + for e in json.loads(seed.CID_SERVICES_JSON.read_text())["services"].values() + }) + plain = _fake_add([]) + overrides = seed.seed_default_artifacts(lambda p: expected.get(p) or plain(p), tmp_path) + + assert overrides["default_manifest"] == "cid:generated/seeded_default_manifest.json" + assert set(overrides) == {"default_manifest", "default_services", "default_abis", "default_requirements", "default_test_dataset"} + assert overrides["default_test_dataset"] == "cid:cache_model_0/dataset/test/test_dataset.pt" + assert set(overrides["default_requirements"]) == {"clients", "auditors", "aggregators", "modelowner"} + + +def test_apply_din_info_overrides_merges(tmp_path): + path = tmp_path / "din_info.json" + path.write_text(json.dumps({"local": {"token": "0xT", "default_manifest": "old"}})) + seed.apply_din_info_overrides(path, "local", {"default_manifest": "new"}) + assert json.loads(path.read_text())["local"] == {"token": "0xT", "default_manifest": "new"} diff --git a/tests/test_validator_register_exit_codes.py b/tests/test_validator_register_exit_codes.py new file mode 100644 index 00000000..2aa889ea --- /dev/null +++ b/tests/test_validator_register_exit_codes.py @@ -0,0 +1,68 @@ +"""`auditor register` / `aggregator register` exit non-zero when registration +fails (a reverted or refused tx, e.g. TA_DualRoleNotAllowed, or too little +stake), so scripts and the integration suite can tell a refusal from success.""" +from types import SimpleNamespace + +import pytest +import typer + +from dincli.cli import aggregator, auditor +from tests._cli_fakes import DummyConsole, FakeContract + +ADDR = "0x000000000000000000000000000000000000A11c" +BIG = 10**30 + + +class _Ctx: + def __init__(self, stake): + self.console = DummyConsole() + self.coordinator = FakeContract("0xC", isDINAggregator=False) + self.auditor = FakeContract("0xA", isRegisteredAuditor=False) + self.stake = FakeContract("0xS", getStake=stake) + + def get_en_w3_account_console(self, model_id=None): + return "local", None, SimpleNamespace(address=ADDR), self.console + + def get_deployed_din_task_coordinator_contract(self, *a, **k): + return self.coordinator + + def get_deployed_din_task_auditor_contract(self, *a, **k): + return self.auditor + + def get_deployed_din_coordinator_contract(self, *a, **k): + return FakeContract("0xD") + + def get_deployed_din_stake_contract(self, *a, **k): + return self.stake + + def get_current_gi_and_state(self, *a, **k): + return 1, 0 + + def validate_gi_ET_curr_GI(self, gi, curr): + return curr + + def validate_GIstate_ET_given_GIstate(self, *a, **k): + pass + + +@pytest.mark.parametrize("module", [auditor, aggregator]) +@pytest.mark.parametrize("receipt,expect_exit", [(None, True), (SimpleNamespace(status=1), False)]) +def test_failed_tx_exits_1(monkeypatch, module, receipt, expect_exit): + monkeypatch.setattr(module, "build_and_send_tx", lambda *a, **k: receipt) + ctx = SimpleNamespace(obj=_Ctx(stake=BIG)) + if expect_exit: + with pytest.raises(typer.Exit) as exc: + module.register(ctx, model_id=0, gi=None) + assert exc.value.exit_code == 1 + else: + module.register(ctx, model_id=0, gi=None) + + +@pytest.mark.parametrize("module", [auditor, aggregator]) +def test_insufficient_stake_exits_1(monkeypatch, module): + sent = [] + monkeypatch.setattr(module, "build_and_send_tx", lambda *a, **k: sent.append(a)) + with pytest.raises(typer.Exit) as exc: + module.register(SimpleNamespace(obj=_Ctx(stake=0)), model_id=0, gi=None) + assert exc.value.exit_code == 1 + assert sent == []