Skip to content

fix(issue-discovery): solving-PR cache poisoned by unscored OSS PRs (token_score=0 blocks valid solves) #1677

Description

@bittensorrider

Description

_build_solving_pr_cache in gittensor/validator/issue_discovery/scan.py pre-populates the cross-miner solving-PR cache from every entry in evaluation.merged_prs, including PRs whose OSS score_pr() call early-returned without ever tokenizing.

Those incomplete PRs keep the ScoredPR defaults (token_score=0, base_score=0, files=None). Once cached, issue discovery treats them as a cache hit and never calls MirrorClient.get_pr_files — so a solving PR that would score on a fresh files fetch stays at token_score=0 and fails min_token_score_for_valid_issue.

This is the OSS → cache sibling of closed #836 (which fixed fabricated zeros on the cache-miss path inside _resolve_solving_pr_score). #836 correctly refuses to cache scoring_data_stored=False on fetch; the pre-populate path still injects the same class of fabricated zero from unscored merged_prs.

Distinct from open #1616 (unique_repos_contributed_to undercount on the same early-return paths) — that bug is about repo attribution; this one is about issue-discovery score resolution via the solving-PR cache.

Steps to Reproduce

  1. Construct a MinerEvaluation whose merged_prs contains a MERGED ScoredPR that never completed OSS scoring, e.g. scoring_data_stored=False caused score_pr to return early:

    • scored.files is None
    • scored.token_score == 0.0
    • scored.base_score == 0.0
    • PR remains in merged_prs (eligibility already passed at load time)
  2. Call _build_solving_pr_cache({uid: evaluation}).

  3. Observe cache[(repo, pr_number)] == CachedSolvingPR(base_score=0.0, token_score=0.0).

  4. Run issue discovery for a miner with a solved issue whose solving_pr.pr_number is that PR. _resolve_solving_pr_score hits the cache and does not call get_pr_files.

  5. Even if the mirror would now return scoring_data_stored=True with real file contents, the issue is treated as below-threshold / unscored.

Representative flow on current test:

OSS score_pr (mirror/scoring.py):
  scoring_data_stored=False and no fixed_base_score → return early
  ScoredPR stays at defaults; PR remains in merged_prs

_build_solving_pr_cache (scan.py):
  caches every merged_prs entry, including the incomplete one
  → cache[(repo, N)] = CachedSolvingPR(0.0, 0.0)

_resolve_solving_pr_score:
  key in cache → HIT → return zeros → never fetch files

valid-solved gate:
  token_score (0) < min_token_score_for_valid_issue → no discovery score

Contrast with the intentional design pinned by test_all_merged_prs_cached_regardless_of_token_score: caching a genuinely tokenized token_score=0 PR is correct (saturation / low-token suppression). Caching an unscored PR that merely looks like zero is not.

Expected Behavior

Only pre-populate the solving-PR cache from PRs whose OSS scoring actually completed, for example:

  • scored.files is a non-empty list (tokenization ran), or
  • scored.base_score > 0 (e.g. fixed_base_score path completed)

Incomplete early-returns (files is None / empty fetch with no fixed base, base_score still 0) must remain cache misses so _resolve_solving_pr_score can fetch files (and still honor #836: do not cache scoring_data_stored=False).

Actual Behavior

Every merged_prs entry is cached, including unscored defaults. Cache hits suppress get_pr_files, under-crediting issue discovery whenever mirror file data was pending/flaky during the OSS pass.

Environment

  • OS: macOS
  • Python version: 3.12.13
  • Commit/Version: test @ 26ebeb8

Additional Context

Affected code:

  • gittensor/validator/oss_contributions/mirror/scoring.pyscore_pr early returns (~139–140, ~147–149, ~174–176)
  • gittensor/validator/issue_discovery/scan.py_build_solving_pr_cache (~397–416), _resolve_solving_pr_score (~618–641)
  • tests/validator/issue_discovery/test_scan.pytest_all_merged_prs_cached_regardless_of_token_score (documents intentional zero caching for scored PRs; needs a sibling test for unscored PRs)

Suggested regression test:

  1. Seed merged_prs with an incomplete ScoredPR (files=None, scores 0).
  2. Build cache → key must be absent.
  3. Drive _resolve_solving_pr_score with a mock get_pr_files that returns real files → must be a miss, fetch, and populate real scores.

I am planning to open a PR targeting test for this fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions