Consistency with mteb: task prompts as judge instructions, model-owned truncation, no hidden defaults - #14
Open
AdnanElAssadi wants to merge 7 commits into
Open
Consistency with mteb: task prompts as judge instructions, model-owned truncation, no hidden defaults#14AdnanElAssadi wants to merge 7 commits into
AdnanElAssadi wants to merge 7 commits into
Conversation
…fixes - Registry v2: alias Nano*/Touche2020Retrieval.v3 names to canonical entries. v1 keys matched full-size BEIR names only, so nano ablation runs silently resolved the generic prompt (different judge condition than full scale) and the Touche2020 entry never fired at all (no such task in mteb 2.15.4). Touche text provenance corrected: it is NanoTouche2020Retrieval's prompt. - Single resolution path (resolve_task_instruction) shared by judge_system() and result-record metadata, so record and judge can never diverge. - GYM_MAX_DOC_CHARS / GYM_MAX_SEQ now key both the verdict hash and the embedding cache (conditional on non-default, existing caches stay valid); previously capped and uncapped runs sharing a cache dir reused each other's embeddings and verdicts. - judge_ prefix on verdict-level reliability fields in result records. - Registry-resolution test incl. verbatim check against installed mteb. Note: aliases change the resolved prompt for nano variants of the 8 registered corpora, so their nano verdict caches are invalidated by design (re-judge is cheap at 50 queries). Full-size caches are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ocal corpora The anchors were produced by embedders that saw mteb's TaskMetadata.prompt, so that text IS the criterion: by default the judge reads the task's own mteb prompt; a task without one falls back to the generic prompt, mirroring mteb's fallback to unprefixed encoding. Wrong or missing prompts get fixed upstream in mteb, never patched locally -- the gym stores no task text and no task lists; tasks are specified directly by mteb task name. Resolution: env override > cfg.judge_instruction > the task's mteb prompt (cfg.judge_instruction_from_task, default on; False = generic control arm) > generic. One path shared by Judge and the result record; the verdict signature hashes the resolved prompt, so any change namespaces its own caches, never silent. Unlabeled corpora via GymConfig.corpus_path (dir of .txt/.md or a .jsonl) + judge_instruction, recorded as config:judge_instruction. judge_ prefix on verdict-level reliability fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A judge model id is experiment-defining state (it enters the config hash); a silent default is how a stale 4B id survived two serving generations. MockClient paths unchanged; the unified API already required the id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oved GYM_MAX_SEQ silently capped fallback-loaded long-context models at 4096 tokens while their mteb anchors were produced at native length, and GYM_MAX_DOC_CHARS char-capped even mteb-path models when set. Both removed with nothing in their place: SentenceTransformer/mteb set max_seq_length from the model's own config, and a model whose config declares no sane limit is a model bug to fix upstream, not to cap locally. MIGRATION: embedding caches written for fallback-loaded models with native max_seq > 4096 (e.g. stella, inf-retriever) were computed under the old cap and are stale -- delete those cache entries before rerunning. Launchers exporting GYM_MAX_SEQ / GYM_MAX_DOC_CHARS now silently no-op; remove them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
spearman_top10 restricts the correlation to the ten officially best models (None under 12 anchored models): whether the ranking holds where selection decisions are made, or the aggregate rho merely separates strong from weak. kendall_ap is the AP rank correlation (Yilmaz et al. 2008), top-weighted, with the official ranking as reference. Both fill the previously-null fields in existing result records on the next rank_agreement pass, no re-judging. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mpt() No default lookup and no flag: judge_instruction is the criterion the judge applies, None is generic, and wanting the task's own mteb prompt is something the run config states via gym.task_prompt(task_name). mteb's prompt field is encoder metadata that only sometimes reads as a task definition (BRIGHT carries encoder prefixes), so the library never repurposes it silently; default behavior is byte-identical to every existing run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.