feat(utilities): add one-time converter for legacy TL-format checkpoints - #1599
Open
LightWork666 wants to merge 2 commits into
Open
feat(utilities): add one-time converter for legacy TL-format checkpoints#1599LightWork666 wants to merge 2 commits into
LightWork666 wants to merge 2 commits into
Conversation
…ct() true inverses state_dict() emits TL-renamed keys, but load_state_dict() only matched raw native names, so a round trip silently loaded nothing and strict=True was silently downgraded to strict=False. Adds the inverse key mapping (including aliased parameters reachable via multiple attribute paths, e.g. GPT-2's split q/k/v views into c_attn) and proper missing/unexpected key accounting that raises under strict=True. Fixes TransformerLensOrg#1587
HookedTransformer checkpoints saved before TransformerBridge existed (OthelloGPT, grokking demos, ARENA content) use property-style keys (blocks.0.attn.W_Q, embed.W_E) and per-head tensor shapes that load_state_dict doesn't recognize natively. convert_tl_checkpoint converts these once into the key/tensor format TransformerBridge.boot_native(cfg).load_state_dict accepts, without teaching load_state_dict a second key convention. Validates per-head attention shapes against cfg before reshaping, since merging/splitting head dims produces a validly-shaped result for any head count, so a mismatched cfg would otherwise silently mis-group heads rather than trip a shape error. Handles GQA's leading-underscore _W_K/_W_V naming. Fixes TransformerLensOrg#1588
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1588.
Depends on #1587 landing first
This issue explicitly can't be finished until #1587 has a stable target format to convert into (see the issue's own comment), so this branch is built on top of my #1587 fix (#1598). Since GitHub can't set a cross-fork branch as a PR's base, the diff below currently includes that commit too — the only new commit here is the last one (adds
transformer_lens/utilities/tl_checkpoint_conversion.pyand its test). Once #1587 (this one or #1595, whichever lands) merges intodev-4.x, I'll rebase and this diff will narrow to just the new file. Flagging this up front rather than leaving it to be discovered in review.What this adds
Historical training-run checkpoints (OthelloGPT, grokking demos, ARENA content) were saved via
HookedTransformer.state_dict()beforeTransformerBridgeexisted, using the old property-style keys (blocks.0.attn.W_Q,embed.W_E, ...) and per-head tensor shapes.bridge.load_state_dictdoesn't recognize these natively, so these checkpoints are currently stranded now thatHookedTransformeris deprecated.convert_tl_checkpoint(state_dict, cfg)intransformer_lens/utilities/tl_checkpoint_conversion.pyconverts one of these old-format state dicts into the key/tensor formatTransformerBridge.boot_native(cfg).load_state_dictaccepts natively — flatnn.Linear-oriented Q/K/V/O weights instead of per-head 3D tensors, and modern key names (blocks.0.attn.q.weightinstead ofblocks.0.attn.W_Q). As requested in the issue, this is a standalone one-time converter, not a second key convention taught toload_state_dictitself: convert once,bridge.load_state_dict(converted), then re-save withbridge.state_dict().Implementation notes
cfgbefore reshaping, not just inferred from tensor rank. This matters because merging/splitting per-head dimensions produces a validly-shaped result for any head count —d_model == n_heads * d_headholds for any wrong factoring of it too — so a mismatchedcfgwouldn't trip a shape error downstream inload_state_dict; it would silently mis-group heads into a model that loads "successfully" but computes wrong attention. The converter checks the real per-head shape againstcfgup front instead._W_K/_W_Vname once grouped-query attention is on, since plainW_K/b_Kbecome expanding non-Parameter properties instead of the raw stored tensor).HookedTransformer's gated MLP keeps live, trainableb_in/b_outparameters. That's a mismatch between the two implementations themselves, unrelated to this converter — the converter still faithfully translates those keys since they're realHookedTransformerparameters, andload_state_dict(strict=True)is what correctly refuses them if they're ever non-zero. I documented this in the test rather than special-casing around it.normalization_type="LNPre", the param-free pre-norm variant recently added to the native bridge) since that's the issue's own named motivating example.Testing
New tests in
tests/unit/model_bridge/test_tl_checkpoint_conversion.py: strict round-trip load, forward-pass logit parity against the sourceHookedTransformer, an independent check that Q/K/V/O land in the correct per-head slots (reading back through the bridge's ownW_Q/etc. properties rather than trusting the converter's own reshape math), acfg-mismatch validation error, an unrecognized-key error, and coverage for GQA, gated MLP,attn_only, andLNPre.uv run mypy .— clean.Also added a short migration-guide section (
docs/source/content/migrating_to_v3.md) since the issue asked for the two-step convert → load → re-save flow to be documented.Per the issue:
load_state_dict(transformer_bridge.py) is completely unchanged — no TL-property key handling was added to it.