Skip to content

feat(utilities): add one-time converter for legacy TL-format checkpoints - #1599

Open
LightWork666 wants to merge 2 commits into
TransformerLensOrg:dev-4.xfrom
LightWork666:fix/tl-checkpoint-conversion
Open

feat(utilities): add one-time converter for legacy TL-format checkpoints#1599
LightWork666 wants to merge 2 commits into
TransformerLensOrg:dev-4.xfrom
LightWork666:fix/tl-checkpoint-conversion

Conversation

@LightWork666

Copy link
Copy Markdown

Fixes #1588.

Depends on #1587 landing first

This issue explicitly can't be finished until #1587 has a stable target format to convert into (see the issue's own comment), so this branch is built on top of my #1587 fix (#1598). Since GitHub can't set a cross-fork branch as a PR's base, the diff below currently includes that commit too — the only new commit here is the last one (adds transformer_lens/utilities/tl_checkpoint_conversion.py and its test). Once #1587 (this one or #1595, whichever lands) merges into dev-4.x, I'll rebase and this diff will narrow to just the new file. Flagging this up front rather than leaving it to be discovered in review.

What this adds

Historical training-run checkpoints (OthelloGPT, grokking demos, ARENA content) were saved via HookedTransformer.state_dict() before TransformerBridge existed, using the old property-style keys (blocks.0.attn.W_Q, embed.W_E, ...) and per-head tensor shapes. bridge.load_state_dict doesn't recognize these natively, so these checkpoints are currently stranded now that HookedTransformer is deprecated.

convert_tl_checkpoint(state_dict, cfg) in transformer_lens/utilities/tl_checkpoint_conversion.py converts one of these old-format state dicts into the key/tensor format TransformerBridge.boot_native(cfg).load_state_dict accepts natively — flat nn.Linear-oriented Q/K/V/O weights instead of per-head 3D tensors, and modern key names (blocks.0.attn.q.weight instead of blocks.0.attn.W_Q). As requested in the issue, this is a standalone one-time converter, not a second key convention taught to load_state_dict itself: convert once, bridge.load_state_dict(converted), then re-save with bridge.state_dict().

Implementation notes

  • Per-head reshapes for Q/K/V/O are validated against cfg before reshaping, not just inferred from tensor rank. This matters because merging/splitting per-head dimensions produces a validly-shaped result for any head count — d_model == n_heads * d_head holds for any wrong factoring of it too — so a mismatched cfg wouldn't trip a shape error downstream in load_state_dict; it would silently mis-group heads into a model that loads "successfully" but computes wrong attention. The converter checks the real per-head shape against cfg up front instead.
  • Handles GQA (HookedTransformer stores K/V under a leading-underscore _W_K/_W_V name once grouped-query attention is on, since plain W_K/b_K become expanding non-Parameter properties instead of the raw stored tensor).
  • I found one genuine pre-existing gap while testing: the native bridge's gated MLP has no bias parameter at all for gate/in/out (matching how real gated-MLP HF architectures like Llama are built), while HookedTransformer's gated MLP keeps live, trainable b_in/b_out parameters. That's a mismatch between the two implementations themselves, unrelated to this converter — the converter still faithfully translates those keys since they're real HookedTransformer parameters, and load_state_dict(strict=True) is what correctly refuses them if they're ever non-zero. I documented this in the test rather than special-casing around it.
  • Verified against OthelloGPT's actual config (normalization_type="LNPre", the param-free pre-norm variant recently added to the native bridge) since that's the issue's own named motivating example.

Testing

New tests in tests/unit/model_bridge/test_tl_checkpoint_conversion.py: strict round-trip load, forward-pass logit parity against the source HookedTransformer, an independent check that Q/K/V/O land in the correct per-head slots (reading back through the bridge's own W_Q/etc. properties rather than trusting the converter's own reshape math), a cfg-mismatch validation error, an unrecognized-key error, and coverage for GQA, gated MLP, attn_only, and LNPre.

uv run mypy . — clean.

Also added a short migration-guide section (docs/source/content/migrating_to_v3.md) since the issue asked for the two-step convert → load → re-save flow to be documented.

Per the issue: load_state_dict (transformer_bridge.py) is completely unchanged — no TL-property key handling was added to it.

…ct() true inverses

state_dict() emits TL-renamed keys, but load_state_dict() only matched
raw native names, so a round trip silently loaded nothing and
strict=True was silently downgraded to strict=False. Adds the inverse
key mapping (including aliased parameters reachable via multiple
attribute paths, e.g. GPT-2's split q/k/v views into c_attn) and
proper missing/unexpected key accounting that raises under strict=True.

Fixes TransformerLensOrg#1587
HookedTransformer checkpoints saved before TransformerBridge existed
(OthelloGPT, grokking demos, ARENA content) use property-style keys
(blocks.0.attn.W_Q, embed.W_E) and per-head tensor shapes that
load_state_dict doesn't recognize natively. convert_tl_checkpoint
converts these once into the key/tensor format
TransformerBridge.boot_native(cfg).load_state_dict accepts, without
teaching load_state_dict a second key convention.

Validates per-head attention shapes against cfg before reshaping,
since merging/splitting head dims produces a validly-shaped result
for any head count, so a mismatched cfg would otherwise silently
mis-group heads rather than trip a shape error. Handles GQA's
leading-underscore _W_K/_W_V naming.

Fixes TransformerLensOrg#1588
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant