docs: add boot_native train-from-scratch recipe to migration guide - #1600
Conversation
jlarson4
left a comment
jlarson4
left a comment
There was a problem hiding this comment.
Hey @priyanka25aug, great work on this! Just a couple small documentation items to update before I merge this. The example itself is well built, thank you for your hard work on this
| import. `cfg.seed` seeds the weight initialiser; omitting it lets the | ||
| global RNG advance normally. Passing a `HookedTransformerConfig` (or any | ||
| other legacy config object) to `boot_native` raises `TypeError` — construct | ||
| a `TransformerBridgeConfig` directly. To re-randomise weights after |
There was a problem hiding this comment.
Under this recipe's own seed=42, init_weights() regenerates identical weights. 0 of 38 parameters changed when I ran it because it rebuilds its generator from cfg.seed. Can you please match the table row's "reinitialize in place" wording (migrating_to_v3.md line 240) and note that re-randomising needs a changed or cleared seed?
There was a problem hiding this comment.
Fixed in 870ccb9 — switched to "reinitialize in place" wording and added a note that init_weights() rebuilds from cfg.seed when set, producing identical weights. The note now says to clear or change cfg.seed first for a genuinely fresh draw.
| ``` | ||
|
|
||
| `boot_native` makes no HuggingFace Hub call and requires no `transformers` | ||
| import. `cfg.seed` seeds the weight initialiser; omitting it lets the |
There was a problem hiding this comment.
HT's init_weights calls torch.manual_seed(cfg.seed) globally, while boot_native forks the RNG, so a training loop migrated verbatim silently loses reproducible data shuffling. Could this note state that difference explicitly?
There was a problem hiding this comment.
Also fixed in 870ccb9 — added an explicit note that boot_native forks the RNG (unlike HookedTransformer.init_weights() which calls torch.manual_seed(cfg.seed) globally), so training loops ported verbatim will silently lose reproducible data shuffling.
…ence Per jlarson4's feedback: - Match 'reinitialize in place' wording from table row - Note that init_weights() with cfg.seed set reproduces identical weights - Note that boot_native forks the RNG (unlike HookedTransformer which calls torch.manual_seed globally), so ported training loops silently lose reproducible data shuffling
|
Thanks for taking care of those changes! Merged now |
2 participants
Closes #1557
The
migrating_to_v3.mdondev-4.xalready has 13 API migration table rows and 4 runnable prose recipes, butboot_native/TransformerBridgeConfigonly appeared as a table row with no runnable before/after snippet.This PR adds the missing
### Build a TL-native model from scratchrecipe, showing the direct replacement forHookedTransformerConfig + HookedTransformer(cfg). Notes included: no HF Hub call,cfg.seedreproducibility,TypeErroron legacy config objects, andbridge.init_weights()for re-randomisation.Base branch:
dev-4.x(targets the v4 preview line where the rest of the migration API section lives).