Skip to content

Commit 54419d9

Browse files
charlesmartin14charlesmartin14
authored
Add AdamW–MuonClip nanoGPT WeightWatcher campaign (#115)
Reproducible two-arm, five-seed FineWeb-Edu nanoGPT campaign with one-call WeightWatcher clip-Xmax/raw-alpha tracking, ERG diagnostics, restart safety, reporting, and Mac /tmp workflow.
1 parent 1533ff2 commit 54419d9

43 files changed

Lines changed: 11494 additions & 334 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 25 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -49,11 +49,14 @@ python -m pip install -e './baseline[experiment]'
4949
jupyter lab baseline/notebooks
5050
```
5151

52-
Long-running outputs should live under `$HOME`, not `/tmp`:
52+
Put generated data and runs beneath an explicit experiment root. The dated
53+
one-head nanoGPT campaign enforces a `/tmp` root and redirects third-party
54+
caches there as well:
5355

5456
```bash
55-
export RG_BASELINE_DATA_DIR="$HOME/rg-optimizer-data"
56-
export RG_BASELINE_RUN_ROOT="$HOME/rg-optimizer-runs"
57+
export RG_BASELINE_EXPERIMENT_ROOT="/tmp/rg-optimizer-baselines"
58+
export RG_BASELINE_DATA_DIR="$RG_BASELINE_EXPERIMENT_ROOT/data"
59+
export RG_BASELINE_RUN_ROOT="$RG_BASELINE_EXPERIMENT_ROOT/runs"
5760
```
5861

5962
### Recommended notebook order
@@ -76,16 +79,26 @@ baseline/notebooks/CIFAR10_ViT_Optimizer_Baselines.ipynb
7679
One-head nanoGPT:
7780

7881
```bash
79-
cd baseline/nanogpt_one_head
80-
bash scripts/setup_mac.sh
81-
bash scripts/prepare_data.sh
82-
bash scripts/smoke_test.sh
83-
84-
export RG_NANOGPT_ONE_HEAD_ROOT="$HOME/rg-nanogpt-one-head"
85-
caffeinate -dimsu bash scripts/run_all_baselines.sh \
86-
2>&1 | tee "$RG_NANOGPT_ONE_HEAD_ROOT/run_all.log"
82+
cd baseline/experiments/nanogpt_one_head_2026_08_21_baseline
83+
export RG_NANOGPT_EXPERIMENT_ROOT="/tmp/rg-nanogpt-one-head-20260821"
84+
mkdir -p "$RG_NANOGPT_EXPERIMENT_ROOT"/{cache/{home,pip,xdg/{cache,config,data,state},matplotlib},tmp}
85+
export HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/home"
86+
export PIP_CACHE_DIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/pip"
87+
export XDG_CACHE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/cache"
88+
export XDG_CONFIG_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/config"
89+
export XDG_DATA_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/data"
90+
export XDG_STATE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/state"
91+
export MPLCONFIGDIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/matplotlib"
92+
export TMPDIR="$RG_NANOGPT_EXPERIMENT_ROOT/tmp"
93+
python -m pip install -e ../../nanogpt_one_head
94+
python scripts/run_experiment.py doctor --device mps
95+
python scripts/run_experiment.py prepare
96+
caffeinate -dimsu python scripts/run_experiment.py run --device mps
8797
```
8898

99+
The full protocol, exact commands, reporting contract, and honest results ledger
100+
are in the [dated experiment folder](baseline/experiments/nanogpt_one_head_2026_08_21_baseline/README.md).
101+
89102
nanochat:
90103

91104
```text
@@ -200,7 +213,7 @@ The committed values above are strong source-backed centers. They become frozen
200213
best baselines only after the bounded qualification protocol:
201214

202215
1. screen the preregistered neighborhood using validation data only;
203-
2. run the finalists with the complete three-seed protocol;
216+
2. run the finalists with the complete five-seed protocol;
204217
3. select the lowest mean best-validation loss;
205218
4. write the full winning configuration and evidence to a lock file;
206219
5. inspect protected-test comparisons only after the lock exists.

‎baseline/.gitignore‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,17 @@ __pycache__/
33
.ipynb_checkpoints/
44
data/
55
runs/
6+
7+
# The dated nanoGPT campaign deliberately archives only small, reviewed run
8+
# records in the repository. Keep ordinary/raw run directories ignored while
9+
# allowing this one experiment's README and timestamped archive directories.
10+
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/
11+
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/README.md
12+
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/*/
13+
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/*/**
14+
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/*.pt
15+
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/*.bin
16+
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/*.partial*
17+
experiments/nanogpt_one_head_2026_08_21_baseline/runs/.*.partial-*/
18+
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/cache/
19+
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/logs/

‎baseline/README.md‎

Lines changed: 24 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -71,21 +71,20 @@ When installing from the repository root, use:
7171
python -m pip install -e './baseline[experiment]'
7272
```
7373

74-
Set persistent locations before running long jobs:
74+
Set one explicit experiment location before running long jobs:
7575

7676
```bash
77-
export RG_BASELINE_DATA_DIR="$HOME/rg-optimizer-data"
78-
export RG_BASELINE_RUN_ROOT="$HOME/rg-optimizer-runs"
77+
export RG_BASELINE_EXPERIMENT_ROOT="/tmp/rg-optimizer-baselines"
78+
export RG_BASELINE_DATA_DIR="$RG_BASELINE_EXPERIMENT_ROOT/data"
79+
export RG_BASELINE_RUN_ROOT="$RG_BASELINE_EXPERIMENT_ROOT/runs"
7980
```
8081

81-
The isolated one-head nanoGPT suite uses:
82+
The dated one-head nanoGPT campaign uses and validates:
8283

8384
```bash
84-
export RG_NANOGPT_ONE_HEAD_ROOT="$HOME/rg-nanogpt-one-head"
85+
export RG_NANOGPT_EXPERIMENT_ROOT="/tmp/rg-nanogpt-one-head-20260821"
8586
```
8687

87-
Do not put long-running results in `/tmp`.
88-
8988
## 1. MNIST / MLP3
9089

9190
Run these notebooks in order:
@@ -243,15 +242,26 @@ that produced that checkpoint.
243242
Mac workflow:
244243

245244
```bash
246-
cd nanogpt_one_head
247-
bash scripts/setup_mac.sh
248-
bash scripts/prepare_data.sh
249-
bash scripts/smoke_test.sh
250-
251-
caffeinate -dimsu bash scripts/run_all_baselines.sh \
252-
2>&1 | tee "$RG_NANOGPT_ONE_HEAD_ROOT/run_all.log"
245+
cd experiments/nanogpt_one_head_2026_08_21_baseline
246+
export RG_NANOGPT_EXPERIMENT_ROOT="/tmp/rg-nanogpt-one-head-20260821"
247+
mkdir -p "$RG_NANOGPT_EXPERIMENT_ROOT"/{cache/{home,pip,xdg/{cache,config,data,state},matplotlib},tmp}
248+
export HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/home"
249+
export PIP_CACHE_DIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/pip"
250+
export XDG_CACHE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/cache"
251+
export XDG_CONFIG_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/config"
252+
export XDG_DATA_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/data"
253+
export XDG_STATE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/state"
254+
export MPLCONFIGDIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/matplotlib"
255+
export TMPDIR="$RG_NANOGPT_EXPERIMENT_ROOT/tmp"
256+
python -m pip install -e ../../nanogpt_one_head
257+
python scripts/run_experiment.py doctor --device mps
258+
python scripts/run_experiment.py prepare
259+
caffeinate -dimsu python scripts/run_experiment.py run --device mps
253260
```
254261

262+
See the [dated experiment README](experiments/nanogpt_one_head_2026_08_21_baseline/README.md)
263+
for the AdamW/MuonClip five-seed protocol, reporting, and archive steps.
264+
255265
## 4. nanochat
256266

257267
Run:

0 commit comments

Comments
 (0)