Cannot reproduce Table 1 on ALFWorld (Qwen3-1.7B): both GRPO and SDAR fall short, and SDAR ≈ GRPO

[nachewigkeit]

Thanks for the great work! I've been trying to reproduce the ALFWorld results for Qwen3-1.7B but cannot match Table 1, and I'd appreciate your guidance on whether my setup is wrong.

Setup

I ran the repo's own scripts unchanged except for path / W&B-key edits — no training hyperparameters were touched:

• SDAR: examples/sdar_trainer/run_alfworld_qwen3.sh
• GRPO: examples/grpo_trainer/run_alfworld_qwen3_1b.sh

Config (as shipped): Qwen3-1.7B, ALFWorld, single 8×A100 node, 150 steps, train_batch_size=16, env.rollout.n=8, max_prompt_length=2048, lr=1e-6, kl_loss_coef=0.01, enable_thinking=False. SDAR extras: sdar_coef=0.01, gate_beta=5.0, skill_all=false, skills_dir=skills/alfworld.

1. Both GRPO and SDAR underperform the paper, and SDAR ≈ GRPO

Using the repo's own scripts unchanged, after 150 steps neither run reaches the numbers reported in Table 1, and the two are nearly indistinguishable from each other (see attached plot — both climb to ~0.35–0.40 and stay interleaved throughout):

• GRPO: final 0.352, max 0.438 — vs the paper's 46.1
• SDAR: final 0.406, max 0.406 — vs the paper's 55.5 (+9.4)

Two problems here: (1) both GRPO and SDAR land well below the paper's scores (GRPO ~8 points under, SDAR even further), so I can't reproduce the reported numbers with the provided scripts; (2) SDAR is within run-to-run noise of GRPO (whichever looks "better" depends on which step you read), so it shows no improvement over the baseline. Are these the exact scripts/hyperparameters used for the Qwen3-1.7B column in Table 1? Any advice to reproduce the result?

2. The SDAR loss scale looks too small to matter

From the SDAR run's logged metrics:

• sdar/loss ≈ 0.041, with sdar/coef = 0.01 → effective contribution ≈ 4e-4
• meanwhile actor/pg_loss ≈ 0.01 and actor/kl_loss ≈ 0.2

So the auxiliary distillation term is 1–2 orders of magnitude smaller than the main objective. Could this scale be too small for the SDAR loss to have any real effect on training?

3. The skill-prompted "teacher" is worse than vanilla

In the paper, table 1 itself shows that for Qwen3-1.7B the Skill-Prompt model is worse than Vanilla on essentially every benchmark (e.g. ALFWorld 9.4 vs 12.5, WebShop-Score 23.0 vs 46.5). If the teacher is a worse policy than the student, how is it expected to provide a useful learning signal? Even though the gating mechanism confines distillation to teacher-endorsed positive-gap tokens, it remains doubtful whether the supervision from a weaker teacher on those positive-gap tokens is actually beneficial.

Thanks for your help!