Teacher repair lifts distilled agents 12.8 success points over the best rival method in all 12 cells, with no task reward (NVIDIA-affiliated preprint)
R2S fixes the scorer, not the student's reach: Qwen3.5-2B hit 77.9% on ALFWorld vs 63.6% for the best rival. Single training run per cell; code and 24 checkpoints are public.
The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks.
Comprehensive take
If your distilled agent stalls instead of finishing, test whether the teacher can finish from the student's states before adding curricula. If it can, repairing the scorer is the lever this paper supports. The evidence is one training run per cell on Qwen models in three replayable text environments, so rerun it on your stack before committing.
Use cases
Teams distilling a large agent into a small one
Diagnose first: run teacher takeovers from the student's highest-disagreement states. If they succeed, the paper's results point to scorer myopia rather than compounding error.
Labs without task rewards
R2S selects takeovers with reward-free checks such as consensus or an LLM judge, so repair and training need no environment reward.
Researchers replicating agentic OPD
Code (github.com/Tree-Shu-Zhao/miles-agent, tag r2s) and 24 checkpoints on Hugging Face allow the twelve-cell comparison to be re-run.
Specs
| Teachers and students | Qwen3.5-9B to Qwen3.5-2B/4B; Qwen3-30B-A3B-Instruct-2507 to Qwen3-4B/8B source |
|---|---|
| Environments | ALFWorld, WebShop, ScienceWorld source |
| Headline result | R2S best in all 12 cells; +12.8 success points on average over best rival per cell source |
| Qwen3.5-2B ALFWorld | R2S 77.9 vs Guided-OPD 63.6 vs vanilla OPD 58.0 (teacher alone 63.1) source |
| Repair cost | 19,200 takeover episodes plus one teacher fine-tune before 150-step distillation source |
| Code and checkpoints | github.com/Tree-Shu-Zhao/miles-agent (tag r2s); 24 checkpoints on Hugging Face source |
Insights
- Axion reading: the finding that a student can outscore its teacher (81.9 vs 63.1 on ALFWorld) suggests the repaired scorer carries information the teacher's actions alone do not.
- Axion caution: one 150-step run per cell, with spread measured over decodes of the final checkpoint, understates training-run variance.
- Axion note: the authors are NVIDIA-affiliated but test only Qwen models; the method requires replayable environments.
The claim
On-policy distillation (OPD) trains a small student agent on its own trajectories while a stronger teacher scores every token it writes. The standard diagnosis when it underperforms is compounding error: the student wanders into states where the teacher's scores are unreliable. Recent methods respond by keeping training inside the teacher's comfort zone, with shorter episodes, teacher-handled turns, replayed prefixes or interventions where the two disagree.
This preprint, from two NVIDIA-affiliated authors, argues that diagnosis is incomplete. They take the states where an untrained student disagrees most with its teacher, late in its episodes, which compounding error predicts should be the worst places. They let the teacher take over eight times from each. The takeovers still finish the task more often than the student did, in all three environments. On ScienceWorld the student's own success from those states was 18.6 percent; the teacher's takeovers reached 27.5 percent and rescued 51.7 percent of lost states. So the teacher can fix these states; its scores just do not say so.
Why scores fail: a myopic scorer
The authors blame next-token scoring itself. A teacher rates each token without seeing where the episode ends, so it lets through safe, delaying moves such as hedging and searching that an outcome reward would punish. Once a finishing takeover is placed in the teacher's context, its probability for the student's hedging and searching words falls two to three times as much as for the rest of the student's thinking. A student trained on the plain scores still runs out of turns in about a third of held-out ALFWorld and ScienceWorld episodes and more than half of WebShop ones, roughly twice as often as with the repaired teacher.
The method: R2S
Rollout-to-Scorer Distillation (R2S) repairs the teacher before distillation. It mines high-disagreement states from the untrained student, runs teacher takeovers from them, keeps states where enough takeovers pass a reward-free check (consensus or an LLM judge), and fine-tunes the teacher to produce its takeover-informed scores on every assistant token from the divergence turn on, without the takeover visible. The frozen repaired teacher then serves as the only scorer in ordinary OPD. No task reward is used in repair or training.
The results
The paper compares R2S with vanilla OPD and three recent methods (TCOD-F2B, Guided-OPD, FutureBridge) across ALFWorld, WebShop and ScienceWorld, with Qwen3.5-9B teaching Qwen3.5-2B and 4B, and Qwen3-30B-A3B-Instruct-2507 teaching Qwen3-4B and 8B. R2S wins all twelve teacher-student-environment cells, by 12.8 success points on average over the best competitor in each cell. Examples from the main table: Qwen3.5-2B on ALFWorld reached 77.9 percent against 63.6 for the best rival (Guided-OPD) and 58.0 for vanilla OPD; the 4B reached 81.9. On WebShop the 2B student scored 57.9 against 35.7 for the best rival. The 4B student on ALFWorld (81.9) also exceeded the 9B teacher acting alone (63.1), a reminder that the teacher is a scorer here, not an upper bound.
What to check before you rely on it
Each cell is one 150-step training run, with the reported spread coming from three decodes of that final checkpoint, not from repeated training runs. All evidence is from text-agent benchmarks and Qwen models, on environments that can be replayed to a given state, which R2S requires. The compute bill is real: 19,200 takeover episodes plus a teacher fine-tune before distillation starts, and some rivals spend theirs inside training instead. The authors work at NVIDIA, but the models are Qwen's and the code is released (github.com/Tree-Shu-Zhao/miles-agent, with 24 checkpoints on Hugging Face), so the numbers can be rerun.
Should you adopt it
If you distil agents from a stronger teacher and see students that stall or exhaust their turn budget, check whether the teacher's takeovers from those states succeed. If they do, the problem is the scorer, not the student's reach, and a repair pass like this is a cheaper fix than another curriculum. If your environment cannot be replayed to arbitrary states, or your agents are not text-based, wait for replication.