Progress Advantage

NeurIPS 2026🏆 Best Paper Award · RLxF @ ICML 2026

Neglected Free Lunch from Post-training:
Progress Advantage for LLM Agents

A process reward model for agents, already inside your RL checkpoint.

Changdae Oh1 Wendi Li1 Seongheon Park1 Samuel Yeh1 Tanwi Mallick2 Sharon Li1

1University of Wisconsin–Madison2Argonne National Laboratory

Paper Code Talk @ Simons Talk @ Cohere BibTeX

01 / Background & motivation

Agents need
step-level feedback.
Getting it is the bottleneck.

An outcome reward tells you whether an agent failed. A process reward model (PRM) tells you where: the signal behind test-time search, runtime monitoring, and failure diagnosis.

PRMs have flourished in math reasoning. For agents, they remain largely out of reach.

1The agent acts: many steps, many observations.

agent actionenvironment feedbackIllustrative

Why PRMs for agents stall

Long horizons

Hundreds of steps per task make step labels prohibitively costly.

Irreversible actions

A sent email can’t be unsent: no rewinding for Monte Carlo rollouts.

Stochastic feedback

Users and tools reply differently every time.

Brittle transfer

Task-specific PRMs rarely generalize to new tasks or environments.

The agents that most need process rewards are precisely those for which PRMs are hardest to build.

02 / Project overviewLimitation

Stochastic environments make the exact reward unrecoverable.

 Alternative

Target the advantage instead: exact, and free from post-training.

 Application

Test-time scaling, uncertainty, and failure attribution.

03 / Limitation

Implicit rewards assume
a deterministic world.

In text reasoning, the log-ratio between an RL-trained policy and its reference is the reward. Agents face stochastic users and tools, and the identity breaks.

Remark 1

✓The value terms telescope away, so the log-ratio is exactly the reward.

r(st, at) = β log π*(at|st)πref(at|st) + V*(st) − 𝔼[V*(st+1)]

Toggle the world. Under stochastic transitions the expected next value no longer matches the realized one, so the exact reward is out of reach from the policy pair alone.

04 / Our suggestion

Stop chasing the reward.
Read off the advantage.

The advantage asks a simpler question: how much better is this action than the policy’s usual choice here? The policy pair answers it exactly, even in a stochastic world.

Proposition 1 · Progress advantage

A*(s, a) = Q*(s, a) − V*(s) = β log π*(a|s)πref(a|s)

π* RL-trained checkpoint, e.g. Qwen3.5-9Bπref its reference, e.g. Qwen3.5-9B-Base

Why it survives stochasticity

The KL-regularized optimum satisfies π*(a|s) = πref(a|s) · exp(A*(s, a)/β). Take the log. The expectation over next states lives inside Q*, so nothing has to cancel and no transition model is needed.

Same tokens, two scorers.

[user] Hi there! My name is Emma Kim, and I’d like to cancel a reservation I have. The reservation number is EHGLP3.

Step 1

Step 3

Confidence alone punishes rare-but-correct tool strings and the very policy terms that decide this task. lowhigh

Recreated from the paper’s τ²-Airline case study, where the agent correctly declines a cancellation that violates airline policy. Shading is approximate.

i

Annotation-free

Uses checkpoint pairs that post-training already leaves behind.

ii

General

Valid for KL-penalized (PPO, GRPO) and clipped (DAPO) RL alike.

iii

Domain-agnostic

Emerges from general post-training; transfers without retraining.

05 / Application

One free signal.
Three jobs.

Five agent benchmarks, four model families, zero reward-model training. Progress advantage beats confidence baselines and surpasses trained reward models.

  1. Sample 8
  2. Score each
  3. Keep the best

Sample eight trajectories, score each by its progress advantage, and keep the best one.

Success rate of the selected trajectory (%)

—

Best-of-8 sampling. Dashed line: Pass@8, the oracle ceiling.

06 / Concluding remark

Post-training already paid for it.
Every RL checkpoint carries a process reward.

No step labels. No reward-model training. Just a checkpoint pair, read the right way.
We hope it opens a scalable path to guiding and monitoring real-world agents.

Explore the code

07 / Acknowledgement

We sincerely thank Jiatong Li, Leitian Tao, Sangyun Lee, and Jiaying Fang for their faithful proofreading and professional feedback on the draft that directly affected the writing and experiment content and sparked future work ideas. This material is based upon work supported by the U.S. Department of Energy, Office of Science, under contract number DE-AC02-06CH11357. Wendi Li, Seongheon Park, Samuel Yeh, and Sharon Li are also supported in part by the AFOSR Young Investigator Program under award number FA9550-23-1-0184, the Office of Naval Research (ONR) under award number N000142612508, National Science Foundation under awards IIS-2237037 and IIS-2331669, Schmidt Sciences Foundation, Open Philanthropy (now Coefficient Giving), Alfred P. Sloan Fellowship, and gifts from Google and Amazon.