RL’s impact on LLM reasoning abilities beyond base models

New research challenges the prevailing assumption that Reinforcement Learning with Verifiable Rewards (RLVR) enhances the reasoning capabilities of large language models. A comprehensive study by researchers from multiple institutions reveals that while RLVR improves sampling efficiency—helping models find correct answers with fewer attempts—it actually narrows the solution space rather than expanding a model’s fundamental reasoning abilities. This distinction matters significantly for AI development strategies, as it suggests that base models already possess more reasoning potential than previously recognized.

The big picture: RLVR-trained reasoning models like OpenAI-o1 and DeepSeek-R1 don’t actually develop new reasoning capabilities but instead optimize the sampling of solutions already present in base models.

• The researchers evaluated models using pass@k metrics, which measure success rates when sampling k different solutions to a problem.
• While RL-trained models excel at small k values (e.g., pass@1), they consistently underperform compared to base models at large k values (e.g., pass@256).
• This surprising finding suggests that reinforcement learning narrows a model’s exploration to favor known high-reward paths rather than expanding its reasoning capacity.

Key revelations: Base models can already solve problems previously thought to require RL training when given sufficient opportunities to explore diverse reasoning paths.

• Manual inspection confirmed that base models contain at least one correct solution per problem across all benchmarks tested.
• All correct solutions generated by RL-trained models already exist within the base model’s output distribution, proving RLVR optimizes sampling rather than creating new reasoning abilities.
• This optimization comes at a cost: RL-trained models have reduced coverage of the solution space compared to base models.

Industry implications: Different training methods produce fundamentally different effects on model capabilities.

• The study found minimal performance differences between various RL algorithms (PPO, GRPO, Reinforce++), suggesting current RLVR approaches remain far from optimal.
• Distillation methods differ fundamentally from RLVR, as they can genuinely introduce new knowledge into models rather than merely optimizing existing capabilities.
• These findings suggest AI developers may achieve better results by sampling extensively from base models rather than focusing exclusively on reinforcement learning techniques.

Why this matters: Understanding the true impact of RLVR on language models requires rethinking AI development strategies and evaluation methods.

• The research challenges how we measure improvement in AI reasoning capabilities, suggesting pass@k metrics with varying k values provide more comprehensive insights than single-attempt evaluations.
• For AI researchers and developers, this indicates that base models may contain more untapped potential than previously recognized when properly sampled.

Between the lines: This study exposes a fundamental tension in AI development between optimization and exploration.

• While RLVR effectively improves a model’s ability to produce correct answers efficiently, it may inadvertently limit the model’s ability to discover novel reasoning approaches or solve a broader range of problems.
• The findings align with growing concerns about the “capabilities ceiling” of current AI training methods and whether alternate approaches may be needed for continued progress.