Claims Browser

All three fusion paradigms improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities

"All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities"

Use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts; and MOPD when preserving domain-specific gains matters more than surpassing teachers

"use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost"

Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one

"Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one"

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@k for large k

"Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$."

Forcing the target model to generate answers based on partial reasoning trajectories from smaller, weaker language models effectively disrupts over-confidence and encourages exploration of distinct reasoning paths

"Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths."

The proposed method consistently outperforms vanilla RLVR across multiple mathematical benchmarks, with performance gains becoming more pronounced as k scales up

"Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage."

The approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting

"Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting."

In DPO, the β parameter non-monotonically affects policy deviation at fixed learning rates, with a dead zone at small β, a peak at intermediate values, and decreasing deviation at larger β

"at a fixed learning rate the achieved policy deviation is non-monotone in $β$: it vanishes in a dead zone at small $β$, reaches a peak at an intermediate value, and decreases again for larger $β$"

The β parameter in DPO entangles two distinct roles: governing the inverse preference-noise scale and rescaling optimization dynamics, which couples this scale with the effective step size

"$β$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size"

Standard DPO loss values are not comparable across different β values, with runs having nearly identical loss curves potentially differing several-fold in KL divergence from the reference model

"standard DPO loss values are not comparable across $β$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model"

The entanglement of β's roles in DPO obscures its function, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling

"This entanglement obscures the role of $β$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling"

A centered-softplus reformulation of DPO is argmin-equivalent for β>0 while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable

"a centered-softplus reformulation that is argmin-equivalent to DPO for $β>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable"

Existing minimax, primal-dual, and fitted fixed-point estimators for marginalized importance weighting can leave residual occupancy-balance violations due to function-class approximation, regularization, or incomplete optimization

"Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations because of function-class approximation, regularization, or incomplete optimization."

Occupancy-balance violations in existing estimators are difficult to diagnose and reduce because objectives lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping

"These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping."

Isotonic Bellman calibration is a one-dimensional, model-agnostic post-processing method that reduces occupancy-balance violations while preserving ranking information in occupancy-ratio estimates

"We introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate."

Isotonic Bellman calibration achieves small calibration error and KL risk within statistical error of the best monotone correction, with guarantees for downstream target-occupancy functionals including policy-value estimation

"isotonic Bellman calibration achieves small calibration error and KL risk within statistical error of the best monotone correction, with guarantees for downstream target-occupancy functionals, including policy-value estimation."

Any fitted ratio with small calibration error performs nearly as well as the best post-processing based on its fitted values, as shown by a calibration-refinement bound

"we derive a calibration-refinement bound showing that any fitted ratio with small calibration error performs nearly as well as the best post-processing based on its fitted values."

Most of modern RL for LLMs is a systems problem balancing off-policy data, training-inference mismatch, and throughput

"Most of modern RL is a systems problem balancing a few problems — how off-policy the data is, training-inference mismatch, and throughput."