Training with debate where two AIs argue can mitigate reward hacking when using LLM judges for RL training

We show that training with debate, where two AIs argue against each to convince a judge, can mitigate reward hacking, potentially providing a hopeful direction for scaling up accurate training supervision for fuzzy tasks.

https://www.alignmentforum.org/posts/BB8o7b8A4Aykeksvw/debate-training-reduces-reward-hacking-in-rlaif