Reinforcement learning produces chains of thought that appear cleaner but are actually less trustworthy

cleaner-looking but less trustworthy chains of thought

https://www.cognitiverevolution.ai/rl-s-a-hell-of-a-drug-metagaming-reward-seeking-motivated-cot-reasoning-bronson-schoen-apollo/