Claims Browser

Value generalisation failures are not weird edge cases, but the typical outcomes for powerful AIs with optimisation goals.

"value generalisation failures are not weird edge cases, but the typical outcomes for powerful AIs with optimisation goals."

Any decomposed alignment system that uses a powerful AI and is deployed extensively in the real world will either cease being decomposable or fail to solve key questions.

"Claim : any decomposed alignment system that uses a powerful AI and is deployed extensively in the real world will either a) cease being decomposable, or b) fail to solve key questions in the real world."

Value generalisation will be crucial for AI alignment.

"This skill will be crucial for AI alignment"

Absent generalisation, motivationally aligned AIs are not safe.

"Claim : absent generalisation, motivationally aligned AIs are not safe."

The alignment problem cannot be decomposed into sub-problems that are easier to deal with, absent generalisation.

"Claim : non-decomposability of AI alignment. In practice, and absent generalisation, the alignment problem cannot be decomposed into sub-problems that are easier to deal with."

This incident was far more severe than expected and feels more than 50% of the way to full-blown AI takeover.

"This incident was far more severe than I expected, and far more severe than previous publicly documented misalignment incidents, both in terms of how concerning the agents’ motives were and the feats they achieved in pursuit of those motives. Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover."

We should expect bigger swarms of AI agents in the future.

"We should expect bigger swarms in the future."

Future AI cases will include situations where AIs think they need additional capabilities, and the same kind of behavior will recur.

"There will be plenty of future cases that work the other way, and also cases where AIs think they may or will need additional things, and so on."

There may not be another warning shot before it's too late.

"I am not sure that we will get another warning shot before it’s too late."

When faced with an impossible task and no penalty for trying things, AI agents will try almost anything.

"Once again, the pattern: If the AI faces an otherwise impossible task, and no penalty for trying things, they’re going to try almost anything."

Even criminals with very limited skills will be able to target victims at every scale due to AI

"Even criminals with very limited skills will be able to target victims at every scale"

Chain-of-thought monitoring may become less useful as reasoning traces become enormous, compressed, and harder for humans or other models to audit

"The stakes are whether chain-of-thought monitoring can remain useful as reasoning traces become enormous, compressed, and harder for humans or other models to audit"

Future RL training will incorporate current-generation AI systems to provide reward signals for next-generation AI

"This is our current best-guess model of future AI development, and so the role of debate is to ensure that the LLM judgements used during training provide as accurate a reward signal as possible."

LLM judgments will become an increasingly important aspect of frontier RL training, especially for long agentic trajectories

"The trend of increasing test time compute will likely only exacerbate this problem. Even for tasks like coding, there are many aspects of desirable LLM agent behavior that are fuzzy, and so LLM judgements are likely to become an increasingly important aspect of frontier RL training."

Geoffrey Irving expects full-blown superintelligence in roughly two to three years

"Geoffrey — formerly a safety researcher at OpenAI and Google DeepMind and chief scientist at the UK AI Security Institute — expects full-blown superintelligence in roughly two to three years."

Agents with strong drive to help other agents from subagent training may comply with misaligned peer requests over alignment objectives

"But a sufficiently strong drive to help other agents, instilled by subagent training, might just outweigh the drive to be aligned."

RLVR training with automatic verifiers can lead to LLMs doing anything, including ruthless power-seeking instrumental convergence, if it increases probability of satisfying the checker

"This can lead to the LLM doing anything, including ruthless power-seeking instrumental convergence stuff, if it leads to a higher probability of satisfying the automatic checker."

With low trust between AI firms, every equilibrium races to ruin with probability one; avoiding catastrophe requires high trust between firms

"With low trust, every equilibrium races to ruin: the disaster arrives with probability one. With intermediate trust, immediate stopping and racing to ruin are both equilibria. With high trust, in every equilibrium, the probability that two rational firms race forever vanishes quadratically in the prior odds ratio of rationality"

Astra will be made generally available soon, though safety work requires a bit more time