OpenAI & Apollo Research Are On The Road To Solving Alignment | Introducing: ‘Stress Testing Deliberative Alignment for Anti-Scheming Training’ | "We developed a training technique that teaches AI models to not engage in ‘scheming’ — secretly pursuing undesirable goals — and studied it rigorously."

Anti Scheming Definition:

We suggest that any training intervention that targets scheming should:

1. Generalize far out of distribution 2. Be robust to evaluation awareness (models realizing when they are and are not being evaluated) 3. Be robust to pre-existing misaligned goals 

From the Paper:

"We study deliberative alignment, an alignment technique where the model learns to cite and apply a specification (“spec”) that contains descriptions of what it is and isn’t supposed to do. In our case, the spec contains rules about not taking deceptive or covert actions and how to handle conflicts when that trades off against other goals."

"We train models on a narrow set of environments to resist the incentives to secretly break rules.

We then test models on 26 evaluations to test generalization beyond the kinds of covert behavior seen in training. The test evaluations are different from training across a variety of axes, including covering a wider range of failure modes like sabotage, reward hacking and sandbagging, and putting models in different deployment settings, ranging from software engineering tasks to corporate environments and game-theoretic problems."

"We find that our anti-scheming training significantly reduces covert action rates across our test environments for both OpenAI o3 (13% → 0.4%) and OpenAI o4-mini (8.7% → 0.3%)."

The Paper

The Official Blogpost

Quick-Read Synopsis of the Findings