Broadness and persistence

As AI systems are deployed in more varied and consequential settings, good behavior cannot be limited to situations represented in a training set. It needs to be broad: generalizing across unfamiliar tasks and domains. It also needs to be persistent: remaining robust when later prompts or fine-tuning push the model toward harmful behavior.

Beneficial RL

Rather than treating alignment only as preventing a model from doing something bad, beneficial RL also reinforces behaviors that actively support good outcomes. Here, beneficial refers to concrete traits that plausibly help across many settings: truthfulness, epistemic humility, metacognitive transparency, corrigibility, risk sensitivity, universal fairness, and concern for human welfare.

These traits are not presented as a final answer to which values AI should embody. They are an empirically tractable starting point for studying whether reinforcing positive behavior can improve alignment more broadly.

Main results

We trained models with reinforcement learning on realistic conversations that exercise these traits across health, science, education, law, engineering, and other domains. The improvements transferred to tasks and domains outside the training distribution. The trained models were also harder to steer toward harmful behavior through adversarial prompting or fine-tuning.

For the full experimental setup, evaluation results, and figures, see the OpenAI Alignment blog post and the paper on arXiv.