Automated alignment is harder than you think

TL;DR

Automated alignment research using AI agents faces significant challenges due to systematic errors in fuzzy tasks, risking misleading safety assessments and potential misdeployment of misaligned AI systems.

Contribution

This paper highlights the difficulties and risks of using AI agents for alignment research, emphasizing the need for reliable training methods like generalisation and scalable oversight.

Findings

AI-generated alignment outputs may contain undetected errors.

Errors are likely to be concentrated among those hard for humans to catch.

Shared training processes may cause outputs to be more correlated than human ones.

Abstract

A leading proposal for aligning artificial superintelligence (ASI) is to use AI agents to automate an increasing fraction of alignment research as capabilities improve. We argue that, even when research agents are not scheming to deliberately sabotage alignment work, this plan could produce compelling but catastrophically misleading safety assessments resulting in the unintentional deployment of misaligned AI. This could happen because alignment research involves many hard-to-supervise fuzzy tasks (tasks without clear evaluation criteria, for which human judgement is systematically flawed). Consequently, research outputs will contain systematic, undetected errors, and even correct outputs could be incorrectly aggregated into overconfident safety assessments. This problem is likely to be worse for automated alignment research than for human-generated alignment research for several reasons: 1) optimisation pressure means agent-generated mistakes are concentrated among those that human reviewers are least likely to catch; 2) agents are likely to produce errors that do not resemble human mistakes; 3) AI-generated alignment solutions may involve arguments humans cannot evaluate; and 4) shared weights, data and training processes may make AI outputs more correlated than human equivalents. Therefore, agents must be trained to reliably perform hard-to-supervise fuzzy tasks. Generalisation and scalable oversight are the leading candidates for achieving this but both face novel challenges in the context of automated alignment.