Causal Analysis of Agent Behavior for AI Safety

TL;DR

This paper presents a methodology for analyzing the causal mechanisms behind AI agent behavior, emphasizing the importance of experiments with manipulations to obtain human-understandable explanations for safe deployment.

Contribution

It introduces a systematic approach for causal analysis of agent behavior, covering six use cases and highlighting the necessity of experimental manipulations over observation alone.

Findings

Pure observation is insufficient for causal understanding.

Systematic experiments with manipulations are essential.

The methodology applies to diverse agent questions.

Abstract

As machine learning systems become more powerful they also become increasingly unpredictable and opaque. Yet, finding human-understandable explanations of how they work is essential for their safe deployment. This technical report illustrates a methodology for investigating the causal mechanisms that drive the behaviour of artificial agents. Six use cases are covered, each addressing a typical question an analyst might ask about an agent. In particular, we show that each question cannot be addressed by pure observation alone, but instead requires conducting experiments with systematically chosen manipulations so as to generate the correct causal evidence.