Upstream and Downstream AI Safety: Both on the Same River?

TL;DR

This paper explores the relationship between upstream safety, focusing on AI capabilities and risks beyond specific applications, and downstream safety, which assesses safety within operational contexts, proposing potential synergies for improved AI safety strategies.

Contribution

It analyzes the characteristics of upstream and downstream safety frameworks and discusses how integrating insights from both can enhance AI safety assessments and interventions.

Findings

Upstream safety addresses AI capabilities and risks beyond specific applications.

Downstream safety focuses on safety within operational contexts.

Potential synergies can improve overall AI safety strategies.

Abstract

Traditional safety engineering assesses systems in their context of use, e.g. the operational design domain (road layout, speed limits, weather, etc.) for self-driving vehicles (including those using AI). We refer to this as downstream safety. In contrast, work on safety of frontier AI, e.g. large language models which can be further trained for downstream tasks, typically considers factors that are beyond specific application contexts, such as the ability of the model to evade human control, or to produce harmful content, e.g. how to make bombs. We refer to this as upstream safety. We outline the characteristics of both upstream and downstream safety frameworks then explore the extent to which the broad AI safety community can benefit from synergies between these frameworks. For example, can concepts such as common mode failures from downstream safety be used to help assess the strength of AI guardrails? Further, can the understanding of the capabilities and limitations of frontier AI be used to inform downstream safety analysis, e.g. where LLMs are fine-tuned to calculate voyage plans for autonomous vessels? The paper identifies some promising avenues to explore and outlines some challenges in achieving synergy, or a confluence, between upstream and downstream safety frameworks.