Modeling Transformative AI Risks (MTAIR) Project -- Summary Report

TL;DR

This report presents a comprehensive model of key hypotheses, uncertainties, and debates surrounding catastrophic risks from advanced AI, aiming to improve understanding and decision-making through an integrated, software-based approach.

Contribution

It introduces a novel, integrated model of AI risk debates based on extensive literature review and expert input, with a software implementation for exploration and planning.

Findings

Mapped key hypotheses and uncertainties in AI risk debates

Developed a software tool for quantitative exploration of AI risk scenarios

Identified critical factors influencing AI safety and failure modes

Abstract

This report outlines work by the Modeling Transformative AI Risk (MTAIR) project, an attempt to map out the key hypotheses, uncertainties, and disagreements in debates about catastrophic risks from advanced AI, and the relationships between them. This builds on an earlier diagram by Ben Cottier and Rohin Shah which laid out some of the crucial disagreements ("cruxes") visually, with some explanation. Based on an extensive literature review and engagement with experts, the report explains a model of the issues involved, and the initial software-based implementation that can incorporate probability estimates or other quantitative factors to enable exploration, planning, and/or decision support. By gathering information from various debates and discussions into a single more coherent presentation, we hope to enable better discussions and debates about the issues involved. The model starts with a discussion of reasoning via analogies and general prior beliefs about artificial intelligence. Following this, it lays out a model of different paths and enabling technologies for high-level machine intelligence, and a model of how advances in the capabilities of these systems might proceed, including debates about self-improvement, discontinuous improvements, and the possibility of distributed, non-agentic high-level intelligence or slower improvements. The model also looks specifically at the question of learned optimization, and whether machine learning systems will create mesa-optimizers. The impact of different safety research on the previous sets of questions is then examined, to understand whether and how research could be useful in enabling safer systems. Finally, we discuss a model of different failure modes and loss of control or takeover scenarios.