Toward Idealized Decision Theory

TL;DR

This paper examines the limitations of current decision theories in aligning advanced AI systems with human interests, proposing new conceptual directions involving policy selection and logical counterfactuals for future research.

Contribution

It identifies shortcomings in standard decision theories and explores innovative concepts like policy selection and logical counterfactuals to guide the development of idealized decision procedures.

Findings

Standard decision theories are insufficient for AI alignment.

Policy selection offers a promising approach for decision-making.

Logical counterfactuals provide new insights into decision processes.

Abstract

This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.