Between Progress and Potential Impact of AI: the Neglected Dimensions

TL;DR

This paper proposes a multidimensional framework for evaluating AI progress by considering task performance alongside development and deployment costs, offering a more comprehensive assessment of AI's societal impact.

Contribution

It introduces a novel framework that incorporates costs and resource use into AI progress evaluation, expanding beyond traditional performance metrics.

Findings

The framework can be collapsed into a utility metric for stakeholder value.

AI progress can be assessed by whether it expands the Pareto surface.

Case studies demonstrate the importance of neglected dimensions in AI evaluation.

Abstract

We reframe the analysis of progress in AI by incorporating into an overall framework both the task performance of a system, and the time and resource costs incurred in the development and deployment of the system. These costs include: data, expert knowledge, human oversight, software resources, computing cycles, hardware and network facilities, and (what kind of) time. These costs are distributed over the life cycle of the system, and may place differing demands on different developers and users. The multidimensional performance and cost space we present can be collapsed to a single utility metric that measures the value of the system for different stakeholders. Even without a single utility function, AI advances can be generically assessed by whether they expand the Pareto surface. We label these types of costs as neglected dimensions of AI progress, and explore them using four case studies: Alpha* (Go, Chess, and other board games), ALE (Atari games), ImageNet (Image classification) and Virtual Personal Assistants (Siri, Alexa, Cortana, and Google Assistant). This broader model of progress in AI will lead to novel ways of estimating the potential societal use and impact of an AI system, and the establishment of milestones for future progress.