Crashing Waves vs. Rising Tides: Preliminary Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks

TL;DR

This paper analyzes the progression of AI automation, finding evidence for a gradual 'rising tide' increase in capabilities across many tasks, rather than abrupt 'crashing waves' surges, based on evaluations of over 3,000 labor market tasks.

Contribution

It provides empirical evidence that AI capabilities are improving steadily across a broad range of tasks, challenging the notion of sudden 'waves' of automation.

Findings

AI models currently complete tasks in 3-4 hours with ~50% success rate in 2024-Q2.

Success rates are projected to reach 80-95% by 2029 if current growth trends continue.

Little evidence was found for abrupt 'crashing waves' in AI capability growth.

Abstract

We propose that AI automation is a continuum between: (i) crashing waves where AI capabilities surge abruptly over small sets of tasks, and (ii) rising tides where the increase in AI capabilities is more continuous and broad-based. We test for these effects in preliminary evidence from an ongoing evaluation of AI capabilities across over 3,000 broad-based tasks derived from the U.S. Department of Labor O*NET categorization that are text-based and thus LLM-addressable. Based on more than 17,000 evaluations by workers from these jobs, we find little evidence of crashing waves (in contrast to recent work by METR), but substantial evidence that rising tides are the primary form of AI automation. AI performance is high and improving rapidly across a wide range of tasks. We estimate that, in 2024-Q2, AI models successfully complete tasks that take humans approximately 3-4 hours with about a 50% success rate, increasing to about 65% by 2025-Q3. If recent trends in AI capability growth persist, this pace of AI improvement implies that LLMs will be able to complete most text-related tasks with success rates of, on average, 80%-95% by 2029 at a minimally sufficient quality level. Achieving near-perfect success rates at this quality level or comparable success rates at superior quality would require several additional years. These AI capability improvements would impact the economy and labor market as organizations adopt AI, which could have a substantially longer timeline.