A Test for Evaluating Performance in Human-Computer Systems

TL;DR

This paper introduces a new test for evaluating performance improvements in human-computer systems using the ratio of means, demonstrating its application through analysis of existing studies and experiments with GPT-3, revealing insights into performance gains and human-computer synergy.

Contribution

The paper develops a novel statistical test for measuring performance improvements in human-computer systems and applies it to multiple scenarios, including analysis of prior studies and experiments with GPT-3.

Findings

Over half of recent studies show no performance improvement.

Maximum observed performance ratio is 1.36 (36% improvement).

GPT-3 enables a 27% speed improvement when used by human programmers.

Abstract

The Turing test for comparing computer performance to that of humans is well known, but, surprisingly, there is no widely used test for comparing how much better human-computer systems perform relative to humans alone, computers alone, or other baselines. Here, we show how to perform such a test using the ratio of means as a measure of effect size. Then we demonstrate the use of this test in three ways. First, in an analysis of 79 recently published experimental results, we find that, surprisingly, over half of the studies find a decrease in performance, the mean and median ratios of performance improvement are both approximately 1 (corresponding to no improvement at all), and the maximum ratio is 1.36 (a 36% improvement). Second, we experimentally investigate whether a higher performance improvement ratio is obtained when 100 human programmers generate software using GPT-3, a massive, state-of-the-art AI system. In this case, we find a speed improvement ratio of 1.27 (a 27% improvement). Finally, we find that 50 human non-programmers using GPT-3 can perform the task about as well as--and less expensively than--the human programmers. In this case, neither the non-programmers nor the computer would have been able to perform the task alone, so this is an example of a very strong form of human-computer synergy.