The Impact of Approximation on Algorithmic Progress

TL;DR

This paper surveys the impact of approximation algorithms across computer science, highlighting their benefits, tradeoffs, and potential to enable faster solutions for complex problems, especially in AI and big data.

Contribution

It provides a comprehensive analysis of 118 key problems, quantifying the historical gains from approximation and identifying its role in advancing computational efficiency.

Findings

Approximately 20% of problems benefit from approximation.

A quarter of intractable problems have polynomial-time approximate algorithms.

Approximation increases linear-time algorithms by 23%, expanding computational possibilities.

Abstract

In nearly every discipline, scientific computations are limited by the cost and speed of computation. For example, the best-known exact algorithms for the canonical Traveling Salesman Problem would take centuries to run on an instance of size 1 million. A natural response to such limits is to try to find new algorithms or to parallelize existing ones, but many algorithms are already at their theoretically-optimal level and parallelization is often impossible or prohibitively expensive. Starting in the 1960's, computer scientists pursued another solution: allowing solutions to have a small amount of error (i.e. approximating them). In this paper, we survey 118 of the most important algorithm problems in computer science, quantifying the gains and tradeoffs from approximation that have been discovered over the history of the field. Overall, only $\approx$20\% of problems have benefited from approximation. However, those with good approximate algorithms can be dramatically faster to compute with little cost to accuracy. For example, a quarter of computationally intractable problems (e.g. those that take exponential time to compute) have polynomial time approximate algorithms. Approximation also increases the number of algorithms that can run in linear time by 23\%, opening up new computational opportunities for those working in the big data regime. This work also sheds light on what should be expected from progress in AI, where approximation is at the heart of how deep learning works.