Power-law Scaling to Assist with Key Challenges in Artificial Intelligence

TL;DR

This paper demonstrates how power-law scaling in deep learning can predict dataset size requirements, improve decision speed, and benchmark training complexity, thereby addressing key challenges in artificial intelligence.

Contribution

It introduces the application of power-law scaling to estimate dataset size, enhance rapid decision-making, and establish benchmarks in AI training complexity.

Findings

Test errors converge as a power-law to zero with database size.

Power-law exponent increases with the number of hidden layers.

Estimated test error approaches state-of-the-art for large epoch numbers.

Abstract

Power-law scaling, a central concept in critical phenomena, is found to be useful in deep learning, where optimized test errors on handwritten digit examples converge as a power-law to zero with database size. For rapid decision making with one training epoch, each example is presented only once to the trained network, the power-law exponent increased with the number of hidden layers. For the largest dataset, the obtained test error was estimated to be in the proximity of state-of-the-art algorithms for large epoch numbers. Power-law scaling assists with key challenges found in current artificial intelligence applications and facilitates an a priori dataset size estimation to achieve a desired test accuracy. It establishes a benchmark for measuring training complexity and a quantitative hierarchy of machine learning tasks and algorithms.