The Bias-Expressivity Trade-off

TL;DR

This paper explores the inherent trade-off between bias and expressivity in learning algorithms, showing how increased bias improves performance but reduces flexibility, with theoretical bounds illustrating these limitations.

Contribution

It introduces an information-theoretic measure of expressivity and derives bounds linking bias and flexibility in algorithms.

Findings

Bias enhances performance over random sampling.

Higher bias reduces algorithm flexibility.

Theoretical bounds quantify the bias-expressivity trade-off.

Abstract

Learning algorithms need bias to generalize and perform better than random guessing. We examine the flexibility (expressivity) of biased algorithms. An expressive algorithm can adapt to changing training data, altering its outcome based on changes in its input. We measure expressivity by using an information-theoretic notion of entropy on algorithm outcome distributions, demonstrating a trade-off between bias and expressivity. To the degree an algorithm is biased is the degree to which it can outperform uniform random sampling, but is also the degree to which is becomes inflexible. We derive bounds relating bias to expressivity, proving the necessary trade-offs inherent in trying to create strongly performing yet flexible algorithms.