Estimating the Fundamental Limits is Easier than Achieving the Fundamental Limits

TL;DR

This paper demonstrates that estimating fundamental data processing limits is significantly easier than designing algorithms to reach those limits, especially in finite spaces, highlighting a gap between estimation and achievement.

Contribution

It reveals that in finite spaces, estimating limits can be done with fewer samples than required to construct explicit algorithms to achieve those limits.

Findings

Estimating limits requires fewer samples than achieving them.

In finite spaces, achieving limits may need n ln n samples.

Constructing explicit algorithms is more sample-intensive than estimation.

Abstract

We show through case studies that it is easier to estimate the fundamental limits of data processing than to construct explicit algorithms to achieve those limits. Focusing on binary classification, data compression, and prediction under logarithmic loss, we show that in the finite space setting, when it is possible to construct an estimator of the limits with vanishing error with $n$ samples, it may require at least $n\ln n$ samples to construct an explicit algorithm to achieve the limits.