Neural Networks Optimally Compress the Sawbridge

TL;DR

This paper characterizes the optimal compression limits for a complex, high-dimensional source modeled as a one-dimensional curve in function space, demonstrating neural networks' superiority over classical methods.

Contribution

It provides a precise theoretical analysis of the entropy-distortion tradeoff for this source and shows neural networks trained with stochastic gradient descent achieve optimal compression.

Findings

Neural-network compressors achieve optimal entropy-distortion tradeoff.

Classical Karhunen-Loève transform-based compressors are suboptimal at high rates.

Numerical and analytical evidence supports neural networks' effectiveness.

Abstract

Neural-network-based compressors have proven to be remarkably effective at compressing sources, such as images, that are nominally high-dimensional but presumed to be concentrated on a low-dimensional manifold. We consider a continuous-time random process that models an extreme version of such a source, wherein the realizations fall along a one-dimensional "curve" in function space that has infinite-dimensional linear span. We precisely characterize the optimal entropy-distortion tradeoff for this source and show numerically that it is achieved by neural-network-based compressors trained via stochastic gradient descent. In contrast, we show both analytically and experimentally that compressors based on the classical Karhunen-Lo\`{e}ve transform are highly suboptimal at high rates.