Information-computation trade-offs in non-linear transforms

TL;DR

This paper investigates the trade-offs between information and computation in non-linear data transformations, introducing new methods for image compression, perceptual enhancement, and universal coding, with implications for various AI tasks.

Contribution

It introduces novel non-linear transforms for compression and denoising, analyzes their properties, and provides a universal coding method, advancing understanding of resource-performance trade-offs.

Findings

INRs and GS have distinct representational and computational trade-offs.

The textual transform improves ultra-low bitrate compression and perceptual quality.

The LZ78-based transform offers universal compression with theoretical guarantees.

Abstract

In this work, we explore the interplay between information and computation in non-linear transform-based compression for broad classes of modern information-processing tasks. We first investigate two emerging nonlinear data transformation frameworks for image compression: Implicit Neural Representations (INRs) and 2D Gaussian Splatting (GS). We analyze their representational properties, behavior under lossy compression, and convergence dynamics. Our results highlight key trade-offs between INR's compact, resolution-flexible neural field representations and GS's highly parallelizable, spatially interpretable fitting, providing insights for future hybrid and compression-aware frameworks. Next, we introduce the textual transform that enables efficient compression at ultra-low bitrate regimes and simultaneously enhances human perceptual satisfaction. When combined with the concept of denoising via lossy compression, the textual transform becomes a powerful tool for denoising tasks. Finally, we present a Lempel-Ziv (LZ78) "transform", a universal method that, when applied to any member of a broad compressor family, produces new compressors that retain the asymptotic universality guarantees of the LZ78 algorithm. Collectively, these three transforms illuminate the fundamental trade-offs between coding efficiency and computational cost. We discuss how these insights extend beyond compression to tasks such as classification, denoising, and generative AI, suggesting new pathways for using non-linear transformations to balance resource constraints and performance.