Opening the Black Box of Learned Image Coders

TL;DR

This paper investigates learned image coders, revealing they develop basis functions similar to traditional transforms, which enhances understanding of their internal mechanisms and guides future improvements.

Contribution

It uncovers the basis functions learned by LICs, providing interpretability and insights into their operation, bridging the gap between black-box models and traditional coding standards.

Findings

LICs learn basis functions akin to orthogonal transforms

Analysis offers insights into LICs' internal representations

Understanding aids future design of learned image codecs

Abstract

End-to-end learned lossy image coders (LICs), as opposed to hand-crafted image codecs, have shown increasing superiority in terms of the rate-distortion performance. However, they are mainly treated as black-box systems and their interpretability is not well studied. In this paper, we show that LICs learn a set of basis functions to transform input image for its compact representation in the latent space, as analogous to the orthogonal transforms used in image coding standards. Our analysis provides insights to help understand how learned image coders work and could benefit future design and development.