S2CFormer: Revisiting the RD-Latency Trade-off in Transformer-based Learned Image Compression

TL;DR

This paper introduces S2CFormer, a new architecture for learned image compression that improves rate-distortion performance while significantly reducing decoding latency by focusing on efficient channel aggregation instead of complex spatial operations.

Contribution

The paper proposes the S2CFormer paradigm, emphasizing channel aggregation over spatial complexity, and introduces variants that achieve state-of-the-art results with faster decoding speeds.

Findings

S2C-Conv and S2C-Attention outperform previous methods in R-D performance.

S2C-Hybrid achieves a better performance-latency trade-off.

Models set new benchmarks on multiple datasets.

Abstract

Transformer-based Learned Image Compression (LIC) suffers from a suboptimal trade-off between decoding latency and rate-distortion (R-D) performance. Moreover, the critical role of the FeedForward Network (FFN)-based channel aggregation module has been largely overlooked. Our research reveals that efficient channel aggregation-rather than complex and time-consuming spatial operations-is the key to achieving competitive LIC models. Based on this insight, we initiate the ``S2CFormer'' paradigm, a general architecture that simplifies spatial operations and enhances channel operations to overcome the previous trade-off. We present two instances of the S2CFormer: S2C-Conv, and S2C-Attention. Both models demonstrate state-of-the-art (SOTA) R-D performance and significantly faster decoding speed. Furthermore, we introduce S2C-Hybrid, an enhanced variant that maximizes the strengths of different S2CFormer instances to achieve a better performance-latency trade-off. This model outperforms all the existing methods on the Kodak, Tecnick, and CLIC Professional Validation datasets, setting a new benchmark for efficient and high-performance LIC. The code is at \href{https://github.com/YunuoChen/S2CFormer}{https://github.com/YunuoChen/S2CFormer}.