Differentiable JPEG: The Devil is in the Details

TL;DR

This paper introduces a novel differentiable JPEG approach that closely mimics the standard JPEG process, enabling its integration into deep learning models and significantly improving performance over previous differentiable approximations.

Contribution

The paper proposes a new differentiable JPEG method that overcomes limitations of prior approaches, accurately modeling JPEG components and enhancing image quality in deep learning applications.

Findings

Achieves PSNR improvements of up to 9.51dB at high compression rates.

Outperforms existing differentiable JPEG methods by 3.47dB PSNR on average.

Demonstrates strong adversarial attack resilience with effective gradient approximation.

Abstract

JPEG remains one of the most widespread lossy image coding methods. However, the non-differentiable nature of JPEG restricts the application in deep learning pipelines. Several differentiable approximations of JPEG have recently been proposed to address this issue. This paper conducts a comprehensive review of existing diff. JPEG approaches and identifies critical details that have been missed by previous methods. To this end, we propose a novel diff. JPEG approach, overcoming previous limitations. Our approach is differentiable w.r.t. the input image, the JPEG quality, the quantization tables, and the color conversion parameters. We evaluate the forward and backward performance of our diff. JPEG approach against existing methods. Additionally, extensive ablations are performed to evaluate crucial design choices. Our proposed diff. JPEG resembles the (non-diff.) reference implementation best, significantly surpassing the recent-best diff. approach by $3.47$dB (PSNR) on average. For strong compression rates, we can even improve PSNR by $9.51$dB. Strong adversarial attack results are yielded by our diff. JPEG, demonstrating the effective gradient approximation. Our code is available at https://github.com/necla-ml/Diff-JPEG.