What's in the Image? Explorable Decoding of Compressed Images

TL;DR

This paper introduces a novel deep learning-based JPEG decompression method that enables users to explore multiple plausible images consistent with a compressed file, enhancing understanding of information loss at low bit rates.

Contribution

It presents the first explorable decompression framework for JPEG images, allowing user-guided exploration of the possible original scenes from compressed data.

Findings

Enables exploration of diverse images from a single JPEG file

Provides a user-friendly interface for image exploration

Demonstrates applications in medical and forensic imaging

Abstract

The ever-growing amounts of visual contents captured on a daily basis necessitate the use of lossy compression methods in order to save storage space and transmission bandwidth. While extensive research efforts are devoted to improving compression techniques, every method inevitably discards information. Especially at low bit rates, this information often corresponds to semantically meaningful visual cues, so that decompression involves significant ambiguity. In spite of this fact, existing decompression algorithms typically produce only a single output, and do not allow the viewer to explore the set of images that map to the given compressed code. In this work we propose the first image decompression method to facilitate user-exploration of the diverse set of natural images that could have given rise to the compressed input code, thus granting users the ability to determine what could and what could not have been there in the original scene. Specifically, we develop a novel deep-network based decoder architecture for the ubiquitous JPEG standard, which allows traversing the set of decompressed images that are consistent with the compressed JPEG file. To allow for simple user interaction, we develop a graphical user interface comprising several intuitive exploration tools, including an automatic tool for examining specific solutions of interest. We exemplify our framework on graphical, medical and forensic use cases, demonstrating its wide range of potential applications.