Towards Task-Compatible Compressible Representations

TL;DR

This paper investigates the limitations of multi-task learnable compression, proposes a new approach to improve representation compatibility across tasks, and demonstrates significant performance gains in image reconstruction and semantic segmentation.

Contribution

The paper introduces a novel method for creating task-compatible compressible representations that enhance multi-task learning performance and compatibility.

Findings

Improved rate-distortion performance in image reconstruction and semantic segmentation.

Enhanced base task performance with the proposed representations.

Representations become simpler and more compatible with downstream tasks.

Abstract

We identify an issue in multi-task learnable compression, in which a representation learned for one task does not positively contribute to the rate-distortion performance of a different task as much as expected, given the estimated amount of information available in it. We interpret this issue using the predictive $\mathcal{V}$-information framework. In learnable scalable coding, previous work increased the utilization of side-information for input reconstruction by also rewarding input reconstruction when learning this shared representation. We evaluate the impact of this idea in the context of input reconstruction more rigorously and extended it to other computer vision tasks. We perform experiments using representations trained for object detection on COCO 2017 and depth estimation on the Cityscapes dataset, and use them to assist in image reconstruction and semantic segmentation tasks. The results show considerable improvements in the rate-distortion performance of the assisted tasks. Moreover, using the proposed representations, the performance of the base tasks are also improved. Results suggest that the proposed method induces simpler representations that are more compatible with downstream processes.