Revisiting Image Reconstruction for Semi-supervised Semantic Segmentation

TL;DR

This paper revisits the classic image reconstruction task as an auxiliary in semi-supervised semantic segmentation, demonstrating its competitiveness with modern methods and proposing modifications to enhance object-background disentanglement.

Contribution

The paper shows that traditional image reconstruction can be effectively integrated into modern semi-supervised segmentation frameworks and introduces a modification to improve object-background separation.

Findings

Reconstruction auxiliary task yields competitive results.

Visualization links feature channels to semantic concepts.

Modified reconstruction improves object-centric segmentation.

Abstract

Autoencoding, which aims to reconstruct the input images through a bottleneck latent representation, is one of the classic feature representation learning strategies. It has been shown effective as an auxiliary task for semi-supervised learning but has become less popular as more sophisticated methods have been proposed in recent years. In this paper, we revisit the idea of using image reconstruction as the auxiliary task and incorporate it with a modern semi-supervised semantic segmentation framework. Surprisingly, we discover that such an old idea in semi-supervised learning can produce results competitive with state-of-the-art semantic segmentation algorithms. By visualizing the intermediate layer activations of the image reconstruction module, we show that the feature map channel could correlate well with the semantic concept, which explains why joint training with the reconstruction task is helpful for the segmentation task. Motivated by our observation, we further proposed a modification to the image reconstruction task, aiming to further disentangle the object clue from the background patterns. From experiment evaluation on various datasets, we show that using reconstruction as auxiliary loss can lead to consistent improvements in various datasets and methods. The proposed method can further lead to significant improvement in object-centric segmentation tasks.