Self-supervised Cross-view Representation Reconstruction for Change Captioning

TL;DR

This paper introduces SCORER, a self-supervised network that learns stable, view-invariant difference representations for change captioning by cross-view contrastive learning and cross-attention reconstruction, achieving state-of-the-art results.

Contribution

The paper proposes a novel self-supervised framework with multi-head token-wise matching and cross-view contrastive learning for stable difference representation in change captioning.

Findings

Achieves state-of-the-art results on four datasets.

Effectively learns view-invariant representations.

Improves caption quality through backward reasoning.

Abstract

Change captioning aims to describe the difference between a pair of similar images. Its key challenge is how to learn a stable difference representation under pseudo changes caused by viewpoint change. In this paper, we address this by proposing a self-supervised cross-view representation reconstruction (SCORER) network. Concretely, we first design a multi-head token-wise matching to model relationships between cross-view features from similar/dissimilar images. Then, by maximizing cross-view contrastive alignment of two similar images, SCORER learns two view-invariant image representations in a self-supervised way. Based on these, we reconstruct the representations of unchanged objects by cross-attention, thus learning a stable difference representation for caption generation. Further, we devise a cross-modal backward reasoning to improve the quality of caption. This module reversely models a ``hallucination'' representation with the caption and ``before'' representation. By pushing it closer to the ``after'' representation, we enforce the caption to be informative about the difference in a self-supervised manner. Extensive experiments show our method achieves the state-of-the-art results on four datasets. The code is available at https://github.com/tuyunbin/SCORER.