Context-aware Difference Distilling for Multi-change Captioning

TL;DR

This paper introduces CARD, a novel network that effectively captures and distills all changes between image pairs for multi-change captioning, outperforming existing methods.

Contribution

The paper proposes a context-aware difference distilling network that explicitly models common and difference features for improved multi-change captioning.

Findings

CARD outperforms state-of-the-art methods on three datasets.

The model effectively captures all genuine changes in image pairs.

Extensive experiments validate the superiority of the proposed approach.

Abstract

Multi-change captioning aims to describe complex and coupled changes within an image pair in natural language. Compared with single-change captioning, this task requires the model to have higher-level cognition ability to reason an arbitrary number of changes. In this paper, we propose a novel context-aware difference distilling (CARD) network to capture all genuine changes for yielding sentences. Given an image pair, CARD first decouples context features that aggregate all similar/dissimilar semantics, termed common/difference context features. Then, the consistency and independence constraints are designed to guarantee the alignment/discrepancy of common/difference context features. Further, the common context features guide the model to mine locally unchanged features, which are subtracted from the pair to distill locally difference features. Next, the difference context features augment the locally difference features to ensure that all changes are distilled. In this way, we obtain an omni-representation of all changes, which is translated into linguistic sentences by a transformer decoder. Extensive experiments on three public datasets show CARD performs favourably against state-of-the-art methods.The code is available at https://github.com/tuyunbin/CARD.