Image Difference Captioning with Pre-training and Contrastive Learning

TL;DR

This paper introduces a novel pre-training and contrastive learning framework for Image Difference Captioning, effectively addressing fine-grained visual differences and data scarcity issues through self-supervised tasks and data expansion strategies.

Contribution

It proposes a new modeling framework with self-supervised tasks, contrastive learning, and data expansion to improve IDC performance under limited supervised data.

Findings

Significant improvements on CLEVR-Change and Birds-to-Words datasets.

Effective alignment of visual differences and text descriptions.

Enhanced performance with data expansion strategies.

Abstract

The Image Difference Captioning (IDC) task aims to describe the visual differences between two similar images with natural language. The major challenges of this task lie in two aspects: 1) fine-grained visual differences that require learning stronger vision and language association and 2) high-cost of manual annotations that leads to limited supervised data. To address these challenges, we propose a new modeling framework following the pre-training-finetuning paradigm. Specifically, we design three self-supervised tasks and contrastive learning strategies to align visual differences and text descriptions at a fine-grained level. Moreover, we propose a data expansion strategy to utilize extra cross-task supervision information, such as data for fine-grained image classification, to alleviate the limitation of available supervised IDC data. Extensive experiments on two IDC benchmark datasets, CLEVR-Change and Birds-to-Words, demonstrate the effectiveness of the proposed modeling framework. The codes and models will be released at https://github.com/yaolinli/IDC.