# Robust Change Captioning

## TL;DR

This paper introduces DUDA, a novel model for robust change captioning that effectively distinguishes relevant scene changes from distractors, localizes changes, and generates accurate descriptions, validated on a new dataset and existing benchmarks.

## Contribution

The paper presents a new Dual Dynamic Attention Model (DUDA) for change captioning, along with a novel CLEVR-Change dataset to evaluate robustness against distractors.

## Key findings

- DUDA outperforms baselines in change captioning and localization.
- The model achieves state-of-the-art results on Spot-the-Diff dataset.
- DUDA effectively handles distractors and various change types.

## Abstract

Describing what has changed in a scene can be useful to a user, but only if generated text focuses on what is semantically relevant. It is thus important to distinguish distractors (e.g. a viewpoint change) from relevant changes (e.g. an object has moved). We present a novel Dual Dynamic Attention Model (DUDA) to perform robust Change Captioning. Our model learns to distinguish distractors from semantic changes, localize the changes via Dual Attention over "before" and "after" images, and accurately describe them in natural language via Dynamic Speaker, by adaptively focusing on the necessary visual inputs (e.g. "before" or "after" image). To study the problem in depth, we collect a CLEVR-Change dataset, built off the CLEVR engine, with 5 types of scene changes. We benchmark a number of baselines on our dataset, and systematically study different change types and robustness to distractors. We show the superiority of our DUDA model in terms of both change captioning and localization. We also show that our approach is general, obtaining state-of-the-art results on the recent realistic Spot-the-Diff dataset which has no distractors.