Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioning

TL;DR

This paper introduces UAV Scene Change Captioning, a new task for describing semantic changes in aerial images from moving UAVs, and proposes a hierarchical dual-change learning method with a novel transformer and calibration technique.

Contribution

It presents a new UAV-SCC task, a hierarchical dual-change collaborative learning framework, and a novel transformer and calibration method for improved change captioning.

Findings

Achieves state-of-the-art performance on UAV-SCC dataset.

Constructed a new benchmark dataset for UAV scene change captioning.

Demonstrates effectiveness of the proposed hierarchical dual-change learning approach.

Abstract

This paper proposes a novel task for UAV scene understanding - UAV Scene Change Captioning (UAV-SCC) - which aims to generate natural language descriptions of semantic changes in dynamic aerial imagery captured from a movable viewpoint. Unlike traditional change captioning that mainly describes differences between image pairs captured from a fixed camera viewpoint over time, UAV scene change captioning focuses on image-pair differences resulting from both temporal and spatial scene variations dynamically captured by a moving camera. The key challenge lies in understanding viewpoint-induced scene changes from UAV image pairs that share only partially overlapping scene content due to viewpoint shifts caused by camera rotation, while effectively exploiting the relative orientation between the two images. To this end, we propose a Hierarchical Dual-Change Collaborative Learning (HDC-CL) method for UAV scene change captioning. In particular, a novel transformer, \emph{i.e.} Dynamic Adaptive Layout Transformer (DALT) is designed to adaptively model diverse spatial layouts of the image pair, where the interrelated features derived from the overlapping and non-overlapping regions are learned within the flexible and unified encoding layer. Furthermore, we propose a Hierarchical Cross-modal Orientation Consistency Calibration (HCM-OCC) method to enhance the model's sensitivity to viewpoint shift directions, enabling more accurate change captioning. To facilitate in-depth research on this task, we construct a new benchmark dataset, named UAV-SCC dataset, for UAV scene change captioning. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on this task. The dataset and code will be publicly released upon acceptance of this paper.