The Change You Want to See (Now in 3D)

TL;DR

This paper introduces a synthetic-data-trained, class-agnostic change detection model for 3D scenes that generalizes well to real-world images without fine-tuning, using a register-and-difference approach with self-supervised embeddings.

Contribution

It presents a novel change detection method that operates on RGB images without needing camera parameters or additional data, trained solely on synthetic data for real-world applicability.

Findings

Effective on real-world images without fine-tuning

Operates directly on RGB images, no extra data needed

Outperforms existing methods on a new real-world dataset

Abstract

The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene acquired from different camera positions and at different temporal instances. The open-set nature of this problem, occlusions/dis-occlusions due to the shift in viewpoint, and the lack of suitable training datasets, presents substantial challenges in devising a solution. To address this problem, we contribute a change detection model that is trained entirely on synthetic data and is class-agnostic, yet it is performant out-of-the-box on real world images without requiring fine-tuning. Our solution entails a "register and difference" approach that leverages self-supervised frozen embeddings and feature differences, which allows the model to generalise to a wide variety of scenes and domains. The model is able to operate directly on two RGB images, without requiring access to ground truth camera intrinsics, extrinsics, depth maps, point clouds, or additional before-after images. Finally, we collect and release a new evaluation dataset consisting of real-world image pairs with human-annotated differences and demonstrate the efficacy of our method. The code, datasets and pre-trained model can be found at: https://github.com/ragavsachdeva/CYWS-3D