Align-Deform-Subtract: An Interventional Framework for Explaining Object Differences

TL;DR

This paper introduces Align-Deform-Subtract (ADS), a novel framework that explains differences between object images by disentangling underlying properties through semantic alignments and counterfactual interventions.

Contribution

ADS is the first interventional framework that uses semantic alignments to iteratively quantify and remove object property differences, providing interpretable explanations.

Findings

Effective in real and synthetic data

Produces disentangled error measures

Enhances understanding of object differences

Abstract

Given two object images, how can we explain their differences in terms of the underlying object properties? To address this question, we propose Align-Deform-Subtract (ADS) -- an interventional framework for explaining object differences. By leveraging semantic alignments in image-space as counterfactual interventions on the underlying object properties, ADS iteratively quantifies and removes differences in object properties. The result is a set of "disentangled" error measures which explain object differences in terms of the underlying properties. Experiments on real and synthetic data illustrate the efficacy of the framework.