Counterfactuals uncover the modular structure of deep generative models

TL;DR

This paper introduces a counterfactual-based framework to uncover modular structures in deep generative models, enabling targeted data manipulations without relying on statistical independence assumptions.

Contribution

It proposes a novel non-statistical approach using counterfactuals to identify disentangled modules within generative models, improving interpretability and controllability.

Findings

Modules enable targeted interventions in generated images

Framework improves style transfer efficiency

Assists in assessing robustness to contextual changes

Abstract

Deep generative models can emulate the perceptual properties of complex image datasets, providing a latent representation of the data. However, manipulating such representation to perform meaningful and controllable transformations in the data space remains challenging without some form of supervision. While previous work has focused on exploiting statistical independence to disentangle latent factors, we argue that such requirement is too restrictive and propose instead a non-statistical framework that relies on counterfactual manipulations to uncover a modular structure of the network composed of disentangled groups of internal variables. Experiments with a variety of generative models trained on complex image datasets show the obtained modules can be used to design targeted interventions. This opens the way to applications such as computationally efficient style transfer and the automated assessment of robustness to contextual changes in pattern recognition systems.