Lost in Latent Space: Disentangled Models and the Challenge of Combinatorial Generalisation

TL;DR

This paper investigates why disentangled generative models struggle with combinatorial generalisation, revealing that encoder failures are a key issue and emphasizing the need for models to understand the generative process.

Contribution

It provides empirical evidence that encoder failures cause poor generalisation in disentangled models and highlights the importance of inverting the generative process for better generalisation.

Findings

Encoder failures lead to poor generalisation.

Successful models often rely on overlapping test conditions.

Understanding the generative process is crucial for generalisation.

Abstract

Recent research has shown that generative models with highly disentangled representations fail to generalise to unseen combination of generative factor values. These findings contradict earlier research which showed improved performance in out-of-training distribution settings when compared to entangled representations. Additionally, it is not clear if the reported failures are due to (a) encoders failing to map novel combinations to the proper regions of the latent space or (b) novel combinations being mapped correctly but the decoder/downstream process is unable to render the correct output for the unseen combinations. We investigate these alternatives by testing several models on a range of datasets and training settings. We find that (i) when models fail, their encoders also fail to map unseen combinations to correct regions of the latent space and (ii) when models succeed, it is either because the test conditions do not exclude enough examples, or because excluded generative factors determine independent parts of the output image. Based on these results, we argue that to generalise properly, models not only need to capture factors of variation, but also understand how to invert the generative process that was used to generate the data.