Boxhead: A Dataset for Learning Hierarchical Representations

TL;DR

This paper introduces Boxhead, a new dataset with hierarchical generative factors to evaluate disentanglement models, revealing that hierarchical models outperform single-layer VAEs in such settings.

Contribution

The paper presents Boxhead, a novel hierarchical dataset for disentanglement evaluation, and demonstrates the effectiveness of hierarchical models over single-layer VAEs.

Findings

Hierarchical models outperform single-layer VAEs in disentangling hierarchical factors.

Boxhead dataset enables evaluation of disentanglement in hierarchical data.

Current methods are mainly tested on toy datasets with independent factors.

Abstract

Disentanglement is hypothesized to be beneficial towards a number of downstream tasks. However, a common assumption in learning disentangled representations is that the data generative factors are statistically independent. As current methods are almost solely evaluated on toy datasets where this ideal assumption holds, we investigate their performance in hierarchical settings, a relevant feature of real-world data. In this work, we introduce Boxhead, a dataset with hierarchically structured ground-truth generative factors. We use this novel dataset to evaluate the performance of state-of-the-art autoencoder-based disentanglement models and observe that hierarchical models generally outperform single-layer VAEs in terms of disentanglement of hierarchically arranged factors.