Towards an Improved Metric for Evaluating Disentangled Representations

TL;DR

This paper critically evaluates existing disentanglement metrics, introduces a new metric called EDI that improves measurement stability, and advocates for its adoption as a standard in disentangled representation evaluation.

Contribution

The paper provides a comprehensive comparison of current metrics and proposes EDI, a novel, more stable metric based on exclusivity and factor-code relationships.

Findings

EDI measures key properties of disentanglement effectively

EDI demonstrates greater stability than existing metrics

The analysis supports adopting EDI as a standard evaluation metric

Abstract

Disentangled representation learning plays a pivotal role in making representations controllable, interpretable and transferable. Despite its significance in the domain, the quest for reliable and consistent quantitative disentanglement metric remains a major challenge. This stems from the utilisation of diverse metrics measuring different properties and the potential bias introduced by their design. Our work undertakes a comprehensive examination of existing popular disentanglement evaluation metrics, comparing them in terms of measuring aspects of disentanglement (viz. Modularity, Compactness, and Explicitness), detecting the factor-code relationship, and describing the degree of disentanglement. We propose a new framework for quantifying disentanglement, introducing a metric entitled \emph{EDI}, that leverages the intuitive concept of \emph{exclusivity} and improved factor-code relationship to minimize ad-hoc decisions. An in-depth analysis reveals that EDI measures essential properties while offering more stability than existing metrics, advocating for its adoption as a standardised approach.