Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representations

TL;DR

This paper introduces iCSNs, a framework that uses prototype representations and interaction to learn and revise visual concepts from images with weak supervision, enhancing interpretability and human interaction.

Contribution

The paper presents a novel interactive learning framework, iCSNs, that leverages prototype representations and concept swapping for improved concept learning and interpretability.

Findings

iCSNs effectively learn concept-grounded representations.

Prototype swapping enables understanding and revision of concepts.

Experiments on ECR dataset demonstrate the approach's effectiveness.

Abstract

Learning visual concepts from raw images without strong supervision is a challenging task. In this work, we show the advantages of prototype representations for understanding and revising the latent space of neural concept learners. For this purpose, we introduce interactive Concept Swapping Networks (iCSNs), a novel framework for learning concept-grounded representations via weak supervision and implicit prototype representations. iCSNs learn to bind conceptual information to specific prototype slots by swapping the latent representations of paired images. This semantically grounded and discrete latent space facilitates human understanding and human-machine interaction. We support this claim by conducting experiments on our novel data set "Elementary Concept Reasoning" (ECR), focusing on visual concepts shared by geometric objects.