SCAN: Learning Hierarchical Compositional Visual Concepts

TL;DR

SCAN introduces a framework for learning hierarchical, compositional visual concepts through unsupervised symbol association, enabling flexible inference and creative recombination of concepts with minimal supervision.

Contribution

It presents a novel approach that learns abstract, hierarchical visual concepts with few symbol-image pairings, surpassing existing models in flexibility and generative capabilities.

Findings

Enables bidirectional image and symbol generation.

Supports manipulation of visual concept hierarchies.

Allows recombination to generate novel visual concepts.

Abstract

The seemingly infinite diversity of the natural world arises from a relatively small set of coherent rules, such as the laws of physics or chemistry. We conjecture that these rules give rise to regularities that can be discovered through primarily unsupervised experiences and represented as abstract concepts. If such representations are compositional and hierarchical, they can be recombined into an exponentially large set of new concepts. This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such abstractions in the visual domain. SCAN learns concepts through fast symbol association, grounding them in disentangled visual primitives that are discovered in an unsupervised manner. Unlike state of the art multimodal generative model baselines, our approach requires very few pairings between symbols and images and makes no assumptions about the form of symbol representations. Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa. It also allows for traversal and manipulation of the implicit hierarchy of visual concepts through symbolic instructions and learnt logical recombination operations. Such manipulations enable SCAN to break away from its training data distribution and imagine novel visual concepts through symbolically instructed recombination of previously learnt concepts.