Concept Decomposition for Visual Exploration and Inspiration

TL;DR

This paper introduces a hierarchical method to decompose visual concepts into aspects using vision-language models, enabling detailed exploration, combination, and generation of new visual ideas from image sets.

Contribution

It presents a novel hierarchical tree-based approach for concept decomposition leveraging large vision-language models and regularizations to guide the process.

Findings

Enables exploration of sub-concepts within visual ideas.

Allows combining aspects to generate new visuals.

Facilitates applying aspects to new designs via natural language.

Abstract

A creative idea is often born from transforming, combining, and modifying ideas from existing visual examples capturing various concepts. However, one cannot simply copy the concept as a whole, and inspiration is achieved by examining certain aspects of the concept. Hence, it is often necessary to separate a concept into different aspects to provide new perspectives. In this paper, we propose a method to decompose a visual concept, represented as a set of images, into different visual aspects encoded in a hierarchical tree structure. We utilize large vision-language models and their rich latent space for concept decomposition and generation. Each node in the tree represents a sub-concept using a learned vector embedding injected into the latent space of a pretrained text-to-image model. We use a set of regularizations to guide the optimization of the embedding vectors encoded in the nodes to follow the hierarchical structure of the tree. Our method allows to explore and discover new concepts derived from the original one. The tree provides the possibility of endless visual sampling at each node, allowing the user to explore the hidden sub-concepts of the object of interest. The learned aspects in each node can be combined within and across trees to create new visual ideas, and can be used in natural language sentences to apply such aspects to new designs.