CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representations

TL;DR

CoCoG introduces a novel AI framework that extracts human-interpretable concepts from visual stimuli, predicts human similarity judgments, and generates controllable visual objects to enhance understanding of human cognition.

Contribution

The paper presents the first AI model capable of extracting human concept representations, predicting human behavior, and generating visual stimuli with controllable concepts.

Findings

Achieves 64.07% accuracy in predicting human similarity judgments

Generates diverse visual objects controlled by concepts

Manipulates human similarity judgments by intervening key concepts

Abstract

A central question for cognitive science is to understand how humans process visual objects, i.e, to uncover human low-dimensional concept representation space from high-dimensional visual stimuli. Generating visual stimuli with controlling concepts is the key. However, there are currently no generative models in AI to solve this problem. Here, we present the Concept based Controllable Generation (CoCoG) framework. CoCoG consists of two components, a simple yet efficient AI agent for extracting interpretable concept and predicting human decision-making in visual similarity judgment tasks, and a conditional generation model for generating visual stimuli given the concepts. We quantify the performance of CoCoG from two aspects, the human behavior prediction accuracy and the controllable generation ability. The experiments with CoCoG indicate that 1) the reliable concept embeddings in CoCoG allows to predict human behavior with 64.07\% accuracy in the THINGS-similarity dataset; 2) CoCoG can generate diverse objects through the control of concepts; 3) CoCoG can manipulate human similarity judgment behavior by intervening key concepts. CoCoG offers visual objects with controlling concepts to advance our understanding of causality in human cognition. The code of CoCoG is available at \url{https://github.com/ncclab-sustech/CoCoG}.