Identifying concept libraries from language about object structure

TL;DR

This paper explores how natural language descriptions can be used to identify meaningful object parts and their relationships, revealing a tradeoff between description conciseness and lexicon size.

Contribution

It introduces a novel method that uses program libraries and machine translation tools to analyze how people describe object parts in natural language.

Findings

Discovered a fundamental tradeoff between description conciseness and lexicon size.

Identified key principles that influence which parts are favored in language.

Developed a framework for analyzing language-based object part concepts.

Abstract

Our understanding of the visual world goes beyond naming objects, encompassing our ability to parse objects into meaningful parts, attributes, and relations. In this work, we leverage natural language descriptions for a diverse set of 2K procedurally generated objects to identify the parts people use and the principles leading these parts to be favored over others. We formalize our problem as search over a space of program libraries that contain different part concepts, using tools from machine translation to evaluate how well programs expressed in each library align to human language. By combining naturalistic language at scale with structured program representations, we discover a fundamental information-theoretic tradeoff governing the part concepts people name: people favor a lexicon that allows concise descriptions of each object, while also minimizing the size of the lexicon itself.