Human-like Content Analysis for Generative AI with Language-Grounded Sparse Encoders

TL;DR

This paper introduces LanSE, a tool that decomposes images into interpretable visual patterns with natural language descriptions to improve analysis of AI-generated content.

Contribution

LanSE is a novel method that automatically identifies and describes visual patterns, enhancing content evaluation and extending to various data modalities.

Findings

Discovered over 5,000 visual patterns with 93% human agreement.

Outperforms existing methods in decomposed evaluation.

First systematic evaluation of physical plausibility in AI content analysis.

Abstract

The rapid development of generative AI has transformed content creation, communication, and human development. However, this technology raises profound concerns in high-stakes domains, demanding rigorous methods to analyze and evaluate AI-generated content. While existing analytic methods often treat images as indivisible wholes, real-world AI failures generally manifest as specific visual patterns that can evade holistic detection and suit more granular and decomposed analysis. Here we introduce a content analysis tool, Language-Grounded Sparse Encoders (LanSE), which decompose images into interpretable visual patterns with natural language descriptions. Utilizing interpretability modules and large multimodal models, LanSE can automatically identify visual patterns within data modalities. Our method discovers more than 5,000 visual patterns with 93\% human agreement, provides decomposed evaluation outperforming existing methods, establishes the first systematic evaluation of physical plausibility, and extends to medical imaging settings. Our method's capability to extract language-grounded patterns can be naturally adapted to numerous fields, including biology and geography, as well as other data modalities such as protein structures and time series, thereby advancing content analysis for generative AI.