Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs

TL;DR

Feynman is a scalable AI agent that generates high-quality, knowledge-infused diagrams and captions, creating a large dataset and benchmark for visual reasoning in vision-language models.

Contribution

We introduce Feynman, a novel diagram generation pipeline that produces well-aligned diagrams and captions, enabling scalable dataset creation and evaluation for vision-language research.

Findings

Generated over 100k diagram-caption pairs.

Created the Diagramma benchmark for visual reasoning.

Demonstrated high-quality, diverse diagram synthesis.

Abstract

Visual design is an essential application of state-of-the-art multi-modal AI systems. Improving these systems requires high-quality vision-language data at scale. Despite the abundance of internet image and text data, knowledge-rich and well-aligned image-text pairs are rare. In this paper, we present a scalable diagram generation pipeline built with our agent, Feynman. To create diagrams, Feynman first enumerates domain-specific knowledge components (''ideas'') and performs code planning based on the ideas. Given the plan, Feynman translates ideas into simple declarative programs and iterates to receives feedback and visually refine diagrams. Finally, the declarative programs are rendered by the Penrose diagramming system. The optimization-based rendering of Penrose preserves the visual semantics while injecting fresh randomness into the layout, thereby producing diagrams with visual consistency and diversity. As a result, Feynman can author diagrams along with grounded captions with very little cost and time. Using Feynman, we synthesized a dataset with more than 100k well-aligned diagram-caption pairs. We also curate a visual-language benchmark, Diagramma, from freshly generated data. Diagramma can be used for evaluating the visual reasoning capabilities of vision-language models. We plan to release the dataset, benchmark, and the full agent pipeline as an open-source project.