Designing Any Imaging System from Natural Language: Agent-Constrained Composition over a Finite Primitive Basis

TL;DR

This paper presents a system that automatically designs imaging setups from natural language descriptions, using autonomous agents and a structured specification format to match expert-level quality across multiple modalities.

Contribution

Introduction of spec.md and three autonomous agents that translate natural language into validated imaging system models, reducing expert effort and enabling cross-modality design.

Findings

Automated pipeline achieves 98.1% accuracy compared to experts.

Successfully designs 10 novel imaging configurations across diverse modalities.

Demonstrates compositional design capabilities beyond single-modality tools.

Abstract

Designing a computational imaging system -- selecting operators, setting parameters, validating consistency -- requires weeks of specialist effort per modality, creating an expertise bottleneck that excludes the broader scientific community from prototyping imaging instruments. We introduce spec.md, a structured specification format, and three autonomous agents -- Plan, Judge, and Execute -- that translate a one-sentence natural-language description into a validated forward model with bounded reconstruction error. A design-to-real error theorem decomposes total reconstruction error into five independently bounded terms, each linked to a corrective action. On 6 real-data modalities spanning all 5 carrier families, the automated pipeline matches expert-library quality (98.1 +/- 4.2%). Ten novel designs -- composing primitives into chains from 3D to 5D -- demonstrate compositional reach beyond any single-modality tool.