SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Art

TL;DR

SciTextures introduces a comprehensive dataset of scientific and artistic visual patterns, models, and code, enabling evaluation of AI's ability to understand and generate complex visual phenomena across disciplines.

Contribution

The paper presents the SciTextures dataset, an extensive collection linking visual patterns with their generative models, and demonstrates AI's capacity to interpret and simulate these patterns.

Findings

VLMs can link visual patterns to underlying models.

AI can infer and recreate mechanisms behind real-world phenomena.

The dataset enables systematic evaluation of visual understanding.

Abstract

The ability to connect visual patterns with the processes that form them represents one of the deepest forms of visual understanding. Textures of clouds and waves, the growth of cities and forests, or the formation of materials and landscapes are all examples of patterns emerging from underlying mechanisms. We present the SciTextures dataset, a large-scale collection of textures and visual patterns from all domains of science, tech, and art, along with the models and code that generate these images. Covering over 1,270 different models and 100,000 images of patterns and textures from physics, chemistry, biology, sociology, technology, mathematics, and art, this dataset offers a way to explore the deep connection between the visual patterns that shape our world and the mechanisms that produce them. Built through an agentic AI pipeline that autonomously collects, implements, and standardizes scientific and generative models. This AI pipeline is also used to autonomously invent and implement novel methods for generating visual patterns and textures. SciTextures enables systematic evaluation of vision language models (VLM's) ability to link visual patterns to the models and code that generate them, and to identify different patterns that emerge from the same underlying process. We also test VLMs ability to infer and recreate the mechanisms behind visual patterns by providing a natural image of a real-world phenomenon and asking the AI to identify and code a model of the process that formed it, then run this code to generate a simulated image that is compared to the reference image. These benchmarks reveal that VLM's can understand and simulate physical systems beyond visual patterns at multiple levels of abstraction. The dataset and code are available at: https://zenodo.org/records/17485502