Towards Open-Ended Visual Scientific Discovery with Sparse Autoencoders

TL;DR

This paper explores how sparse autoencoders can facilitate open-ended scientific discovery by extracting meaningful features from foundation model representations across various scientific domains, demonstrated through ecological imagery.

Contribution

It introduces a domain-agnostic method using sparse autoencoders for uncovering unknown patterns in foundation model representations, advancing beyond targeted structure extraction.

Findings

SAEs align with semantic concepts in segmentation tasks

Surface fine-grained anatomical structures without labels

Applicable across diverse scientific data types

Abstract

Scientific archives now contain hundreds of petabytes of data across genomics, ecology, climate, and molecular biology that could reveal undiscovered patterns if systematically analyzed at scale. Large-scale, weakly-supervised datasets in language and vision have driven the development of foundation models whose internal representations encode structure (patterns, co-occurrences and statistical regularities) beyond their training objectives. Most existing methods extract structure only for pre-specified targets; they excel at confirmation but do not support open-ended discovery of unknown patterns. We ask whether sparse autoencoders (SAEs) can enable open-ended feature discovery from foundation model representations. We evaluate this question in controlled rediscovery studies, where the learned SAE features are tested for alignment with semantic concepts on a standard segmentation benchmark and compared against strong label-free alternatives on concept-alignment metrics. Applied to ecological imagery, the same procedure surfaces fine-grained anatomical structure without access to segmentation or part labels, providing a scientific case study with ground-truth validation. While our experiments focus on vision with an ecology case study, the method is domain-agnostic and applicable to models in other sciences (e.g., proteins, genomics, weather). Our results indicate that sparse decomposition provides a practical instrument for exploring what scientific foundation models have learned, an important prerequisite for moving from confirmation to genuine discovery.