The Vendiscope: An Algorithmic Microscope For Data Collections

TL;DR

The Vendiscope is a novel algorithmic microscope that uses diversity metrics to analyze large data collections, revealing redundancies, model limitations, and memorization patterns across various scientific domains.

Contribution

It introduces the Vendiscope, a new computational tool that extends microscopy to data analysis using differentiable diversity metrics for high-resolution insights.

Findings

Identified over 200 million near-duplicate protein sequences in the protein universe.

Discovered that AlphaFold struggles with proteins contributing most to diversity.

Found that most crystals in the Materials Project are near-duplicates, affecting ML performance.

Abstract

The evolution of microscopy, beginning with its invention in the late 16th century, has continuously enhanced our ability to explore and understand the microscopic world, enabling increasingly detailed observations of structures and phenomena. In parallel, the rise of data-driven science has underscored the need for sophisticated methods to explore and understand the composition of complex data collections. This paper introduces the Vendiscope, the first algorithmic microscope designed to extend traditional microscopy to computational analysis. The Vendiscope leverages the Vendi scores -- a family of differentiable diversity metrics rooted in ecology and quantum mechanics -- and assigns weights to data points based on their contribution to the overall diversity of the collection. These weights enable high-resolution data analysis at scale. We demonstrate this across biology, materials science, and machine learning (ML). We analyzed the $250$ million protein sequences in the protein universe, discovering that over $200$ million are near-duplicates and that AlphaFold fails on proteins with Gene Ontology (GO) functions that contribute most to diversity. Applying the Vendiscope to the Materials Project database led to similar findings: more than $85\%$ of the crystals with formation energy data are near-duplicates and ML models perform poorly on materials that enhance diversity. Additionally, the Vendiscope can be used to study phenomena such as memorization in generative models. We used the Vendiscope to identify memorized training samples from $13$ different generative models and found that the best-performing ones often memorize the training samples that contribute least to diversity. Our findings demonstrate that the Vendiscope can serve as a powerful tool for data-driven science.