SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations

TL;DR

SEMASIA is a comprehensive dataset of latent representations from around 1,700 pretrained vision models, enabling analysis, benchmarking, and understanding of semantic structures across diverse models and datasets.

Contribution

It introduces SEMASIA, a large-scale, metadata-rich benchmark dataset for analyzing and aligning latent spaces of diverse pretrained vision models.

Findings

Consistent semantic clustering across models and datasets.

Benchmarking of alignment methods using reconstruction error.

Regression analysis linking training data complexity and model scale to embedding properties.

Abstract

Latent representations learned by neural networks often exhibit semantic structure, where concept similarity is reflected by geometric proximity in embedding space. However, comparing such spaces across models remains difficult: changes in architecture, pretraining data, objective, or random seed can yield embeddings with similar content but incompatible geometry. This latent space alignment problem is central to interpretability, transfer and multimodal learning, federated systems, and semantic communication; however, progress remains limited by the lack of large-scale, model-diverse, and metadata-rich benchmarks. To address this gap, we introduce SEMASIA, a large-scale collection of latent representations extracted from approximately 1,700 pretrained vision models across eight standard image-classification benchmarks. SEMASIA pairs embeddings with structured metadata describing architectures, training regimes, pretraining sources, and model scale. We demonstrate three applications of the resource. First, we analyze the conceptual organization of individual latent spaces, showing consistent prototype-like clustering and hierarchical semantic neighborhoods across models and datasets. Second, we benchmark supervised alignment mappings between latent spaces using reconstruction error and downstream task performance. Third, we perform a large-scale regression analysis of how pretraining-data complexity, specialization, transfer learning, augmentation, and model scale relate to geometric and probing properties of embeddings. By coupling representational scale with standardized metadata, SEMASIA provides a reproducible foundation for studying latent geometry, evaluating alignment methods, and developing next-generation heterogeneous and interoperable AI systems.