# The cell as a token: high-dimensional geometry in language models and cell embeddings

## TL;DR

This paper compares how language models and cell embeddings use high-dimensional spaces, suggesting insights from language models can improve cell analysis.

## Contribution

It introduces a novel perspective by linking language model advancements to single-cell data analysis techniques.

## Key findings

- Token context influences embedding space geometry in both language and cell data.
- Low-dimensional manifolds affect the robustness and interpretation of embedding spaces.
- Language model techniques like interpretability probes can enhance virtual cell model training.

## Abstract

Single-cell sequencing technology maps cells to a high-dimensional space encoding their internal activity. Recently-proposed virtual cell models extend this concept, enriching cells’ representations based on patterns learned from pretraining on vast cell atlases.

This review explores how advances in understanding the structure of natural language embeddings informs ongoing efforts to analyze single-cell datasets. Both fields process unstructured data by partitioning datasets into tokens embedded within a high-dimensional vector space. We discuss how the context of tokens influences the geometry of embedding space, and how low-dimensional manifolds shape this space’s robustness and interpretation. We highlight how new developments in foundation models for language, such as interpretability probes and in-context reasoning, can inform efforts to construct cell atlases and train virtual cell models.

Code is available at https://github.com/williamgilpin/celltoken.