SOM-VQ: Topology-Aware Tokenization for Interactive Generative Models

TL;DR

SOM-VQ introduces a topology-aware tokenization method that enhances interpretability and human control in discrete generative models by preserving semantic neighborhood structure in the token space.

Contribution

It combines vector quantization with Self-Organizing Maps to create discrete codebooks with explicit low-dimensional topology, enabling intuitive manipulation and improved learnability.

Findings

Produces more learnable token sequences in evaluated domains.

Enables intuitive human-in-the-loop control through geometric manipulation.

Demonstrates effective control in human motion generation tasks.

Abstract

Vector-quantized representations enable powerful discrete generative models but lack semantic structure in token space, limiting interpretable human control. We introduce SOM-VQ, a tokenization method that combines vector quantization with Self-Organizing Maps to learn discrete codebooks with explicit low-dimensional topology. Unlike standard VQ-VAE, SOM-VQ uses topology-aware updates that preserve neighborhood structure: nearby tokens on a learned grid correspond to semantically similar states, enabling direct geometric manipulation of the latent space. We demonstrate that SOM-VQ produces more learnable token sequences in the evaluated domains while providing an explicit navigable geometry in code space. Critically, the topological organization enables intuitive human-in-the-loop control: users can steer generation by manipulating distances in token space, achieving semantic alignment without frame-level constraints. We focus on human motion generation - a domain where kinematic structure, smooth temporal continuity, and interactive use cases (choreography, rehabilitation, HCI) make topology-aware control especially natural - demonstrating controlled divergence and convergence from reference sequences through simple grid-based sampling. SOM-VQ provides a general framework for interpretable discrete representations applicable to music, gesture, and other interactive generative domains.