ElasticTok: Adaptive Tokenization for Image and Video

TL;DR

ElasticTok introduces an adaptive tokenization method for images and videos that dynamically allocates tokens based on data complexity, improving efficiency and encoding quality for long sequences.

Contribution

It proposes a novel adaptive tokenization technique conditioned on prior frames, enabling variable-length encoding for more efficient processing of visual data.

Findings

Effective in reducing token usage while maintaining quality

Improves processing of complex and simple data dynamically

Enhances scalability for long video sequences

Abstract

Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents.