Glyph: Scaling Context Windows via Visual-Text Compression

TL;DR

Glyph introduces a visual-text compression framework that renders long texts into images for processing by vision-language models, enabling efficient long-context understanding with significant token compression and speed improvements.

Contribution

This work presents a novel visual-text compression method using image rendering and VLMs, allowing scalable long-context modeling beyond traditional token-based approaches.

Findings

Achieves 3-4x token compression with maintained accuracy

Enables 4x faster prefilling and decoding

Supports 1M-token-level tasks with a 128K-context VLM

Abstract

Large language models (LLMs) increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level brings prohibitive computational and memory costs, limiting the practicality of long-context LLMs. In this work, we take a different perspective-visual context scaling-to tackle this challenge. Instead of extending token-based sequences, we propose Glyph, a framework that renders long texts into images and processes them with vision-language models (VLMs). This approach substantially compresses textual input while preserving semantic information, and we further design an LLM-driven genetic search to identify optimal visual rendering configurations for balancing accuracy and compression. Through extensive experiments, we demonstrate that our method achieves 3-4x token compression while maintaining accuracy comparable to leading LLMs such as Qwen3-8B on various long-context benchmarks. This compression also leads to around 4x faster prefilling and decoding, and approximately 2x faster SFT training. Furthermore, under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks. In addition, the rendered text data benefits real-world multimodal tasks, such as document understanding. Our code and model are released at https://github.com/thu-coai/Glyph.