Efficient Context Scaling with LongCat ZigZag Attention

TL;DR

This paper presents LongCat ZigZag Attention (LoZA), a sparse attention method that enhances existing models to efficiently handle extremely long contexts up to 1 million tokens, improving speed and reasoning capabilities.

Contribution

The paper introduces LoZA, a novel sparse attention scheme that transforms full-attention models into efficient sparse models suitable for very long contexts.

Findings

LoZA achieves significant speed-ups in long-context scenarios.

LoZA enables models to process up to 1 million tokens.

LoZA improves long-term reasoning and agentic capabilities.

Abstract

We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.