Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing

TL;DR

This paper introduces MegaBeam-Mistral-7B, a compact 7-billion-parameter language model capable of processing 512,000 tokens of context, enabling efficient long-range reasoning and practical long-context applications without extensive fine-tuning.

Contribution

The work presents a novel 7B model supporting 512K tokens, demonstrating competitive long-range reasoning and practical utility in long-context tasks, with open-source release and broad accessibility.

Findings

Outperforms on HELMET in in-context learning

Shows robust retrieval and tracing on RULER

Achieves competitive long-range reasoning on BABILong

Abstract

We present MegaBeam-Mistral-7B, a language model that supports 512K-token context length. Our work addresses practical limitations in long-context training, supporting real-world tasks such as compliance monitoring and verification. Evaluated on three long-context benchmarks, our 7B-parameter model demonstrates superior in-context learning performance on HELMET and robust retrieval and tracing capability on RULER. It is currently the only open model to achieve competitive long-range reasoning on BABILong at 512K context length without RAG or targeted fine-tuning. Released as fully open source under the Apache 2.0 license, the model has been downloaded over 100,000 times on Hugging Face. Model available at: https://huggingface.co/aws-prototyping/MegaBeam-Mistral-7B-512k