HiCI: Hierarchical Construction-Integration for Long-Context Attention

TL;DR

HiCI introduces a hierarchical attention module inspired by cognitive theories, significantly improving long-context language modeling efficiency and performance by constructing and integrating segment-level representations.

Contribution

The paper presents HiCI, a novel hierarchical attention mechanism that enhances long-context modeling with minimal additional parameters, outperforming existing models on multiple benchmarks.

Findings

Extended context from 4K to 100K tokens with minimal parameter increase.

Achieved performance comparable to proprietary models on topic retrieval.

Surpassed GPT-3.5-Turbo-16K on code comprehension tasks.

Abstract

Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches. Drawing on cognitive theories of discourse comprehension, we propose HiCI (Hierarchical Construction--Integration), a hierarchical attention module that constructs segment-level representations, integrates them into a shared global context, and broadcasts both to condition segment-level attention. We validate HiCI through parameter-efficient adaptation of LLaMA-2 with only <5.5% additional parameters, extending context from 4K to 100K tokens (7B) and 64K tokens (13B). Across language modeling, retrieval, and instruction-following benchmarks, HiCI yields consistent improvements over strong baselines, including matching proprietary models on topic retrieval and surpassing GPT-3.5-Turbo-16K on code comprehension. These results demonstrate the effectiveness of explicit hierarchical structuring as an inductive bias for long-context modeling.