Functional Component Ablation Reveals Specialization Patterns in Hybrid Language Model Architectures

TL;DR

This study investigates the functional roles of components in hybrid language models, revealing that both attention and state space modules are essential, with the linear attention backbone being primary, and early layers being most critical.

Contribution

The paper introduces a functional ablation framework for hybrid language models, demonstrating the importance of both components and their positional significance, with implications for model design and robustness.

Findings

Both attention and SSM/linear attention components are essential.

Linear attention or SSM forms the primary backbone, with removal causing >35,000x perplexity increase.

Early layers are disproportionately critical, with importance decreasing in later layers.

Abstract

Hybrid language models combining attention with state space models (SSMs) or linear attention offer improved efficiency, but whether both components are genuinely utilized remains unclear. We present a functional component ablation framework applied to two sub-1B hybrid models -- Qwen3.5-0.8B (sequential: Gated DeltaNet + softmax attention) and Falcon-H1-0.5B (parallel: Mamba-2 + attention) -- with a pure Transformer control (Qwen2.5-0.5B). Through group ablations, layer-wise sweeps, positional ablations, matched random controls, and perplexity analysis across five benchmarks, we establish four findings: (1) both component types are essential and neither is bypassed; (2) the alternative component (linear attention or SSM) is the primary language modeling backbone, causing >35,000x perplexity degradation when removed versus ~82x for attention; (3) component importance follows a positional gradient, with early layers being disproportionately critical; and (4) hybrid architectures exhibit 20-119x greater resilience to random layer removal than pure Transformers, revealing built-in functional redundancy between component types. These results provide actionable guidance for hybrid model compression, architecture design, and fault-tolerant deployment.