Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios

TL;DR

This paper presents a novel hybrid Mamba-Transformer model for extractive summarization that efficiently processes full documents in low-resource settings, outperforming existing models in quality and speed.

Contribution

It introduces the first hybrid architecture combining transformers with state space models for extractive summarization, enabling linear-time processing of long documents.

Findings

Achieves +0.23 ROUGE-1 on ArXiv compared to BERTSUM

Significant improvements across all tested datasets (p < 0.001)

Increases inference speed by 24-27% on CNN/DailyMail

Abstract

Extractive summarization of long documents is bottlenecked by quadratic complexity, often forcing truncation and limiting deployment in resource-constrained settings. We introduce the first Mamba-Transformer hybrid for extractive summarization, combining the semantic strength of pre-trained transformers with the linear-time processing of state space models. Leveraging Mamba's ability to process full documents without truncation, our approach preserves context while maintaining strong summarization quality. The architecture includes: (1) a transformer encoder for sentence-level semantics, (2) a Mamba state space model to capture inter-sentence dependencies efficiently, and (3) a linear classifier for sentence relevance prediction. Across news, argumentative, and scientific domains under low-resource conditions, our method achieves: (1) large gains over BERTSUM and MATCHSUM, including +0.23 ROUGE-1 on ArXiv and statistically significant improvements on all datasets (p < 0.001); (2) consistent advantages across domains, strongest on the longest documents; (3) robust performance with limited training data; and (4) 24-27% faster inference on news summarization (CNN/DailyMail). We introduce the first hybrid Transformer-state space architecture for summarization, showing significant ROUGE improvements in low-resource scenarios.