Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking

TL;DR

This paper introduces a zero-shot method for long document classification that reduces input size by ranking and selecting key sentences, maintaining accuracy while improving efficiency.

Contribution

It presents a novel sentence ranking approach using TF-IDF to enable efficient zero-shot classification of long documents without changing model architecture.

Findings

Retaining top 50% sentences maintains accuracy

Inference time reduced by up to 35%

Effective for long Marathi news articles

Abstract

Transformer-based models like BERT excel at short text classification but struggle with long document classification (LDC) due to input length limitations and computational inefficiencies. In this work, we propose an efficient, zero-shot approach to LDC that leverages sentence ranking to reduce input context without altering the model architecture. Our method enables the adaptation of models trained on short texts, such as headlines, to long-form documents by selecting the most informative sentences using a TF-IDF-based ranking strategy. Using the MahaNews dataset of long Marathi news articles, we evaluate three context reduction strategies that prioritize essential content while preserving classification accuracy. Our results show that retaining only the top 50\% ranked sentences maintains performance comparable to full-document inference while reducing inference time by up to 35\%. This demonstrates that sentence ranking is a simple yet effective technique for scalable and efficient zero-shot LDC.