Domain-specific long text classification from sparse relevant information

TL;DR

This paper introduces a hierarchical model designed to improve classification of long, domain-specific documents with sparse relevant information, outperforming larger language models in medical document retrieval tasks.

Contribution

The paper presents a novel hierarchical approach that leverages a short list of target terms to enhance relevance detection in long, domain-specific texts, especially when relevant signals are weak.

Findings

Outperforms larger language models in medical document retrieval

Effective in scenarios with sparse relevant information

Validated on English and French medical datasets

Abstract

Large Language Models have undoubtedly revolutionized the Natural Language Processing field, the current trend being to promote one-model-for-all tasks (sentiment analysis, translation, etc.). However, the statistical mechanisms at work in the larger language models struggle to exploit the relevant information when it is very sparse, when it is a weak signal. This is the case, for example, for the classification of long domain-specific documents, when the relevance relies on a single relevant word or on very few relevant words from technical jargon. In the medical domain, it is essential to determine whether a given report contains critical information about a patient's condition. This critical information is often based on one or few specific isolated terms. In this paper, we propose a hierarchical model which exploits a short list of potential target terms to retrieve candidate sentences and represent them into the contextualized embedding of the target term(s) they contain. A pooling of the term(s) embedding(s) entails the document representation to be classified. We evaluate our model on one public medical document benchmark in English and on one private French medical dataset. We show that our narrower hierarchical model is better than larger language models for retrieving relevant long documents in a domain-specific context.