SciGisPy: a Novel Metric for Biomedical Text Simplification via Gist Inference Score

TL;DR

SciGisPy is a new evaluation metric for biomedical text simplification that measures how well simplified texts preserve core meanings by facilitating gist inference, outperforming existing metrics.

Contribution

Introduces SciGisPy, a domain-specific extension of Gist Inference Score, tailored for biomedical text simplification evaluation.

Findings

SciGisPy achieves 84% accuracy in identifying simplified texts.

Outperforms original GIS with significant improvements.

Better captures essential biomedical content in simplification evaluation.

Abstract

Biomedical literature is often written in highly specialized language, posing significant comprehension challenges for non-experts. Automatic text simplification (ATS) offers a solution by making such texts more accessible while preserving critical information. However, evaluating ATS for biomedical texts is still challenging due to the limitations of existing evaluation metrics. General-domain metrics like SARI, BLEU, and ROUGE focus on surface-level text features, and readability metrics like FKGL and ARI fail to account for domain-specific terminology or assess how well the simplified text conveys core meanings (gist). To address this, we introduce SciGisPy, a novel evaluation metric inspired by Gist Inference Score (GIS) from Fuzzy-Trace Theory (FTT). SciGisPy measures how well a simplified text facilitates the formation of abstract inferences (gist) necessary for comprehension, especially in the biomedical domain. We revise GIS for this purpose by introducing domain-specific enhancements, including semantic chunking, Information Content (IC) theory, and specialized embeddings, while removing unsuitable indexes. Our experimental evaluation on the Cochrane biomedical text simplification dataset demonstrates that SciGisPy outperforms the original GIS formulation, with a significant increase in correctly identified simplified texts (84% versus 44.8%). The results and a thorough ablation study confirm that SciGisPy better captures the essential meaning of biomedical content, outperforming existing approaches.