Making Knowledge Accessible: Divergent Readability-Accuracy Strategies of Mistral and QWen in Biomedical Text Simplification

TL;DR

This study compares two large language models, Mistral and QWen, in biomedical text simplification, analyzing their strategies and performance against human benchmarks.

Contribution

It provides an empirical analysis of how instruction-tuned LLMs differ in simplifying biomedical text, highlighting operational strategies and metric redundancies.

Findings

Mistral improves readability while maintaining discourse fidelity (BERTScore 0.91).

QWen achieves readability gains but shows a trade-off with accuracy (BERTScore 0.89).

Strong redundancies found among 21 evaluation metrics.

Abstract

The growing public demand for accessible biomedical information calls for scalable text simplification. While large language models (LLMs) offer solutions, they too struggle with balancing improved readability against preservation of meaning. This report empirically compares how two LLMs - instruction-tuned Mistral-Small 3 24B and the reasoning-augmented QWen2.5 32B- navigate this trade-off in biomedical text simplification, benchmarked against human performance. Our analysis highlights how each model applies distinct operational strategies when simplifying biomedical text. Mistral exhibits a tempered lexical simplification approach that consistently enhances readability across multiple metrics while preserving discourse fidelity (BERTScore: 0.91, statistically comparable to that of humans). In comparison, QWen also attains enhanced readability performance and a reasonable BERTScore of 0.89, but presents a disconnect in balancing between readability and accuracy. Additionally, a comprehensive correlation analysis of a suite of 21 metrics confirms strong functional redundancies in metrics and informs adaptation requirements.