Profiling German Text Simplification with Interpretable Model-Fingerprints

TL;DR

This paper presents the Simplification Profiler, a diagnostic toolkit that creates interpretable fingerprints of text simplification models, enabling detailed analysis of model behavior and differences across configurations without relying on extensive human ratings.

Contribution

The paper introduces a novel, interpretable fingerprinting approach for analyzing LLM-based text simplification models, especially useful in data-scarce, diverse language contexts.

Findings

Fingerprint features can classify model configurations with up to 71.9% accuracy.

The profiler distinguishes between prompting strategies and prompt engineering effects.

The approach outperforms simple baselines by over 48 percentage points.

Abstract

While Large Language Models (LLMs) produce highly nuanced text simplifications, developers currently lack tools for a holistic, efficient, and reproducible diagnosis of their behavior. This paper introduces the Simplification Profiler, a diagnostic toolkit that generates a multidimensional, interpretable fingerprint of simplified texts. Multiple aggregated simplifications of a model result in a model's fingerprint. This novel evaluation paradigm is particularly vital for languages, where the data scarcity problem is magnified when creating flexible models for diverse target groups rather than a single, fixed simplification style. We propose that measuring a model's unique behavioral signature is more relevant in this context as an alternative to correlating metrics with human preferences. We operationalize this with a practical meta-evaluation of our fingerprints' descriptive power, which bypasses the need for large, human-rated datasets. This test measures if a simple linear classifier can reliably identify various model configurations by their created simplifications, confirming that our metrics are sensitive to a model's specific characteristics. The Profiler can distinguish high-level behavioral variations between prompting strategies and fine-grained changes from prompt engineering, including few-shot examples. Our complete feature set achieves classification F1-scores up to 71.9 %, improving upon simple baselines by over 48 percentage points. The Simplification Profiler thus offers developers a granular, actionable analysis to build more effective and truly adaptive text simplification systems.