Statistical Measures for Explainable Aspect-Based Sentiment Analysis: A Case Study on Environmental Discourse in Reddit

TL;DR

This paper presents a statistical, model-agnostic framework with metrics like entropy and divergence to evaluate the interpretability and trustworthiness of aspect-based sentiment analysis models, demonstrated through a Reddit environmental discourse case study.

Contribution

It introduces a novel set of statistical indicators for assessing ABSA models' transparency and reliability without relying on labeled data, enhancing interpretability in real-world applications.

Findings

Statistical indicators effectively diagnose model certainty and variability.

Metrics are validated through bootstrap resampling and sensitivity analysis.

Framework is computationally efficient and complements traditional validation methods.

Abstract

Aspect-Based Sentiment Analysis (ABSA) provides a fine-grained understanding of opinions by linking sentiment to specific aspects in text. While transformer-based models excel at this task, their black-box nature limits their interpretability, posing risks in real-world applications without labeled data. This paper introduces a statistical, model-agnostic framework to assess the behavioral transparency and trustworthiness of ABSA models. Our framework relies on several metrics, such as the entropy of polarity distributions, soft-count-based dominance scores, and sentiment divergence between sources, whose robustness is validated through bootstrap resampling and sensitivity analysis. A case study on environmentally focused Reddit communities illustrates how the proposed indicators provide interpretable diagnostics of model certainty, decisiveness, and cross-source variability. The results show that statistical indicators computed on soft outputs can complement traditional approaches, offering a computationally efficient methodology for validating, monitoring, and interpreting ABSA models in contexts where labeled data are unavailable.