Sentiment analysis of research attention: the Altmetric proof of concept

TL;DR

This paper introduces an AI-based sentiment analysis framework to better understand how research is discussed on social media, improving on traditional citation-based metrics.

Contribution

A novel AI-driven sentiment analysis framework for altmetric data, improving alignment with human judgments through iterative model refinement.

Findings

The AI model achieved an F1 score of 0.577, outperforming the baseline ML2024 model.

The framework captures nuanced sentiment across seven levels, enhancing context-aware altmetric analysis.

Iterative expert evaluation significantly improved model precision and recall for social media research mentions.

Abstract

Traditional bibliometric approaches to research impact assessment have predominantly relied on citation counts, overlooking the qualitative dimensions of how research is received and discussed. Altmetrics have expanded this perspective by capturing mentions across diverse platforms, yet most analyses remain limited to quantitative measures, failing to account for sentiment. This study aimed to introduce a novel artificial intelligence-driven sentiment analysis framework designed to evaluate the tone and intent behind research mentions on social media, with a primary focus on X (formerly Twitter). Our approach leverages a bespoke sentiment classification system, spanning seven levels from strong negative to strong positive, to capture the nuanced ways in which research is endorsed, critiqued, or debated. Using a machine learning model trained on 5,732 manually curated labels (ML2024) as a baseline (F1 score = 0.419), we developed and refined a Large Language Model (LLM)-based classification system through three iterative rounds of expert evaluation. The final AI-driven model demonstrated improved alignment with human assessments, achieving an F1 score of 0.577, significantly enhancing precision and recall over traditional methods. These findings underscore the potential of advanced AI methodologies in altmetric analysis, offering a richer, more context-aware understanding of research reception. This study laid the foundation for integrating sentiment analysis into Altmetric platforms, providing researchers, institutions, and policymakers with deeper insights into the societal discourse surrounding scientific outputs.

Figures