Explainable assessment of financial experts' credibility by classifying social media forecasts and checking the predictions with actual market data

TL;DR

This paper presents a novel method for assessing the credibility of financial social media forecasts by combining NLP, machine learning, and market data verification, providing continuous scores and explanations.

Contribution

It introduces a continuous credibility scoring system that verifies predictions with actual market data and offers natural language explanations, advancing beyond binary classification methods.

Findings

Credibility scores correlate with social media metrics.

The system effectively verifies forecasts with market data.

Explanations improve trustworthiness of credibility assessments.

Abstract

Social media include diverse interaction metrics related to user popularity, the most evident example being the number of user followers. The latter has raised concerns about the credibility of the posts by the most popular creators. However, most existing approaches to assess credibility in social media strictly consider this problem a binary classification, often based on a priori information, without checking if actual real-world facts back the users' comments. In addition, they do not provide automatic explanations of their predictions to foster their trustworthiness. In this work, we propose a credibility assessment solution for financial creators in social media that combines Natural Language Processing and Machine Learning. The reputation of the contributors is assessed by automatically classifying their forecasts on asset values by type and verifying these predictions with actual market data to approximate their probability of success. The outcome of this verification is a continuous credibility score instead of a binary result, an entirely novel contribution by this work. Moreover, social media metrics (i.e., user context) are exploited by calculating their correlation with the credibility rankings, providing insights on the interest of the end-users in financial posts and their forecasts (i.e., drop or rise). Finally, the system provides natural language explanations of its decisions based on a model-agnostic analysis of relevant features.