Enhancing Financial Market Predictions: Causality-Driven Feature Selection

TL;DR

This paper presents the FinSen dataset and a novel causality-driven feature selection method that, combined with sentiment analysis and calibration techniques, significantly improves the accuracy and reliability of financial market predictions.

Contribution

It introduces the FinSen dataset and a Focal Calibration Loss, advancing the integration of causality, sentiment analysis, and calibration in financial forecasting models.

Findings

Expected Calibration Error reduced to 3.34% with DAN 3

Enhanced prediction accuracy using causally validated sentiment scores

Demonstrated effectiveness of combining sentiment analysis with calibration techniques

Abstract

This paper introduces the FinSen dataset that revolutionizes financial market analysis by integrating economic and financial news articles from 197 countries with stock market data. The dataset's extensive coverage spans 15 years from 2007 to 2023 with temporal information, offering a rich, global perspective with 160,000 records on financial market news. Our study leverages causally validated sentiment scores and LSTM models to enhance market forecast accuracy and reliability. Utilizing the FinSen dataset, we introduce an innovative Focal Calibration Loss, reducing Expected Calibration Error (ECE) to 3.34 percent with the DAN 3 model. This not only improves prediction accuracy but also aligns probabilistic forecasts closely with real outcomes, crucial for the financial sector where predicted probability is paramount. Our approach demonstrates the effectiveness of combining sentiment analysis with precise calibration techniques for trustworthy financial forecasting where the cost of misinterpretation can be high. Finsen Data can be found at [this github URL](https://github.com/EagleAdelaide/FinSen_Dataset.git).