NIFTY Financial News Headlines Dataset

TL;DR

The NIFTY Financial News Headlines dataset is a comprehensive resource designed to enhance financial market forecasting research using large language models, offering tailored versions for supervised fine-tuning and alignment tasks.

Contribution

Introduces the NIFTY dataset with two versions for different LLM training approaches, including curated data, metadata, and utilities, to support financial forecasting research.

Findings

Demonstrates applications in stock price movement prediction.

Shows the role of LLM embeddings in information richness.

Provides high-quality, filtered financial news data.

Abstract

We introduce and make publicly available the NIFTY Financial News Headlines dataset, designed to facilitate and advance research in financial market forecasting using large language models (LLMs). This dataset comprises two distinct versions tailored for different modeling approaches: (i) NIFTY-LM, which targets supervised fine-tuning (SFT) of LLMs with an auto-regressive, causal language-modeling objective, and (ii) NIFTY-RL, formatted specifically for alignment methods (like reinforcement learning from human feedback (RLHF)) to align LLMs via rejection sampling and reward modeling. Each dataset version provides curated, high-quality data incorporating comprehensive metadata, market indices, and deduplicated financial news headlines systematically filtered and ranked to suit modern LLM frameworks. We also include experiments demonstrating some applications of the dataset in tasks like stock price movement and the role of LLM embeddings in information acquisition/richness. The NIFTY dataset along with utilities (like truncating prompt's context length systematically) are available on Hugging Face at https://huggingface.co/datasets/raeidsaqur/NIFTY.