HARLF: Hierarchical Reinforcement Learning and Lightweight LLM-Driven Sentiment Integration for Financial Portfolio Optimization

TL;DR

This paper introduces a hierarchical reinforcement learning framework that integrates lightweight LLMs for sentiment analysis to optimize financial portfolios, achieving significant returns and outperforming benchmarks.

Contribution

It presents a novel hierarchical architecture combining LLM-driven sentiment analysis with reinforcement learning for portfolio optimization, emphasizing scalability and reproducibility.

Findings

26% annualized return on test data

Sharpe ratio of 1.2 indicating good risk-adjusted returns

Outperforms equal-weighted and S&P 500 benchmarks

Abstract

This paper presents a novel hierarchical framework for portfolio optimization, integrating lightweight Large Language Models (LLMs) with Deep Reinforcement Learning (DRL) to combine sentiment signals from financial news with traditional market indicators. Our three-tier architecture employs base RL agents to process hybrid data, meta-agents to aggregate their decisions, and a super-agent to merge decisions based on market data and sentiment analysis. Evaluated on data from 2018 to 2024, after training on 2000-2017, the framework achieves a 26% annualized return and a Sharpe ratio of 1.2, outperforming equal-weighted and S&P 500 benchmarks. Key contributions include scalable cross-modal integration, a hierarchical RL structure for enhanced stability, and open-source reproducibility.