Hindsight Preference Optimization for Financial Time Series Advisory

TL;DR

This paper introduces Hindsight Preference Optimization, a method using retrospective outcomes to train language models for financial advisory, improving accuracy and quality in stock market predictions.

Contribution

It proposes a novel reinforcement learning approach that leverages outcome-based feedback to enhance financial advisory models without human annotations.

Findings

A 4B model outperforms a 235B teacher in accuracy.

The method improves advisory quality and decision-making.

Retrospective outcome evaluation enhances model training.

Abstract

Time series models predict numbers; decision-makers need advisory -- directional signals with reasoning, actionable suggestions, and risk management. Training language models for such predictive advisory faces a fundamental challenge: quality depends on outcomes unknown at prediction time. We bridge two ideas from reinforcement learning -- using information unavailable during execution to retrospectively generate training signal, and preference alignment -- and propose Hindsight Preference Optimization: observed outcomes let an LLM judge rank candidate advisories on dimensions that scalar metrics cannot capture, producing preference pairs for DPO without human annotation. We apply this to Vision-Language-Model-based predictive advisories on S&P 500 equity time series, demonstrated by a 4B model outperforming its 235B teacher on both accuracy and advisory quality.