Plan Before You Trade: Inference-Time Optimization for RL Trading Agents

TL;DR

This paper introduces FPILOT, an inference-time optimization framework for RL trading agents that leverages price forecasts to improve trading performance without retraining.

Contribution

FPILOT enables inference-time policy adaptation using price forecasts, improving trading outcomes across multiple algorithms without retraining the agents.

Findings

FPILOT improves total return and risk-adjusted metrics across five algorithms.

Stochastic policies benefit more from FPILOT than deterministic ones.

Performance gains increase with the quality of the forecaster.

Abstract

Reinforcement learning agents for portfolio management are typically trained and deployed as static policies, with no mechanism for using price forecasts at inference time. We propose $\text{FPILOT}$ (**Fin**ancial **P**lugin **I**nference-time **L**earning for **O**ptimal **T**rading), a plugin inference-time optimization framework inspired by Model Predictive Control (MPC). Our key structural insight is that future prices mostly do not depend on one agent's portfolio allocation, so a suitable predictive model can produce a multi-step price trajectory without iterative action-conditioned rollouts as in typical reinforcement learning. At each decision step, we use the forecaster's predicted price trajectory to construct an allocation-based imagined return objective, and optimize the policy at inference-time before executing one step of the trade. Our framework is compatible with any pre-trained agent and adapts the policy to the forecaster's predictions without any retraining. Evaluated across five policy learning algorithms on the TradeMaster DJ30 benchmark, $\text{FPILOT}$ produces consistent improvements in total return and return-based risk-adjusted metrics (Sharpe, Sortino, Calmar), with stochastic policies benefiting more than deterministic ones. Further, using synthetic forecasts at calibrated quality levels, we show that gains consistently improve with forecaster quality, suggesting that our performance will improve based on advances in financial forecasting.