Covariance-Aware Transformers for Quadratic Programming and Decision Making

TL;DR

This paper demonstrates how transformer models, enhanced with covariance information, can directly solve quadratic programming problems and improve decision-making tasks like portfolio optimization, outperforming traditional methods.

Contribution

It introduces a covariance-aware transformer framework that can solve various quadratic programs and improves decision-making models by explicitly incorporating second-order statistics.

Findings

Transformers can solve unconstrained quadratic programs via linear attention.

The proposed method outperforms classical predict-then-optimize approaches in portfolio optimization.

Explicit covariance integration enhances transformer performance in decision-making tasks.

Abstract

We explore the use of transformers for solving quadratic programs and how this capability benefits decision-making problems that involve covariance matrices. We first show that the linear attention mechanism can provably solve unconstrained QPs by tokenizing the matrix variables (e.g.~$A$ of the objective $\frac{1}{2}x^\top Ax+b^\top x$) row-by-row and emulating gradient descent iterations. Furthermore, by incorporating MLPs, a transformer block can solve (i) $\ell_1$-penalized QPs by emulating iterative soft-thresholding and (ii) $\ell_1$-constrained QPs when equipped with an additional feedback loop. Our theory motivates us to introduce Time2Decide: a generic method that enhances a time series foundation model (TSFM) by explicitly feeding the covariance matrix between the variates. We empirically find that Time2Decide uniformly outperforms the base TSFM model for the classical portfolio optimization problem that admits an $\ell_1$-constrained QP formulation. Remarkably, Time2Decide also outperforms the classical "Predict-then-Optimize (PtO)" procedure, where we first forecast the returns and then explicitly solve a constrained QP, in suitable settings. Our results demonstrate that transformers benefit from explicit use of second-order statistics, and this can enable them to effectively solve complex decision-making problems, like portfolio construction, in one forward pass.