LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimization

TL;DR

This paper introduces a pretrain-then-finetune approach using Transformers for small-data, large-scale decision problems, leveraging synthetic domain-informed data and theoretical error analysis to improve operational decision-making.

Contribution

It proposes a novel Transformer-based pipeline with domain-specific design and provides the first comprehensive error analysis with nonasymptotic guarantees.

Findings

Pretraining injects domain knowledge and enables high-capacity models.

Fine-tuning adapts models to real data and improves accuracy.

Transfer learning effectiveness increases with more data.

Abstract

We consider small-data, large-scale decision problems in which a firm must make many operational decisions simultaneously (e.g., across a large product portfolio) while observing only a few, potentially noisy, data points per instance. Inspired by the success of large language models (LLMs), we propose a pretrain-then-finetune approach built on a designed Transformer model to address this challenge. The model is first pretrained on large-scale, domain-informed synthetic data that encode managerial knowledge and structural features of the decision environment, and is then fine-tuned on real observations. This new pipeline offers two complementary advantages: pretraining injects domain knowledge into the learning process and enables the training of high-capacity models using abundant synthetic data, while finetuning adapts the pretrained model to the operational environment and improves alignment with the true data-generating regime. While we have leveraged the Transformer's state-of-the-art representational capacity, particularly its attention mechanism, to efficiently extract cross-task structure, our approach is not an off-the-shelf application. Instead, it relies on problem-specific architectural design and a tailored training procedure to match the decision setting. Theoretically, we develop the first comprehensive error analysis regarding Transformer learning in relevant contexts, establishing nonasymptotic guarantees that validate the method's effectiveness. Critically, our analysis reveals how pretraining and fine-tuning jointly determine performance, with the dominant contribution governed by whichever is more favorable. In particular, finetuning exhibits an economies-of-scale effect, whereby transfer learning becomes increasingly effective as the number of instances grows.