Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data

TL;DR

This paper introduces Type-Compliant Adaptation Cascades (TACs), a framework that improves the reliability and compliance of complex LLM workflows by learning typed probabilistic programs, significantly outperforming prompt-optimization baselines.

Contribution

The paper presents TACs, a novel approach that recasts workflow adaptation as learning typed probabilistic programs, enabling principled training and improved task compliance.

Findings

TACs outperform state-of-the-art prompt-optimization baselines.

Significant improvements on structured tasks, e.g., FinQA and MGSM.

Theoretical proof that optimization bias vanishes with learning type compliance.

Abstract

Reliably composing Large Language Models (LLMs) for complex, multi-step workflows remains a significant challenge. The dominant paradigm -- optimizing discrete prompts in a pipeline -- is notoriously brittle and struggles to enforce the formal compliance required for structured tasks. We introduce Type-Compliant Adaptation Cascades (TACs), a framework that recasts workflow adaptation as learning typed probabilistic programs. TACs treat the entire workflow, which is composed of parameter-efficiently adapted LLMs and deterministic logic, as an unnormalized joint distribution. This enables principled, gradient-based training even with latent intermediate structures. We provide theoretical justification for our tractable optimization objective, proving that the optimization bias vanishes as the model learns type compliance. Empirically, TACs significantly outperform state-of-the-art prompt-optimization baselines. Gains are particularly pronounced on structured tasks, improving FinQA from $12.0\%$ to $24.7\%$ for a Qwen 3 8B model, MGSM-SymPy from $57.1\%$ to $75.9\%$ for a Gemma 2 27B model, MGSM from $1.6\%$ to $27.3\%$, and MuSR from $36.5\%$ to $62.6\%$ for a Gemma 7B model. TACs offer a robust and theoretically grounded paradigm for developing reliable, task-compliant LLM systems.