LLM REgression with a Latent Iterative State Head

TL;DR

RELISH is a lightweight, iterative architecture that enables large language models to predict scalar text regression targets directly, outperforming prior methods across multiple datasets and models.

Contribution

It introduces RELISH, a novel method for direct scalar prediction in LLMs using an iterative latent state, with high efficiency and improved accuracy over existing approaches.

Findings

RELISH outperforms prior LLM regression baselines on five datasets.

It requires only 3.4-3.7M trainable parameters, less than LoRA-based methods.

RELISH is effective across four different LLM backbones and two training regimes.

Abstract

We present RELISH (REgression with a Latent Iterative State Head), a novel, lightweight architecture designed for text regression with large language models. Rather than decoding numeric targets as text or aggregating multiple generated outputs, RELISH predicts scalar values directly from frozen LLM representations by iteratively refining a learned latent state through cross-attention over token-level representations, and then mapping the final state to a point estimate with a linear regressor. Across five datasets, four LLM backbones, and two LLM training regimes, RELISH consistently outperforms prior baselines from all three major LLM regression families, including autoregressive decoding, regression-aware inference, and existing predictive head methods. Despite these gains, RELISH remains highly parameter-efficient, requiring only 3.4-3.7M trainable parameters across frozen LLM backbones (only 0.01-0.04% additional overhead), far less than LoRA-based alternatives that grow with model size (0.26-0.42%).