# Reasoning-Intensive Regression

## TL;DR

This paper introduces a new benchmark for reasoning-intensive regression tasks using LLMs, highlighting the limitations of existing methods and proposing MENTAT, a lightweight ensemble approach that significantly improves performance.

## Contribution

The paper establishes a new RiR benchmark, evaluates existing methods, and proposes MENTAT, a novel ensemble technique that outperforms prompting and fine-tuning baselines.

## Key findings

- MENTAT achieves up to 65% improvement over baselines.
- Prompting and fine-tuning often struggle in RiR tasks.
- The benchmark reveals challenges in current LLM approaches for RiR.

## Abstract

AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e., deducing subtle numerical scores from text. Unlike standard language regression tasks such as sentiment or similarity analysis, RiR often appears instead in ad-hoc applications such as rubric-based scoring, modeling dense rewards in complex environments, or domain-specific retrieval, where much deeper analysis of context is required while only limited task-specific training data and computation are available. We cast four realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and fine-tuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances.