Quantile Regression with Large Language Models for Price Prediction

TL;DR

This paper introduces a novel quantile regression method using Large Language Models to produce full predictive distributions for price estimation tasks, demonstrating significant improvements over traditional methods.

Contribution

It proposes a new quantile regression approach for LLMs that enhances probabilistic predictions and systematically compares various LLM architectures and training strategies.

Findings

Mistral-7B with quantile heads outperforms traditional methods.

LLM-assisted label correction achieves human-level accuracy.

Model architecture and data scaling significantly impact performance.

Abstract

Large Language Models (LLMs) have shown promise in structured prediction tasks, including regression, but existing approaches primarily focus on point estimates and lack systematic comparison across different methods. We investigate probabilistic regression using LLMs for unstructured inputs, addressing challenging text-to-distribution prediction tasks such as price estimation where both nuanced text understanding and uncertainty quantification are critical. We propose a novel quantile regression approach that enables LLMs to produce full predictive distributions, improving upon traditional point estimates. Through extensive experiments across three diverse price prediction datasets, we demonstrate that a Mistral-7B model fine-tuned with quantile heads significantly outperforms traditional approaches for both point and distributional estimations, as measured by three established metrics each for prediction accuracy and distributional calibration. Our systematic comparison of LLM approaches, model architectures, training approaches, and data scaling reveals that Mistral-7B consistently outperforms encoder architectures, embedding-based methods, and few-shot learning methods. Our experiments also reveal the effectiveness of LLM-assisted label correction in achieving human-level accuracy without systematic bias. Our curated datasets are made available at https://github.com/vnik18/llm-price-quantile-reg/ to support future research.