Predicting Through Generation: Why Generation Is Better for Prediction

TL;DR

This paper demonstrates that token-level generation is more effective for prediction tasks than pooled representations, supported by theoretical and empirical evidence, and introduces PredGen, a framework addressing key challenges to improve structured prediction performance.

Contribution

The paper introduces PredGen, a novel end-to-end framework that enhances generation-based prediction by addressing exposure bias and format mismatch, with a new alignment loss for better accuracy.

Findings

PredGen outperforms standard baselines on multiple benchmarks.

Token-level generation retains more mutual information for prediction.

Scheduled sampling reduces exposure bias effectively.

Abstract

This paper argues that generating output tokens is more effective than using pooled representations for prediction tasks because token-level generation retains more mutual information. Since LLMs are trained on massive text corpora using next-token prediction, generation aligns naturally with their learned behavior. Using the Data Processing Inequality (DPI), we provide both theoretical and empirical evidence supporting this claim. However, autoregressive models face two key challenges when used for prediction: (1) exposure bias, where the model sees ground truth tokens during training but relies on its own predictions during inference, leading to errors, and (2) format mismatch, where discrete tokens do not always align with the tasks required output structure. To address these challenges, we introduce PredGen(Predicting Through Generating), an end to end framework that (i) uses scheduled sampling to reduce exposure bias, and (ii) introduces a task adapter to convert the generated tokens into structured outputs. Additionally, we introduce Writer-Director Alignment Loss (WDAL), which ensures consistency between token generation and final task predictions, improving both text coherence and numerical accuracy. We evaluate PredGen on multiple classification and regression benchmarks. Our results show that PredGen consistently outperforms standard baselines, demonstrating its effectiveness in structured prediction tasks.