Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models

TL;DR

This paper introduces HIA, a tuning-free inference-time alignment method for LLMs that balances alignment quality and computational cost using heuristic rewards and filtering, outperforming traditional search methods.

Contribution

HIA is a novel, tuning-free approach that uses heuristic reward models and filtering to efficiently align LLMs at inference time without fine-tuning.

Findings

HIA outperforms baseline methods in multi-objective tasks under the same inference budget.

HIA remains effective with as few as one or two response queries.

HIA offers a practical, scalable solution for personalized LLM deployment.

Abstract

Aligning LLMs with user preferences is crucial for real-world use but often requires costly fine-tuning or expensive inference, forcing trade-offs between alignment quality and computational cost. Existing inference-time methods typically ignore this balance, focusing solely on the optimized policy's performance. We propose HIA (Heuristic-Guided Inference-time Alignment), a tuning-free, black-box-compatible approach that uses a lightweight prompt optimizer, heuristic reward models, and two-stage filtering to reduce inference calls while preserving alignment quality. On real-world prompt datasets, HelpSteer and ComPRed, HIA outperforms best-of-N sampling, beam search, and greedy search baselines in multi-objective, goal-conditioned tasks under the same inference budget. We also find that HIA is effective under low-inference budgets with as little as one or two response queries, offering a practical solution for scalable, personalized LLM deployment.