Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral

TL;DR

This paper introduces Kad, a proxy-based test-time alignment framework for large language models that uses a knapsack approximation to efficiently decide when to defer token generation to a smaller aligned model, improving performance and speed.

Contribution

Kad presents a novel token-specific cascading approach that reduces test-time alignment to a 0-1 knapsack problem, enabling efficient alignment with reduced computational costs.

Findings

Improves task performance of LLMs with test-time alignment.

Speeds up decoding by deferring tokens using knapsack approximation.

Demonstrates effectiveness on large language models.

Abstract

Several previous works concluded that the largest part of generation capabilities of large language models (LLM) are learned (early) during pre-training. However, LLMs still require further alignment to adhere to downstream task requirements and stylistic preferences, among other desired properties. As LLMs continue to scale in terms of size, the computational cost of alignment procedures increase prohibitively. In this work, we propose a novel approach to circumvent these costs via proxy-based test-time alignment, i.e. using guidance from a small aligned model. Our approach can be described as a token-specific cascading method, where the token-specific deferral rule is reduced to 0-1 knapsack problem. In this setting, we derive primal and dual approximations of the optimal deferral decision. We experimentally show the benefits of our method both in task performance and speculative decoding speed.