On Bits and Bandits: Quantifying the Regret-Information Trade-off

TL;DR

This paper explores the fundamental trade-off between information gained and regret suffered in sequential decision-making, using information-theoretic bounds to quantify and improve performance in tasks like question-answering with large language models.

Contribution

It introduces the first Bayesian regret lower bounds based on information accumulation and derives regret upper bounds, linking information measured in bits to regret in reward.

Findings

Information-theoretic bounds effectively quantify the regret-information trade-off.

Bayesian regret lower bounds depend on the amount of information accumulated.

Application to large language models improves question-answering performance.

Abstract

In many sequential decision problems, an agent performs a repeated task. He then suffers regret and obtains information that he may use in the following rounds. However, sometimes the agent may also obtain information and avoid suffering regret by querying external sources. We study the trade-off between the information an agent accumulates and the regret it suffers. We invoke information-theoretic methods for obtaining regret lower bounds, that also allow us to easily re-derive several known lower bounds. We introduce the first Bayesian regret lower bounds that depend on the information an agent accumulates. We also prove regret upper bounds using the amount of information the agent accumulates. These bounds show that information measured in bits, can be traded off for regret, measured in reward. Finally, we demonstrate the utility of these bounds in improving the performance of a question-answering task with large language models, allowing us to obtain valuable insights.