Aligning AI Agents via Information-Directed Sampling

TL;DR

This paper introduces a new bandit alignment framework addressing long-term AI alignment by balancing exploration of environment and human preferences, demonstrating the effectiveness of information-directed sampling over naive methods.

Contribution

It extends classic multi-armed bandit problems to include human preferences and costs, proposing an information-directed sampling approach for better AI alignment.

Findings

Naive exploration algorithms perform poorly in the alignment setting.

Information-directed sampling achieves lower regret in the toy problem.

Current algorithms like Thompson sampling are inadequate for this alignment task.

Abstract

The staggering feats of AI systems have brought to attention the topic of AI Alignment: aligning a "superintelligent" AI agent's actions with humanity's interests. Many existing frameworks/algorithms in alignment study the problem on a myopic horizon or study learning from human feedback in isolation, relying on the contrived assumption that the agent has already perfectly identified the environment. As a starting point to address these limitations, we define a class of bandit alignment problems as an extension of classic multi-armed bandit problems. A bandit alignment problem involves an agent tasked with maximizing long-run expected reward by interacting with an environment and a human, both involving details/preferences initially unknown to the agent. The reward of actions in the environment depends on both observed outcomes and human preferences. Furthermore, costs are associated with querying the human to learn preferences. Therefore, an effective agent ought to intelligently trade-off exploration (of the environment and human) and exploitation. We study these trade-offs theoretically and empirically in a toy bandit alignment problem which resembles the beta-Bernoulli bandit. We demonstrate while naive exploration algorithms which reflect current practices and even touted algorithms such as Thompson sampling both fail to provide acceptable solutions to this problem, information-directed sampling achieves favorable regret.