Analysis of Search Heuristics in the Multi-Armed Bandit Setting

TL;DR

This paper analyzes how different search heuristics perform in the Multi-Armed Bandit setting, focusing on their ability to identify the Condorcet winner through dueling comparisons.

Contribution

It demonstrates the limitations of evolutionary algorithms in finding the Condorcet winner and proposes a simple EDA that performs better in this task.

Findings

Evolutionary algorithms rarely identify the Condorcet winner in stationary distribution.

A Max-Min Ant System-based EDA effectively maintains the Condorcet winner with high probability.

Repeated duels can improve the Condorcet winner probability for (1+1) EA.

Abstract

We consider the classic Multi-Armed Bandit setting to understand the exploration/exploitation tradeoffs made by different search heuristics. Since many search heuristics work by comparing different options (in evolutionary algorithms called "individuals"; in the Bandit literature called "arms"), we work with the "Dueling Bandits" setting. In each iteration, a comparison between different arms can be made; in the binary stochastic setting, each arm has a fixed winning probability against any other arm. A Condorcet winner is any arm that beats every other arm with a probability strictly higher than $1/2$. We show that evolutionary algorithms are rather bad at identifying the Condorcet winner: Even if the Condorcet winner beats every other arm with a probability $1-p$, the (1+1) EA, in its stationary distribution, chooses the Condorcet winner only with constant probability if $p=\Omega(1/n)$. By contrast, we show that a simple EDA (based on the Max-Min Ant System with iteration-best update) will choose the Condorcet winner in its maintained distribution with probability $1-\Theta(p)$. As a remedy for the (1+1) EA, we show how repeated duels can significantly boost the probability of the Condorcet winner in the stationary distribution.