Two-Armed Bandit Problem, Data Processing, and Parallel Version of the Mirror Descent Algorithm

TL;DR

This paper improves the theoretical understanding and efficiency of the mirror descent algorithm in two-armed bandit problems, introducing a parallel version that reduces processing time and risk, especially when methods have similar efficiencies.

Contribution

It introduces a parallel version of the mirror descent algorithm for two-armed bandit problems, significantly reducing risk and processing time, with theoretical and simulation validation.

Findings

Parallel MDA reduces total processing time independent of data volume.

Parallel MDA achieves smaller minimax risk compared to the ordinary version.

Effectiveness of the parallel approach depends on the similarity of method efficiencies.

Abstract

We consider the minimax setup for the two-armed bandit problem as applied to data processing if there are two alternative processing methods available with different a priori unknown efficiencies. One should determine the most effective method and provide its predominant application. To this end we use the mirror descent algorithm (MDA). It is well-known that corresponding minimax risk has the order $N^{1/2}$ with $N$ being the number of processed data. We improve significantly the theoretical estimate of the factor using Monte-Carlo simulations. Then we propose a parallel version of the MDA which allows processing of data by packets in a number of stages. The usage of parallel version of the MDA ensures that total time of data processing depends mostly on the number of packets but not on the total number of data. It is quite unexpectedly that the parallel version behaves unlike the ordinary one even if the number of packets is large. Moreover, the parallel version considerably improves control performance because it provides significantly smaller value of the minimax risk. We explain this result by considering another parallel modification of the MDA which behavior is close to behavior of the ordinary version. Our estimates are based on invariant descriptions of the algorithms. All estimates are obtained by Monte-Carlo simulations. It's worth noting that parallel version performs well only for methods with close efficiencies. If efficiencies differ significantly then one should use the combined algorithm which at initial sufficiently short control horizon uses ordinary version and then switches to the parallel version of the MDA.