# Parameterized Exploration

## TL;DR

Parameterized Exploration (PE) is a new method for tuning exploration schedules in decision problems, considering time horizon and knowledge state, leading to improved performance across various bandit and MDP tasks.

## Contribution

Introduces Parameterized Exploration, a simple family of methods that adapt exploration based on problem horizon and knowledge, outperforming un-tuned methods in multiple settings.

## Key findings

- PE outperforms un-tuned exploration methods in bandit and MDP tasks.
- Performance of PE depends on the accuracy of the dynamics model.
- PE adapts exploration to the decision problem's characteristics.

## Abstract

We introduce Parameterized Exploration (PE), a simple family of methods for model-based tuning of the exploration schedule in sequential decision problems. Unlike common heuristics for exploration, our method accounts for the time horizon of the decision problem as well as the agent's current state of knowledge of the dynamics of the decision problem. We show our method as applied to several common exploration techniques has superior performance relative to un-tuned counterparts in Bernoulli and Gaussian multi-armed bandits, contextual bandits, and a Markov decision process based on a mobile health (mHealth) study. We also examine the effects of the accuracy of the estimated dynamics model on the performance of PE.