# Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization

## TL;DR

This paper introduces a Bayesian optimization method that efficiently combines simulation data and physical experiments to optimize control policies in robotics, reducing the number of costly physical trials needed.

## Contribution

It extends Entropy Search to integrate multiple information sources, enabling cost-effective policy tuning by leveraging simulations alongside physical experiments.

## Key findings

- The method finds effective control policies with fewer physical experiments.
- It successfully combines simulation and real-world data for optimization.
- Application to a cart-pole system demonstrates practical benefits.

## Abstract

In practice, the parameters of control policies are often tuned manually. This is time-consuming and frustrating. Reinforcement learning is a promising alternative that aims to automate this process, yet often requires too many experiments to be practical. In this paper, we propose a solution to this problem by exploiting prior knowledge from simulations, which are readily available for most robotic platforms. Specifically, we extend Entropy Search, a Bayesian optimization algorithm that maximizes information gain from each experiment, to the case of multiple information sources. The result is a principled way to automatically combine cheap, but inaccurate information from simulations with expensive and accurate physical experiments in a cost-effective manner. We apply the resulting method to a cart-pole system, which confirms that the algorithm can find good control policies with fewer experiments than standard Bayesian optimization on the physical system only.