Model-free reinforcement learning with noisy actions for automated experimental control in optics

TL;DR

This paper demonstrates that model-free reinforcement learning can efficiently control complex optical systems directly through experiments, achieving high coupling efficiency faster than humans and reducing the need for detailed system modeling.

Contribution

It introduces a model-free RL approach using sample-efficient algorithms for optical system control, outperforming human experts in speed and efficiency without relying on system simulations.

Findings

RL agents achieve 90% coupling efficiency.

CrossQ outperforms other RL algorithms in speed and training time.

Direct experimental training can replace detailed system modeling.

Abstract

Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training on simulations. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC), Truncated Quantile Critics (TQC), or CrossQ, our agents learn to couple with 90% efficiency. A human expert reaches this efficiency, but the RL agents are quicker. In particular, the CrossQ agent outperforms the other agents in coupling speed while requiring only half the training time. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL's potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.