Steering undulatory micro-swimmers in a fluid flow through reinforcement learning

TL;DR

This paper explores how reinforcement learning can be used to develop navigation strategies for undulatory micro-swimmers in complex fluid flows, addressing challenges of non-Markovian dynamics and chaos.

Contribution

It introduces a novel approach using multiple Q-learning runs to find effective policies for microswimmer navigation in non-homogeneous flows.

Findings

Standard methods often fail to converge due to chaotic dynamics.

Multiple independent Q-learning runs can produce robust, efficient navigation policies.

The approach allows detailed analysis of policy properties and robustness.

Abstract

This work aims at finding optimal navigation policies for thin, deformable microswimmers that progress in a viscous fluid by propagating a sinusoidal undulation along their slender body. These active filaments are embedded in a prescribed, non-homogeneous flow, in which their swimming undulations have to compete with the drifts, strains, and deformations inflicted by the outer velocity field. Such an intricate situation, where swimming and navigation are tightly bonded, is addressed using various methods of reinforcement learning. Each swimmer has only access to restricted information on its configuration and has to select accordingly an action among a limited set. The optimisation problem then consists in finding the policy leading to the most efficient displacement in a given direction. It is found that usual methods do not converge and this pitfall is interpreted as a combined consequence of the non-Markovianity of the decision process, together with the highly chaotic nature of the dynamics, which is responsible for high variability in learning efficiencies. Still, we provide an alternative method to construct efficient policies, which is based on running several independent realisations of Q-learning. This allows the construction of a set of admissible policies whose properties can be studied in detail and compared to assess their efficiency and robustness.