Optimal navigation in two-dimensional flows: control theory and reinforcement learning

TL;DR

This paper explores optimal navigation strategies in complex two-dimensional fluid flows, comparing control theory and reinforcement learning methods, and demonstrating the robustness of learned policies in turbulent environments.

Contribution

It introduces reinforcement learning approaches for robust navigation in complex flows and shows their effectiveness and generalization capabilities compared to traditional control solutions.

Findings

Reinforcement learning achieves successful navigation in complex flows.

Agents trained on coarse flow data generalize well to full turbulence.

Control theory solutions become unstable in chaotic regimes.

Abstract

Zermelo's navigation problem seeks the trajectory of minimal travel time between two points in a fluid flow. We address this problem for an agent -- such as a micro-robot or active particle -- that is advected by a two-dimensional flow, self-propels at a fixed speed smaller than or comparable to the characteristic flow velocity, and can steer its direction. The flows considered span increasing levels of complexity, from steady solid-body rotation to the Taylor-Green flow and fully developed turbulence in the inverse cascade regime. Although optimal control theory provides time-minimizing trajectories, these solutions become unstable in chaotic regimes realized for complex background flows. To design robust navigation strategies under such conditions, we apply reinforcement learning. Both action-value (Q-learning) and policy-gradient (one-step actor-critic) methods achieve successful navigation with comparable performance. Crucially, we show that agents trained on coarse-grained turbulent flows -- retaining only large-scale features -- generalize effectively to the full velocity field. This robustness to incomplete flow information is essential for practical navigation in real-world oceanic and atmospheric environments.