Long-distance migration with minimal energy consumption in a thermal turbulent environment

TL;DR

This paper uses reinforcement learning to train an autonomous agent to efficiently migrate long distances in turbulent thermal environments by leveraging flow currents, demonstrating energy savings and policy robustness across different flow conditions.

Contribution

The study develops a reinforcement learning approach enabling agents to utilize flow currents for energy-efficient long-distance migration in turbulent convection cells, with policies transferable to larger and more complex environments.

Findings

Smart agents save energy by utilizing flow currents.

Optimized policies extend to larger convection cells with different flow modes.

Energy savings increase with larger aspect ratio (Γ) of the environment.

Abstract

We adopt the reinforcement learning algorithm to train the self-propelling agent migrating long-distance in a thermal turbulent environment. We choose the Rayleigh-B\'enard turbulent convection cell with an aspect ratio ($\Gamma$, which is defined as the ratio between cell length and cell height) of 2 as the training environment. Our results showed that, compared to a naive agent that moves straight from the origin to the destination, the smart agent can learn to utilize the carrier flow currents to save propelling energy. We then apply the optimal policy obtained from the $\Gamma=2$ cell and test the smart agent migrating in convection cells with $\Gamma$ up to 32. In a larger $\Gamma$ cell, the dominant flow modes of horizontally stacked rolls are less stable, and the energy contained in higher-order flow modes increases. We found that the optimized policy can be successfully extended to convection cells with a larger $\Gamma$. In addition, the ratio of propelling energy consumed by the smart agent to that of the naive agent decreases with the increase of $\Gamma$, indicating more propelling energy can be saved by the smart agent in a larger $\Gamma$ cell. We also evaluate the optimized policy when the agents are being released from the randomly chosen origin, which aims to test the robustness of the learning framework, and possible solutions to improve the success rate are suggested. This work has implications for long-distance migration problems, such as unmanned aerial vehicles patrolling in a turbulent convective environment, where planning energy-efficient trajectories can be beneficial to increase their endurance.