ConcertoRL: An Innovative Time-Interleaved Reinforcement Learning Approach for Enhanced Control in Direct-Drive Tandem-Wing Vehicles

TL;DR

ConcertoRL introduces a novel time-interleaved reinforcement learning method that combines classical controllers with RL to improve control accuracy and stability in insect-scale tandem-wing vehicles, demonstrating significant performance gains.

Contribution

The paper presents a new algorithm, ConcertoRL, which integrates classical control with reinforcement learning using a time-interleaved mechanism and experience organization for enhanced control and training stability.

Findings

Performance improved by approximately 70% over non-RL scenarios.

Achieved a 50% increase in efficiency compared to reference controllers.

Demonstrated the effectiveness of the combined control approach in experimental tests.

Abstract

In control problems for insect-scale direct-drive experimental platforms under tandem wing influence, the primary challenge facing existing reinforcement learning models is their limited safety in the exploration process and the stability of the continuous training process. We introduce the ConcertoRL algorithm to enhance control precision and stabilize the online training process, which consists of two main innovations: a time-interleaved mechanism to interweave classical controllers with reinforcement learning-based controllers aiming to improve control precision in the initial stages, a policy composer organizes the experience gained from previous learning to ensure the stability of the online training process. This paper conducts a series of experiments. First, experiments incorporating the time-interleaved mechanism demonstrate a substantial performance boost of approximately 70% over scenarios without reinforcement learning enhancements and a 50% increase in efficiency compared to reference controllers with doubled control frequencies. These results highlight the algorithm's ability to create a synergistic effect that exceeds the sum of its parts.