Autonomous Goal Detection and Cessation in Reinforcement Learning: A Case Study on Source Term Estimation

TL;DR

This paper introduces the AGDC module that enables reinforcement learning algorithms to autonomously detect and stop goals in environments with limited feedback, demonstrated through improved performance on source term estimation tasks.

Contribution

The paper presents a novel AGDC module that enhances RL algorithms with self-feedback for autonomous goal detection and cessation, especially in environments with sparse feedback signals.

Findings

AGDC significantly improves success rate in source term estimation.

Enhanced RL algorithms outperform traditional statistical methods.

AGDC reduces search time and traveled distance in experiments.

Abstract

Reinforcement Learning has revolutionized decision-making processes in dynamic environments, yet it often struggles with autonomously detecting and achieving goals without clear feedback signals. For example, in a Source Term Estimation problem, the lack of precise environmental information makes it challenging to provide clear feedback signals and to define and evaluate how the source's location is determined. To address this challenge, the Autonomous Goal Detection and Cessation (AGDC) module was developed, enhancing various RL algorithms by incorporating a self-feedback mechanism for autonomous goal detection and cessation upon task completion. Our method effectively identifies and ceases undefined goals by approximating the agent's belief, significantly enhancing the capabilities of RL algorithms in environments with limited feedback. To validate effectiveness of our approach, we integrated AGDC with deep Q-Network, proximal policy optimization, and deep deterministic policy gradient algorithms, and evaluated its performance on the Source Term Estimation problem. The experimental results showed that AGDC-enhanced RL algorithms significantly outperformed traditional statistical methods such as infotaxis, entrotaxis, and dual control for exploitation and exploration, as well as a non-statistical random action selection method. These improvements were evident in terms of success rate, mean traveled distance, and search time, highlighting AGDC's effectiveness and efficiency in complex, real-world scenarios.