When to Localize? A Risk-Constrained Reinforcement Learning Approach

TL;DR

This paper introduces RiskRL, a reinforcement learning approach that helps robots decide when to localize to balance localization costs and failure risks, improving success rates in unseen environments.

Contribution

It proposes a novel risk-constrained RL framework using particle filtering and Soft Actor-Critic to optimize localization timing without full model knowledge.

Findings

RiskRL increases success rates by at least 26% in unseen environments.

It effectively balances localization costs and failure risks.

The method outperforms previous POMDP-based approaches in efficiency.

Abstract

In a standard navigation pipeline, a robot localizes at every time step to lower navigational errors. However, in some scenarios, a robot needs to selectively localize when it is expensive to obtain observations. For example, an underwater robot surfacing to localize too often hinders it from searching for critical items underwater, such as black boxes from crashed aircraft. On the other hand, if the robot never localizes, poor state estimates cause failure to find the items due to inadvertently leaving the search area or entering hazardous, restricted areas. Motivated by these scenarios, we investigate approaches to help a robot determine "when to localize?" We formulate this as a bi-criteria optimization problem: minimize the number of localization actions while ensuring the probability of failure (due to collision or not reaching a desired goal) remains bounded. In recent work, we showed how to formulate this active localization problem as a constrained Partially Observable Markov Decision Process (POMDP), which was solved using an online POMDP solver. However, this approach is too slow and requires full knowledge of the robot transition and observation models. In this paper, we present RiskRL, a constrained Reinforcement Learning (RL) framework that overcomes these limitations. RiskRL uses particle filtering and recurrent Soft Actor-Critic network to learn a policy that minimizes the number of localizations while ensuring the probability of failure constraint is met. Our numerical experiments show that RiskRL learns a robust policy that leads to at least a 26% increase in success rates when traversing unseen test environments.