Optimizing Sensor Redundancy in Sequential Decision-Making Problems

TL;DR

This paper presents a method to optimize backup sensor configurations in reinforcement learning applications, balancing expected performance and costs using a quadratic approximation and Tabu Search, validated across multiple environments.

Contribution

It introduces a second-order approximation approach combined with a meta-heuristic optimization to select cost-effective sensor setups in real-world RL tasks.

Findings

Quadratic approximation closely matches actual expected returns.

Optimized sensor configurations improve performance-cost trade-offs.

Method effective across diverse simulation and robotic environments.

Abstract

Reinforcement Learning (RL) policies are designed to predict actions based on current observations to maximize cumulative future rewards. In real-world applications (i.e., non-simulated environments), sensors are essential for measuring the current state and providing the observations on which RL policies rely to make decisions. A significant challenge in deploying RL policies in real-world scenarios is handling sensor dropouts, which can result from hardware malfunctions, physical damage, or environmental factors like dust on a camera lens. A common strategy to mitigate this issue is the use of backup sensors, though this comes with added costs. This paper explores the optimization of backup sensor configurations to maximize expected returns while keeping costs below a specified threshold, C. Our approach uses a second-order approximation of expected returns and includes penalties for exceeding cost constraints. We then optimize this quadratic program using Tabu Search, a meta-heuristic algorithm. The approach is evaluated across eight OpenAI Gym environments and a custom Unity-based robotic environment (RobotArmGrasping). Empirical results demonstrate that our quadratic program effectively approximates real expected returns, facilitating the identification of optimal sensor configurations.