ReSPEC: A Framework for Online Multispectral Sensor Reconfiguration in Dynamic Environments

TL;DR

ReSPEC introduces an adaptive framework that dynamically reconfigures multispectral sensors in robotic systems using reinforcement learning, optimizing resource use while maintaining perception accuracy in changing environments.

Contribution

The paper presents a novel integrated framework combining sensing, learning, and actuation for real-time sensor reconfiguration based on environmental conditions.

Findings

Reduces GPU load by 29.3%

Maintains 94.7% of baseline accuracy

Demonstrates effective resource-aware sensing

Abstract

Multi-sensor fusion is central to robust robotic perception, yet most existing systems operate under static sensor configurations, collecting all modalities at fixed rates and fidelity regardless of their situational utility. This rigidity wastes bandwidth, computation, and energy, and prevents systems from prioritizing sensors under challenging conditions such as poor lighting or occlusion. Recent advances in reinforcement learning (RL) and modality-aware fusion suggest the potential for adaptive perception, but prior efforts have largely focused on re-weighting features at inference time, ignoring the physical cost of sensor data collection. We introduce a framework that unifies sensing, learning, and actuation into a closed reconfiguration loop. A task-specific detection backbone extracts multispectral features (e.g. RGB, IR, mmWave, depth) and produces quantitative contribution scores for each modality. These scores are passed to an RL agent, which dynamically adjusts sensor configurations, including sampling frequency, resolution, sensing range, and etc., in real time. Less informative sensors are down-sampled or deactivated, while critical sensors are sampled at higher fidelity as environmental conditions evolve. We implement and evaluate this framework on a mobile rover, showing that adaptive control reduces GPU load by 29.3\% with only a 5.3\% accuracy drop compared to a heuristic baseline. These results highlight the potential of resource-aware adaptive sensing for embedded robotic platforms.