Learning to Select Camera Views: Efficient Multiview Understanding at Few Glances

TL;DR

This paper introduces MVSelect, a reinforcement learning-based method for selecting optimal camera views in multiview systems, reducing computational costs while maintaining high performance in classification and detection tasks.

Contribution

The paper presents a novel view selection approach that jointly trains with task networks, enabling efficient multiview understanding with fewer views and insights into camera layout optimization.

Findings

Achieves high accuracy with only 2-3 views out of N

Reduces computational costs significantly

Identifies cameras that can be turned off with minimal performance loss

Abstract

Multiview camera setups have proven useful in many computer vision applications for reducing ambiguities, mitigating occlusions, and increasing field-of-view coverage. However, the high computational cost associated with multiple views poses a significant challenge for end devices with limited computational resources. To address this issue, we propose a view selection approach that analyzes the target object or scenario from given views and selects the next best view for processing. Our approach features a reinforcement learning based camera selection module, MVSelect, that not only selects views but also facilitates joint training with the task network. Experimental results on multiview classification and detection tasks show that our approach achieves promising performance while using only 2 or 3 out of N available views, significantly reducing computational costs. Furthermore, analysis on the selected views reveals that certain cameras can be shut off with minimal performance impact, shedding light on future camera layout optimization for multiview systems. Code is available at https://github.com/hou-yz/MVSelect.