IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognition

TL;DR

IDSelect is a reinforcement learning-based system that adaptively chooses the most efficient models per modality for video-based person recognition, significantly reducing computational costs while maintaining or improving accuracy.

Contribution

It introduces a novel RL-based cost-aware selector that dynamically chooses models per modality, surpassing fixed ensembles in efficiency and accuracy.

Findings

Achieves 95.9% accuracy with 92.4% less computation on CCVID.

Reduces computation by 41.3% on MEVID while maintaining performance.

Outperforms strong baselines in efficiency and accuracy.

Abstract

Video-based person recognition achieves robust identification by integrating face, body, and gait. However, current systems waste computational resources by processing all modalities with fixed heavyweight ensembles regardless of input complexity. To address these limitations, we propose IDSelect, a reinforcement learning-based cost-aware selector that chooses one pre-trained model per modality per-sequence to optimize the accuracy-efficiency trade-off. Our key insight is that an input-conditioned selector can discover complementary model choices that surpass fixed ensembles while using substantially fewer resources. IDSelect trains a lightweight agent end-to-end using actor-critic reinforcement learning with budget-aware optimization. The reward balances recognition accuracy with computational cost, while entropy regularization prevents premature convergence. At inference, the policy selects the most probable model per modality and fuses modality-specific similarities for the final score. Extensive experiments on challenging video-based datasets demonstrate IDSelect's superior efficiency: on CCVID, it achieves 95.9% Rank-1 accuracy with 92.4% less computation than strong baselines while improving accuracy by 1.8%; on MEVID, it reduces computation by 41.3% while maintaining competitive performance.