Few-shot Vision-based Human Activity Recognition with MLLM-based Visual Reinforcement Learning

TL;DR

This paper introduces FAVOR, a few-shot human activity recognition method using multimodal large language models and visual reinforcement learning, significantly improving generalization and explainability in limited data scenarios.

Contribution

It extends reinforcement learning to multimodal large language models for human activity recognition, enhancing few-shot learning, reasoning, and interpretability.

Findings

Outperforms existing methods on four HAR datasets

Improves few-shot recognition accuracy

Enables explainable activity inference

Abstract

Reinforcement learning in large reasoning models enables learning from feedback on their outputs, making it particularly valuable in scenarios where fine-tuning data is limited. However, its application in multi-modal human activity recognition (HAR) domains remains largely underexplored. Our work extends reinforcement learning to the human activity recognition domain with multimodal large language models. By incorporating visual reinforcement learning in the training process, the model's generalization ability on few-shot recognition can be greatly improved. Additionally, visual reinforcement learning can enhance the model's reasoning ability and enable explainable analysis in the inference stage. We name our few-shot human activity recognition method with visual reinforcement learning FAVOR. Specifically, our approach first utilizes a multimodal large language model (MLLM) to generate multiple candidate responses for the human activity image, each containing reasoning traces and final answers. These responses are then evaluated using reward functions, and the MLLM model is subsequently optimized using the Group Relative Policy Optimization (GRPO) algorithm. In this way, the MLLM model can be adapted to human activity recognition with only a few samples. Extensive experiments on four human activity recognition datasets and five different settings demonstrate the superiority of the proposed method.