Can VLMs be used on videos for action recognition? LLMs are Visual Reasoning Coordinators

TL;DR

This paper explores using a coordinated system of vision-language models and large language models to recognize actions in surveillance videos with limited temporal information, demonstrating promising results but highlighting the need for stronger temporal cues.

Contribution

It introduces a novel framework where LLMs coordinate multiple VLMs for action recognition in videos, extending previous commonsense reasoning capabilities to temporal video analysis.

Findings

LLMs can effectively coordinate VLMs for action recognition in videos.

The approach performs well with minimal temporal information.

Enhancing temporal signals could improve accuracy.

Abstract

Recent advancements have introduced multiple vision-language models (VLMs) demonstrating impressive commonsense reasoning across various domains. Despite their individual capabilities, the potential of synergizing these complementary VLMs remains underexplored. The Cola Framework addresses this by showcasing how a large language model (LLM) can efficiently coordinate multiple VLMs through natural language communication, leveraging their distinct strengths. We have verified this claim on the challenging A-OKVQA dataset, confirming the effectiveness of such coordination. Building on this, our study investigates whether the same methodology can be applied to surveillance videos for action recognition. Specifically, we explore if leveraging the combined knowledge base of VLMs and LLM can effectively deduce actions from a video when presented with only a few selectively important frames and minimal temporal information. Our experiments demonstrate that LLM, when coordinating different VLMs, can successfully recognize patterns and deduce actions in various scenarios despite the weak temporal signals. However, our findings suggest that to enhance this approach as a viable alternative solution, integrating a stronger temporal signal and exposing the models to slightly more frames would be beneficial.