Top-down Activity Representation Learning for Video Question Answering

TL;DR

This paper introduces a novel top-down activity representation learning method for VideoQA, converting long-term videos into spatial images and fine-tuning multimodal models to better capture hierarchical activities and contextual events.

Contribution

It proposes a new approach that leverages spatial image domain conversion and fine-tuning of multimodal models to improve VideoQA performance on complex hierarchical activities.

Findings

Achieved 78.4% accuracy on the STAR task, surpassing previous state-of-the-art by 2.8 points.

Effectively captures non-continuous contextual events in videos.

Demonstrates the benefit of converting long-term video sequences into spatial images for VideoQA.

Abstract

Capturing complex hierarchical human activities, from atomic actions (e.g., picking up one present, moving to the sofa, unwrapping the present) to contextual events (e.g., celebrating Christmas) is crucial for achieving high-performance video question answering (VideoQA). Recent works have expanded multimodal models (e.g., CLIP, LLaVA) to process continuous video sequences, enhancing the model's temporal reasoning capabilities. However, these approaches often fail to capture contextual events that can be decomposed into multiple atomic actions non-continuously distributed over relatively long-term sequences. In this paper, to leverage the spatial visual context representation capability of the CLIP model for obtaining non-continuous visual representations in terms of contextual events in videos, we convert long-term video sequences into a spatial image domain and finetune the multimodal model LLaVA for the VideoQA task. Our approach achieves competitive performance on the STAR task, in particular, with a 78.4% accuracy score, exceeding the current state-of-the-art score by 2.8 points on the NExTQA task.