Grid-LOGAT: Grid Based Local and Global Area Transcription for Video Question Answering

TL;DR

Grid-LoGAT introduces a grid-based approach for extracting detailed local and global visual information from videos, enhancing VideoQA performance while maintaining privacy by processing data on edge devices and in the cloud.

Contribution

The paper presents a novel grid-based visual prompting method for VideoQA, improving transcript quality and accuracy over previous non-grid approaches.

Findings

Outperforms state-of-the-art methods on NExT-QA and STAR-QA datasets.

Achieves 65.9% and 50.11% accuracy respectively.

Surpasses non-grid version by 24 points on localization questions.

Abstract

In this paper, we propose a Grid-based Local and Global Area Transcription (Grid-LoGAT) system for Video Question Answering (VideoQA). The system operates in two phases. First, extracting text transcripts from video frames using a Vision-Language Model (VLM). Next, processing questions using these transcripts to generate answers through a Large Language Model (LLM). This design ensures image privacy by deploying the VLM on edge devices and the LLM in the cloud. To improve transcript quality, we propose grid-based visual prompting, which extracts intricate local details from each grid cell and integrates them with global information. Evaluation results show that Grid-LoGAT, using the open-source VLM (LLaVA-1.6-7B) and LLM (Llama-3.1-8B), outperforms state-of-the-art methods with similar baseline models on NExT-QA and STAR-QA datasets with an accuracy of 65.9% and 50.11% respectively. Additionally, our method surpasses the non-grid version by 24 points on localization-based questions we created using NExT-QA. (This paper is accepted by IEEE ICIP 2025.)