GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearables

TL;DR

GLIMPSE introduces a hybrid approach for real-time text recognition and visual question answering on wearables, balancing high-resolution OCR with low-resolution streaming to save power while maintaining accuracy.

Contribution

The paper presents a novel hybrid architecture that performs selective high-resolution OCR on-device and low-resolution visual streaming, enabling efficient VQA on resource-limited wearables.

Findings

Achieves 72% accuracy on text VQA benchmark

Uses only 49% of the power of full-resolution streaming

Maintains coherent temporal context in real-time processing

Abstract

Video Large Language Models (Video LLMs) have shown remarkable progress in understanding and reasoning about visual content, particularly in tasks involving text recognition and text-based visual question answering (Text VQA). However, deploying Text VQA on wearable devices faces a fundamental tension: text recognition requires high-resolution video, but streaming high-quality video drains battery and causes thermal throttling. Moreover, existing models struggle to maintain coherent temporal context when processing text across multiple frames in real-time streams. We observe that text recognition and visual reasoning have asymmetric resolution requirements - OCR needs fine detail while scene understanding tolerates coarse features. We exploit this asymmetry with a hybrid architecture that performs selective high-resolution OCR on-device while streaming low-resolution video for visual context. On a benchmark of text-based VQA samples across five task categories, our system achieves 72% accuracy at 0.49x the power consumption of full-resolution streaming, enabling sustained VQA sessions on resource-constrained wearables without sacrificing text understanding quality.