SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses

TL;DR

This paper introduces SUPERGLASSES, a real-world VQA benchmark for smart glasses, and proposes SUPERLENS, a retrieval-augmented multimodal agent that significantly improves performance.

Contribution

It provides the first comprehensive, real-world VQA dataset for smart glasses and develops SUPERLENS, a novel multimodal agent that outperforms existing models.

Findings

26 VLMs evaluated reveal significant performance gaps.

SUPERLENS achieves state-of-the-art results, surpassing GPT-4o by 2.19%.

The dataset and benchmark are publicly available for further research.

Abstract

The rapid advancement of AI-powered smart glasses-one of the hottest wearable devices-has unlocked new frontiers for multimodal interaction, with Visual Question Answering (VQA) over external knowledge sources emerging as a core application. Existing Vision Language Models (VLMs) adapted to smart glasses are typically trained and evaluated on traditional multimodal datasets; however, these datasets lack the variety and realism needed to reflect smart glasses usage scenarios and diverge from their specific challenges, where accurately identifying the object of interest must precede any external knowledge retrieval. To bridge this gap, we introduce SUPER- GLASSES, the first comprehensive VQA benchmark built on real-world data entirely collected by smart glasses devices. SUPERGLASSES comprises 2,422 egocentric image-question pairs spanning 14 image domains and 8 query categories, enriched with full search trajectories and reasoning annotations. We evaluate 26 representative VLMs on this benchmark, revealing significant performance gaps. To address the limitations of existing models, we further propose the SUPERLENS, a multimodal smart glasses agent that enables retrieval-augmented answer generation by integrating automatic object detection, query decoupling, and multimodal web search. SUPERLENS achieves state-of-the-art performance, outperforming GPT-4o by 2.19%, underscoring the need for task-specific solutions in smart glasses VQA. Our dataset is publicly available at https://huggingface.co/datasets/xandery/SuperGlasses.