Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge

TL;DR

This paper explores the use of multimodal large language models for real-time episodic memory question answering on edge devices, balancing privacy, latency, and accuracy.

Contribution

It introduces an asynchronous pipeline integrating streaming constraints for edge-based memory question answering with promising experimental results.

Findings

Edge implementation achieves 51.76% accuracy on a consumer GPU.

Scaling to enterprise hardware improves accuracy to 54.40%.

Edge solutions are competitive with cloud-based approaches.

Abstract

We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering. While cloud offloading is common, it raises privacy and latency concerns for wearable assistants, hence we investigate implementation on the edge. We integrated streaming constraints into our question answering pipeline, which is structured into two asynchronous threads: a Descriptor Thread that continuously converts video into a lightweight textual memory, and a Question Answering (QA) Thread that reasons over the textual memory to answer queries. Experiments on the QAEgo4D-Closed benchmark analyze the performance of Multimodal Large Language Models (MLLMs) within strict resource boundaries, showing promising results also when compared to clound-based solutions. Specifically, an end-to-end configuration running on a consumer-grade 8GB GPU achieves 51.76% accuracy with a Time-To-First-Token (TTFT) of 0.41s. Scaling to a local enterprise-grade server yields 54.40% accuracy with a TTFT of 0.88s. In comparison, a cloud-based solution obtains an accuracy of 56.00%. These competitive results highlight the potential of edge-based solutions for privacy-preserving episodic memory retrieval.