MemexQA: Visual Memex Question Answering

TL;DR

MemexQA introduces a new multimodal dataset and a unified neural network architecture for answering questions about personal photo and video collections to aid memory recall.

Contribution

The paper presents the MemexQA dataset and MemexNet model, enabling end-to-end question answering on personal multimedia collections, advancing memory-related AI applications.

Findings

MemexNet outperforms baseline models on MemexQA.

Achieves state-of-the-art results on the new dataset.

Demonstrates scalability across TextQA and VideoQA tasks.

Abstract

This paper proposes a new task, MemexQA: given a collection of photos or videos from a user, the goal is to automatically answer questions that help users recover their memory about events captured in the collection. Towards solving the task, we 1) present the MemexQA dataset, a large, realistic multimodal dataset consisting of real personal photos and crowd-sourced questions/answers, 2) propose MemexNet, a unified, end-to-end trainable network architecture for image, text and video question answering. Experimental results on the MemexQA dataset demonstrate that MemexNet outperforms strong baselines and yields the state-of-the-art on this novel and challenging task. The promising results on TextQA and VideoQA suggest MemexNet's efficacy and scalability across various QA tasks.