HOME-KGQA: A Benchmark Dataset for Multimodal Knowledge Graph Question Answering on Household Daily Activities

TL;DR

HOME-KGQA introduces a challenging multimodal knowledge graph question answering benchmark focused on household activities, emphasizing multi-hop, spatiotemporal, and multimodal reasoning to advance real-world AI applications.

Contribution

It presents a novel, complex KGQA dataset with multi-level reasoning and multimodal grounding, addressing limitations of existing benchmarks for real-world scenarios.

Findings

LLM-based KGQA methods perform poorly on HOME-KGQA compared to existing datasets.

HOME-KGQA includes multi-hop, spatiotemporal, and multimodal questions.

The dataset exposes significant challenges for deploying KGQA in real-world environments.

Abstract

Large Language Models (LLMs) provide flexible natural language processing capabilities, while knowledge graphs (KGs) offer explicit and structured knowledge. Integrating these two in a complementary manner enables the development of reliable and verifiable AI systems. In particular, knowledge graph question answering (KGQA) has attracted attention as a means to reduce LLM hallucinations and to leverage knowledge beyond the training data. However, existing KGQA benchmark datasets are biased toward encyclopedic knowledge, limited to a single modality, and lack fine-grained spatiotemporal data, which limits their applicability to real-world scenarios targeted by Embodied AI. We introduce HOME-KGQA, a novel KGQA benchmark dataset built on a multimodal KG of daily household activities. HOME-KGQA consists of complex, multi-hop natural language questions paired with graph database query languages. Compared to existing benchmarks, it includes more challenging questions that involve multi-level spatiotemporal reasoning, multimodal grounding, and aggregate functions. Experimental results show that the LLM-based KGQA methods fail to achieve performance comparable to that on existing datasets when evaluated on HOME-KGQA. This highlights significant challenges that should be addressed for the real-world deployment of KGQA systems. Our dataset is available at https://github.com/aistairc/home-kgqa