Comprehensive Evaluation for a Large Scale Knowledge Graph Question Answering Service

TL;DR

This paper presents Chronos, a scalable and comprehensive evaluation framework for large-scale knowledge graph question answering systems, addressing industry-specific challenges and enabling data-driven improvements.

Contribution

The paper introduces Chronos, a novel evaluation framework tailored for industry-scale KGQA systems, capable of assessing multiple components and guiding system enhancements.

Findings

Chronos effectively evaluates KGQA systems at scale.

It supports diverse datasets and pre-release performance measurement.

Provides insights for data-driven system improvements.

Abstract

Question answering systems for knowledge graph (KGQA), answer factoid questions based on the data in the knowledge graph. KGQA systems are complex because the system has to understand the relations and entities in the knowledge-seeking natural language queries and map them to structured queries against the KG to answer them. In this paper, we introduce Chronos, a comprehensive evaluation framework for KGQA at industry scale. It is designed to evaluate such a multi-component system comprehensively, focusing on (1) end-to-end and component-level metrics, (2) scalable to diverse datasets and (3) a scalable approach to measure the performance of the system prior to release. In this paper, we discuss the unique challenges associated with evaluating KGQA systems at industry scale, review the design of Chronos, and how it addresses these challenges. We will demonstrate how it provides a base for data-driven decisions and discuss the challenges of using it to measure and improve a real-world KGQA system.