Report on the First Knowledge Graph Reasoning Challenge 2018 -- Toward the eXplainable AI System

TL;DR

This paper summarizes the 2018 Knowledge Graph Reasoning Challenge focused on explainable AI, highlighting various reasoning techniques, evaluation metrics, and results in the context of Sherlock Holmes story analysis.

Contribution

It presents the first challenge on knowledge graph reasoning for explainability, including methodology, evaluation, and diverse approaches proposed by participants.

Findings

Constraint satisfaction approach won first prize

SPARQL and rules approach was highly effective

Word embeddings and multi-agent models were innovative solutions

Abstract

A new challenge for knowledge graph reasoning started in 2018. Deep learning has promoted the application of artificial intelligence (AI) techniques to a wide variety of social problems. Accordingly, being able to explain the reason for an AI decision is becoming important to ensure the secure and safe use of AI techniques. Thus, we, the Special Interest Group on Semantic Web and Ontology of the Japanese Society for AI, organized a challenge calling for techniques that reason and/or estimate which characters are criminals while providing a reasonable explanation based on an open knowledge graph of a well-known Sherlock Holmes mystery story. This paper presents a summary report of the first challenge held in 2018, including the knowledge graph construction, the techniques proposed for reasoning and/or estimation, the evaluation metrics, and the results. The first prize went to an approach that formalized the problem as a constraint satisfaction problem and solved it using a lightweight formal method; the second prize went to an approach that used SPARQL and rules; the best resource prize went to a submission that constructed word embedding of characters from all sentences of Sherlock Holmes novels; and the best idea prize went to a discussion multi-agents model. We conclude this paper with the plans and issues for the next challenge in 2019.