Semantic Answer Type and Relation Prediction Task (SMART 2021)

TL;DR

The paper introduces the SMART 2021 challenge focusing on answer type and relation prediction for knowledge base question answering, providing datasets, evaluation metrics, and insights to advance semantic web research.

Contribution

It presents the second edition of the SMART challenge, detailing task definitions, datasets, and evaluation methods for answer type and relation prediction in KBQA.

Findings

Established benchmark datasets for answer type and relation prediction.

Defined evaluation metrics for the SMART challenge.

Facilitated progress in semantic web question answering systems.

Abstract

Each year the International Semantic Web Conference organizes a set of Semantic Web Challenges to establish competitions that will advance state-of-the-art solutions in some problem domains. The Semantic Answer Type and Relation Prediction Task (SMART) task is one of the ISWC 2021 Semantic Web challenges. This is the second year of the challenge after a successful SMART 2020 at ISWC 2020. This year's version focuses on two sub-tasks that are very important to Knowledge Base Question Answering (KBQA): Answer Type Prediction and Relation Prediction. Question type and answer type prediction can play a key role in knowledge base question answering systems providing insights about the expected answer that are helpful to generate correct queries or rank the answer candidates. More concretely, given a question in natural language, the first task is, to predict the answer type using a target ontology (e.g., DBpedia or Wikidata. Similarly, the second task is to identify relations in the natural language query and link them to the relations in a target ontology. This paper discusses the task descriptions, benchmark datasets, and evaluation metrics. For more information, please visit https://smart-task.github.io/2021/.