SemEval-2015 Task 3: Answer Selection in Community Question Answering

TL;DR

This paper presents SemEval-2015 Task 3, a challenge on answer selection in community question answering, involving classification and yes/no answering tasks across English and Arabic domains, with a large labeled dataset and multiple participating systems.

Contribution

It introduces a new benchmark for answer selection in cQA with multilingual data, crowdsourced annotations, and a competitive evaluation framework.

Findings

Best systems achieved macro F1 scores of 57.19 and 63.7 in English subtasks.

Arabic subtask A achieved a macro F1 score of 78.55.

Thirteen teams participated, demonstrating the challenge's research interest.

Abstract

Community Question Answering (cQA) provides new interesting research directions to the traditional Question Answering (QA) field, e.g., the exploitation of the interaction between users and the structure of related posts. In this context, we organized SemEval-2015 Task 3 on "Answer Selection in cQA", which included two subtasks: (a) classifying answers as "good", "bad", or "potentially relevant" with respect to the question, and (b) answering a YES/NO question with "yes", "no", or "unsure", based on the list of all answers. We set subtask A for Arabic and English on two relatively different cQA domains, i.e., the Qatar Living website for English, and a Quran-related website for Arabic. We used crowdsourcing on Amazon Mechanical Turk to label a large English training dataset, which we released to the research community. Thirteen teams participated in the challenge with a total of 61 submissions: 24 primary and 37 contrastive. The best systems achieved an official score (macro-averaged F1) of 57.19 and 63.7 for the English subtasks A and B, and 78.55 for the Arabic subtask A.