A strong baseline for question relevancy ranking

TL;DR

This paper introduces a simple, fast, language-independent baseline model for question relevancy ranking that outperforms complex systems and even Google search rankings in shared tasks.

Contribution

The authors propose a multi-task feed forward network using 14 distance measures as features, providing a strong, efficient baseline for question relevancy ranking.

Findings

Outperforms state-of-the-art shared task systems

Faster training with simple features

Surpasses Google search rankings in relevancy retrieval

Abstract

The best systems at the SemEval-16 and SemEval-17 community question answering shared tasks -- a task that amounts to question relevancy ranking -- involve complex pipelines and manual feature engineering. Despite this, many of these still fail at beating the IR baseline, i.e., the rankings provided by Google's search engine. We present a strong baseline for question relevancy ranking by training a simple multi-task feed forward network on a bag of 14 distance measures for the input question pair. This baseline model, which is fast to train and uses only language-independent features, outperforms the best shared task systems on the task of retrieving relevant previously asked questions.