Unsupervised Question Duplicate and Related Questions Detection in e-learning platforms

TL;DR

This paper introduces QDup, an unsupervised tool that efficiently detects near-duplicate and related questions in e-learning platforms, aiding in maintaining diverse and non-redundant question repositories.

Contribution

The paper presents a novel unsupervised hybrid approach combining statistical and neural methods for question duplicate detection without supervised data.

Findings

QDup accurately detects near-duplicate questions.

QDup efficiently suggests related questions from large repositories.

The tool operates with remarkable speed and accuracy.

Abstract

Online learning platforms provide diverse questions to gauge the learners' understanding of different concepts. The repository of questions has to be constantly updated to ensure a diverse pool of questions to conduct assessments for learners. However, it is impossible for the academician to manually skim through the large repository of questions to check for duplicates when onboarding new questions from external sources. Hence, we propose a tool QDup in this paper that can surface near-duplicate and semantically related questions without any supervised data. The proposed tool follows an unsupervised hybrid pipeline of statistical and neural approaches for incorporating different nuances in similarity for the task of question duplicate detection. We demonstrate that QDup can detect near-duplicate questions and also suggest related questions for practice with remarkable accuracy and speed from a large repository of questions. The demo video of the tool can be found at https://www.youtube.com/watch?v=loh0_-7XLW4.