Identifying Experts in Question & Answer Portals: A Case Study on Data Science Competencies in Reddit

TL;DR

This paper presents a semi-supervised method for identifying data science experts, non-experts, and out-of-scope comments on Reddit, utilizing NLP and user features, with a novel coding approach and user classification.

Contribution

It introduces a novel manual coding scheme including out-of-scope comments and a semi-supervised model combining labeled and unlabeled data for expert identification.

Findings

NLP and user features are most effective for classification

Model generalizes well within the domain

Different user types in Reddit are identified

Abstract

The irreplaceable key to the triumph of Question & Answer (Q&A) platforms is their users providing high-quality answers to the challenging questions posted across various topics of interest. From more than a decade, the expert finding problem attracted much attention in information retrieval research. Based on the encountered gaps in the expert identification across several Q&A portals, we inspect the feasibility of identifying data science experts in Reddit. Our method is based on the manual coding results where two data science experts labelled not only expert and non-expert comments, but also out-of-scope comments, which is a novel contribution to the literature, enabling the identification of more groups of comments across web portals. We present a semi-supervised approach which combines 1,113 labelled comments with 100,226 unlabelled comments during training. The proposed model uses the activity behaviour of every user, including Natural Language Processing (NLP), crowdsourced and user feature sets. We conclude that the NLP and user feature sets contribute the most to the better identification of these three classes. It means that this method can generalise well within the domain. Finally, we make a novel contribution by presenting different types of users in Reddit, which opens many future research directions.