Modeling Proficiency with Implicit User Representations

TL;DR

This paper proposes a new problem of proficiency modeling on social media, aiming to identify users' expertise levels across topics using unsupervised user embeddings, which can improve content filtering and ranking.

Contribution

It introduces the proficiency modeling problem, explores five different user embedding approaches, and evaluates them on two real-world benchmarks.

Findings

Advanced user modeling improves proficiency prediction accuracy.

Unsupervised embeddings effectively capture user engagement levels.

Multiple approaches offer varying trade-offs in proficiency estimation.

Abstract

We introduce the problem of proficiency modeling: Given a user's posts on a social media platform, the task is to identify the subset of posts or topics for which the user has some level of proficiency. This enables the filtering and ranking of social media posts on a given topic as per user proficiency. Unlike experts on a given topic, proficient users may not have received formal training and possess years of practical experience, but may be autodidacts, hobbyists, and people with sustained interest, enabling them to make genuine and original contributions to discourse. While predicting whether a user is an expert on a given topic imposes strong constraints on who is a true positive, proficiency modeling implies a graded scoring, relaxing these constraints. Put another way, many active social media users can be assumed to possess, or eventually acquire, some level of proficiency on topics relevant to their community. We tackle proficiency modeling in an unsupervised manner by utilizing user embeddings to model engagement with a given topic, as indicated by a user's preference for authoring related content. We investigate five alternative approaches to model proficiency, ranging from basic ones to an advanced, tailored user modeling approach, applied within two real-world benchmarks for evaluation.