Do Subjectivity and Objectivity Always Agree? A Case Study with Stack Overflow Questions

TL;DR

This study investigates the alignment between subjective user votes and objective quality metrics on Stack Overflow questions, revealing discrepancies and proposing machine learning models that improve question quality classification.

Contribution

The paper compares subjective and objective quality assessments on Stack Overflow and develops machine learning models that outperform existing approaches in classifying questions.

Findings

Four objective metrics agree with subjective evaluations.

Two metrics do not agree with subjective evaluations.

Machine learning models achieve 76%-87% accuracy in classifying questions.

Abstract

In Stack Overflow (SO), the quality of posts (i.e., questions and answers) is subjectively evaluated by users through a voting mechanism. The net votes (upvotes - downvotes) obtained by a post are often considered an approximation of its quality. However, about half of the questions that received working solutions got more downvotes than upvotes. Furthermore, about 18% of the accepted answers (i.e., verified solutions) also do not score the maximum votes. All these counter-intuitive findings cast doubts on the reliability of the evaluation mechanism employed at SO. Moreover, many users raise concerns against the evaluation, especially downvotes to their posts. Therefore, rigorous verification of the subjective evaluation is highly warranted to ensure a non-biased and reliable quality assessment mechanism. In this paper, we compare the subjective assessment of questions with their objective assessment using 2.5 million questions and ten text analysis metrics. According to our investigation, four objective metrics agree with the subjective evaluation, two do not agree, one either agrees or disagrees, and the remaining three neither agree nor disagree with the subjective evaluation. We then develop machine learning models to classify the promoted and discouraged questions. Our models outperform the state-of-the-art models with a maximum of about 76% - 87% accuracy.