Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation

TL;DR

This paper introduces a causal framework called Counterfactual Voting Adjustment (CVA) to correct biases in helpfulness voting on online platforms, improving content quality assessment and ranking accuracy.

Contribution

The paper presents CVA, a novel causal method that models and adjusts for position and herding biases in helpfulness votes, enhancing fairness and accuracy in content quality evaluation.

Findings

CVA accurately recovers true content quality in experiments.

Reranking with CVA aligns better with user sentiment and GPT-4 evaluations.

Embeddings reveal behavioral patterns of expert user groups across communities.

Abstract

Efficient access to high-quality information is vital for online platforms. To promote more useful information, users not only create new content but also evaluate existing content, often through helpfulness voting. Although aggregated votes help service providers rank their user content, these votes are often biased by disparate accessibility per position and the cascaded influence of prior votes. For a fairer assessment of information quality, we propose the Counterfactual Voting Adjustment (CVA), a causal framework that accounts for the context in which individual votes are cast. Through preliminary and semi-synthetic experiments, we show that CVA effectively models the position and herding biases, accurately recovering the predefined content quality. In a real experiment, we demonstrate that reranking content based on the learned quality by CVA exhibits stronger alignment with both user sentiment and quality evaluation assessed by GPT-4o, outperforming system rankings based on aggregated votes and model-based rerankings without causal inference. Beyond the individual quality inference, our embeddings offer comparative insights into the behavioral dynamics of expert user groups across 120 major StackExchange communities.