VASTU: Value-Aligned Social Toolkit for Online Content Curation

TL;DR

VASTU is a benchmark dataset and evaluation framework for detecting community-valued content in online social communities, demonstrating that community-specific models, especially fine-tuned transformers, outperform global models in content curation tasks.

Contribution

Introduces VASTU, a comprehensive benchmark with a dataset and evaluation framework for community-specific content valuation, enabling systematic comparison of detection approaches.

Findings

Community-specific models outperform global models.

Fine-tuned transformers achieve the highest AUROC (0.72).

Chain-of-thought prompting offers no benefit in this task.

Abstract

Detecting what content communities value is a foundational challenge for social computing systems -- from feed curation and content ranking to moderation tools and personalized recommendation systems. Yet existing approaches remain fragmented across methodological paradigms, and it remains unclear which methods best capture community-specific notions of value. We introduce VASTU (Value-Aligned Social Toolkit for Online Content Curation), a benchmark and evaluation framework for systematically comparing approaches to detecting community-valued content. VASTU includes a dataset of 75,000 comments from 15 diverse Reddit communities, annotated with community approval labels and rich linguistic features. Using VASTU, we evaluate feature-based models, transformers, prompted and fine-tuned language models under global versus community-specific training regimes. We find that community-specific models consistently outperform global approaches, with fine-tuned transformers achieving the strongest performance (0.72 AUROC). Notably, fine-tuned SLMs (0.65 AUROC) substantially outperform prompted LLMs (0.60 AUROC) despite being 100 times smaller. Counterintuitively, chain-of-thought prompting provides no benefit, and reasoning models perform the worst (0.53 AUROC), suggesting this task requires learning community norms rather than test-time reasoning. By releasing VASTU, we provide a standardized benchmark to advance research on value-aligned sociotechnical systems.