Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement

TL;DR

This study compares aggregate and personalized judging approaches for evaluating business ideas with expert disagreement, finding personalized judges better match individual evaluator judgments and suggesting new evaluation methods.

Contribution

It introduces a dataset of expert scores and demonstrates that personalized judges outperform aggregate models in aligning with individual evaluator judgments.

Findings

Personalized judges align more closely with individual evaluator scores.

Expert disagreement is higher on fine-grained scores but lower on coarse selections.

Evaluator agreement correlates with similarity of reasoning only in personalized models.

Abstract

Evaluating LLM-generated business ideas is often harder to scale than generating them. Unlike standard NLP benchmarks, business idea evaluation relies on multi-dimensional criteria such as feasibility, novelty, differentiation, user need, and market size, and expert judgments often disagree. This paper studies a methodological question raised by such disagreement: should an automatic judge approximate an aggregate consensus, or model evaluators individually? We introduce PBIG-DATA, a dataset of approximately 3,000 individual scores across 300 patent-grounded product ideas, provided by domain experts on six business-oriented dimensions: specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Analyses show substantial expert disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection, suggesting structured heterogeneity rather than random noise. We then compare three judge configurations: a rubric-only zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator's scoring history. Across dimensions and model sizes, personalized judges align more closely with the corresponding evaluator than aggregate judges, and evaluator agreement correlates with similarity of judge-generated reasoning only under personalized conditioning. These results indicate that pooled labels can be a fragile target in pluralistic evaluation settings and motivate evaluator-conditioned judge designs for business idea assessment.