COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding
Xiaozhe Li, Tianyi Lyu, Siyi Yang, Yizhao Yang, Yuxi Gong, Jinxuan Huang, Ligao Zhang, Zhuoyi Huang, Qingwen Liu

TL;DR
COIN-BENCH is a new benchmark designed to evaluate large language models on their ability to understand and synthesize collective human intent from complex, multi-source discussions in real-world scenarios.
Contribution
This paper introduces COIN-BENCH, a hierarchical, real-world benchmark with a novel evaluation pipeline for assessing LLMs' collective intent understanding capabilities.
Findings
Current models handle surface-level aggregation well.
Models struggle with deep causal reasoning.
COIN-BENCH sets new standards for intent analysis evaluation.
Abstract
Understanding human intent is a high-level cognitive challenge for Large Language Models (LLMs), requiring sophisticated reasoning over noisy, conflicting, and non-linear discourse. While LLMs excel at following individual instructions, their ability to distill Collective Intent - the process of extracting consensus, resolving contradictions, and inferring latent trends from multi-source public discussions - remains largely unexplored. To bridge this gap, we introduce COIN-BENCH, a dynamic, real-world, live-updating benchmark specifically designed to evaluate LLMs on collective intent understanding within the consumer domain. Unlike traditional benchmarks that focus on transactional outcomes, COIN-BENCH operationalizes intent as a hierarchical cognitive structure, ranging from explicit scenarios to deep causal reasoning. We implement a robust evaluation pipeline that combines a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Explainable Artificial Intelligence (XAI) · AI in Service Interactions
