Aligned Textual Scoring Rules
Yuxuan Lu, Yifan Wu, Jason Hartline, Michael J. Curry

TL;DR
This paper introduces the Aligned Scoring Rule (ASR), a novel method for eliciting human-preferred textual predictions by optimizing scoring rules to better align with human judgments while ensuring properness.
Contribution
The paper proposes the Aligned Scoring Rule (ASR), which improves alignment with human preferences in textual scoring by minimizing mean squared error with a reference score.
Findings
ASR outperforms previous methods in aligning with human preferences.
ASR maintains properness while improving alignment.
Experimental results demonstrate the effectiveness of ASR in real-world settings.
Abstract
Scoring rules elicit probabilistic predictions from a strategic agent by scoring the prediction against a ground truth state. A scoring rule is proper if, from the agent's perspective, reporting the true belief maximizes the expected score. With the development of language models, Wu and Hartline (2024) proposes a reduction from textual information elicitation to the numerical (i.e. probabilistic) information elicitation problem, which achieves provable properness for textual elicitation. However, not all proper scoring rules are well aligned with human preference over text. Our paper designs the Aligned Scoring rule (ASR) for text by optimizing and minimizing the mean squared error between a proper scoring rule and a reference score (e.g. human score). Our experiments show that our ASR outperforms previous methods in aligning with human preference while maintaining properness.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Sentiment Analysis and Opinion Mining · Natural Language Processing Techniques
