From Appearance to Essence: Comparing Truth Discovery Methods without Using Ground Truth
Xiu Susie Fang, Quan Z. Sheng, Xianzhi Wang, Wei Emma Zhang, and Anne H.H. Ngu

TL;DR
This paper introduces CompTruthHyp, a novel approach to compare truth discovery methods without relying on ground truth, enabling more practical evaluation in real-world scenarios where ground truth is unavailable.
Contribution
The paper proposes a new method, CompTruthHyp, for evaluating truth discovery techniques without ground truth, and compares twelve existing methods using this approach.
Findings
CompTruthHyp effectively ranks truth discovery methods without ground truth.
The approach performs well on both real-world and synthetic datasets.
It provides a practical solution for method evaluation in real-world applications.
Abstract
Truth discovery has been widely studied in recent years as a fundamental means for resolving the conflicts in multi-source data. Although many truth discovery methods have been proposed based on different considerations and intuitions, investigations show that no single method consistently outperforms the others. To select the right truth discovery method for a specific application scenario, it becomes essential to evaluate and compare the performance of different methods. A drawback of current research efforts is that they commonly assume the availability of certain ground truth for the evaluation of methods. However, the ground truth may be very limited or even out-of-reach in practice, rendering the evaluation biased by the small ground truth or even unfeasible. In this paper, we present CompTruthHyp, a general approach for comparing the performance of truth discovery methods without…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Data Visualization and Analytics · Data-Driven Disease Surveillance
