Truth Discovery Algorithms: An Experimental Evaluation

TL;DR

This paper systematically compares 12 state-of-the-art truth discovery algorithms through extensive experiments on synthetic and real data, analyzing their efficiency, usability, and scalability to guide future research.

Contribution

It provides a comprehensive review, reference implementations, and an experimental framework for evaluating truth discovery methods under diverse scenarios.

Findings

Initialization and parameter settings significantly affect algorithm performance.

Scalability varies widely among the evaluated methods.

The experimental framework enables thorough comparison across multiple data scenarios.

Abstract

A fundamental problem in data fusion is to determine the veracity of multi-source data in order to resolve conflicts. While previous work in truth discovery has proved to be useful in practice for specific settings, sources' behavior or data set characteristics, there has been limited systematic comparison of the competing methods in terms of efficiency, usability, and repeatability. We remedy this deficit by providing a comprehensive review of 12 state-of-the art algorithms for truth discovery. We provide reference implementations and an in-depth evaluation of the methods based on extensive experiments on synthetic and real-world data. We analyze aspects of the problem that have not been explicitly studied before, such as the impact of initialization and parameter setting, convergence, and scalability. We provide an experimental framework for extensively comparing the methods in a wide range of truth discovery scenarios where source coverage, numbers and distributions of conflicts, and true positive claims can be controlled and used to evaluate the quality and performance of the algorithms. Finally, we report comprehensive findings obtained from the experiments and provide new insights for future research.