Are Fact-Checking Tools Helpful? An Exploration of the Usability of Google Fact Check

TL;DR

This study evaluates Google Fact Check's ability to retrieve relevant fact-checking results for COVID-19 misinformation, revealing limited coverage but reliable results, and suggests input wording adjustments to improve retrieval effectiveness.

Contribution

It provides an empirical analysis of Google Fact Check's performance and usability in retrieving fact-checks for COVID-19 claims, highlighting areas for optimization.

Findings

Google Fact Check retrieved 15.8% of claims.

Retrieved results are relatively reliable.

Claim wording affects retrieval outcomes.

Abstract

Fact-checking-specific search tools such as Google Fact Check are a promising way to combat misinformation on social media, especially during events bringing significant social influence, such as the COVID-19 pandemic and the U.S. presidential elections. However, the usability of such an approach has not been thoroughly studied. We evaluated the performance of Google Fact Check by analyzing the retrieved fact-checking results regarding 1,000 COVID-19-related false claims and found it able to retrieve the fact-checking results for 15.8% of the input claims, and the rendered results are relatively reliable. We also found that the false claims receiving different fact-checking verdicts (i.e., "False," "Partly False," "True," and "Unratable") tend to reflect diverse emotional tones, and fact-checking sources tend to check the claims in different lengths and using dictionary words to various extents. Claim variations addressing the same issue yet described differently are likely to retrieve distinct fact-checking results. We suggest that the quantities of the retrieved fact-checking results could be optimized and that slightly adjusting input wording may be the best practice for users to retrieve more useful information. This study aims to contribute to the understanding of state-of-the-art fact-checking tools and information integrity.