"How trustworthy is this research?" Designing a Tool to Help Readers Understand Evidence and Uncertainty in Science Journalism

TL;DR

This research designs and evaluates a tool that helps science news readers assess the strength and uncertainty of scientific evidence, aiming to improve understanding and differentiate science from pseudoscience.

Contribution

It introduces the Scientific Evidence Indicator, a novel tool using publication metadata to communicate evidence strength in science journalism.

Findings

Helped readers recognize peer-reviewed studies

Demonstrated some success in conveying evidence strength

Identified challenges in fostering in-depth understanding

Abstract

This article reports on a Research through Design study exploring how to design a tool for helping readers of science journalism understand the strength and uncertainty of scientific evidence in news stories about health science, using both textual and visual information. A central aim has been to teach readers about criteria for assessing scientific evidence, in particular in order to help readers differentiate between science and pseudoscience. Working in a research-in-the-wild collaboration with a website for popular science, the study presents the design and evaluation of the Scientific Evidence Indicator, which uses metadata about scientific publications to present an assessment of evidence strength to the readers. Evaluations of the design demonstrate some success in helping readers recognize whether studies have undergone scientific peer review or not, but point to challenges in facilitating a more in-depth understanding. Insights from the study point to a potential for developing similar tools aimed at journalists rather than directly at audiences.