Development and validation of a tool for detecting misinformation risk in diet, nutrition, and health content (Diet-MisRAT)

TL;DR

This paper introduces Diet-MisRAT, a tool to assess the risk of misinformation in diet and health content, validated by experts and AI.

Contribution

The novel contribution is the development and validation of Diet-MisRAT, a graded risk assessment tool for diet-related misinformation.

Findings

Diet-MisRAT evaluates content across four risk dimensions and provides five-tier risk estimates.

Validation showed strong alignment with expert benchmarks and high performance by ChatGPT in risk detection.

The tool offers a scalable alternative to binary misinformation detection.

Abstract

Misinformation in diet and nutrition is recognised as a major public health threat, with the potential to misguide dietary choices and contribute to preventable harm. To address this, we developed a Misinformation Risk Assessment Model (MisRAM), grounded in the World Health Organization’s hazard risk assessment principles. MisRAM conceptualises misleading content traits as stratifiable agents of informational adverse effects, weighed by their severity and likelihood of increasing recipient susceptibility. Building on this model, we designed the Diet-Nutrition Misinformation Risk Assessment Tool (Diet-MisRAT), a structured instrument that evaluates medium-to-long form content across four risk dimensions (inaccuracy, incompleteness, deceptiveness, health harm), yielding five-tier risk estimates from very low to very high. Validation involved five rounds: expert reviewers, trainee dietitians, postgraduate nutrition students, highly experienced nutrition professionals, and zero-shot prompt-based generative-AI risk detection. Results showed strong to very strong alignment with expert-derived benchmarks, supporting the tool’s interpretability and concurrent validity. ChatGPT demonstrated high test–retest reliability, accuracy, precision, sensitivity, and F1 scores under blinded untuned conditions, suggesting that adequately constructed, expert-designed prompting tools may help overcome training-dataset limitations. Diet-MisRAT offers a scalable, graded alternative to binary detection. Domain-calibrated risk stratification could guide proportionate interventions in content oversight, regulation, education, misinformation inoculation, and infodemic mitigation.

The online version contains supplementary material available at 10.1038/s41598-026-40534-2.

Figures