A comment-driven evidence appraisal approach for decision-making when only uncertain evidence available

TL;DR

This study investigates using comments from scientific publications as a quick, reliable tool for evidence appraisal in decision-making, especially when evidence is uncertain or incomplete, demonstrated through COVID-19 drug analysis.

Contribution

It introduces evidence-comment networks (ECNs) as a novel method to leverage comments for evidence validation, showing early and aligned insights with official guidelines.

Findings

Comment-derived evidence assertions matched WHO guidelines for 5 out of 6 drugs.

Comment sentiment aligned with clinical guideline recommendations.

Half of critical comments appeared 4.5 months before guidelines were published.

Abstract

Purpose: To explore whether comments could be used as an assistant tool for heuristic decision-making, especially in cases where missing, incomplete, uncertain, or even incorrect evidence is acquired. Methods: Six COVID-19 drug candidates were selected from WHO clinical guidelines. Evidence-comment networks (ECNs) were completed of these six drug candidates based on evidence-comment pairs from all PubMed indexed COVID-19 publications with formal published comments. WHO guidelines were utilized to validate the feasibility of comment-derived evidence assertions as a fast decision supporting tool. Results: Out of 6 drug candidates, comment-derived evidence assertions of leading subgraphs of 5 drugs were consistent with WHO guidelines, and the overall comment sentiment of 6 drugs was aligned with WHO clinical guidelines. Additionally, comment topics were in accordance with the concerns of guidelines and evidence appraisal criteria. Furthermore, half of the critical comments emerged 4.5 months earlier than the date guidelines were published. Conclusions: Comment-derived evidence assertions have the potential as an evidence appraisal tool for heuristic decisions based on the accuracy, sensitivity, and efficiency of evidence-comment networks. In essence, comments reflect that academic communities do have a self-screening evaluation and self-purification (argumentation) mechanism, thus providing a tool for decision makers to filter evidence.