BioACE: An Automated Framework for Biomedical Answer and Citation Evaluations

TL;DR

BioACE is an automated framework designed to evaluate the quality of biomedical answers and citations generated by large language models, considering multiple aspects like correctness and completeness, and validated against human assessments.

Contribution

This work introduces BioACE, a novel automated evaluation framework for biomedical answer and citation quality, integrating multiple assessment metrics and extensive experimental validation.

Findings

BioACE effectively correlates with human evaluations.

Automated methods outperform some existing approaches.

The framework supports comprehensive biomedical answer assessment.

Abstract

With the increasing use of large language models (LLMs) for generating answers to biomedical questions, it is crucial to evaluate the quality of the generated answers and the references provided to support the facts in the generated answers. Evaluation of text generated by LLMs remains a challenge for question answering, retrieval-augmented generation (RAG), summarization, and many other natural language processing tasks in the biomedical domain, due to the requirements of expert assessment to verify consistency with the scientific literature and complex medical terminology. In this work, we propose BioACE, an automated framework for evaluating biomedical answers and citations against the facts stated in the answers. The proposed BioACE framework considers multiple aspects, including completeness, correctness, precision, and recall, in relation to the ground-truth nuggets for answer evaluation. We developed automated approaches to evaluate each of the aforementioned aspects and performed extensive experiments to assess and analyze their correlation with human evaluations. In addition, we considered multiple existing approaches, such as natural language inference (NLI) and pre-trained language models and LLMs, to evaluate the quality of evidence provided to support the generated answers in the form of citations into biomedical literature. With the detailed experiments and analysis, we provide the best approaches for biomedical answer and citation evaluation as a part of BioACE (https://github.com/deepaknlp/BioACE) evaluation package.