EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering

TL;DR

EvidenceMap enables small biomedical language models to analyze evidence explicitly, improving answer quality and accuracy without large models, by learning supportive, logical, and summarization aspects of evidence.

Contribution

The paper introduces EvidenceMap, a novel approach for small models to learn evidence analysis, enhancing biomedical question answering performance.

Findings

Outperforms larger RAG models by 19.9% in reference-based quality.

Achieves 5.7% higher accuracy than baseline.

Uses only 66M parameters for fine-tuning.

Abstract

When addressing professional questions in the biomedical domain, humans typically acquire multiple pieces of information as evidence and engage in multifaceted analysis to provide high-quality answers. Current LLM-based question answering methods lack a detailed definition and learning process for evidence analysis, leading to the risk of error propagation and hallucinations while using evidence. Although increasing the parameter size of LLMs can alleviate these issues, it also presents challenges in training and deployment with limited resources. In this study, we propose EvidenceMap, which aims to enable a tiny pre-trained language model to explicitly learn multiple aspects of biomedical evidence, including supportive evaluation, logical correlation and content summarization, thereby latently guiding a small generative model (around 3B parameters) to provide textual responses. Experimental results demonstrate that our method, learning evidence analysis by fine-tuning a model with only 66M parameters, exceeds the RAG method with an 8B LLM by 19.9% and 5.7% in reference-based quality and accuracy, respectively.