Using Pretrained Large Language Model with Prompt Engineering to Answer Biomedical Questions

TL;DR

This paper presents a biomedical question-answering system leveraging pre-trained large language models with prompt engineering, evaluated on BioASQ 2024 tasks, demonstrating competitive retrieval and answering performance.

Contribution

It introduces a two-level retrieval and QA system based on prompt engineering and post-processing, comparing multiple LLMs for biomedical question answering.

Findings

Achieved 0.14 MAP on document retrieval

Achieved 0.96 F1 on yes/no questions

Demonstrated effectiveness of prompt engineering techniques

Abstract

Our team participated in the BioASQ 2024 Task12b and Synergy tasks to build a system that can answer biomedical questions by retrieving relevant articles and snippets from the PubMed database and generating exact and ideal answers. We propose a two-level information retrieval and question-answering system based on pre-trained large language models (LLM), focused on LLM prompt engineering and response post-processing. We construct prompts with in-context few-shot examples and utilize post-processing techniques like resampling and malformed response detection. We compare the performance of various pre-trained LLM models on this challenge, including Mixtral, OpenAI GPT and Llama2. Our best-performing system achieved 0.14 MAP score on document retrieval, 0.05 MAP score on snippet retrieval, 0.96 F1 score for yes/no questions, 0.38 MRR score for factoid questions and 0.50 F1 score for list questions in Task 12b.