Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models

TL;DR

This paper introduces a retrieval-augmented framework with tool calling for large language models to improve medication consultation accuracy, addressing hallucinations and knowledge gaps in medical AI applications.

Contribution

It proposes a novel Distill-Retrieve-Read framework utilizing tool calling, enhancing evidence retrieval in medical LLMs over previous methods.

Findings

Improved evidence retrieval accuracy in medical question-answering.

The new framework outperforms previous retrieval methods.

Demonstrated effectiveness on the MedicineQA benchmark.

Abstract

Large-scale language models (LLMs) have achieved remarkable success across various language tasks but suffer from hallucinations and temporal misalignment. To mitigate these shortcomings, Retrieval-augmented generation (RAG) has been utilized to provide external knowledge to facilitate the answer generation. However, applying such models to the medical domain faces several challenges due to the lack of domain-specific knowledge and the intricacy of real-world scenarios. In this study, we explore LLMs with RAG framework for knowledge-intensive tasks in the medical field. To evaluate the capabilities of LLMs, we introduce MedicineQA, a multi-round dialogue benchmark that simulates the real-world medication consultation scenario and requires LLMs to answer with retrieved evidence from the medicine database. MedicineQA contains 300 multi-round question-answering pairs, each embedded within a detailed dialogue history, highlighting the challenge posed by this knowledge-intensive task to current LLMs. We further propose a new \textit{Distill-Retrieve-Read} framework instead of the previous \textit{Retrieve-then-Read}. Specifically, the distillation and retrieval process utilizes a tool calling mechanism to formulate search queries that emulate the keyword-based inquiries used by search engines. With experimental results, we show that our framework brings notable performance improvements and surpasses the previous counterparts in the evidence retrieval process in terms of evidence retrieval accuracy. This advancement sheds light on applying RAG to the medical domain.