Adapting LLMs for the Medical Domain in Portuguese: A Study on Fine-Tuning and Model Evaluation

TL;DR

This paper evaluates Portuguese medical large language models, fine-tuning them with datasets translated from English, and compares their performance, highlighting challenges like catastrophic forgetting and the need for better evaluation protocols.

Contribution

It demonstrates the effectiveness of the InternLM2 model for medical tasks in Portuguese and analyzes the impact of fine-tuning methods and dataset translation on model performance.

Findings

InternLM2 achieved the best overall performance.

DrBode models showed catastrophic forgetting of medical knowledge.

Low inter-rater agreement indicates need for better evaluation methods.

Abstract

This study evaluates the performance of large language models (LLMs) as medical agents in Portuguese, aiming to develop a reliable and relevant virtual assistant for healthcare professionals. The HealthCareMagic-100k-en and MedQuAD datasets, translated from English using GPT-3.5, were used to fine-tune the ChatBode-7B model using the PEFT-QLoRA method. The InternLM2 model, with initial training on medical data, presented the best overall performance, with high precision and adequacy in metrics such as accuracy, completeness and safety. However, DrBode models, derived from ChatBode, exhibited a phenomenon of catastrophic forgetting of acquired medical knowledge. Despite this, these models performed frequently or even better in aspects such as grammaticality and coherence. A significant challenge was low inter-rater agreement, highlighting the need for more robust assessment protocols. This work paves the way for future research, such as evaluating multilingual models specific to the medical field, improving the quality of training data, and developing more consistent evaluation methodologies for the medical field.