SuryaKiran at MEDIQA-Sum 2023: Leveraging LoRA for Clinical Dialogue Summarization

TL;DR

This paper evaluates the effectiveness of Low Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, for clinical dialogue summarization, demonstrating comparable performance to full model fine-tuning.

Contribution

It provides the first comprehensive evaluation of LoRA for clinical dialogue summarization, showing its efficiency and effectiveness compared to traditional fine-tuning methods.

Findings

LoRA performs on par with end-to-end fine-tuning.

LoRA reduces resource requirements significantly.

Effective for both Subtask A and B in ImageCLEFmedical.

Abstract

Finetuning Large Language Models helps improve the results for domain-specific use cases. End-to-end finetuning of large language models is time and resource intensive and has high storage requirements to store the finetuned version of the large language model. Parameter Efficient Fine Tuning (PEFT) methods address the time and resource challenges by keeping the large language model as a fixed base and add additional layers, which the PEFT methods finetune. This paper demonstrates the evaluation results for one such PEFT method Low Rank Adaptation (LoRA), for Clinical Dialogue Summarization. The evaluation results show that LoRA works at par with end-to-end finetuning for a large language model. The paper presents the evaluations done for solving both the Subtask A and B from ImageCLEFmedical {https://www.imageclef.org/2023/medical}