Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis

TL;DR

This paper presents a novel method using fine-tuned large language models with Retrieval Augmented Generation to automate and improve the accuracy of scientific meta-analyses, reducing human effort and error.

Contribution

The study introduces a fine-tuning approach with a new loss metric and prompt engineering to enhance LLM performance in automated meta-analysis tasks.

Findings

Fine-tuned LLMs generate 87.6% relevant meta-analysis content.

Relevancy irrelevance reduced from 4.56% to 1.9%.

Effective in low-resource environments.

Abstract

This study investigates the automation of meta-analysis in scientific documents using large language models (LLMs). Meta-analysis is a robust statistical method that synthesizes the findings of multiple studies support articles to provide a comprehensive understanding. We know that a meta-article provides a structured analysis of several articles. However, conducting meta-analysis by hand is labor-intensive, time-consuming, and susceptible to human error, highlighting the need for automated pipelines to streamline the process. Our research introduces a novel approach that fine-tunes the LLM on extensive scientific datasets to address challenges in big data handling and structured data extraction. We automate and optimize the meta-analysis process by integrating Retrieval Augmented Generation (RAG). Tailored through prompt engineering and a new loss metric, Inverse Cosine Distance (ICD), designed for fine-tuning on large contextual datasets, LLMs efficiently generate structured meta-analysis content. Human evaluation then assesses relevance and provides information on model performance in key metrics. This research demonstrates that fine-tuned models outperform non-fine-tuned models, with fine-tuned LLMs generating 87.6% relevant meta-analysis abstracts. The relevance of the context, based on human evaluation, shows a reduction in irrelevancy from 4.56% to 1.9%. These experiments were conducted in a low-resource environment, highlighting the study's contribution to enhancing the efficiency and reliability of meta-analysis automation.