Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

TL;DR

Sci-LoRA is a novel model that dynamically combines multiple domain-specific LoRAs to improve cross-domain lay paraphrasing, making scientific information accessible across diverse fields without explicit domain labels.

Contribution

It introduces a mixture of LoRAs approach with dynamic weighting for multi-domain scientific paraphrasing, enhancing adaptability and performance without requiring domain annotations.

Findings

Outperforms state-of-the-art large language models in twelve domains

Demonstrates flexible generalization across multiple scientific fields

Achieves significant improvements on five public datasets

Abstract

Lay paraphrasing aims to make scientific information accessible to audiences without technical backgrounds. However, most existing studies focus on a single domain, such as biomedicine. With the rise of interdisciplinary research, it is increasingly necessary to comprehend knowledge spanning multiple technical fields. To address this, we propose Sci-LoRA, a model that leverages a mixture of LoRAs fine-tuned on multiple scientific domains. In particular, Sci-LoRA dynamically generates and applies weights for each LoRA, enabling it to adjust the impact of different domains based on the input text, without requiring explicit domain labels. To balance domain-specific knowledge and generalization across various domains, Sci-LoRA integrates information at both the data and model levels. This dynamic fusion enhances the adaptability and performance across various domains. Experimental results across twelve domains on five public datasets show that Sci-LoRA significantly outperforms state-of-the-art large language models and demonstrates flexible generalization and adaptability in cross-domain lay paraphrasing.