Enhancing News Summarization with ELearnFit through Efficient In-Context Learning and Efficient Fine-Tuning

TL;DR

This paper introduces ELearnFit, a combined approach of in-context learning and parameter-efficient fine-tuning for news summarization using large language models, demonstrating improved performance and practical optimization techniques.

Contribution

The paper presents a novel combination of ELearn and EFit methods, showing that fine-tuning the first layer and increasing prompt shots enhances news summarization quality.

Findings

Increasing shots in prompts improves summary quality.

Fine-tuning the first layer yields better results than other layers.

ELearnFit outperforms individual ELearn or EFit models.

Abstract

With the deluge of information delivered by the daily news cycle, there is a growing need to effectively and efficiently summarize news feeds for quick consumption. We leverage large language models (LLMs), with their advanced learning and generative abilities as compared to conventional language models, to generate concise and coherent summaries for news articles from the XSum dataset. Our paper focuses on two key aspects of LLMs: Efficient in-context Learning (ELearn) and Parameter Efficient Fine-tuning (EFit). Under ELearn, we find that increasing the number of shots in prompts and utilizing simple templates generally improve the quality of summaries. We also find that utilizing relevant examples in few-shot learning for ELearn does not improve model performance. In addition, we studied EFit using different methods and demonstrate that fine-tuning the first layer of LLMs produces better outcomes as compared to fine-tuning other layers or utilizing LoRA. We also find that leveraging more relevant training samples using selective layers does not result in better performance. By combining ELearn and EFit, we create a new model (ELearnFit) that leverages the benefits of both few-shot learning and fine-tuning and produces superior performance to either model alone. We also use ELearnFit to highlight the trade-offs between prompting and fine-tuning, especially for situations where only a limited number of annotated samples are available. Ultimately, our research provides practical techniques to optimize news summarization during the prompting and fine-tuning stages and enhances the synthesis of news articles.