Assessing the Performance of Chinese Open Source Large Language Models in Information Extraction Tasks

TL;DR

This study evaluates the zero-shot and few-shot performance of Chinese open-source large language models on information extraction tasks, comparing them with ChatGPT to identify strengths and limitations.

Contribution

It provides a comprehensive analysis of Chinese open-source LLMs in IE tasks, highlighting their capabilities and gaps compared to ChatGPT under zero-shot and few-shot settings.

Findings

Chinese open-source LLMs show promising zero-shot IE performance

Few-shot experiments improve model accuracy significantly

Open-source models still lag behind ChatGPT in IE tasks

Abstract

Information Extraction (IE) plays a crucial role in Natural Language Processing (NLP) by extracting structured information from unstructured text, thereby facilitating seamless integration with various real-world applications that rely on structured data. Despite its significance, recent experiments focusing on English IE tasks have shed light on the challenges faced by Large Language Models (LLMs) in achieving optimal performance, particularly in sub-tasks like Named Entity Recognition (NER). In this paper, we delve into a comprehensive investigation of the performance of mainstream Chinese open-source LLMs in tackling IE tasks, specifically under zero-shot conditions where the models are not fine-tuned for specific tasks. Additionally, we present the outcomes of several few-shot experiments to further gauge the capability of these models. Moreover, our study includes a comparative analysis between these open-source LLMs and ChatGPT, a widely recognized language model, on IE performance. Through meticulous experimentation and analysis, we aim to provide insights into the strengths, limitations, and potential enhancements of existing Chinese open-source LLMs in the domain of Information Extraction within the context of NLP.