TechGPT-2.0: A large language model project to solve the task of knowledge graph construction

TL;DR

TechGPT-2.0 is a large language model tailored for knowledge graph construction, offering enhanced domain capabilities and robustness, with open-source weights and detailed training procedures for research use.

Contribution

Introduces TechGPT-2.0, a large language model optimized for knowledge graph tasks, with domain-specific enhancements and open-source resources for the Chinese NLP community.

Findings

Robust performance in NER and RTE tasks across multiple domains

Enhanced handling of hallucinations and lengthy texts

Open-source model weights and detailed training process

Abstract

Large language models have exhibited robust performance across diverse natural language processing tasks. This report introduces TechGPT-2.0, a project designed to enhance the capabilities of large language models specifically in knowledge graph construction tasks, including named entity recognition (NER) and relationship triple extraction (RTE) tasks in NLP applications. Additionally, it serves as a LLM accessible for research within the Chinese open-source model community. We offer two 7B large language model weights and a QLoRA weight specialized for processing lengthy texts.Notably, TechGPT-2.0 is trained on Huawei's Ascend server. Inheriting all functionalities from TechGPT-1.0, it exhibits robust text processing capabilities, particularly in the domains of medicine and law. Furthermore, we introduce new capabilities to the model, enabling it to process texts in various domains such as geographical areas, transportation, organizations, literary works, biology, natural sciences, astronomical objects, and architecture. These enhancements also fortified the model's adeptness in handling hallucinations, unanswerable queries, and lengthy texts. This report provides a comprehensive and detailed introduction to the full fine-tuning process on Huawei's Ascend servers, encompassing experiences in Ascend server debugging, instruction fine-tuning data processing, and model training. Our code is available at https://github.com/neukg/TechGPT-2.0