LLM-IE: A Python Package for Generative Information Extraction with Large Language Models

TL;DR

LLM-IE is a Python package that simplifies biomedical information extraction using large language models, featuring an interactive agent for schema and prompt design, and demonstrating strong performance on benchmark datasets.

Contribution

The paper introduces LLM-IE, a novel Python toolkit that streamlines the development of biomedical information extraction pipelines with an interactive LLM agent for schema and prompt management.

Findings

Sentence-based prompting achieves best performance

System evaluation includes intuitive visualization

Package is adopted in internal healthcare NLP projects

Abstract

Objectives: Despite the recent adoption of large language models (LLMs) for biomedical information extraction, challenges in prompt engineering and algorithms persist, with no dedicated software available. To address this, we developed LLM-IE: a Python package for building complete information extraction pipelines. Our key innovation is an interactive LLM agent to support schema definition and prompt design. Materials and Methods: The LLM-IE supports named entity recognition, entity attribute extraction, and relation extraction tasks. We benchmarked on the i2b2 datasets and conducted a system evaluation. Results: The sentence-based prompting algorithm resulted in the best performance while requiring a longer inference time. System evaluation provided intuitive visualization. Discussion: LLM-IE was designed from practical NLP experience in healthcare and has been adopted in internal projects. It should hold great value to the biomedical NLP community. Conclusion: We developed a Python package, LLM-IE, that provides building blocks for robust information extraction pipeline construction.