LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs

TL;DR

This paper compares large language models and graph-based parsers for relation extraction, showing that parsers outperform LLMs as graph complexity increases, especially in complex linguistic scenarios.

Contribution

The study demonstrates that graph-based parsers outperform LLMs in relation extraction tasks involving complex linguistic graphs, highlighting their suitability for such scenarios.

Findings

Graph-based parser outperforms LLMs as relation complexity increases.

LLMs underperform on datasets with large, complex sentence graphs.

Graph parsers are more efficient for relation extraction in complex linguistic contexts.

Abstract

Relation extraction represents a fundamental component in the process of creating knowledge graphs, among other applications. Large language models (LLMs) have been adopted as a promising tool for relation extraction, both in supervised and in-context learning settings. However, in this work we show that their performance still lags behind much smaller architectures when the linguistic graph underlying a text has great complexity. To demonstrate this, we evaluate four LLMs against a graph-based parser on six relation extraction datasets with sentence graphs of varying sizes and complexities. Our results show that the graph-based parser increasingly outperforms the LLMs, as the number of relations in the input documents increases. This makes the much lighter graph-based parser a superior choice in the presence of complex linguistic graphs.