Empirical Analysis of Dialogue Relation Extraction with Large Language Models

TL;DR

This paper evaluates large language models for dialogue relation extraction, demonstrating that scaling up models significantly improves performance and helps address challenges like long, sparse multi-turn information and partial dialogue understanding.

Contribution

It provides an empirical analysis of LLMs in DRE, showing their effectiveness in overcoming key challenges and outperforming existing methods in various settings.

Findings

Scaling up models boosts DRE performance.

LLMs have smaller performance drops from full to partial dialogues.

LLMs outperform current state-of-the-art in both full-shot and few-shot settings.

Abstract

Dialogue relation extraction (DRE) aims to extract relations between two arguments within a dialogue, which is more challenging than standard RE due to the higher person pronoun frequency and lower information density in dialogues. However, existing DRE methods still suffer from two serious issues: (1) hard to capture long and sparse multi-turn information, and (2) struggle to extract golden relations based on partial dialogues, which motivates us to discover more effective methods that can alleviate the above issues. We notice that the rise of large language models (LLMs) has sparked considerable interest in evaluating their performance across diverse tasks. To this end, we initially investigate the capabilities of different LLMs in DRE, considering both proprietary models and open-source models. Interestingly, we discover that LLMs significantly alleviate two issues in existing DRE methods. Generally, we have following findings: (1) scaling up model size substantially boosts the overall DRE performance and achieves exceptional results, tackling the difficulty of capturing long and sparse multi-turn information; (2) LLMs encounter with much smaller performance drop from entire dialogue setting to partial dialogue setting compared to existing methods; (3) LLMs deliver competitive or superior performances under both full-shot and few-shot settings compared to current state-of-the-art; (4) LLMs show modest performances on inverse relations but much stronger improvements on general relations, and they can handle dialogues of various lengths especially for longer sequences.