Can LLMs Extract Frame-Semantic Arguments?

TL;DR

This paper evaluates large language models' ability to extract frame-semantic arguments, highlighting the importance of input formats, model size, and fine-tuning, while identifying current limitations in out-of-domain generalization.

Contribution

It provides a comprehensive analysis of LLMs for frame-semantic argument extraction, introduces a novel frame identification method, and demonstrates the impact of input representation and model size.

Findings

JSON representations improve performance

Larger models generally perform better

Fine-tuning smaller models yields competitive results

Abstract

Frame-semantic parsing is a critical task in natural language understanding, yet the ability of large language models (LLMs) to extract frame-semantic arguments remains underexplored. This paper presents a comprehensive evaluation of LLMs on frame-semantic argument identification, analyzing the impact of input representation formats, model architectures, and generalization to unseen and out-of-domain samples. Our experiments, spanning models from 0.5B to 78B parameters, reveal that JSON-based representations significantly enhance performance, and while larger models generally perform better, smaller models can achieve competitive results through fine-tuning. We also introduce a novel approach to frame identification leveraging predicted frame elements, achieving state-of-the-art performance on ambiguous targets. Despite strong generalization capabilities, our analysis finds that LLMs still struggle with out-of-domain data.