Exploring Language Model Generalization in Low-Resource Extractive QA

TL;DR

This paper examines how large language models perform in zero-shot extractive question answering across different domains like medicine and law, revealing limitations in handling domain-specific knowledge and dataset demands.

Contribution

It provides empirical insights into the challenges LLMs face in domain generalization for extractive QA, highlighting the impact of dataset characteristics and model scaling.

Findings

LLMs struggle with long answer spans in closed domains

Weaknesses in word sense discrimination affect performance

Scaling models does not always improve cross-domain generalization

Abstract

In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize to domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to explain the performance gap empirically. Our findings suggest that: (a) LLMs struggle with dataset demands of closed domains such as retrieving long answer spans; (b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; (c) Scaling model parameters is not always effective for cross domain generalization; and (d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.