Do LLMs Really Adapt to Domains? An Ontology Learning Perspective

TL;DR

This paper investigates whether large language models truly adapt to specific domains or merely learn lexical senses, revealing that they often rely on senses rather than reasoning, but fine-tuning enhances their domain adaptation capabilities.

Contribution

It introduces a controlled experiment using WordNet to differentiate between reasoning and sense learning in LLMs for ontology learning tasks.

Findings

LLMs leverage lexical senses rather than reasoning over semantic relationships.

Fine-tuning improves LLM performance on lexical semantic tasks with domain-specific data.

Pre-trained LLMs can be adapted for ontology learning through fine-tuning.

Abstract

Large Language Models (LLMs) have demonstrated unprecedented prowess across various natural language processing tasks in various application domains. Recent studies show that LLMs can be leveraged to perform lexical semantic tasks, such as Knowledge Base Completion (KBC) or Ontology Learning (OL). However, it has not effectively been verified whether their success is due to their ability to reason over unstructured or semi-structured data, or their effective learning of linguistic patterns and senses alone. This unresolved question is particularly crucial when dealing with domain-specific data, where the lexical senses and their meaning can completely differ from what a LLM has learned during its training stage. This paper investigates the following question: Do LLMs really adapt to domains and remain consistent in the extraction of structured knowledge, or do they only learn lexical senses instead of reasoning? To answer this question and, we devise a controlled experiment setup that uses WordNet to synthesize parallel corpora, with English and gibberish terms. We examine the differences in the outputs of LLMs for each corpus in two OL tasks: relation extraction and taxonomy discovery. Empirical results show that, while adapting to the gibberish corpora, off-the-shelf LLMs do not consistently reason over semantic relationships between concepts, and instead leverage senses and their frame. However, fine-tuning improves the performance of LLMs on lexical semantic tasks even when the domain-specific terms are arbitrary and unseen during pre-training, hinting at the applicability of pre-trained LLMs for OL.