Transformer-based Language Models for Reasoning in the Description Logic ALCQ

TL;DR

This paper evaluates the reasoning capabilities of transformer-based language models, including fine-tuned DeBERTa and GPT-3.5/4, on a large, complex dataset derived from the expressive description logic ALCQ, revealing their potential and limitations.

Contribution

It introduces DELTA$_D$, a large and complex dataset for reasoning in ALCQ, and systematically assesses transformer models' reasoning abilities on this challenging benchmark.

Findings

DeBERTa fine-tuned on DELTA$_D$ masters entailment checking.

GPT-3.5 and GPT-4 improve with few-shot prompting.

Models show promising reasoning skills but face challenges with increased complexity.

Abstract

Recent advancements in transformer-based language models have sparked research into their logical reasoning capabilities. Most of the benchmarks used to evaluate these models are simple: generated from short (fragments of) first-order logic sentences with only a few logical operators and quantifiers. We construct the natural language dataset, DELTA$_D$, using the expressive description logic language $\mathcal{ALCQ}$. DELTA$_D$ comprises 384K examples and increases in two dimensions: i) reasoning depth, and ii) linguistic complexity. In this way, we systematically investigate the logical reasoning capabilities of a supervised fine-tuned DeBERTa-based model and two large language models (GPT-3.5, GPT-4) with few-shot prompting. We show that the DeBERTa-based model fine-tuned on our dataset can master the entailment checking task. Moreover, the performance of GPTs can improve significantly even when a small number of samples is provided (9 shots). We open-source our code and datasets.