To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment

TL;DR

This paper demonstrates that zero-shot pretrained language models can outperform fine-tuned models in legal case entailment tasks, especially with limited labeled data, highlighting their robustness and transfer capabilities.

Contribution

It shows that zero-shot models can achieve top performance in legal entailment without domain-specific fine-tuning, challenging traditional adaptation approaches.

Findings

Zero-shot models outperformed fine-tuned models in COLIEE 2021.

Limited labeled data favors models with minimal adaptation.

Zero-shot models exhibit greater robustness to data distribution changes.

Abstract

There has been mounting evidence that pretrained language models fine-tuned on large and diverse supervised datasets can transfer well to a variety of out-of-domain tasks. In this work, we investigate this transfer ability to the legal domain. For that, we participated in the legal case entailment task of COLIEE 2021, in which we use such models with no adaptations to the target domain. Our submissions achieved the highest scores, surpassing the second-best team by more than six percentage points. Our experiments confirm a counter-intuitive result in the new paradigm of pretrained language models: given limited labeled data, models with little or no adaptation to the target task can be more robust to changes in the data distribution than models fine-tuned on it. Code is available at https://github.com/neuralmind-ai/coliee.