On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education

TL;DR

This paper evaluates the capabilities of open-source foundational LLMs in German legal education, highlighting their strengths, limitations, and proposing a retrieval-augmented approach to enhance legal analysis tasks.

Contribution

It introduces a retrieval-augmented generation method to improve LLM performance in legal analysis and evaluates their effectiveness on argument mining and essay scoring tasks.

Findings

LLMs possess sufficient instruction and legal background knowledge for some educational tasks.

Performance drops significantly in complex legal analysis tasks and with complete legal opinions.

Retrieval-augmented prompting substantially improves predictions in high data scenarios.

Abstract

We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case.