Do Charge Prediction Models Learn Legal Theory?

TL;DR

This paper investigates whether current charge prediction models in legal AI understand and incorporate legal theories, proposing principles for trustworthy models and evaluating existing models against these principles.

Contribution

It introduces a framework to assess if charge prediction models learn legal theories and evaluates existing models, revealing gaps in sensitivity and presumption of innocence.

Findings

Models meet the selective principle on benchmark data.

Most models lack sensitivity to legal theories.

Many models do not uphold presumption of innocence.

Abstract

The charge prediction task aims to predict the charge for a case given its fact description. Recent models have already achieved impressive accuracy in this task, however, little is understood about the mechanisms they use to perform the judgment.For practical applications, a charge prediction model should conform to the certain legal theory in civil law countries, as under the framework of civil law, all cases are judged according to certain local legal theories. In China, for example, nearly all criminal judges make decisions based on the Four Elements Theory (FET).In this paper, we argue that trustworthy charge prediction models should take legal theories into consideration, and standing on prior studies in model interpretation, we propose three principles for trustworthy models should follow in this task, which are sensitive, selective, and presumption of innocence.We further design a new framework to evaluate whether existing charge prediction models learn legal theories. Our findings indicate that, while existing charge prediction models meet the selective principle on a benchmark dataset, most of them are still not sensitive enough and do not satisfy the presumption of innocence. Our code and dataset are released at https://github.com/ZhenweiAn/EXP_LJP.