An empirical evaluation of using ChatGPT to summarize disputes for recommending similar labor and employment cases in Chinese

TL;DR

This study evaluates the effectiveness of using GPT-3.5 and GPT-4 generated dispute summaries to improve a hybrid case similarity recommendation system for Chinese labor and employment litigations.

Contribution

It introduces a hybrid classification approach that incorporates GPT-generated dispute summaries, demonstrating improved performance over previous methods using court-prepared disputes.

Findings

GPT-4 generated disputes improved classification results

The hybrid approach outperformed previous cluster-only systems

ChatGPT-generated disputes yielded satisfactory classification performance

Abstract

We present a hybrid mechanism for recommending similar cases of labor and employment litigations. The classifier determines the similarity based on the itemized disputes of the two cases, that the courts prepared. We cluster the disputes, compute the cosine similarity between the disputes, and use the results as the features for the classification tasks. Experimental results indicate that this hybrid approach outperformed our previous system, which considered only the information about the clusters of the disputes. We replaced the disputes that were prepared by the courts with the itemized disputes that were generated by GPT-3.5 and GPT-4, and repeated the same experiments. Using the disputes generated by GPT-4 led to better results. Although our classifier did not perform as well when using the disputes that the ChatGPT generated, the results were satisfactory. Hence, we hope that the future large-language models will become practically useful.