Are LLM-based methods good enough for detecting unfair terms of service?

TL;DR

This study evaluates the effectiveness of large language models in detecting unfair terms in online privacy policies, revealing that current models perform only marginally better than random chance and require significant improvements.

Contribution

The paper introduces a new dataset of questions on privacy policies and benchmarks multiple LLMs, including ChatGPT, for their ability to identify unfair terms.

Findings

Open-source models outperform some commercial models.

ChatGPT4 achieves the highest accuracy among tested models.

All models perform only slightly better than random chance.

Abstract

Countless terms of service (ToS) are being signed everyday by users all over the world while interacting with all kinds of apps and websites. More often than not, these online contracts spanning double-digit pages are signed blindly by users who simply want immediate access to the desired service. What would normally require a consultation with a legal team, has now become a mundane activity consisting of a few clicks where users potentially sign away their rights, for instance in terms of their data privacy, to countless online entities/companies. Large language models (LLMs) are good at parsing long text-based documents, and could potentially be adopted to help users when dealing with dubious clauses in ToS and their underlying privacy policies. To investigate the utility of existing models for this task, we first build a dataset consisting of 12 questions applied individually to a set of privacy policies crawled from popular websites. Thereafter, a series of open-source as well as commercial chatbots such as ChatGPT, are queried over each question, with the answers being compared to a given ground truth. Our results show that some open-source models are able to provide a higher accuracy compared to some commercial models. However, the best performance is recorded from a commercial chatbot (ChatGPT4). Overall, all models perform only slightly better than random at this task. Consequently, their performance needs to be significantly improved before they can be adopted at large for this purpose.