Predicting potentially abusive clauses in Chilean terms of services with natural language processing

TL;DR

This paper introduces a new dataset and methodology for detecting and classifying potentially abusive clauses in Chilean Terms of Service using NLP, addressing language and jurisdiction gaps in legal analysis.

Contribution

It presents the first Spanish-language multi-label classification dataset for legal clauses and evaluates transformer models for this task in the Chilean legal context.

Findings

Transformer models achieve up to 89% macro-F1 in detection.

Detection tasks have higher performance than classification.

Language-specific pre-training improves model effectiveness.

Abstract

This study addresses the growing concern of information asymmetry in consumer contracts, exacerbated by the proliferation of online services with complex Terms of Service that are rarely even read. Even though research on automatic analysis methods is conducted, the problem is aggravated by the general focus on English-language Machine Learning approaches and on major jurisdictions, such as the European Union. We introduce a new methodology and a substantial dataset addressing this gap. We propose a novel annotation scheme with four categories and a total of 20 classes, and apply it on 50 online Terms of Service used in Chile. Our evaluation of transformer-based models highlights how factors like language- and/or domain-specific pre-training, few-shot sample size, and model architecture affect the detection and classification of potentially abusive clauses. Results show a large variability in performance for the different tasks and models, with the highest macro-F1 scores for the detection task ranging from 79% to 89% and micro-F1 scores up to 96%, while macro-F1 scores for the classification task range from 60% to 70% and micro-F1 scores from 64% to 80%. Notably, this is the first Spanish-language multi-label classification dataset for legal clauses, applying Chilean law and offering a comprehensive evaluation of Spanish-language models in the legal domain. Our work lays the ground for future research in method development for rarely considered legal analysis and potentially leads to practical applications to support consumers in Chile and Latin America as a whole.