ContextLens: Modeling Imperfect Privacy and Safety Context for Legal Compliance

TL;DR

ContextLens is a semi-rule-based framework using LLMs to improve legal compliance assessment by grounding context and identifying known and unknown factors, especially in ambiguous real-world situations.

Contribution

It introduces a novel semi-rule-based approach that leverages LLMs to explicitly identify and reason about incomplete and ambiguous legal contexts for compliance.

Findings

Significantly improves LLMs' compliance assessment accuracy.

Outperforms existing baselines without requiring training.

Effectively identifies ambiguous and missing context factors.

Abstract

Individuals' concerns about data privacy and AI safety are highly contextualized and extend beyond sensitive patterns. Addressing these issues requires reasoning about the context to identify and mitigate potential risks. Though researchers have widely explored using large language models (LLMs) as evaluators for contextualized safety and privacy assessments, these efforts typically assume the availability of complete and clear context, whereas real-world contexts tend to be ambiguous and incomplete. In this paper, we propose ContextLens, a semi-rule-based framework that leverages LLMs to ground the input context in the legal domain and explicitly identify both known and unknown factors for legal compliance. Instead of directly assessing safety outcomes, our ContextLens instructs LLMs to answer a set of crafted questions that span over applicability, general principles and detailed provisions to assess compliance with pre-defined priorities and rules. We conduct extensive experiments on existing compliance benchmarks that cover the General Data Protection Regulation (GDPR) and the EU AI Act. The results suggest that our ContextLens can significantly improve LLMs' compliance assessment and surpass existing baselines without any training. Additionally, our ContextLens can further identify the ambiguous and missing factors.