TL;DR
This paper introduces ACCESS DENIED INC, a benchmark environment for evaluating sensitivity awareness in large language models, aiming to improve privacy compliance in corporate data handling.
Contribution
It presents the first benchmark environment for assessing sensitivity awareness in LLMs, addressing privacy challenges beyond simple filtering methods.
Findings
Models show significant variation in handling unauthorized data requests.
Sensitivity awareness can improve privacy compliance in LLMs.
Benchmark provides a foundation for future sensitivity-aware AI development.
Abstract
Large language models (LLMs) are increasingly becoming valuable to corporate data management due to their ability to process text from various document formats and facilitate user interactions through natural language queries. However, LLMs must consider the sensitivity of information when communicating with employees, especially given access restrictions. Simple filtering based on user clearance levels can pose both performance and privacy challenges. To address this, we propose the concept of sensitivity awareness (SA), which enables LLMs to adhere to predefined access rights rules. In addition, we developed a benchmarking environment called ACCESS DENIED INC to evaluate SA. Our experimental findings reveal significant variations in model behavior, particularly in managing unauthorized data requests while effectively addressing legitimate queries. This work establishes a foundation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
