AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMs

TL;DR

AC-LoRA is a system that enables access control-aware multi-modal large language models, maintaining data privacy and strong information isolation without additional training, while matching or surpassing existing methods.

Contribution

It introduces AC-LoRA, a novel approach for access control in multi-modal LLMs that uses permissioned adapters and similarity-based retrieval without extra training.

Findings

AC-LoRA matches or exceeds state-of-the-art performance.

Provides strong information isolation guarantees.

Applicable across different modalities.

Abstract

Corporate LLMs are gaining traction for efficient knowledge dissemination and management within organizations. However, as current LLMs are vulnerable to leaking sensitive information, it has proven difficult to apply them in settings where strict access control is necessary. To this end, we design AC-LoRA, an end-to-end system for access control-aware corporate LLM chatbots that maintains a strong information isolation guarantee. AC-LoRA maintains separate LoRA adapters for permissioned datasets, along with the document embedding they are finetuned on. AC-LoRA retrieves a precise set of LoRA adapters based on the similarity score with the user query and their permission. This similarity score is later used to merge the responses if more than one LoRA is retrieved, without requiring any additional training for LoRA routing. We provide an end-to-end prototype of AC-LoRA, evaluate it on two datasets, and show that AC-LoRA matches or even exceeds the performance of state-of-the-art LoRA mixing techniques while providing strong isolation guarantees. Furthermore, we show that AC-LoRA design can be directly applied to different modalities.