PASTA: A Scalable Framework for Multi-Policy AI Compliance Evaluation

TL;DR

PASTA is a scalable, multi-policy AI compliance evaluation framework that combines model-card descriptions, normalization, LLM-powered assessments, and interpretable outputs, enabling rapid, cost-effective, and user-friendly governance.

Contribution

The paper introduces PASTA, a novel scalable framework integrating multiple innovations for multi-policy AI compliance evaluation, addressing existing limitations of single-policy approaches.

Findings

PASTA's judgments align closely with human experts ($\rho \geq .626$).

Evaluates five policies in under two minutes at about $3.

Practitioners find outputs easy-to-understand and actionable.

Abstract

AI compliance is becoming increasingly critical as AI systems grow more powerful and pervasive. Yet the rapid expansion of AI policies creates substantial burdens for resource-constrained practitioners lacking policy expertise. Existing approaches typically address one policy at a time, making multi-policy compliance costly. We present PASTA, a scalable compliance tool integrating four innovations: (1) a comprehensive model-card format supporting descriptive inputs across development stages; (2) a policy normalization scheme; (3) an efficient LLM-powered pairwise evaluation engine with cost-saving strategies; and (4) an interface delivering interpretable evaluations via compliance heatmaps and actionable recommendations. Expert evaluation shows PASTA's judgments closely align with human experts ($\rho \geq .626$). The system evaluates five major policies in under two minutes at approximately \$3. A user study (N = 12) confirms practitioners found outputs easy-to-understand and actionable, introducing a novel framework for scalable automated AI governance.