Towards Provably Secure Generative AI: Reliable Consensus Sampling

TL;DR

This paper introduces Reliable Consensus Sampling (RCS), a new algorithm for generative AI that offers provable security, improved robustness against malicious manipulation, and better utility without abstention, supported by theoretical guarantees and extensive experiments.

Contribution

The paper proposes RCS, a novel primitive that enhances security and robustness of generative AI, eliminating abstention and providing theoretical risk control.

Findings

RCS significantly improves robustness against adversarial models.

RCS maintains utility comparable to existing methods.

Theoretical guarantees ensure controllable risk thresholds.

Abstract

Existing research on generative AI security is primarily driven by mutually reinforcing attack and defense methodologies grounded in empirical experience. This dynamic frequently gives rise to previously unknown attacks that can circumvent current detection and prevention. This necessitates the continual updating of security mechanisms. Constructing generative AI with provable security and theoretically controllable risk is therefore necessary. Consensus Sampling (CS) is a promising algorithm toward provably secure AI. It controls risk by leveraging overlap in model output probabilities. However, we find that CS relies on frequent abstention to avoid unsafe outputs, which reduces utility. Moreover, CS becomes highly vulnerable when unsafe models are maliciously manipulated. To address these issues, we propose a new primitive called Reliable Consensus Sampling (RCS), that traces acceptance probability to tolerate extreme adversarial behaviors, improving robustness. RCS also eliminates the need for abstention entirely. We further develop a feedback algorithm to continuously and dynamically enhance the safety of RCS. We provide theoretical guarantees that RCS maintains a controllable risk threshold. Extensive experiments show that RCS significantly improves robustness and utility while maintaining latency comparable to CS. We hope this work contributes to the development of provably secure generative AI.