BlueGlass: A Framework for Composite AI Safety

TL;DR

BlueGlass is a unified framework that integrates diverse AI safety tools to enhance safety analysis and robustness of AI systems, demonstrated through vision-language model evaluations.

Contribution

Introduces BlueGlass, a novel framework enabling the integration of multiple safety tools for comprehensive AI safety workflows.

Findings

Distributional evaluation reveals performance trade-offs and failure modes.

Probe-based analysis uncovers shared hierarchical learning.

Sparse autoencoders identify interpretable concepts.

Abstract

As AI systems become increasingly capable and ubiquitous, ensuring the safety of these systems is critical. However, existing safety tools often target different aspects of model safety and cannot provide full assurance in isolation, highlighting a need for integrated and composite methodologies. This paper introduces BlueGlass, a framework designed to facilitate composite AI safety workflows by providing a unified infrastructure enabling the integration and composition of diverse safety tools that operate across model internals and outputs. Furthermore, to demonstrate the utility of this framework, we present three safety-oriented analyses on vision-language models for the task of object detection: (1) distributional evaluation, revealing performance trade-offs and potential failure modes across distributions; (2) probe-based analysis of layer dynamics highlighting shared hierarchical learning via phase transition; and (3) sparse autoencoders identifying interpretable concepts. More broadly, this work contributes foundational infrastructure and findings for building more robust and reliable AI systems.