A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

TL;DR

This paper introduces an automated framework leveraging advanced LLMs like GPT-4 to measure responsible AI harms in generative AI applications, facilitating more effective harm detection and promoting responsible AI use.

Contribution

The paper presents a novel automated framework for measuring AI harms that combines technical and sociotechnical expertise, adaptable to new harm areas.

Findings

Demonstrated the framework through case studies on LLMs

Showed the framework's ability to identify violations of RAI principles

Enabled future harm measurement in diverse domains

Abstract

We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services. Our framework for automatically measuring harms from LLMs builds on existing technical and sociotechnical expertise and leverages the capabilities of state-of-the-art LLMs, such as GPT-4. We use this framework to run through several case studies investigating how different LLMs may violate a range of RAI-related principles. The framework may be employed alongside domain-specific sociotechnical expertise to create measurements for new harm areas in the future. By implementing this framework, we aim to enable more advanced harm measurement efforts and further the responsible use of LLMs.