Report on NSF Workshop on Science of Safe AI

TL;DR

This report summarizes a workshop on AI safety, emphasizing the need for developing trustworthy, safe AI systems through new theories, methods, and tools, especially for autonomous and interactive AI applications.

Contribution

It presents a new research agenda focused on foundational theories, methods, and tools to enhance the safety and trustworthiness of AI systems.

Findings

Identified key challenges in AI safety and trustworthiness.

Highlighted the importance of developing theoretical foundations.

Outlined future research directions for safe AI development.

Abstract

Recent advances in machine learning, particularly the emergence of foundation models, are leading to new opportunities to develop technology-based solutions to societal problems. However, the reasoning and inner workings of today's complex AI models are not transparent to the user, and there are no safety guarantees regarding their predictions. Consequently, to fulfill the promise of AI, we must address the following scientific challenge: how to develop AI-based systems that are not only accurate and performant but also safe and trustworthy? The criticality of safe operation is particularly evident for autonomous systems for control and robotics, and was the catalyst for the Safe Learning Enabled Systems (SLES) program at NSF. For the broader class of AI applications, such as users interacting with chatbots and clinicians receiving treatment recommendations, safety is, while no less important, less well-defined with context-dependent interpretations. This motivated the organization of a day-long workshop, held at University of Pennsylvania on February 26, 2025, to bring together investigators funded by the NSF SLES program with a broader pool of researchers studying AI safety. This report is the result of the discussions in the working groups that addressed different aspects of safety at the workshop. The report articulates a new research agenda focused on developing theory, methods, and tools that will provide the foundations of the next generation of AI-enabled systems.