Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development

TL;DR

This paper discusses the Amazon Nova AI Challenge focused on advancing secure AI in software development, highlighting novel safety alignment techniques, adversarial testing, and collaborative efforts to improve AI safety standards.

Contribution

It introduces new methods for safety alignment, robust guardrails, and adversarial testing in AI-assisted software development, supported by a comprehensive challenge framework.

Findings

Development of state-of-the-art safety techniques

Creation of a custom baseline AI model

Successful adversarial safety evaluations

Abstract

AI systems for software development are rapidly gaining prominence, yet significant challenges remain in ensuring their safety. To address this, Amazon launched the Trusted AI track of the Amazon Nova AI Challenge, a global competition among 10 university teams to drive advances in secure AI. In the challenge, five teams focus on developing automated red teaming bots, while the other five create safe AI assistants. This challenge provides teams with a unique platform to evaluate automated red-teaming and safety alignment methods through head-to-head adversarial tournaments where red teams have multi-turn conversations with the competing AI coding assistants to test their safety alignment. Along with this, the challenge provides teams with a feed of high quality annotated data to fuel iterative improvement. Throughout the challenge, teams developed state-of-the-art techniques, introducing novel approaches in reasoning-based safety alignment, robust model guardrails, multi-turn jail-breaking, and efficient probing of large language models (LLMs). To support these efforts, the Amazon Nova AI Challenge team made substantial scientific and engineering investments, including building a custom baseline coding specialist model for the challenge from scratch, developing a tournament orchestration service, and creating an evaluation harness. This paper outlines the advancements made by university teams and the Amazon Nova AI Challenge team in addressing the safety challenges of AI for software development, highlighting this collaborative effort to raise the bar for AI safety.