Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering

TL;DR

This paper analyzes the risks of LLM-assisted code generation in software engineering and proposes the SAFE-AI Framework to enhance safety, transparency, and accountability through integrated guardrails, verification, and explainability.

Contribution

It introduces the SAFE-AI Framework and a taxonomy of AI behaviors, offering a comprehensive approach to mitigate risks and guide responsible AI integration in software engineering.

Findings

Identification of key risks like insecure code and hallucinations

Proposal of the SAFE-AI Framework with integrated safety mechanisms

Discussion of open problems and future research directions

Abstract

The integration of Large Language Models (LLMs) into software engineering has revolutionized code generation, enabling unprecedented productivity through promptware and autonomous AI agents. However, this transformation introduces significant risks, including insecure code generation, hallucinated outputs, irreversible actions, and a lack of transparency and accountability. Incidents like the Replit database deletion underscore the urgent need for robust safety and governance mechanisms. This paper comprehensively analyzes the inherent challenges of LLM-assisted code generation, such as vulnerability inheritance, overtrust, misinterpretation, and the absence of standardized validation and rollback protocols. To address these, we propose the SAFE-AI Framework, a holistic approach emphasizing Safety, Auditability, Feedback, and Explainability. The framework integrates guardrails, sandboxing, runtime verification, risk-aware logging, human-in-the-loop systems, and explainable AI techniques to mitigate risks while fostering trust and compliance. We introduce a novel taxonomy of AI behaviors categorizing suggestive, generative, autonomous, and destructive actions to guide risk assessment and oversight. Additionally, we identify open problems, including the lack of standardized benchmarks for code specific hallucinations and autonomy levels, and propose future research directions for hybrid verification, semantic guardrails, and proactive governance tools. Through detailed comparisons of autonomy control, prompt engineering, explainability, and governance frameworks, this paper provides a roadmap for responsible AI integration in software engineering, aligning with emerging regulations like the EU AI Act and Canada's AIDA to ensure safe, transparent, and accountable AI-driven development.