The Impact of AI-Assisted Development on Software Security: A Study of Gemini and Developer Experience

TL;DR

This study investigates how AI tools like Google's Gemini influence software security, finding that developer experience significantly enhances security more than the AI tool itself, with no significant security differences observed across different Gemini versions.

Contribution

It provides empirical evidence on the limited impact of Gemini on security and highlights the importance of developer experience in secure software development.

Findings

Developer experience improves code security significantly.

No significant security difference between no AI, free, and paid Gemini.

AI tools alone do not substitute for developer expertise.

Abstract

The ongoing shortage of skilled developers, particularly in security-critical software development, has led organizations to increasingly adopt AI-powered development tools to boost productivity and reduce reliance on limited human expertise. These tools, often based on large language models, aim to automate routine tasks and make secure software development more accessible and efficient. However, it remains unclear how developers' general programming and security-specific experience, and the type of AI tool used (free vs. paid) affect the security of the resulting software. Therefore, we conducted a quantitative programming study with software developers (n=159) exploring the impact of Google's AI tool Gemini on code security. Participants were assigned a security-related programming task using either no AI tools, the free version, or the paid version of Gemini. While we did not observe significant differences between using Gemini in terms of secure software development, programming experience significantly improved code security and cannot be fully substituted by Gemini.