Evaluating the Impact of ChatGPT on Exercises of a Software Security Course

TL;DR

This study evaluates ChatGPT's ability to identify and fix web application vulnerabilities in a software security course, revealing its strengths and limitations in educational and practical security tasks.

Contribution

It provides an empirical analysis of ChatGPT's effectiveness in vulnerability detection and fixing, highlighting its potential as a supportive tool in security education.

Findings

ChatGPT identified 20 of 28 inserted vulnerabilities

Reported three false positives and found four additional vulnerabilities

Made nine satisfactory recommendations for fixing vulnerabilities

Abstract

Along with the development of large language models (LLMs), e.g., ChatGPT, many existing approaches and tools for software security are changing. It is, therefore, essential to understand how security-aware these models are and how these models impact software security practices and education. In exercises of a software security course at our university, we ask students to identify and fix vulnerabilities we insert in a web application using state-of-the-art tools. After ChatGPT, especially the GPT-4 version of the model, we want to know how the students can possibly use ChatGPT to complete the exercise tasks. We input the vulnerable code to ChatGPT and measure its accuracy in vulnerability identification and fixing. In addition, we investigated whether ChatGPT can provide a proper source of information to support its outputs. Results show that ChatGPT can identify 20 of the 28 vulnerabilities we inserted in the web application in a white-box setting, reported three false positives, and found four extra vulnerabilities beyond the ones we inserted. ChatGPT makes nine satisfactory penetration testing and fixing recommendations for the ten vulnerabilities we want students to fix and can often point to related sources of information.