A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality

TL;DR

This study explores both perceptions and practical effectiveness of ChatGPT in software security tasks, revealing positive views but limited industry applicability due to generic responses.

Contribution

It provides a dual analysis of ChatGPT's perceived benefits and practical limitations in software security, highlighting areas for future specialized LLM development.

Findings

Security practitioners see ChatGPT as beneficial for vulnerability detection and penetration testing.

ChatGPT responses are often generic and may not be suitable for industry use.

Practical deployment shows limitations in ChatGPT's current capabilities for security tasks.

Abstract

Artificial Intelligence (AI) advancements have enabled the development of Large Language Models (LLMs) that can perform a variety of tasks with remarkable semantic understanding and accuracy. ChatGPT is one such LLM that has gained significant attention due to its impressive capabilities for assisting in various knowledge-intensive tasks. Due to the knowledge-intensive nature of engineering secure software, ChatGPT's assistance is expected to be explored for security-related tasks during the development/evolution of software. To gain an understanding of the potential of ChatGPT as an emerging technology for supporting software security, we adopted a two-fold approach. Initially, we performed an empirical study to analyse the perceptions of those who had explored the use of ChatGPT for security tasks and shared their views on Twitter. It was determined that security practitioners view ChatGPT as beneficial for various software security tasks, including vulnerability detection, information retrieval, and penetration testing. Secondly, we designed an experiment aimed at investigating the practicality of this technology when deployed as an oracle in real-world settings. In particular, we focused on vulnerability detection and qualitatively examined ChatGPT outputs for given prompts within this prominent software security task. Based on our analysis, responses from ChatGPT in this task are largely filled with generic security information and may not be appropriate for industry use. To prevent data leakage, we performed this analysis on a vulnerability dataset compiled after the OpenAI data cut-off date from real-world projects covering 40 distinct vulnerability types and 12 programming languages. We assert that the findings from this study would contribute to future research aimed at developing and evaluating LLMs dedicated to software security.