Bringing Order Amidst Chaos: On the Role of Artificial Intelligence in Secure Software Engineering

TL;DR

This paper analyzes the role of AI in secure software engineering, highlighting challenges, limitations, and opportunities for improving vulnerability detection and prediction through empirical evaluation and evidence-based techniques.

Contribution

It provides a comprehensive analysis of AI's effectiveness in SSE, emphasizing domain-specific factors and proposing methods to enhance vulnerability prediction accuracy.

Findings

Static analysis tools have limitations in vulnerability detection.

Coverage gaps exist in SASTT for different vulnerability types.

Just-in-time modeling improves defect prediction accuracy.

Abstract

Context. Developing secure and reliable software remains a key challenge in software engineering (SE). The ever-evolving technological landscape offers both opportunities and threats, creating a dynamic space where chaos and order compete. Secure software engineering (SSE) must continuously address vulnerabilities that endanger software systems and carry broader socio-economic risks, such as compromising critical national infrastructure and causing significant financial losses. Researchers and practitioners have explored methodologies like Static Application Security Testing Tools (SASTTs) and artificial intelligence (AI) approaches, including machine learning (ML) and large language models (LLMs), to detect and mitigate these vulnerabilities. Each method has unique strengths and limitations. Aim. This thesis seeks to bring order to the chaos in SSE by addressing domain-specific differences that impact AI accuracy. Methodology. The research employs a mix of empirical strategies, such as evaluating effort-aware metrics, analyzing SASTTs, conducting method-level analysis, and leveraging evidence-based techniques like systematic dataset reviews. These approaches help characterize vulnerability prediction datasets. Results. Key findings include limitations in static analysis tools for identifying vulnerabilities, gaps in SASTT coverage of vulnerability types, weak relationships among vulnerability severity scores, improved defect prediction accuracy using just-in-time modeling, and threats posed by untouched methods. Conclusions. This thesis highlights the complexity of SSE and the importance of contextual knowledge in improving AI-driven vulnerability and defect prediction. The comprehensive analysis advances effective prediction models, benefiting both researchers and practitioners.