LLMs in Coding and their Impact on the Commercial Software Engineering Landscape

TL;DR

Large-language models are transforming software engineering by increasing efficiency but also introducing security and privacy risks, requiring new safety practices and regulations.

Contribution

This paper highlights the risks of LLMs in coding and proposes safety practices for firms to mitigate these issues.

Findings

10% prompts leak private data

42% generated snippets hide security flaws

Models can agree with incorrect ideas (sycophancy)

Abstract

Large-language-model coding tools are now mainstream in software engineering. But as these same tools move human effort up the development stack, they present fresh dangers: 10% of real prompts leak private data, 42% of generated snippets hide security flaws, and the models can even ``agree'' with wrong ideas, a trait called sycophancy. We argue that firms must tag and review every AI-generated line of code, keep prompts and outputs inside private or on-premises deployments, obey emerging safety regulations, and add tests that catch sycophantic answers -- so they can gain speed without losing security and accuracy.