HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding

TL;DR

HiLDe introduces a human-in-the-loop decoding approach for AI code generation, enabling users to influence model decisions, resulting in fewer vulnerabilities and better goal alignment in security-critical tasks.

Contribution

The paper presents HiLDe, a novel interactive technique allowing users to guide LLM code generation, improving security and alignment in critical programming tasks.

Findings

Participants generated fewer vulnerabilities with HiLDe.

HiLDe improved alignment of code with user goals.

User engagement increased control over code quality.

Abstract

While AI programming tools hold the promise of increasing programmers' capabilities and productivity to a remarkable degree, they often exclude users from essential decision-making processes, causing many to effectively "turn off their brains" and over-rely on solutions provided by these systems. These behaviors can have severe consequences in critical domains, like software security. We propose Human-in-the-loop Decoding, a novel interaction technique that allows users to observe and directly influence LLM decisions during code generation, in order to align the model's output with their personal requirements. We implement this technique in HiLDe, a code completion assistant that highlights critical decisions made by the LLM and provides local alternatives for the user to explore. In a within-subjects study (N=18) on security-related tasks, we found that HiLDe led participants to generate significantly fewer vulnerabilities and better align code generation with their goals compared to a traditional code completion assistant.