Toward Imitating Visual Attention of Experts in Software Development Tasks

TL;DR

This paper proposes a neural autonomous agent framework that mimics expert programmers' visual attention during source code reading to improve software development tasks.

Contribution

It introduces a novel imitation learning-based framework for creating autonomous agents that replicate expert eye-movements in software development.

Findings

Framework enables agents to imitate expert visual attention.

Discussion of challenges in implementing IL-based agents for software tasks.

Potential applications in issue localization, comment, and code generation.

Abstract

Expert programmers' eye-movements during source code reading are valuable sources that are considered to be associated with their domain expertise. We advocate a vision of new intelligent systems incorporating expertise of experts for software development tasks, such as issue localization, comment generation, and code generation. We present a conceptual framework of neural autonomous agents based on imitation learning (IL), which enables agents to mimic the visual attention of an expert via his/her eye movement. In this framework, an autonomous agent is constructed as a context-based attention model that consists of encoder/decoder network and trained with state-action sequences generated by an experts' demonstration. Challenges to implement an IL-based autonomous agent specialized for software development task are discussed in this paper.