KOALA: a Configurable Tool for Collecting IDE Data When Solving Programming Tasks

TL;DR

KOALA is a configurable IDE plugin that collects detailed student programming activity data, including code snapshots, IDE actions, hotkeys, and focus changes, to aid research and education.

Contribution

It introduces KOALA, a flexible tool for comprehensive data collection in JetBrains IDEs, addressing limitations of existing tools with customizable features.

Findings

Collected data from 28 students across two courses.

Highlighted insights into students' programming behaviors.

Demonstrated KOALA's effectiveness in capturing detailed IDE interactions.

Abstract

Collecting data of students solving programming tasks is incredibly valuable for researchers and educators. It allows verifying that the students correctly apply the features and concepts they are taught, or finding students' misconceptions. However, existing data collection tools have limitations, e.g., no control over the granularity of the collected code, not collecting the specific events of the programming environment used, and overall being hard to configure. To overcome these limitations, we propose KOALA, a convenient and highly configurable tool for collecting code snapshots and feature usage from students solving programming tasks in JetBrains IDEs. The plugin can be installed in IDEs and configured to provide the students with the necessary tasks, enable or disable certain IDE features like code completion, and run surveys. During problem solving, the plugin collects code snapshots at the configured granularity, all IDE actions like running and debugging, as well as some data not collected in prior works, like employed hotkeys and switching focus between files. The collected data is sent to the server that comes with the tool, where it is stored and can be converted to the standardized ProgSnap2 format. To showcase the tool, we collected data from 28 students solving tasks in two courses within the IDE, highlighting some insights from this data.