Fingerprinting AI Coding Agents on GitHub

TL;DR

This study analyzes behavioral signatures of AI coding agents on GitHub, achieving high accuracy in identifying which AI generated specific pull requests, revealing distinct patterns for different tools.

Contribution

First comprehensive fingerprinting of AI coding agents using 41 features across multiple tools with high identification accuracy.

Findings

97.2% F1-score in multi-class agent identification

Distinct behavioral signatures for different AI agents

AI-generated code exhibits detectable patterns

Abstract

AI coding agents are reshaping software development through both autonomous and human-mediated pull requests (PRs). When developers use AI agents to generate code under their own accounts, code authorship attribution becomes critical for repository governance, research validity, and understanding modern development practices. We present the first study on fingerprinting AI coding agents, analyzing 33,580 PRs from five major agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, Claude Code) to identify behavioral signatures. With 41 features spanning commit messages, PR structure, and code characteristics, we achieve 97.2% F1-score in multi-class agent identification. We uncover distinct fingerprints: Codex shows unique multiline commit patterns (67.5% feature importance), and Claude Code exhibits distinctive code structure (27.2% importance of conditional statements). These signatures reveal that AI coding tools produce detectable behavioral patterns, suggesting potential for identifying AI contributions in software repositories.