Towards a Benchmark for Dependency Decision-Making

TL;DR

This paper introduces DepDec-Bench, a benchmark for evaluating dependency decision-making in AI coding agents, emphasizing security, efficiency, and policy compliance beyond mere functional correctness.

Contribution

It presents a new benchmark and evaluation framework based on real-world dependency change data, highlighting security and policy considerations often overlooked.

Findings

Agents often select vulnerable dependency versions.

Dependency decisions can have negative security impacts.

Benchmark evaluates safe, disciplined dependency management.

Abstract

AI coding agents increasingly modify real software repositories and make dependency decisions, including adding, removing, or updating third-party packages. These choices can materially affect security posture and maintenance burden, yet repository-level evaluations largely emphasize test passing and executability without explicitly scoring whether systems (i) reuse existing dependencies, (ii) avoid unnecessary additions, or (iii) select versions that satisfy security and policy constraints. We propose DepDec-Bench, a benchmark for evaluating dependency decision-making beyond functional correctness. To ground DepDec-Bench in real-world behavior, we conduct a preliminary study of 117,062 dependency changes from agent- and human-authored pull requests across seven ecosystems. We show that coding agents frequently make dependency decisions with security consequences that remain invisible to test-focused evaluation: agents select PR-time known-vulnerable versions (2.46%) and exhibit net-negative security impact overall (net impact -98 vs. +1,316 for humans). These observations inform DepDec-Bench task families and metrics that evaluate safe version selection, reuse discipline, and restraint against dependency bloat alongside test passing.