CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval

TL;DR

This paper introduces CoQuIR, a large-scale, multilingual benchmark for evaluating code retrieval systems on quality dimensions like correctness, security, and maintainability, highlighting current models' limitations and proposing methods to improve quality awareness.

Contribution

We present CoQuIR, the first comprehensive benchmark for quality-aware code retrieval, along with evaluation metrics and initial training strategies to enhance model awareness of code quality.

Findings

Top models often fail to identify insecure or buggy code.

Training with synthetic datasets improves quality-aware metrics.

Downstream tasks confirm the benefits of quality-focused retriever training.

Abstract

Code retrieval is essential in modern software development, as it boosts code reuse and accelerates debugging. However, current benchmarks primarily emphasize functional relevance while neglecting critical dimensions of software quality. Motivated by this gap, we introduce CoQuIR, the first large-scale, multilingual benchmark specifically designed to evaluate quality-aware code retrieval across four key dimensions: correctness, efficiency, security, and maintainability. CoQuIR provides fine-grained quality annotations for 42,725 queries and 134,907 code snippets in 11 programming languages, and is accompanied by two quality-centric evaluation metrics: Pairwise Preference Accuracy and Margin-based Ranking Score. Using CoQuIR, we benchmark 23 retrieval models, covering both open-source and proprietary systems, and find that even top-performing models frequently fail to distinguish buggy or insecure code from their more robust counterparts. Furthermore, we conduct preliminary investigations into training methods that explicitly encourage retrievers to recognize code quality. Using synthetic datasets, we demonstrate promising improvements in quality-aware metrics across various models, without sacrificing semantic relevance. Downstream code generation experiments further validate the effectiveness of our approach. Overall, our work highlights the importance of integrating quality signals into code retrieval systems, laying the groundwork for more trustworthy and robust software development tools.