Evaluating How Developers Use General-Purpose Web-Search for Code Retrieval

TL;DR

This study investigates how effectively developers use general-purpose search engines like Google for code-related searches, revealing that code searches tend to require more effort and are less optimized compared to non-code searches.

Contribution

The paper introduces a novel classifier for distinguishing code-related from non-code-related search queries and analyzes a large dataset to evaluate search effort differences.

Findings

Code searches require more effort than non-code searches.

Google's search performance is less effective for code-related queries.

A high-accuracy classifier was developed to identify code intent in queries.

Abstract

Search is an integral part of a software development process. Developers often use search engines to look for information during development, including reusable code snippets, API understanding, and reference examples. Developers tend to prefer general-purpose search engines like Google, which are often not optimized for code related documents and use search strategies and ranking techniques that are more optimized for generic, non-code related information. In this paper, we explore whether a general purpose search engine like Google is an optimal choice for code-related searches. In particular, we investigate whether the performance of searching with Google varies for code vs. non-code related searches. To analyze this, we collect search logs from 310 developers that contains nearly 150,000 search queries from Google and the associated result clicks. To differentiate between code-related searches and non-code related searches, we build a model which identifies the code intent of queries. Leveraging this model, we build an automatic classifier that detects a code and non-code related query. We confirm the effectiveness of the classifier on manually annotated queries where the classifier achieves a precision of 87%, a recall of 86%, and an F1-score of 87%. We apply this classifier to automatically annotate all the queries in the dataset. Analyzing this dataset, we observe that code related searching often requires more effort (e.g., time, result clicks, and query modifications) than general non-code search, which indicates code search performance with a general search engine is less effective.