BotHawk: An Approach for Bots Detection in Open Source Software Projects

TL;DR

This paper introduces BotHawk, a highly accurate model for detecting various types of bots in open-source software projects, addressing challenges like impersonation and security risks.

Contribution

The study presents a new dataset of nearly 20,000 accounts and a novel detection model that outperforms existing methods in identifying diverse OSS bots.

Findings

BotHawk achieves an AUC of 0.947 and F1-score of 0.89.

Four types of bots are identified based on behavior analysis.

Key features include followers, repositories, and tags.

Abstract

Social coding platforms have revolutionized collaboration in software development, leading to using software bots for streamlining operations. However, The presence of open-source software (OSS) bots gives rise to problems including impersonation, spamming, bias, and security risks. Identifying bot accounts and behavior is a challenging task in the OSS project. This research aims to investigate bots' behavior in open-source software projects and identify bot accounts with maximum possible accuracy. Our team gathered a dataset of 19,779 accounts that meet standardized criteria to enable future research on bots in open-source projects. We follow a rigorous workflow to ensure that the data we collect is accurate, generalizable, scalable, and up-to-date. We've identified four types of bot accounts in open-source software projects by analyzing their behavior across 17 features in 5 dimensions. Our team created BotHawk, a highly effective model for detecting bots in open-source software projects. It outperforms other models, achieving an AUC of 0.947 and an F1-score of 0.89. BotHawk can detect a wider variety of bots, including CI/CD and scanning bots. Furthermore, we find that the number of followers, number of repositories, and tags contain the most relevant features to identify the account type.