Randomized Controlled Trials for Security Copilot for IT Administrators

TL;DR

This study demonstrates that Microsoft's Security Copilot significantly improves IT administrators' accuracy and speed across various tasks, with higher benefits for complex scenarios, indicating GAI tools can enhance productivity in IT management.

Contribution

First randomized controlled trial evaluating the impact of Security Copilot on IT admin productivity, showing substantial improvements in accuracy and efficiency.

Findings

34.53% improvement in accuracy across tasks

29.79% reduction in task completion time

Greater benefits observed for complex tasks

Abstract

As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft's Security Copilot ("Copilot") on information technology administrators ("IT admins") through randomized controlled trials. Participants were divided into treatment and control groups, with the former granted access to Copilot within Microsoft's Entra and Intune admin centers. Across three IT admin scenarios - sign-in troubleshooting, device policy management, and device troubleshooting - Copilot users demonstrated substantial improvements in both accuracy and speed. Across all scenarios and tasks, Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time. We also find that the productivity benefits vary by task type, with more complex tasks showing greater improvement. In free response tasks, Copilot users identified 146.07% more relevant facts and reduced task completion time by 61.14%. Subject satisfaction with Copilot was high, with participants reporting reduced effort and a strong preference for using the tool in future tasks. These findings suggest that GAI tools like Copilot can significantly enhance the productivity and efficiency of IT admins, especially in scenarios requiring information synthesis and complex decision-making.