Empirical Studies on Adversarial Reverse Engineering with Students

TL;DR

This paper investigates the feasibility and methodology of using students as participants in empirical reverse engineering studies, providing guidelines to ensure valid, ethical, and meaningful research outcomes.

Contribution

It offers a comprehensive framework for conducting empirical reverse engineering experiments with students, including training, challenge design, and validity considerations.

Findings

Students can effectively participate with proper training

Guidelines improve research validity and ethics

Recommendations enhance reproducibility of studies

Abstract

Empirical research in reverse engineering and software protection is crucial for evaluating the efficacy of methods designed to protect software against unauthorized access and tampering. However, conducting such studies with professional reverse engineers presents significant challenges, including access to professionals and affordability. This paper explores the use of students as participants in empirical reverse engineering experiments, examining their suitability and the necessary training; the design of appropriate challenges; strategies for ensuring the rigor and validity of the research and its results; ways to maintain students' privacy, motivation, and voluntary participation; and data collection methods. We present a systematic literature review of existing reverse engineering experiments and user studies, a discussion of related work from the broader domain of software engineering that applies to reverse engineering experiments, an extensive discussion of our own experience running experiments ourselves in the context of a master-level software hacking and protection course, and recommendations based on this experience. Our findings aim to guide future empirical studies in RE, balancing practical constraints with the need for meaningful, reproducible results.