Challenges and Future Directions in Agentic Reverse Engineering Systems

TL;DR

This paper analyzes the limitations of agentic large language model systems in complex reverse engineering tasks and discusses future research directions to address these challenges.

Contribution

It provides an analysis of current agentic reverse engineering systems, identifies key limitations, and outlines future research directions for security improvements.

Findings

Existing systems struggle with obfuscation and architecture complexity.

Token constraints limit system performance in reverse engineering.

Current systems lack robust program guardrails.

Abstract

Agentic systems built on large language models (LLMs) are increasingly being used for complex security tasks, including binary reverse engineering (RE). Despite recent growth in popularity and capability, these systems continue to face limitations in realistic settings. Cutting-edge systems still fail in complex RE scenarios that involve obfuscation, timing, and unique architecture. In this work, we examine how agentic systems perform reverse engineering tasks with static, dynamic, and hybrid agents. Through an analysis of existing agentic tool usage, we identify several limitations, including token constraints, struggles with obfuscation, and a lack of program guardrails. From these findings, we outline current challenges and position future directions for system designers to overcome from a security perspective.