Asm2SrcEval: Evaluating Large Language Models for Assembly-to-Source Code Translation

TL;DR

This paper systematically evaluates five large language models on assembly-to-source code translation, analyzing their performance across multiple metrics and identifying key strengths and limitations for practical applications.

Contribution

It introduces the first comprehensive benchmark for assessing large language models on assembly-to-source translation, providing detailed performance insights and qualitative analyses.

Findings

Models vary in text similarity and fluency metrics.

Trade-offs exist between accuracy and inference speed.

Challenges include control flow and identifier reconstruction.

Abstract

Assembly-to-source code translation is a critical task in reverse engineering, cybersecurity, and software maintenance, yet systematic benchmarks for evaluating large language models on this problem remain scarce. In this work, we present the first comprehensive evaluation of five state-of-the-art large language models on assembly-to-source translation. We assess model performance using a diverse set of metrics capturing lexical similarity (BLEU, ROUGE, and METEOR), semantic alignment (BERTScore), fluency (Perplexity), and efficiency (time prediction). Our results reveal clear trade-offs: while certain models excel in text similarity metrics, others demonstrate lower perplexity or faster inference times. We further provide qualitative analyses of typical model successes and failure cases, highlighting challenges such as control flow recovery and identifier reconstruction. Taken together, our benchmark offers actionable insights into the strengths and limitations of current large language models for program translation, establishing a foundation for future research in combining accuracy with efficiency for real-world applications.