Angora: Efficient Fuzzing by Principled Search

TL;DR

Angora is a novel mutation-based fuzzer that significantly improves software testing efficiency by increasing branch coverage through innovative constraint-solving techniques without relying on symbolic execution.

Contribution

It introduces scalable byte-level taint tracking, context-sensitive branch counting, gradient descent search, and input length exploration to enhance fuzzing performance.

Findings

Found almost all injected bugs in LAVA-M dataset

Detected more bugs than existing fuzzers in multiple programs

Achieved higher branch coverage and bug detection rates

Abstract

Fuzzing is a popular technique for finding software bugs. However, the performance of the state-of-the-art fuzzers leaves a lot to be desired. Fuzzers based on symbolic execution produce quality inputs but run slow, while fuzzers based on random mutation run fast but have difficulty producing quality inputs. We propose Angora, a new mutation-based fuzzer that outperforms the state-of-the-art fuzzers by a wide margin. The main goal of Angora is to increase branch coverage by solving path constraints without symbolic execution. To solve path constraints efficiently, we introduce several key techniques: scalable byte-level taint tracking, context-sensitive branch count, search based on gradient descent, and input length exploration. On the LAVA-M data set, Angora found almost all the injected bugs, found more bugs than any other fuzzer that we compared with, and found eight times as many bugs as the second-best fuzzer in the program who. Angora also found 103 bugs that the LAVA authors injected but could not trigger. We also tested Angora on eight popular, mature open source programs. Angora found 6, 52, 29, 40 and 48 new bugs in file, jhead, nm, objdump and size, respectively. We measured the coverage of Angora and evaluated how its key techniques contribute to its impressive performance.

Figures

Tables

Table 1. TABLE I : Bugs found on the LAVA-M data set by different fuzzers. Note that und more bugs than listed by LAVA authors.

Program	Listed	Bugs found by each fuzzer
	bugs		AFL	FUZZER	SES	VUzzer	Steelix
uniq	$28$	$29$	$9$	$7$	$0$	$27$	$7$
base64	$44$	$48$	$0$	$7$	$9$	$17$	$43$
md5sum	$57$	$57$	$0$	$2$	$0$	Fail	$28$
who	$2136$	$1541$	$1$	$0$	$18$	$50$	$194$

Table 2. TABLE II : Transforming comparisons into constraints. a 𝑎 a and b 𝑏 b represent arbitrary expressions.

Comparison	f	Constraint
	$f=a-b$
	$f=a-b$
$a>b$	$f=b-a$
$a>=b$	$f=b-a$
$a==b$	$f=abs\left(a-b\right)$	$f==0$
$a!=b$	$f=-abs\left(a-b\right)$

Table 3. TABLE IV : Bugs found by d the corresponding running time on the LAVA-M data set. Listed bugs are in LAVA’s validated_bugs file. Unlisted bugs were not triggered when LAVA’s authors prepared the data set.

Program	Listed	Found bugs		Time (min)
	bugs	Listed	Unlisted
uniq	$28$	$28$	$1$	$10$
base64	$44$	$44$	$4$	$10$
md5sum	$57$	$57$	$0$	$10$
who	$2136$	$1443$	$98$	$45$

Table 4. TABLE V : Comparison of d AFL on real world programs

Program	Argument	Size	Line coverage			Branch coverage			Unique crashes
		(kB)	AFL		Increase	AFL		Increase	AFL
file-5.32		$617$	$2070$	$2534$	$21.2\%$	$1462$	$1899$	$29.9\%$	$0$	$6$
jhead-3.00		$120$	$347$	$789$	$127.4\%$	$218$	$532$	$144.0\%$	$19$	$52$
xmlwf(expat)-2.2.5		$791$	$1980$	$2025$	$2.3\%$	$2905$	$3158$	$8.7\%$	$0$	$0$
djpeg(ijg)-v9b		$790$	$5401$	$5509$	$2.0\%$	$1677$	$1782$	$6.3\%$	$0$	$0$
readpng(libpng)-1.6.34		$972$	$1592$	$1799$	$13.0\%$	$872$	$1007$	$15.5\%$	$0$	$0$
nm-2.29	-C	$6252$	$6372$	$7721$	$21.2\%$	$4105$	$4693$	$14.3\%$	$12$	$29$
objdump-2.29	-x	$9063$	$3448$	$6216$	$80.3\%$	$2071$	$3393$	$63.8\%$	$4$	$40$
size-2.29		$6207$	$2839$	$4832$	$70.2\%$	$1792$	$2727$	$52.2\%$	$6$	$48$

Table 5. TABLE VI : Comparison of non-context-sensitive branch count vs.  context-sensitive branch count on the program file

Metric	Non-context-sensitive	Context-sensitive
Line coverage	$2416$	$2534$
Branch coverage	$1788$	$1899$
Unique crashes	$0$	$6$

Table 6. TABLE VII : Impact of incorporating context sensitivity on unique branches

Program	Unique branches		Ratio
	Context insensitive	Context sensitive
file	$3578$	$\mathrm{13\hspace{0.17em}554}$	$3.79$
jhead	$1049$	$6914$	$6.59$
xmlwf	$5531$	$\mathrm{11\hspace{0.17em}746}$	$2.20$
djpeg	$3819$	$\mathrm{13\hspace{0.17em}787}$	$3.61$
readpng	$7130$	$\mathrm{27\hspace{0.17em}577}$	$3.87$
nm	$9029$	$\mathrm{65\hspace{0.17em}131}$	$7.21$
objdump	$8113$	$\mathrm{40\hspace{0.17em}539}$	$5.00$
size	$5964$	$\mathrm{40\hspace{0.17em}148}$	$6.73$

Table 7. TABLE VIII : Percentage of solved constraints in conditional statements using three strategies

Program	Random	Magic bytes	Gradient descent
		+ random
file	$63.5\%$	$76.0\%$	$87.1\%$
jhead	$86.9\%$	$87.1\%$	$97.6\%$
xmlwf	$77.4\%$	$81.4\%$	$97.0\%$
djpeg	$66.1\%$	$73.6\%$	$78.3\%$
readpng	$19.9\%$	$23.7\%$	$24.5\%$
nm	$57.5\%$	$66.4\%$	$80.2\%$
objdump	$47.0\%$	$54.9\%$	$56.3\%$
size	$44.1\%$	$52.4\%$	$54.3\%$

Table 8. TABLE IX : Comparison of input length exploration vs. other tools’ random strategy. The total columns report how many times the strategies created a longer input, respectively. The useful columns report how many of these inputs successfully explored new branches, respectively. The two rightmost columns report the average lengths of the inputs in the useful columns, respectively.

Program	Longer inputs				Average length
	Random				Random	missing
	Useful	Total	Useful	Total
file	185	79k	251	3342	889.9	399.0
jhead	0	66k	0	26	0.0	0.0
xmlwf	277	143k	588	2196	190.3	128.9
djpeg	32	106k	474	3476	846.6	283.6
readpng	46	35k	43	152	2242.7	363.1
nm	17	170k	19	872	771.7	248.0
objdump	44	214k	60	1614	1271.6	496.0
size	27	197k	33	1482	1584.5	949.7

Table 9. TABLE X : Inputs tested per second

Program	AFL	file	971.17	791.73
jhead	2684.45	2648.91
xmlwf	2225.07	2206.24
djpeg	1439.94	1185.52
readpng	3374.43	2881.72
nm	1633.72	1045.35
objdump	1882.05	1192.04
size	1671.95	1174.55

Full text

1 Introduction

Fuzzing is a popular technique for finding software bugs. Coverage-based fuzzers face the key challenge of how to create inputs to explore program states. Some fuzzers use symbolic execution to solve path constraints [5, 8], but symbolic execution is slow and cannot solve many types of constraints efficiently [6]. To avoid these problems, AFL uses no symbolic execution or any heavy weight program analysis [1]. It instruments the program to observe which inputs explore new program branches, and keeps these inputs as seeds for further mutation. AFL incurs low overhead on program execution, but most of the inputs that it creates are ineffective (i.e., they fail to explore new program states) because it blindly mutates the input without taking advantage of the data flow in the program. Several fuzzers added heuristics to AFL to solve simple predicates, such as “magic bytes” [25, 19], but they cannot solve other path constraints.

We designed and implemented a fuzzer, called , that explores the states of a program by solving path constraints without using symbolic execution. acks the unexplored branches and tries to solve the path constraints on these branches. We introduced the following techniques to solve path constraints efficiently.