FiFTy: Large-scale File Fragment Type Identification using Neural Networks

TL;DR

FiFTy is a neural network-based tool for large-scale file type identification that outperforms previous methods in speed and accuracy, using a novel dataset with 75 diverse file types.

Contribution

The paper introduces FiFTy, a neural network approach that eliminates explicit feature extraction for file type identification, achieving higher accuracy and speed.

Findings

Average accuracy of 77.5% on 75 file types

Processing speed of approximately 38 sec/GB

Outperforms the previous state-of-the-art by over an order of magnitude

Abstract

We present FiFTy, a modern file type identification tool for memory forensics and data carving. In contrast to previous approaches based on hand-crafted features, we design a compact neural network architecture, which uses a trainable embedding space, akin to successful natural language processing models. Our approach dispenses with explicit feature extraction which is a bottleneck in legacy systems. We evaluate the proposed method on a novel dataset with 75 file types - the most diverse and balanced dataset reported to date. FiFTy consistently outperforms all baselines in terms of speed, accuracy and individual misclassification rates. We achieved an average accuracy of 77.5% with processing speed of approx 38 sec/GB, which is better and more than an order of magnitude faster than the previous state-of-the-art tool - Sceadan (69% at 9 min/GB). Our tool and the corresponding dataset are available publicly online.

Tables

Table 1. TABLE I: Quantitative review of related work in reverse chronological order; Reported metrics correspond to: our experiments on the same dataset (75 file-types; 512 and 4,096-byte inputs) and hardware (top 4 rows); and results reported in original publications (all remaining rows)

Year	Classifier	Features	Feature dim.	#File types	Block size	Accuracy	JPEG Acc.	Speed [ms/block]†	Speed [min/GB]†	Training Time‡	Remarks	Ref.
2019	CNN	raw bytes	n/a	751	512	65.6	83.5	0.04	1.50	7.51 hours	FiFTy	Self
			n/a	751	4096	77.5	86.3	0.15	0.64	12.53 hours
2019	CNN	co-occurence matrix	(128, 128)	751	512	64.4	77.0	0.84	29.36	8.67 hours	NN-CO	Self
				751	4096	75.3	92.5	1.05	4.59	9.04 hours
2019	NN	global stat. features	14	751	512	45.4	53.2	0.61	21.32	2.88 hours	NN-GF	Self
				751	4096	46.8	48.6	2.67	11.67	10.53 hours
2019	SVM	statistical: unigram, bigram, entropy, etc	65792	751	512	57.3	81.5	1.48	51.7	44 hours	Sceadan [1]; our evaluation	Self
				751	4096	69.0	91.8	2.06	9.0	10 days*
2018	CNN	1-hot enc. bytes	8192	42	512	73.4	96.5	n/a	n/a	7 hours	Few & very different file types	[14]
	RNN			42	512	98.04	96.5	n/a	n/a	135 hours
2018	CNN	image pixels	(64, 64)	162	4096	70.9	15	n/a	n/a	n/a		[15]
2018	SVM	dictionary of N-Grams	1024 + 1296 + 1444	182	512	61.3	65.0	n/a	n/a	235 ms/block	Feature extraction is the bottleneck	[4]
2016	K-means	unigrams, bigrams, and global statistics (mean, entropy, complexity, etc.)	n/a	522,3 (6 classes)	512	74.0	n/a	n/a	n/a	n/a	Focus on exploratory heirarchical clustering	[20]
2015	SVM	BFD, entropy	n/a	122	512	67.4	69.2	n/a	n/a	n/a	Rudimentary analysis	[19]
2014	KNN	image pixels	512 ($32\times 32$)	292	1024	39.7	22	n/a	n/a	0.001 ms/block	Includes various classifiers & grid search	[9]
2013	SVM	statistical: unigram, bigram, entropy, etc	65792	382,3	512	73.8	81	0.30	10.6	n/a	Sceadan; as reported by authors	[1]
2012	SVM	unigram, bigram, entropy, Hamming weight, complexity, longest cont. streak	$256+{256}^2+6$	242	512	49.1	17.4	n/a	n/a	n/a	Poor results for all high-entropy file types (incl. JPG)	[2]
2008	FLD	longest common substrings, other statistical measures	21	43	512	86	83.6	n/a	n/a	n/a	Pair-wise analysis	[8]
			-	43	896	88.3	92	n/a	n/a	n/a
2007	FLD	1-gram, entropy, Kolmogorov complexity	n/a	113	4096	45	98	n/a	n/a	n/a		[3]
2006	VQ	rate of change and binary freq. distribution	n/a	43	512	n/a	99.2	0.02	0.6	n/a	JPEG-specific method (byte markers); high FPR for other types	[7]
Dataset: 1 - the proposed FFT-75 dataset[21] ; 2 - a public dataset GovDocs1 [22] ; 3 - Private dataset; * ref. section IV-C; † - Computed on Tesla V100 (Top 4 rows) ; ‡ - Computed on Tesla P40 (Top 4)

Table 2. TABLE II: Model architectures obtained with TPE for each scenario and block size.

Scenario	Model Description	# Params
	FiFTy models for 512-byte blocks
#1	E (64) - C1D (128, 27) - MP (4) - AP - D (0.1) - F (256) - F (75)	289,995
#2	E (48) - C1D (128, 11) - MP (4) - C1D (128, 11) - MP (4) - AP - D (0.1) - F (64) - F (11)	269,323
#3	E (64) - C1D (128, 27) - MP (2) - C1D (128, 27) - MP (2) - AP - D (0.1) - F (64) - F (25)	690,073
#4	E (48) - C1D (128, 19) - MP (4) - C1D (128, 19) - MP (4) - AP - D (0.1) - F (256) - F (5)	474,885
#5	E (64) - C1D (128, 35) - MP (8) - AP - D (0.1) - F (256) - F (2)	336,770
#6	E (32) - C1D (128, 11) - MP (6) - C1D (128, 11) - MP (6) - AP - D (0.1) - F (64) - F (2)	242,114
	FiFTy models for 4,096-byte blocks
#1	E (32) - C1D (128, 19) - MP (4) - C1D (128, 19) - MP (4) - AP - D (0.1) - F (256) - F (75)	449,867
#2	E (32) - C1D (128, 27) - MP (8) - C1D (128, 27) - MP (8) - AP - D (0.1) - F (256) - F (11)	597,259
#3	E (32) - C1D (128, 11) - MP (6) - C1D (128, 11) - MP (6) - C1D (128, 11) - MP (6) - AP - D (0.1) - F (256) - F (25)	453,529
#4	E (64) - C1D (128, 27) - MP (6) - C1D (128, 27) - MP (6) - AP - D (0.1) - F (32) - F (5)	684,485
#5	E (48) - C1D (32, 35) - MP (6) - C1D (32, 35) - MP (6) - C1D (32, 35) - MP (6) - AP - D (0.1) - F (16) - F (2)	138,386
#6	E (16) - C1D (128, 35)- MP (8) - C1D (128, 35) - MP (8) - AP - D (0.1) - F (128) - F (2)	666,242
Baseline	Baseline neural network models (scenario #1 for 512 and 4,096-byte blocks)
NN-CO	C2D (48, 3) - C2D (48, 3) - C2D (48, 3) - C2D (48, 3) - V - F (64) - F(75)	44,304,571
NN-GF	F (256) - F (256) - F (256) - F (75)	157,771
Layers: [E] embedding ($⟨$embedding vector length$⟩$); [C1D/C2D] convolution ($⟨$filter size, stride$⟩$); [MP] 1-D max-pooling ($⟨$size$⟩$) ; [AP] average pooling; [F] fully connected ($⟨$# units$⟩$); [D] dropout ($⟨$dropout value$⟩$); [V] flatten.

Table 3. TABLE III: Detailed results for all 6 scenarios for 512 and 4096-byte blocks.

Scen.	#Classes	Classes	Composition1	Accuracy		JPEG Acc.		Train. time [h]		Inf. time [$\frac{\text{ms}}{\text{block}}$]		#Samples
				512	4096	512	4096	512	4096	512	4096
#1	75	All	75	65.6	77.5	83.5	86.3	7.51	12.53	0.043	0.146	7500k
#2	11	Bitmaps + RAW + Vector + Video + Archives + Exe + Office + Published + Human Readable + Audio + Other	(6) + (11) + (3) + (7) + (13) + (4) + (7) + (4) + (9) + (7) + (4)	78.9	89.8	-	-	1.98	7.55	0.036	0.294	1935k
#3	25	Bitmaps + RAW + Video + other	6 + 11 + 7 + (51)	87.9	94.6	93.3	98.9	4.26	10.16	0.063	0.227	2300k
#4	5	JPEG + RAW + Video + 5_Bitmaps + Other	(1) + (11) + (7) + (5) + (51)	90.2	94.1	98.6	99.1	0.50	4.95	0.043	0.400	1054k
#5	2	JPEG + non-jpegs	(1) + (74)	99.0	99.2	99.3	99.2	0.32	1.89	0.045	0.134	1036k
#6	2	JPEG + non-jpegs	(1) + (11 + 3 + 2)	99.3	99.6	99.5	99.7	0.41	4.34	0.047	0.259	1000k
1 Values in paranthesis represent the number of file types grouped into one class; values without paranthesis represent separate classes for each file type.

Table 4. TABLE IV: Computation time for global statistical & bigram features.

Feature	Description	Runtime [ms/block]
		4096	512
Kolgomorov Complexity	reduction of block length after bzip2/gzip compression [27, 28]	0.856	0.238
Arithmetic Mean	self explanatory	0.003	0.001
Geometric Mean	self explanatory	0.063	0.008
Harmonic Mean	self explanatory	0.048	0.007
Standard Deviation	self explanatory	0.046	0.006
Mean absolute deviation	standard deviation using abs. value of difference instead of squares	0.077	0.009
Hamming Weight	average number of set bits in the block	0.015	0.002
Kurtosis of Byte Value	measure of peakedness in the byte value distribution graph	0.110	0.015
Skewness	Measure of asymmetry of the byte value distribution graph	0.113	0.017
Longest Byte streak	length of longest streak of repeating bytes in the block.	0.981	0.156
Low ASCII Range Freq	frequency of bytes in range: 0x00 - 0x1F	0.006	0.001
Med ASCII Range Freq	frequency of bytes in range: 0x20 - 0x7F	0.013	0.002
High ASCII Range Freq	frequency of bytes in range: 0x80 - 0xFF	0.018	0.002
Shannon Entropy	$\Sigma \frac{1}{p}lo{g}_2\frac{1}{p}$	0.289	0.115
Total feature extraction time for NN-GF		2.636	0.579
Bigram matrix	byte co-occurrence matrix (NN-CO)	0.567	0.370

Table 5. TABLE V: Inference Time (ms/block) of FiFTy for various scenarios tested on a Tesla V100, P40, K80 GPU and 4 E5-2690 CPUs.

Scenario	512-byte blocks				4,096-byte blocks
GPUs	V100	P40	K80	CPU	V100	P40	K80	CPU
#1	0.043	0.058	0.181	3.102	0.146	0.256	0.947	16.836
#2	0.036	0.047	0.126	1.907	0.163	0.294	1.014	20.519
#3	0.063	0.086	0.353	5.694	0.119	0.227	0.628	11.526
#4	0.043	0.057	0.191	2.859	0.245	0.400	1.705	33.900
#5	0.045	0.057	0.232	3.893	0.097	0.134	0.541	12.569
#6	0.047	0.059	0.105	1.330	0.137	0.259	0.901	17.519

Table 6. TABLE VI: Accuracy on GovDocs and DFRWS for all scenarios

Data\Scenario	#1	#2	#3	#4	#5	#6
GovDocs (4096)	55.9%	45.2%	83.9%	88.0%	88.3%	82.3%
DFRWS (512)	41.4%	46.5%	60.8%	71.9%	86.8%	78.1%

Table 7. TABLE VII: File-type breakdown of classification accuracy for all models; misclassification reported for FiFTy on 4,096-bytes inputs.

		512 bytes				4096 bytes				Misclassification
Filetype	Tag	NN-CO	NN-GF	Sceadan	FiFTy	NN-CO	NN-GF	Sceadan	FiFTy	Top 1	Top 2
ARW	Raw	97.8	88.6	97.3	97.8	98.8	84.7	97.5	98.9	JPG (0.4)	MOV (0.1)
CR2	Raw	82.4	43.1	58.2	86.7	95.3	30.0	88.9	94.8	NEF (1.1)	DNG (0.6)
DNG	Raw	81.6	18.1	47.7	82.7	96.7	54.6	75.3	96.1	CR2 (2.0)	NEF (0.4)
GPR	Raw	98.7	89.8	98.1	99.2	99.5	87.2	99.3	99.9	CR2 (0.1)	PEF (0.1)
NEF	Raw	89.3	59.6	82.5	87.7	89.0	36.1	88.3	95.3	NRW (0.9)	JPG (0.9)
NRW	Raw	96.1	89.4	93.9	96.9	96.7	68.7	92.2	97.7	RAF (0.9)	3FR (0.3)
ORF	Raw	84.0	37.7	63.7	86.3	97.6	51.9	87.3	96.3	FLAC (1.4)	NEF (0.7)
PEF	Raw	96.1	84.3	88.8	95.1	98.9	77.5	96.8	99.1	NEF (0.1)	ORF (0.1)
RAF	Raw	97.9	95.4	96.6	98.3	91.3	34.0	73.6	87.1	FLAC (4.6)	3FR (4.1)
RW2	Raw	96.6	92.4	95.5	96.5	97.9	89.3	95.9	97.9	JPG (1.0)	PPT (0.2)
3FR	Raw	99.4	96.0	96.6	99.6	92.6	81.0	97.2	99.5	DNG (0.2)	TIFF (0.2)
JPG	Bitmap	77.5	53.2	83.8	83.5	92.5	48.6	91.8	86.3	PPT (3.9)	PPTX (2.3)
TIFF	Bitmap	90.0	34.5	83.8	96.1	98.6	57.9	97.7	99.0	APK (0.2)	PPT (0.2)
HEIC	Bitmap	2.0	1.8	2.9	31.7	4.0	0.0	5.2	49.5	MOV (15.3)	XZ (11.6)
BMP	Bitmap	97.5	90.1	96.1	98.0	99.4	91.3	98.7	98.4	3FR (0.5)	DLL (0.3)
GIF	Bitmap	84.5	42.3	72.3	93.5	98.5	54.3	96.2	99.1	MSI (0.1)	OGV (0.1)
PNG	Bitmap	47.6	1.5	21.7	67.2	72.9	0.4	69.5	88.0	PKG (1.7)	RAR (1.2)
AI	Vector	25.5	12.6	17.4	23.3	50.9	10.6	31.7	57.7	EPUB (6.9)	PKG (5.8)
EPS	Vector	98.8	95.2	97.8	98.7	98.0	88.2	97.4	95.8	AI (3.4)	PDF (0.4)
PSD	Vector	93.7	86.2	91.2	95.3	95.6	75.3	94.4	95.9	AI (2.1)	JPG (0.4)
MOV	Video	8.7	3.6	6.1	6.1	0.0	0.0	4.8	18.5	HEIC (43.1)	RAR (13.0)
MP4	Video	14.8	0.0	2.8	1.6	62.2	7.0	45.5	71.8	HEIC (11.2)	XZ (2.9)
3GP	Video	91.6	23.1	79.3	85.6	98.8	32.1	98.7	98.1	AVI (0.6)	DMG (0.2)
AVI	Video	13.8	1.7	14.7	10.9	64.9	32.6	66.4	67.1	HEIC (13.5)	MOV (6.2)
MKV	Video	90.5	23.5	82.3	88.0	99.1	57.5	98.9	98.4	OGV (0.4)	DMG (0.2)
OGV	Video	67.4	4.6	37.3	57.3	94.5	59.0	76.0	94.9	OGG (1.3)	MSI (0.8)
WEBM	Video	39.6	21.4	33.3	39.8	71.5	71.6	71.2	78.1	HEIC (10.0)	MOV (3.2)
APK	Archive	32.5	1.5	10.0	29.9	48.3	9.0	29.0	49.7	PKG (11.5)	PPT (3.5)
JAR	Archive	36.3	20.4	28.9	41.2	74.8	53.4	70.2	73.8	GZ (6.1)	RPM (4.1)
MSI	Archive	5.1	0.3	3.3	10.1	26.6	1.5	7.6	60.1	PKG (5.2)	EPUB (3.0)
DMG	Archive	7.0	1.1	3.4	18.2	17.4	0.4	9.1	19.4	HEIC (17.4)	PKG (14.6)
7Z	Archive	0.9	1.4	2.8	0.0	53.2	0.0	4.6	0.1	HEIC (47.8)	MOV (15.4)
BZ2	Archive	8.6	1.2	2.5	13.9	75.8	4.3	74.8	80.6	DMG (8.5)	HEIC (3.2)
DEB	Archive	9.9	7.0	2.9	13.8	1.5	0.0	3.9	0.8	HEIC (38.8)	RAR (10.6)
GZ	Archive	11.3	0.0	3.4	13.2	53.0	4.9	19.1	42.1	PKG (17.9)	RPM (11.6)
PKG	Archive	5.6	1.9	4.5	4.5	41.0	0.8	14.8	62.7	GZ (6.5)	APK (5.7)
RAR	Archive	13.0	4.2	3.0	24.0	18.0	1.7	3.9	46.6	ZIP (6.7)	HEIC (6.3)
RPM	Archive	7.9	2.6	5.4	13.6	10.7	0.5	8.9	21.9	PKG (12.8)	GZ (10.1)
XZ	Archive	0.0	19.4	2.9	0.0	0.0	1.3	5.0	12.1	HEIC (45.6)	MOV (15.4)
ZIP	Archive	16.2	1.8	9.5	13.8	36.6	9.7	22.2	34.2	RAR (21.6)	M4A (5.5)
EXE	Executable	9.2	0.8	2.4	0.3	4.3	0.3	5.4	2.7	HEIC (39.7)	MOV (13.0)
MACH-O	Executable	93.6	75.0	90.0	92.6	95.7	71.8	93.7	95.0	DLL (2.1)	TXT (0.6)
ELF	Executable	87.6	65.4	83.7	85.9	92.2	54.9	89.3	86.4	DLL (4.6)	MACH-O(2.1)
DLL	Executable	91.7	67.8	86.2	91.1	94.4	65.7	86.4	91.7	TXT (2.8)	ELF (0.6)
DOC	Office	91.4	44.8	70.5	87.2	91.6	12.4	77.6	88.9	PPT (2.2)	APK (1.2)
DOCX	Office	14.7	0.7	6.3	19.4	49.9	0.1	31.7	60.6	EPUB (20.7)	RAR (8.7)
KEY	Office	35.8	4.5	23.6	44.1	56.7	10.3	44.3	43.1	PPTX (19.9)	PPT (13.2)
PPT	Office	44.2	10.9	23.6	41.3	58.4	15.7	32.5	53.5	PPTX (6.2)	PKG (3.8)
PPTX	Office	38.3	14.4	24.0	44.4	32.7	2.6	20.2	36.0	PPT (14.6)	KEY (11.5)
XLS	Office	99.4	98.7	99.1	99.3	99.2	96.7	99.0	99.3	DLL (0.3)	MSI (0.1)
XLSX	Office	96.8	65.6	92.7	95.2	97.4	20.7	91.3	97.2	PKG (0.6)	AI (0.5)
DJVU	Published	18.0	3.5	10.5	24.6	34.8	84.2	24.2	30.4	HEIC (33.7)	MOV (12.5)
EPUB	Published	29.3	2.0	9.2	20.7	74.3	3.3	56.0	79.7	DOCX (10.5)	PKG (2.0)
MOBI	Published	72.5	72.8	71.5	72.4	74.4	69.6	72.9	73.7	EPUB (19.2)	DOCX (2.4)
PDF	Published	21.9	11.5	20.0	23.1	48.3	11.6	38.1	45.8	HEIC (13.1)	MOV (4.7)
MD	Human-readable	98.0	77.9	93.9	97.1	99.9	86.6	98.9	97.4	HTML (1.7)	TEX (0.5)
RTF	Human-readable	99.5	90.2	99.7	99.7	100.0	87.7	100.0	99.8	TXT (0.1)	DOC (0.1)
TXT	Human-readable	95.0	85.9	95.4	93.7	92.9	84.1	92.5	93.0	DOCX (2.1)	EPUB (2.0)
TEX	Human-readable	97.8	66.3	93.0	97.6	98.8	58.1	97.2	98.3	TXT (1.6)	RTF (0.1)
JSON	Human-readable	99.8	84.3	99.5	99.5	100.0	81.5	99.8	99.8	TXT (0.1)	HTML (0.1)
HTML	Human-readable	98.4	74.2	94.9	97.5	99.5	59.1	98.9	99.6	TXT (0.2)	MD (0.1)
XML	Human-readable	100.0	99.3	99.9	100.0	100.0	99.9	100.0	100.0	-	-
LOG	Human-readable	100.0	99.3	99.9	99.9	99.9	91.2	99.9	99.9	TXT (0.1)	TXT (0.1)
CSV	Human-readable	99.9	92.1	99.6	99.6	99.0	87.3	98.4	99.7	TXT (0.1)	TXT (0.1)
AIFF	Audio	95.7	96.2	94.9	99.4	94.5	95.0	98.4	98.8	TXT (0.8)	TXT (0.8)
FLAC	Audio	66.9	18.5	13.2	74.2	87.8	76.2	85.3	97.9	ORF (0.4)	ORF (0.4)
M4A	Audio	92.6	34.6	86.7	94.3	96.5	29.6	98.4	98.2	HEIC (0.3)	MOV (0.3)
MP3	Audio	94.0	70.8	85.1	94.4	98.9	77.6	98.8	98.9	APK (0.2)	APK (0.2)
OGG	Audio	84.2	55.3	72.3	90.4	95.4	37.3	93.3	95.5	FLAC (2.3)	FLAC (2.3)
WAV	Audio	99.9	99.9	99.8	100.0	74.0	58.2	97.6	98.7	TXT (0.6)	APK (0.4)
WMA	Audio	99.2	73.8	98.7	98.2	99.9	95.7	99.9	99.9	3GP (0.1)	APK (0.1)
PCAP	Misc	55.4	32.9	48.6	51.0	95.6	69.6	94.7	95.4	HEIC (0.9)	MOV (0.4)
TTF	Misc	98.1	86.8	96.3	98.1	99.2	90.2	98.4	99.3	DLL (0.3)	DLL (0.3)
DWG	Misc	96.5	90.1	93.0	96.2	98.8	67.5	97.0	96.9	BMP (0.4)	BMP (0.4)
SQLITE	Misc	98.9	87.2	97.1	98.6	99.5	91.3	98.6	99.8	DLL (0.1)	DLL (0.1)
-	Total Wins	32	2	3	38	27	1	4	43	-	-
Note: Ties were broken depending upon runtime speed.

Equations

$$f:B\in {\mathbb{Z}}_{255}^n\to T\in \{\mathrm{JPG},\mathrm{PPT},...,\mathrm{PNG}\}$$

$$\hat{\theta }=\underset{\theta \in \Theta }{\mathrm{argmax}}EI(\theta )=\underset{\theta \in \Theta }{\mathrm{argmax}}\frac{p_1(\theta )}{p_2(\theta )},$$

Full text

I Introduction

Memory forensics and data carving heavily rely on data/file-type identification from small chunks of memory (Fig. 1). Many existing techniques are based on simple heuristics (like common byte markers) or hand-engineered features (unigrams [1, 2, 3], bigrams [1, 2], Shannon entropy [4, 1, 2, 3] or Koglomorov complexity [4, 1, 2, 3], etc. - see Section II for more details). Such an approach suffers from multiple scalability issues, ranging from a limited number of supported file-types, deteriorating classification accuracy, up to slow processing speed. Feature extraction alone is responsible for a bulk of the processing time [1]. (Our evaluation corroborates this; see Tab. IV.) Some studies consider boosting, and compute individual features only when necessary [5].

Many file carvers rely on simple heuristics and entropy thresholding to find JPEG blocks in raw disk images [6]. Although more sophisticated methods exist they often focus on distinguishing file-types that are very different, e.g., JPG, CSV, HTML, DOC, XLS [7, 3, 1, 8, 9]. Many of these file-types exhibit distinct byte frequency statistics and are also unlikely to occur in memory images from cameras and IoT devices. The complexity of the problem leads to trade-offs which can compromise the speed [4] or accuracy [1, 9]. While allowing for rapid triage, reliance on simple heuristics leads to excessive false positive matches, which need to be considered and discarded at later stages. This bottleneck affects even the most recent carvers [10, 6, 11, 12].

To address these limitations, we explore modern neural networks (NNs) to dispense with the explicit feature computation. Our search for compact architectures delivering good balance between accuracy and speed resulted in FiFTy. It is a scalable and fast file-type identification tool comprising of models based on six real-world scenarios (see Tab. III) and two common block sizes (512 and 4,096 bytes).

Our models are 1-D convolutional neural networks (CNNs) that take blocks of raw bytes as input, and embed them in a trainable latent space - akin to word embeddings that proved successful in natural language processing [13]. The resulting classifier consistently outperforms state-of-the-art models reported in the literature and can be an order of magnitude faster. We base our results on a novel dataset with 75 diverse and representative file-types (nearly twice as much as the largest prior work [1]). Our model achieves an average accuracy of 77.5% (using memory blocks of 4,096 bytes) with processing speed of $\approx$38 sec/GB. The previous state-of-the art system, Sceadan [1], achieved 69% accuracy and required 9 min/GB in the same conditions (see detailed breakdown in Tab. I).

While several recent studies started to explore deep learning models for the problem, their solutions have fallen short. Hiester [14] considered fully connected, convolutional and recurrent networks, but studied only 4 very different file-types (JPEG, GIF, XML and CSV) which can be easily distinguished using simple techniques employing statistical features. The proposed models convert input byte blocks to 1-hot encoding, which leads to unnecessary dimensionality explosion. Chen et al. [15] took a different approach, and reshaped 512-byte blocks as $64\times 64$ grayscale images. Despite using modern CNNs and a small 16-type dataset, their approached delivered sub-par performance with accuracy of only 71%.

Our study is significantly broader in scope. In addition to the largest reported dataset (Tab. I), we also study several smaller scenarios with application-specific selection of file-types. We focus on graphic and photographic data, which occur in photo carving applications. Our selection reflects the most popular files, that can be found on SD cards from modern cameras, smartphones and camera-enabled IoT devices. We also include multiple previously unstudied types like HEIC/HEIF (high efficiency image file format [16]) and numerous RAW photo varieties (including GPR from GoPro action cameras). To accommodate variations of cluster size in different file systems, we consider both 512 and 4,096-byte inputs.

To ensure fair comparison, we evaluate our approach against three strong baselines: (1) a state-of-the-art file-type classifier, Sceadan [1]; (2) a fully connected neural network trained on commonly used hand-crafted global block features, denoted as NN-GF; (3) a convolutional neural network trained on byte co-occurrences, denoted as NN-CO. The comparison was done on the same datasets, and using the same computational resources (using GPU acceleration, when possible). The proposed approach consistently outperformed all baselines with respect to speed, overall accuracy, and individual misclassification rates.

The contributions of our work includes:

• •

A compact neural network architecture for file-type classification with state-of-the-art performance and runtime; we also explore the hyperparameter space using the Tree-structured Parzen Estimator (TPE) [17].

• •

Detailed evaluation of the proposed approach against three strong baselines, including a state-of-the-art tool based on support vector machines [1], and modern neural networks trained on hand-crafted features.

• •

Detailed analysis and discussion of several classification scenarios relevant to photo carving applications.

• •

A ready-to-use, open-source implementation (Python 3) available at https://github.com/mittalgovind/fifty and via pip3 install fifty.

• •

A novel data-set (75 popular file-types) publicly available at http://dx.doi.org/10.21227/kfxw-8084.

The paper is organized as follows. Section II reviews related work. In Section III, we describe the proposed model, discuss key design decisions and hyper-parameter selection. We report the results of experimental evaluation in Section IV and conclude in Section V.

II Related Work

Data carving is a technique used to identify and extract specific file-types from memory blocks, without assuming explicit access to file-system metadata. The simplest carving methods rely on starting and ending signatures (e.g., magic numbers). However, they are inadequate when dealing with heavily fragmented file-systems or in the presence of address space randomization (often used to alleviate memory cell wear in modern SSDs). Similar techniques are also applicable to memory forensics, which involves analyzing random access memory (RAM) dumps. In such cases, it is helpful to use a file-type classifier that can blindly identify the data-type of small memory blocks.

Most of file-type classifiers relied on hand-crafted heuristic features, and were limited to very small datasets. Only recently researchers began exploring (deep) neural networks, but the reported solutions were still limited in both scope and performance. Tab. I summarizes the previous approaches, their evaluation scenarios, and the reported performance. The table is divided into two sections: the top 4 rows summarize results of our evaluation of FiFTy and three baseline methods (Sceadan, NN-GF, NN-CO) in the same conditions (all 75 file-types; 4,096-byte blocks; same hardware); the remaining rows summarize results, as reported in their original publications.

II-A Hand-crafted Features

Karresand et al. [7] developed Oscar, a classifier based on two features: rate of change (RoC) of consecutive bytes, and binary frequency distribution (BFD). They defined a 1-norm metric as the distance of these features from pre-computed centroids. This approach worked well for JPEG photos due to byte-stuffing [18], a technique involving insertion of 0x00 after 0xFF. The RoC distribution captured this feature easily and thus lead to near-perfect detection of JPEG files (99.2%) but with high false positive rates for other high-entropy filetypes, e.g., Windows executables and ZIP.

Veenman [3] trained a supervised classifier on a dataset described by 1-grams, entropy, and Kolmogorov complexity. They employed a two-stage strategy with multi-class and binary classification (one-against-all) using the Fischer linear discriminant (FLD) aiming to maximize inter-class distances. While this method yielded high accuracy for JPEG files (98%), it averaged to only 45% on a dataset with 11 filetypes. Calhoun et al. [8] also used the FLD, but included additional features, e.g., longest common strings and subsequences. While they report good results, and consider various feature combinations, the study was limited to only 4 filetypes (JPEG, PDF, GIF and BMP).

Fitzgerald et al. [2] used unigrams and bigrams along with entropy and complexity measures to train a support vector machine (SVM) based classifier. They used the bag-of-words model (here, bag-of-bytes) to classify text documents (file fragment). This method worked well for text formats but failed for all high entropy filetypes.

Beebe et al. [1] developed Sceadan, a tool based on various statistical features, including unigrams, bigrams, entropy, longest streak, etc. The features were originally extracted from 512-byte blocks, and organized into four sets: unigram, bigram, all other global features and a subset of global features. These feature sets were then used for classification using SVMs with linear and radial basis function (RBF) kernels. The authors concluded that increasing the number of types of features had a negative effect on the classification accuracy, and that unigrams & bigrams worked best as they are. Until now, Sceadan remains the state-of-the-art open source tool and the corresponding paper is the most comprehensive study to date - it covered 38 diverse filetypes and reported accuracy of 73.8%. We used Sceadan as an important baseline in our experiments. In Tab. I, we collect both the results reported by the authors, as well as the results from our evaluation (with both 512 and 4096-byte blocks). More information about the experiment is available in Section  IV-B.

Xu et al. [9] converted 512-byte blocks into $32\times 32$ px grayscale images and extracted a 512-D GIST image descriptors based on Gabor filters. Finally, they experimented with various classification algorithms, (e.g., Naive-Bayes, SVMs, k-nearest neighbours, SMO, J48) and concluded that KNN performed best both in terms of speed and accuracy.

Zheng et al. [19] also relied on SVMs but employed only two features (binary frequency distribution and Shannon entropy). They explicitly differentiate between file-types (that could contain other embedded files, e.g., PDF and DOC) and data-types (that are often embedded within other files, e.g., JPEG photos), but ultimately deal only with filetypes without any embedded foreign objects.

Beebe et al. [20] experimented with clustering of 52 filetypes, and proposed a two-layer hierarchical structure where filetypes are first grouped into 6 classes and classifiers are trained separately for each class. The authors used the same features as all other studies, including unigrams, bigrams and global statistical features (mean, entropy, complexity, etc.). The paper focuses on exploratory analysis based on K-means clustering, and reports file-level accuracy of 74.1%. The utilized classification strategy is not reported.

II-B Automatic Feature Extraction

Wang et al. [4] used sparse coding for automatic feature extraction. They computed higher-order $N$-grams ($N$ = 4, 8, 16, 32, 64) and stored them in dictionaries. The dictionaries are subsequently averaged and concatenated with bigram and unigram frequencies. Similarly to other studies, feature extraction takes most of the processing time and makes the approach infeasible for high-volume applications. The reported accuracy was also sub-par, reaching on average 61.3% for 18 filetypes.

Hiester [14] considered fully connected, convolutional and recurrent neural networks with raw bytes fed directly as model inputs (1-hot encoded). The study included only 4 very different filetypes (JPEG, GIF, XML and CSV) which are easy to distinguish with existing techniques. The adopted 1-hot encoding contributes to excessive dimensionality and slow training. Only the recurrent network achieved satisfactory accuracy, and did so at excessive computational effort. The presented approach is not scalable to real-world applications.

[FIGURE:]

Chen et al. [15] took an approach similar to Xu et al. [9] by converting 512-byte fragments into $64\times 64$ images. Despite using a CNN architecture, the model delivered sub-par performance with accuracy of only 71% among 16 filetypes. Moreover, the network failed to classify even very distinct types like HTML or plain text files.

III Proposed Method

This section discusses our motivations for the approach, use-cases, description of the general architecture of models included in FiFTy and the dataset used.

III-A Formal Problem Statement

We address the problem of inferring the file-type of a block of bytes in the absence of any side information, i.e., we aim to obtain a function $f$ that maps a single block $B$ of $n$ bytes to its file-type label $T$:

[TABLE]

where $\mathbb{Z}_{255}=[\texttt{0, \ldots, 255}]$. We will model function $f$ as a neural network, which allows for jointly learning a suitable feature representation, and the final classifier.

We focus on the more common file-type classification and not on data-type classification. The latter is more granular, and involves complex combinations where one data type might be embedded into another (e.g., images embedded into PDF documents, base64-encoded images embedded into HTML, or sections of structured data within human-readable text). This leads to prohibitive complexity of training data preparation, more complex carving logic, and still does not solve the necessity to handle heterogeneous blocks with multiple data types (especially for larger blocks). Hence, file-type classification is often the best choice in practice (cf. Section II).

III-B Design Objectives

In this work, we focus on three objectives – speed, accuracy and generality – to make design choices. We aim to meet these objectives by relying on modern neural networks, which can automatically learn effective feature representations from raw bytes and dispense with explicit feature computation - a major bottleneck in previous solutions [1] (Tab. IV). Moreover, NN models can take advantage of advanced accelerators (GPUs, TPUs, VPUs) and can be easily deployed in desktop, server, mobile and edge environments [23]. We used Keras which abstracts model definition and allows to exchange computational backends [24]. In our experiments, we used Python 3.6 and the Tensorflow (1.5.0) backend. All experiments were run on a server with 4 Intel Xeon CPUs (E5-2680 v4 @ 2.40GHz) and a single Tesla V100 GPU. Our models achieve nearly identical runtime regardless of the number of output classes (Tab. III).

To facilitate representation learning, we collected a novel dataset with 75 filetypes including: 6 bitmap photo formats, 11 RAW photo formats, 7 video formats, 7 audio formats, and many others. Hence, not only is it the largest reported dataset, but also the only one uniquely suited for data carving applications. However, to make our discussion as general as possible, we consider 6 application scenarios with different levels of granularity (see Section IV-A).

III-C General Model Architecture

We feed raw byte blocks as the input to our models. Instead of one-hot encoding, we use a trainable embedding layer in a manner similar to natural language processing (NLP) [13]. In our work, we treat file-fragments as a sequence of bytes and intend to capture characteristic byte transitions patterns, inherent to many file-types.

Embedding layers are commonly used for learning a fixed-length real-valued representation of variable-length discrete words, while maintaining their linguistic context [13]. In NLP, word embeddings can learn relationships between concepts, and allow for meaningful arithmetics. In our work, we use the embedding layer mainly for their efficiency in converting sparse binary features (1-hot encoded words in the dictionary) into compact real-valued representations. This allowed us to reduce a single byte (256 binary features in 1-hot encoding) into 16-64 real-valued features, which are more meaningful for machine learning models. The following layers will further reduce this dimensionality by pooling spatially-adjacent features of intermediate representations.

Our architecture is 1-dimensional and relies on convolutions, followed by max-pooling to gradually reduce the size of the problem. We refer to these units as convolutional blocks and vary their number as a tunable hyper-parameter. Our final vectorized representation is obtained by average pooling with dropout. In the last step, two fully connected layers map the obtained feature vectors into class probabilities. All hidden layers are activated by Leaky Rectified Linear Units (LeakyReLU) with $\alpha=0.3$.

The overall architecture is shown in Fig. 1. We used the same general architecture for all tested scenarios (Section IV-A; Tab. III) and only vary hyper-parameters (Section III-D). The model shown in the figure corresponds to hyper-parameters optimized for scenario #3 with block size of 512 bytes. The reported numbers indicate how dimensionality changes with successive processing steps, and how parametrization is allocated throughout the model.

We also experimented with recurrent architectures using GRU units placed after the last convolutional layer. While initially this seemed to be more accurate, careful hyper-parameter tuning eventually reduced performance gap to 1%, which no longer justified nearly 40$\times$ increase in processing time. While probably better performance could be obtained, longer processing time makes it difficult to explore the hyper-parameter space. As a result, we currently believe that carefully tuned 1-D convolutional models are the best choice with respect to the accuracy-speed trade-off.

III-D Design Space and Hyper-parameter Optimization

We use the same general network architecture described in Section III-C for all our experiments, and only tune selected hyper-parameters to adjust the model to the scenario at hand. The scenarios differ in the number of output classes and reflect various application needs, ranging from general-purpose to increasingly specialized photo carving (Section IV-A).

III-D1 Hyper-parameter Definition

After hand tuning the general architecture, we defined 6 hyper-parameters which will be optimized automatically using the Tree-structured Parzen Estimator (TPE) [17]. For each dimension, we chose a few possible values spanning a wide range of reasonable choices.

• •

# dense units: the number of units in the hidden dense layer; $\{16,32,64,128,256\}$;

• •

embedding size: dimensionality of the trainable real-valued embedding space; $\{16,32,48,64\}$;

• •

conv. kernel: the size of the convolution kernel (same for all layers); $\{16,32,64,128\}$;

• •

conv. stride: convolution stride (same for all layers): $\{3,11,18,27,35\}$;

• •

# conv. blocks: the number of convolutional blocks (conv + max-pooling); $\{1,2,3\}$;

• •

max-pooling size: max pooling window size; $\{2,4,6,8\}$.

In a separate, prior experiment, we established that adoption of dropout is beneficial before the hidden dense layer (with dropping probability of 0.1).

In Fig. 3, we graph validation loss and accuracy achieved by the same model versus the amount of data used for training. We observe that even using less than 20% data gave us proportional results that could be used to compare the models. Therefore, to speed up the hyper-parameter space exploration, we used 10% of the training data and 40% of the validation data instead of using the full set of data everytime.

We sampled network configurations using the TPE (we used the implementation available in the hyperopt package [25]). In total, we collected 225 models for each of the 12 scenarios (6 base scenarios with 512 and 4,096-byte inputs, respectively). The first 20 configurations are fully random, and the remaining are sampled to maximize model performance. Most scenarios completed within a day, except for the largest two (scenario #1 with 75-classes for both 512 and 4096-byte inputs) that took up to 36 hours. Finally, we retrained the best candidate models on the entire dataset, and used a separate hold-out test set for the final evaluation.

III-D2 Tree-structured Parzen Estimator

TPE is a sequential model-based optimization (SMBO) algorithm [26] proposed by Bergstra et al. [17], which models the probability that given a scoring metric $L$, a hyperparameter value $\theta$ leads to a better performance, denoted as $p(\theta|L)$. The algorithm samples subsequent values for evaluation based on historical performances.

The algorithm can be summarized as follows:

Initialize the model with $N_{\alpha}$ random samples from the hyper-parameter space $S$. 2. 2.

Divide the sampled models into two groups based on a threshold on the performance quantile. 3. 3.

Estimate the probability density functions for top performers ($p_{1}$) and the rest ($p_{2}$). 4. 4.

The next sample $\hat{\theta}$ is determined using the expected improvement:

[TABLE]

where $p_{1}(\theta)$ and $p_{2}(\theta)$ are the group membership probabilities defined above. 5. 5.

Evaluate the network defined by $\hat{\theta}$ and repeat steps 2 to 5 until $N$ samples are collected.

Fig. 2 shows the modeled probabilities and the expected improvement (EI) for an example hyper-parameter from our design space (conv. kernel). Fig. 2(a) illustrates the initialization step of TPE. It collected 20 random samples and divided it into two equal groups using a quantile (here, 0.5). The EI is calculated and the value of 64 maximizes it. Fig. 2(b) illustrates the distributions after 200 samples have been collected, showing that the worse sample distribution is moving towards the better sample distribution (here, towards higher kernel size).

III-D3 Final Models and Key Observations

In Fig. 4, we show the impact of the hyper-parameters on the validation accuracy for 4,096-byte inputs (see supplementary materials for the 512-byte version). Each plot shows a violin plot with the distributions of validation accuracies (top), and the expected improvement from the TPE. The best hyper-parameters are marked in dark green. Note that these values correspond to the best sampled network, and due to non-trivial interactions and stochastic nature of the process, may not necessarily correspond to seemingly best values suggested by TPE. The key observations are as follows:

The convolution kernel size and stride seem to have the stringest and most consistent effect. Larger values lead to better performance regardless of the scenario. 2. 2.

A single convolutional layer is typically not enough, two or three blocks are needed for best performance. 3. 3.

In most scenarios, larger embedding sizes seem to be preferred, but good network with good performance can also be obtained for smaller embeddings - especially for smaller number of classes. 4. 4.

For 4,096-byte blocks, stronger max-pooling seems to lead to better performance.

Tab. II shows the final architectures obtained for all scenarios for both 512 and 4096-byte inputs.

IV Experimental Evaluation

In this section, we present the results of our experimental evaluation. First, we introduce a novel dataset and distinguish several evaluation scenarios. Then, we describe 3 reference models, including a state-of-the-art tool Sceadan and two alternative NN-based baselines. Finally, we report our results discuss the key limitations and causes of errors.

IV-A Scenarios and Dataset

Previous works relied mostly on small-scale private datasets or on GovDocs [22] - an unbalanced corpus with 20 filetypes comprising 99.3% and 43 filetypes comprising the remaining 0.7% of the dataset. FiFTy comes with a novel, large and balanced corpus, named FFT-75, comprising of 75 different filetypes with emphasis on multimedia types which dominate disk images extracted from contemporary devices.

We prepared the dataset by downloading public files from various public Internet sources and from our own collection. Instead of sharing the actual files, we sampled 102,400 small blocks for each filetype to obtain a balanced dataset well-suited for classification. Large and complex files cannot be recovered from the obtained blocks. For tiny text files potentially smaller than block size, we made sure to rely on open-source materials that explicitly permit sharing. To accommodate various file-system cluster sizes, we consider both 512 and 4,096-byte blocks. The blocks are shuffled to evenly distribute the filetypes within the training (80%), validation (10%) and hold-out testing (10%) subsets.

The included file-types are grouped by use-cases that include bitmaps (6 formats), RAW photographs (11), vector graphics (3), videos (7), archives (13), executable files (4), office documents (7), rich publications (4), human-readable text files (9), audios (7) and 4 others. More information, including detailed accuracy and misclassification rates are collected in Tab. VII. To the best of our knowledge, the obtained corpus is the largest and most comprehensive to date.

We distinguish 6 scenarios that correspond to various use cases and have different granularity (Tab. III). We focus on photo carving applications, where scenarios #3 to #6 are the most relevant:

• •

#1 (All; 75 classes): All filetypes are separate classes; this is the most generic case and can be aggregated into more specialized use-cases.

• •

#2 (Use-specific; 11): Filetypes are grouped into 11 classes according to their use (see tags in 2nd column of Tab. VII); this information may be useful for more-detailed, hierarchical classification or for determining the primary use of an unknown device.

• •

#3 (Media Carver - Photos & Videos; 25): Every filetype tagged as a bitmap (6), RAW photo (11) or video (7) is considered as a separate class; all remaining types are grouped into one other class.

• •

#4 (Coarse Photo Carver; 5): Separate classes for different photographic types: JPEG, 11 RAW images, 7 videos, 5 remaining bitmaps are grouped into one separate class per category; all remaining types are grouped into one other class.

• •

#5 (Specialized JPEG Carver; 2): JPEG is a separate class and the remaining 74 filetypes are grouped into one other class; scenario intended for analyzing disk images from generic devices.

• •

#6 (Camera-Specialized JPEG Carver; 2): JPEG is a separate class and the remaining photographic/video types (11 RAW images, 3GP, MOV, MKV, TIFF and HEIC) are grouped into one other class; scenario intended for analyzing SD cards from digital cameras.

Tab. III shows the class composition of each scenario, overall accuracy, JPEG accuracy, training and inference time along with number of samples in the particular dataset for all considered scenarios. The table shows that in general the accuracy improves as the number of classes decreases - except for scenario #2 because it is more difficult than scenario #3. Scenario #2 groups filetypes by use, and not by internal structure, which may exhibit completely different statistics, e.g., the bitmap class consists of TIFF, HEIC, BMP, GIF, PNG and JPEG which include images with no compression, as well as lossless or lossy compression. These distinct characteristics can be easily treated as separate classes in scenario #3.

IV-B Baseline Methods

To validate our approach, we compare FiFTy against three baseline methods: the state-of-the-art tool Sceadan, and two neural networks trained on global statistical features (NN-GF) and co-occurrence features (NN-CO). We test all methods on the largest scenario (#1) for both 512 and 4,096-byte blocks. For Sceadan, we used the open-source implementation and the provided training scripts [29]. For the remaining two baselines, we used the same TPE-based procedure (Section III-D) to optimize the models and their hyper-parameters. The final network architectures are collected at the bottom of Tab. II.

IV-B1 NN-CO

The co-occurrence or bigram features were commonly used for file-type classification [4, 1, 2, 7]. The feature is constructed from a matrix of byte co-occurrences, i.e., a matrix of shape (256, 256) with each element $a_{ij}$ corresponding to the frequency of occurrence of byte value $j$ after byte value $i$. Bigram features lead to good classification accuracy, but tends to be very sparse - especially for 512-byte blocks - and relatively slow to compute (Tab. IV). As a result, they tend to be a performance bottleneck [1]. Due to the sparsity and excessive memory requirements, we down-sampled the matrices to (128, 128) by average pooling. The NN-CO model comprises four 2-dimensional convolutional layers, and two dense layers operating on a flattened 1-dim representation. We keep the same kernel size and stride parameters for all layers, and treat them as hyper-parameters. Additional hyper-parameters include number of convolutional layers, presence of max pooling and the number of units in the hidden dense layer.

IV-B2 NN-GF

We use 14 global statistical features (Tab. IV), commonly used in previous studies, as the input to the final baseline. Each feature maps an input block into a single scalar value. The features vary widely with respect to their computational effort, and also lead to performance bottlenecks in high-volume environments. The numbers reported in Tab. IV correspond to feature extraction in Python 3.6 performed by calling low-level routines from popular external libraries (we relied on scipy 1.3.0 and numpy 1.16.4). This makes the runtime comparable to low-level C/C++ code. The classifier consists of four dense layers, and the number of units in all hidden layers was used as separate hyper-parameters.

IV-C Evaluation Results

Tab. I summarizes the overall accuracy and processing speed for FiFTy and all of the baselines on scenario #1 with 75 file-types. We report both the average accuracy and the JPEG identification accuracy. FiFTy clearly outperforms all of the other methods - both with respect to accuracy and runtime. (Due to clearly inferior performance, we omit NN-GF in the following discussion.) On 4,096-byte blocks, FiFTy achieved average accuracy of 77.5%, compared to 75.3% and 69.0% for the runner-ups (NN-CO and Sceadan) which were 7$\times$ and 14$\times$ slower, respectively. On the smaller 512-byte blocks, the accuracy deteriorates to 65.6% vs. 64.4% and 57.3% with an even steeper speed-up gains - 20$\times$ and 34$\times$.

In terms of training time, FiFTy was comparable to NN-CO and significantly faster than Sceadan, which took 44 hours for 512-byte inputs and over 10 days for 4096-byte inputs. Our model required $\approx$ 8 and 12 hours to train the final model on the entire dataset (on the slower P40 GPU), and between 24 - 36 hours for hyper-parameter search (depending on the scenario at hand). If needed, faster training times are possible - a sub-set of the training set and a slightly simpler model can cut down the training time to around 1-2 h with only minor degradation in accuracy (Fig. 3).

Fig. 5 compares confusion matrices for all of the models. Due to a large number of classes, the included figure depicts groups of file-types and marks selected individual file-types of interest. For detailed file-type-level results, please refer to supplementary materials or Table VII which summarizes the most common mistakes. FiFTy delivered the best accuracy for 43 out of 75 individual file-types. The NN-CO was the winner for 27 and Sceadan for 4 (for 4,096 blocks). Whenever FiFTy lost, the difference was marginal.

IV-C1 Ensemble of FiFTy and NN baselines

We consider ensemble learning to assess whether inclusion of hand-engineered features could improve the accuracy. By averaging the probabilities prodicted by FiFTy, NN-CO, and NN-GF we can achieve 78.7% on 4,096-byte inputs - a marginal improvement not worth the enormous computational effort. We also tried training a simple combination NN, but did not observe any further improvements.

IV-C2 Additional Scenarios

Tab. III summarizes FiFTy evaluation results for all test scenarios. As the number of classes decreases, we obtain increasingly more reliable results, reaching 94.6 % accuracy for 25 multimedia file-types and 99.7% accuracy for the most constrained JPEG-vs-others scenario. To the best of our knowledge, these are the best such results to date. The best previously reported JPEG identification accuracy was 99.2% on a dataset with only 4 very different file-types [7]. Our classifier considered 16 other photographic and video types in the alternative class, which makes it suitable for photo carving applications. Even for the detailed multimedia scenario #3 (25 file-types), we achieved JPEG classification accuracy of 98.9%.

IV-C3 Runtime Evaluation

The classifier comprising the heart of FiFTy is a neural network implemented in a high-level framework Keras. The model can take advantage of GPUs to speed up computations. In this experiment, we assess the inference speed on several popular server GPUs (nVidia Tesla V100, P40 and K80). The results are collected in Tab. V. Even with parallel processing, general-purpose CPUs are not competitive for high-volume applications. Adoption of GPUs significantly speeds-up the computations. On the fastest considered GPU and with 4,096-byte blocks, we achieved the speed of 214 Mbps for the largest scenario with 75 classes, and up to 322 Mbps for the simplest 2-class problem.

IV-D Misclassification Analysis

The last two columns in Tab. VII show a misclassification breakdown for all of the considered file-types. For brevity, we show only top-2 types most confused with each given file-type (for scenario #1/4096). There are several reasons for such errors. We list the most interesting observations below:

• •

HEIC & MOV: HEIF is a new image format used in the recent iPhones [30]. It is essentially a container for still photos encoded using the intra-frame mode of the HEVC video codec [31]. This leads to misclassification errors, since HEIC photographs are essentially small movies and tend to be confused with actual MOV videos. Moreover, due to high entropy, both formats are sometimes confused with archives - especially the XZ and RAR formats.

• •

Archives: Data compression (whether lossy or lossless) reduces redundancy and stores the same information using fewer bits. Archive file-types are characterized by high entropies, thus minimizing statistical artifacts which could be exploited to identify them. Almost all included types suffer from this problem. Although the proposed FiFTy improves upon previous methods, the accuracy still averages around only 39%. The only format with reasonable accuracy was BZ2, which obtained 80%.

• •

Compound filetypes: Many filetypes, e.g., PDF, PPT, KEY, DOC, MOBI are containers and may have embedded objects of other types, e.g., JPEG or other bitmaps, audio, video, etc. This confuses all classifiers, but seems to be mostly a data labeling problem. In practice, such segments should probably be carved using dedicated tools for the embedded objects. However, the container file should most likely be carved as well. More complex logic will be needed both for carving such data as well as properly labeling the training data for file-type classifiers.

• •

Presentation File-Types: We observed a cluster of mutual misclassifications for common presentation file-types (PPT, PPTX and KEY). Interestingly, we didn’t observe simialar relationships between document and spreadsheet types. To some extend the problem may stem from file-type conversion (see below).

• •

Errors stemming from type conversion: Some files in our dataset were generated by converting from other files of a similar type (to reach the required number of samples). We observed elevated mutual misclassification rates for some of the pairs (WAV & AIFF, KEY & PPTX, PPT & PPTX and MOBI & EPUB). Other pairs (RTF & TXT, HTML & MD or TEX & MD) remain unaffected. More information about the conversion can be found in the documentation of the dataset.

IV-E Generalization

We tested generalization capabilities of FiFTy on the GovDocs [22] corpus and a file-system memory dump from a DFRWS111http://old.dfrws.org/2006 carving challenge.

GovDocs

GovDocs [22] is a large corpus of files submitted to US government websites. We took a portion of that corpus with sufficient number of testing examples and overlap with our file-types: CSV, GIF, GZ, HTML, JPG, LOG, XLS, PPT, DOC, RTF, PDF, MP3, PNG and TXT. We used 4,096-byte blocks for preparing a shuffled disk image. Table VI summarizes the validation accuracy for all scenarios, and Fig. 6ab show the confusion tables for two example scenarios.

Please note that we report accuracy on random memory blocks derived from the corpus and not on entire files as is sometimes done by other studies. Moreover, the displayed rows may not add up to 100% because we only show the common classes between GovDocs and scenario #1. For example, none of DOCX, PPTX and XLSX filetypes are present in GovDocs. Therefore, if a PNG block is predicted to be an APK block, then it is missing in the presented confusion matrix. Similarly for scenario #4, there are no raw images or video files, therefore their rows are empty. Absent classes can potentially indicate classification errors.

The key observations about the mistakes made by the classifier are as follows:

• •

In scenario #1, JPG is confused with PDF, PPT and DOC because most of these files had JPG images embedded in them. Similar problems occur for PNG.

• •

In scenario #1, RTF and TXT files are confused with other file-types because the samples in Govdocs were very often structured and mislabeled (e.g., tabular data, or postscript).

• •

In scenario #1, CSV could be confused with HTML because the CSV samples have more quotation marks (0x22) than commas (0x2c), which is similar to HTML files.

DFRWS

We took a disk image from the DFRWS-2006 challenge. The file system has cluster size of 512 bytes and contained XLS, GZ, JPG, HTML, DOC and TXT files deliberately shuffled to make carving harder. Table VI summarizes the validation accuracy for all scenarios, and Fig 6cd show the results on two example scenarios.

A key observation for the above two samples is that although the classes of both scenarios #5 and #6 are the same, we see different accuracies because the other class consists of different sets of file-types (refer Sec. IV-A). It is important to note that the classifier for scenario #6 was not trained on any other file-types in DFRWS image, except JPG. Scenario #5 - other class consists of all 74 file-types and scenario #6 - other class consists of 16 multimedia file-types. Classifier for scenario #5 learned better what is not JPG and for #6 learned better what is JPG.

Table VI presents a summary of results on the above two samples for all the six scenarios. We observe that the tool gives all results comparable results to the original testing accuracy on FFT-75 for scenarios #3 - #6. All these scenarios are related to multimedia file-types. Hence, the tool does show evidence that it is efficient and accurate asset for multimedia carving and related applications.

IV-F Limitations and Future Work

The presented neural network architecture reflects both the effects of hands-on experimentation and automatic hyper-parameter tuning. Out of necessity, we explored only a small fraction of the design space. It is certainly possible to expect that better architectures could be found using modern architecture search [32] or even random sampling methods [33]. These methods still remain an area of active research, and its application for the problem at hand is left for future work.

The FFT-75 dataset used for training suffers from some dependency problems. There are a lot of DOC, PDF and PPT files that contain images embedded in them. This makes it difficult for the classifier to learn and generalize. An approach to solving this could be to carve the images and other embeddable data-types out of these files and train on the dataset prepared from these files.

Our current hyper-parameter search was driven mainly by validation accuracy. This sometimes leads to surprisingly high model complexity for seemingly simpler problems (Tab. III). In the future work, we’re planning to investigate the relationship between model complexity (e.g., parameter count), computational effort and accuracy. The problem requires a separate treatment since it heavily depends on the target runtime environment, and both hardware platforms and tensor libraries can result in widely different behavior [34].

Additional improvements in classification accuracy can be expected by adopting more complex logic to deal with embedded filetypes as well as with the dependencies between labeling decisions among neighboring blocks. For example, random field models could be used to reach an agreement regarding a larger chunk of data based on weak predictions of individual blocks [35]. This information could further be combined with the locations of known headers/footers.

Finally, we emphasize that in real applications additional care may be needed for certain filetypes to accommodate the expected content characteristics. For example, statistical properties of video files may depend not only on the container, but also on the audio/video codecs. In our study, we relied on material that we captured using common camera-enabled devices (e.g., GoPro, dash cameras). We used default codec settings which should be most applicable in common media carving applications. Our dataset included mainly MPEG4, H.264, H.265, VP8, VP9 and OGV codecs with different distribution across containers. We leave a more detailed study of this phenomenon for future work.

V Conclusions

Our study addressed the problem of large-scale file fragment type identification. We explored neural network models, which dispense with explicit feature engineering - a bottleneck from both development and computation perspectives. Our architecture is based on 1-D convolutions. We accept raw blocks of bytes as input, and embed them in a trainable real-valued latent space. By changing a few hyper-parameters, our architecture can be easily adapted to various applications. We used the Tree-structured Parzen Estimator [17] to analyze the impact of hyper-parameters and find their optimal values. The resulting models are included in FiFTy, an open source tool for file-type identification.

Our evaluation is based on a novel dataset with 75 popular file-types (available publicly at [21]). We focused on multimedia and photographic types which commonly occur in data carving applications. Based on the obtained results, we can conclude that neural networks are an extremely effective and flexible tool for file-type identification. The trained classifiers consistently outperformed all baseline methods. Moreover, thanks to modern software frameworks and hardware accelerators, classification can run an order of magnitude faster.