PRNU-Based Source Device Attribution for YouTube Videos

TL;DR

This paper introduces a new PRNU-based method for source device attribution of YouTube videos that accounts for video compression effects, significantly improving accuracy over existing frame-based approaches.

Contribution

It proposes a novel PRNU fingerprint estimation technique that considers compression effects, enhancing source attribution accuracy for compressed videos.

Findings

Method outperforms existing frame-based techniques on YouTube videos.

Effective on both I frames and all frames, especially in compressed videos.

Demonstrates robustness against video re-compression and low bit-rate scenarios.

Abstract

Photo Response Non-Uniformity (PRNU) is a camera imaging sensor imperfection which has earned a great interest for source device attribution of digital videos. A majority of recent researches about PRNU-based source device attribution for digital videos do not take into consideration the effects of video compression on the PRNU noise in video frames, but rather consider video frames as isolated images of equal importance. As a result, these methods perform poorly on re-compressed or low bit-rate videos. This paper proposes a novel method for PRNU fingerprint estimation from video frames taking into account the effects of video compression on the PRNU noise in these frames. With this method, we aim to determine whether two videos from unknown sources originate from the same device or not. Experimental results on a large set of videos show that the method we propose is more effective than existing frame-based methods that use either only I frames or all (I-B-P) frames, especially on YouTube videos.

Figures

Tables

Table 1. Table 1: Different cases of fingerprint and noise estimation in the frame-based approach

Case	Fingerprint estimated from	Video noise estimated from
${\text{C}}_1$	I frames	I frames
${\text{C}}_2$	I + P + B frames	I + P + B frames

Table 2. Table 2: The set of native non-stabilized videos used in the experiments (by natural scenes we mean outdoor or indoor scenes).

ID	Resolution	Brand	Container	H.264 Profile	#Flat	#Natural	Total
D01	720p	S. Galaxy S3 Mini	MP4	Baseline	10	12	22
D03	1080p	Huawei P9	MP4	Constrained B.	7	12	19
D07	720p	Lenovo P70A	3GP	Baseline	7	13	20
D08	720p	S. Galaxy Tab 3	MP4	Constrained B.	13	24	37
D09	720p	A. iPhone 4	MOV	Baseline	7	12	19
D11	1080p	S. Galaxy S3	MP4	Baseline	7	12	19
D13	720p	A. iPad 2	MOV	Baseline	4	12	16
D16	1080p	Huawei P9 Lite	MP4	Constrained B.	7	12	19
D17	1080p	M. Lumia 640 LTE	MP4	Main	4	6	10
D21	1080p	Wiko Ridge 4G	MP4	Baseline	4	7	11
D22	720p	S. Galaxy Trend Plus	MP4	Baseline	4	12	16
D24	1080p	X. Redmi Note 3	MP4	Baseline	7	12	19
D26	720p	S. Galaxy S3 Mini	MP4	Baseline	4	12	16
D27	1080p	S. Galaxy S5	MP4	High	7	12	19
D28	1080p	Huawei P8	MP4	Constrained B.	7	12	19
D30	1080p	Huawei Honor 5c	MP4	Constrained B.	7	12	19
D31	1080p	S. Galaxy S4 Mini	MP4	High	7	12	19
D33	720p	Huawei Ascend	MP4	Constrained B.	7	12	19
D35	720p	S. Galaxy Tab A	MP4	Baseline	4	12	16
Total					124	230	354

Table 3. Table 3: Average video bit-rates and average number of frames (I,B,P) of native videos per device in the VISION dataset

ID	Resolution	Container	# Frames	# I frames	# P frames	# B frames	Bit-rate (Kbps)
D01	720p	MP4	2112	71	2041	0	3341
D03	1080p	MP4	2163	70	2093	0	16567
D07	720p	3GP	2125	71	2054	0	8212
D08	720p	MP4	1912	64	1847	0	11459
D09	720p	MOV	2027	46	1567	414	6877
D11	1080p	MP4	2187	73	2114	0	27466
D13	720p	MOV	2133	62	1872	198	9407
D16	1080p	MP4	2155	70	2085	0	16110
D17	1080p	MP4	2138	47	879	1211	11056
D21	1080p	MP4	2032	68	1963	0	20004
D22	720p	MP4	2174	73	2101	0	12025
D24	1080p	MP4	2031	68	1963	0	19994
D26	720p	MP4	2161	64	1931	167	10901
D27	1080p	MP4	2088	70	2017	0	17010
D28	1080p	MP4	2152	70	2082	0	15942
D30	1080p	MP4	2162	70	2092	0	17096
D31	1080p	MP4	2198	74	2092	0	17005
D33	720p	MP4	2022	68	1954	0	8032
D35	720p	MP4	2150	66	1939	145	10903

Table 4. Table 4: Average video bit-rates and average number of frames (I,B,P) of YouTube videos per device in the VISION dataset

ID	Resolution	H.264 Profile	# Frames	# I frames	# P frames	# B frames	Bitrate (Kbps)
D01	720p	Main	2113	22	1366	726	1741
D03	1080p	High	2161	52	1520	589	3341
D07	720p	Main	2153	42	1307	803	1401
D08	720p	Main	1920	31	1216	674	1923
D09	720p	Main	2115	50	1675	390	1582
D11	1080p	High	2187	22	1177	988	3332
D13	720p	Main	2128	35	1416	677	1906
D16	1080p	High	2141	49	1508	584	3176
D17	1080p	High	2125	53	1411	661	3654
D21	1080p	High	2174	97	1511	565	3050
D22	720p	Main	2172	27	1415	730	1997
D24	1080p	High	2029	39	1228	763	3228
D26	720p	Main	2156	24	1421	712	1973
D27	1080p	High	2088	46	1429	613	3210
D28	1080p	High	2152	21	1288	843	3835
D30	1080p	High	2151	34	1339	779	3608
D31	1080p	High	2198	31	1296	871	3668
D33	720p	Main	2080	20	1568	492	1988
D35	720p	Main	2155	29	1359	767	1876

Table 5. Table 5: Test scenarios used to evaluate the efficiency of the frame-based and the block-based methods

Scenario	Reference video	Query video
Scenario-1	Native-flat	Native-natural
Scenario-2	Native-natural	Native-natural
Scenario-3	Native-flat	YouTube-natural
Scenario-4	Native-natural	YouTube-natural
Scenario-5	YouTube-flat	YouTube-natural
Scenario-6	YouTube-natural	YouTube-natural

Table 6. Table 6: AUC values for source device attribution of native videos

	Flat vs. Natural		Natural vs. Natural
Methods	720p	1080p	720p	1080p
I-frames	1.00	0.99	0.99	1.00
All-frames	1.00	1.00	0.99	0.99
Proposed method	1.00	1.00	1.00	1.00

Table 7. Table 7: AUC values for source attribution of YouTube videos with native reference videos

	Native (flat) vs. YouTube (natural)		Native (natural) vs. YouTube (natural)
Methods	720p	1080p	720p	1080p
I-frames	0.99	0.99	0.97	0.94
All-frames	0.99	0.99	0.98	0.98
Proposed method	1.00	1.00	0.99	1.00

Table 8. Table 8: AUC values for source attribution of YouTube videos with YouTube reference videos

	YouTube (flat) vs. YouTube (natural)		YouTube (natural) vs. YouTube (natural)
Methods	720p	1080p	720p	1080p
I-frames	0.90	0.86	0.78	0.69
All-frames	0.95	0.91	0.89	0.83
Proposed method	0.95	0.98	0.97	0.98

Table 9. Table 9: AUC values with regard to motion in YouTube-natural videos

	(still vs. still)		(move vs. move)
Methods	720p	1080p	720p	1080p
I-frames	0.83	0.63	0.69	0.63
All-frames	0.87	0.74	0.86	0.87
Proposed method	0.94	0.95	0.98	0.98

Equations

$$\mathbf{I}={\mathbf{I}}^{(0)}+{\mathbf{I}}^{(0)}\mathbf{K}+\mathbf{\Psi }$$

$$\mathbf{F}=\frac{\sum _{k=1}^d{\mathbf{W}}_k{\mathbf{I}}_k}{\sum _{k=1}^d({\mathbf{I}}_k{)}^2}$$

$$\begin{array}[]{lcl}H_{0}:\mathbf{F}\neq\mathbf{W}\\ H_{1}:\mathbf{F}=\mathbf{W}\end{array}$$

$$\rho (r,c)=\frac{\sum _{i=1}^m\sum _{j=1}^n(\mathbf{F}(i,j)-\overline{\mathbf{F}})(\mathbf{W}(i+r-1,j+c-1)-\overline{\mathbf{W}})}{||\mathbf{F}-\overline{\mathbf{F}}||||\mathbf{W}-\overline{\mathbf{W}}||}$$

$$\mathrm{PCE}(\rho )=\frac{{\rho }_{peak}^2}{\frac{1}{mn-|S|}\sum _{r,c\notin S}\rho (r,c{)}^2}$$

$${\mathbf{B}}_{cur\delta }={\mathbf{B}}_{cur}-{\tilde{\mathbf{B}}}_{ref}$$

$${\hat{\mathbf{B}}}_{cur\delta }=\mathrm{Quant}[\mathrm{Scale}[\mathrm{DCT}({\mathbf{B}}_{cur\delta })]]$$

$${\tilde{\mathbf{B}}}_{cur}={\tilde{\mathbf{B}}}_{ref}+{\tilde{\mathbf{B}}}_{cur\delta }={\tilde{\mathbf{B}}}_{ref}+{\mathrm{DCT}}^{-1}[\mathrm{Scale}({\hat{\mathbf{B}}}_{cur\delta })]$$

$$\displaystyle M_{k}(r,c)=\begin{cases}0{\textrm{ , if DCT-AC coefficients at $(r,c)$ are all {zero}}}\\ 1{\textrm{ , {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}otherwise}}}\end{cases}$$

$${\mathbf{F}}_v=\frac{\sum _{k=1}^l{\mathbf{W}}_k{\mathbf{I}}_k{\mathbf{M}}_k}{\sum _{k=1}^l({\mathbf{I}}_k{\mathbf{M}}_k{)}^2+\text{J}}$$

Full text

††journal: Digital Investigationmytitlenotemytitlenotefootnotetext: © 2019. This manuscript version is made available under the CC-BY-NC-ND 4.0 license  http://creativecommons.org/licenses/by-nc-nd/4.0/

1 Introduction

Digital media (images and videos) are increasingly becoming a popular means for sharing information due to the explosion of smart-phone and tablet sales. Nowadays, people are increasingly using their smart-phones (which they mostly always have at their fingertips) to capture daily life scenes and share them through social media (like Facebook, YouTube, etc.). Apart from being a formidable means to communicate or share emotions, digital media can also be used to perpetrate crimes such as movie piracy, terrorist propaganda or child pornography. Furthermore, digital images or videos can be used as legal evidence during a trial in a court of justice. For these reasons, multimedia forensics is increasingly attracting the attention of forensic scientists and government agencies.

Identifying the device from which a digital media originates can sometimes be very crucial to an investigation. For instance, it can mean that the owner of the camera witnessed the scene that was captured and that he was at the place where the footage was taken. Determining the source device of an image or video during a trial in a court of justice or digital investigation can help to incriminate a suspect (for instance, a pedophile) possessing or sharing photos/videos acquired with the same camera device. Sometimes the camera device is not physically available. In this type of situation, query/input images or videos can be matched with another set of images or videos seized during the investigation to check whether their source cameras are the same or not. Such an analysis can be done through meta-data (EXIF) of the subjected images or videos. However, sometimes EXIF data are removed by third-party applications either intentionally or while sharing their contents on a social network. A different (yet more effective) source camera attribution method consists of examining specific noise patterns that are present in acquired images or videos due to sensor imperfections. The idea of using CCD (Charge-Coupled Device) sensors’ imperfections to perform video source identification first originated from K. Kurosawa late in 1999. In [1], he showed that dark currents in the CCD chips of camcorders form a fixed noise pattern which is added to recorded videotapes. This fixed noise pattern is used as a ”fingerprint” to identify the source device of a given recording.

Jan Lukas et al. in [2] showed that, as an intrinsic, natural, and unique camera fingerprint, the Photo-Response Non-Uniformity (PRNU) noise in digital images could effectively be used to perform digital image source attribution and forgery detection. This seminal work on PRNU-based image source attribution was followed by many others which established the PRNU as one of the most promising and powerful imaging sensor characteristics which can be exploited for image source attribution.

Chen et al. in [3] investigated the video source device attribution problem and showed that PRNU could effectively be used to identify the source camcorder of a subjected digital video (even for low-resolution cases) by estimating the PRNU fingerprint or sensor pattern noise (SPN) from individual video frames given that enough frames are available (a video clip of ten minutes was sufficient to identify the source device of low-resolution videos such as 264$\times$352 pixels). Dai-Kyung et al. in [4] improved the results in [3] by applying a MACE (Minimum Average Correlation Energy) filter to the reference PRNU fingerprint while testing its similarity with a query video’s sensor pattern noise (SPN). Through this, an improvement of up to 10% of the decision accuracy was achieved compared to Chen’s method for relatively small video resolutions such as 128$\times$128 pixels.

W. van Houten and Z. Geradts in [5] investigated the usage of PRNU for source attribution of YouTube videos. A set of webcams and codecs were used to record and encode videos. These videos were later uploaded to and downloaded from YouTube. SPN was then estimated from downloaded videos and used for source device attribution. Even though this work gave good results, its findings, which date to 2009, are out of date because the video cameras (notably handheld devices) used by YouTube users have considerably evolved since then.

Louis Javier et al. proposed in [6] a video source identification scheme based on the usage of PRNU and Support Vector Machines (SVM). A set of 5 smart-phones from 5 different brands were used to acquire videos used in training and testing steps. A total of 81 features, which are the SPN wavelet components, were used to feed the SVM classifier. Only native videos (videos taken from the acquisition devices without any post-processing) which have been cropped to various resolutions were used in the experiments. It was reported that the proposed classification scheme had an accuracy of about 87% to 90% depending on the video resolution. In general, features-based classification algorithms are not suitable for source attribution problems because there could be millions of devices that have the same brand and model. In such cases, a forensic examiner should model each of these devices as a separate class which is not practical in real life scenarios.

Massimo et al. in [7] proposed a “hybrid” approach to video source attribution. Retaking the idea of utilizing still images for camera fingerprint estimation that was previously introduced in [8], in [7], they established camera specific transfer functions between fingerprints estimated from images and video frames from the same camera for a broad set of smart-phone and tablet cameras (this is called image-to-video matching). These transfer functions consist of crop and scale parameters that best match these two fingerprints that have different resolutions and aspect ratios. The camera-specific transfer function is applied to the fingerprint estimated from still images before correlating it with the SPN estimated from the frames of a (non-stabilized) query video for source device attribution. This approach also solves the problem of estimating PRNU fingerprint of cameras featuring digital video stabilization (like iPhones and some Android smartphones) since this camera feature misaligns the PRNU noise from one frame to another as it has been stated in [8]. Massimo et al. also discussed how to link a Facebook account to a YouTube account by correlating two PRNU fingerprint estimates obtained from a query video downloaded from YouTube and images shared on a specific Facebook account, but the accuracy of by their method was very low.

The method in [7] has good identification results on native videos but source attribution accuracy for YouTube videos are not as high as for native camera outputs. Moreover, this method cannot be used to perform video-to-video device linking for cameras featuring video stabilization. Furthermore, in the case of Facebook-shared images, estimating a fingerprint using images from an unknown source is not realistic since it is assumed that they all come from the same device, which may not always be the case.

In video source device attribution, it is very crucial to estimate the PRNU fingerprint accurately. The original version (native camera output) of a query video is not always accessible during an investigation. In some cases, only a resized, re-compressed, or cropped version of the video is available for forensic examination. In most of the previous researches related to PRNU-based source device attribution for digital videos, the effects of video compression are not taken into account when estimating the PRNU fingerprint from video frames. Some authors like Samet et al. in [8] just assume that I frames are the best to be used, others like Dasara et al. in [9] give equal importance to I, P and B frames, and use all video frames for fingerprint estimation. Accordingly, they reported that a low accuracy in source attribution is obtained when performed on videos re-compressed by YouTube or Whatsapp (compared to their native version). Thus, it is obvious that video compression significantly affects or degrades the PRNU noise in video frames. This fact has to be taken into account when estimating the PRNU fingerprint from highly compressed videos.

In this study, we show the limits of the above-mentioned approaches (utilizing I or all frames) for fingerprint estimation by testing them under different scenarios (Table 1) for native and YouTube video cases. We will call these approaches “frame-based” in the rest of the paper.

In the paper, we will briefly describe the H.264/AVC video compression standard, then study the operations applied on an encoded frame block and investigate how PRNU noise is affected locally by these operations. Accordingly, we will propose a novel method for PRNU fingerprint estimation which takes into account the effects of video compression on PRNU noise in video frames. We call this method “block-based” approach since it relies on block-wise in-frame noise analysis. The proposed and frame-based methods are tested with a wide range of videos available in the VISION database [9] acquired from various smart-phones and tablets (Table 2).

We will particularly test a scenario where two query videos (native or YouTube) are compared with each other based on their estimated SPN to determine whether they originate from the same source or not. It should be noted that neither EXIF nor any side information except the estimated SPN of videos is used in the analysis.

The rest of the paper is organized as follows: Section 2 introduces PRNU-based image source camera attribution for still images. Section 3 presents the principles of H.264/AVC video compression and its impact on the PRNU noise estimation from video frames. Section 4 introduces the frame-based and the block-based approaches for source video device attribution in detail. Section 5 provides the details of the experimental setup. The experimental results are presented in Section 6. Finally, Section 7 concludes the paper and discusses future works.

2 PRNU-based Source Camera Attribution

The camera sensor is at the heart of the image acquisition process. It is made of a large number of small photo-detectors called pixels. Pixels use the photoelectric effect to convert incident light (photons) to electrons. For a given intensity of light falling on a pixel, the amount of electrons generated depends on the pixel’s physical dimensions and silicon homogeneity. Because of the imperfections of the manufacturing process and the non-homogeneity naturally present in the silicon, all the pixels of a sensor will never have the same photo-response characteristics. This phenomenon is called Photo-Response Non-Uniformity, and it is inevitable for all type of camera sensors (both CCD and CMOS).

Let the PRNU of an imaging sensor be represented by a matrix K, having the same dimensions with the sensor. A simplified and linearized imaging sensor model [10] can be written as:

[TABLE]

where I represents the sensor output, $\mathbf{I}^{(0)}$ is the ideal sensor output in the absence of any noise, $\mathbf{I}^{(0)}\mathbf{K}$ is the sensor’s PRNU, and $\mathbf{\Psi}$ is the temporal random noise comprising of thermal noise, shot noise, and other noise components. The matrices in (1) have the same size. Throughout the paper, all the mentioned matrix operations are element-wise. The PRNU noise is non-temporal, random, and unique to each camera sensor. It is pretty robust to lossy JPEG compression [11] and global image scaling [12]. These properties make PRNU a reliable quantity (intrinsic camera fingerprint) which can be used to perform many digital image forensic tasks such as source device identification, device linking, and forgery detection [10].

2.1 PRNU fingerprint estimation

The PRNU noise pattern of an imaging sensor can be estimated through a set of images of the same camera device. Having a number $d$ of images of the same camera, the camera PRNU fingerprint is estimated with a maximum likelihood estimator [10] as follows:

[TABLE]

where $\mathbf{I}_{k}$ is the $k$ th image acquired from the same camera device. $\mathbf{W}_{k}=\mathbf{I}_{k}-\textrm{Denoise}(\mathbf{I}_{k})$ is the difference between the original image $\mathbf{I}_{k}$ and its denoised version. The denoised version of the image $\mathbf{I}_{k}$ is obtained using a wavelet-based denoising filter as described in [13]. For color images, three fingerprints corresponding to the three color channels (red, green, and blue) are estimated separately then combined like in a generic RGB to gray conversion [10].

The estimated fingerprint $\mathbf{{F}}$ is made of two components: the reference pattern (RP) and the linear pattern (LP). The linear pattern contains all the noise components that are systematically present in an image due to artifacts introduced by Color Filter Array (CFA) interpolation, JPEG compression, and post-processing operations performed in the image acquisition pipeline. Contrary to the reference pattern, the linear pattern is common to the cameras of the same model; thus, it has to be removed from the fingerprint to achieve an accurate source attribution even with cameras of the same model. The linear pattern can be used to identify a camera model as it was done in [14]. Removing the linear pattern from the fingerprint is a straightforward task since it appears periodically in $\mathbf{{F}}$. In [10], the linear pattern is removed from the fingerprint by subtracting the averages of each row and column from the corresponding element in $\mathbf{{F}}$. The estimated fingerprint is then filtered with a Wiener filter in the DFT domain to meet the zero mean Gaussian white noise model hypothesis.

It has been stated in [10] that 20 to 50 natural (any content) images are enough to obtain a good estimate of a camera’s fingerprint. However, a fingerprint with similar accuracy can also be obtained using 10 to 25 images with flat content (blue skies or flat walls).

2.2 Source camera attribution

Source camera attribution of a query image having an estimated PRNU noise $\mathbf{W}$ and a camera having a PRNU fingerprint F is formulated as a two-channel hypothesis testing problem as follows [12]:

[TABLE]

This hypothesis can be tested by taking normalized cross-correlation of the noise and the fingerprint estimates as:

[TABLE]

where $\mathbf{\overline{F}}$ and $\mathbf{\overline{W}}$ represent the averages of $\mathbf{F}$ and $\mathbf{W}$, respectively. The operator $||\text{ }||$ is the Euclidean norm, r and c are circular shift parameters ranging from 1 to m and 1 to n, respectively. We assume that $\mathbf{F}$ and $\mathbf{W}$ have the same size $m\times n$. The Peak to Correlation Energy (PCE), a resolution independent similarity metric, is computed from normalized cross correlation (NCC) as follows:

[TABLE]

where $\rho_{peak}$ is the maximum value of NCC matrix, $S$ is a small region surrounding $\rho_{peak}$ and $|S|$ is the cardinality of $S$. When matrix resolutions of the fingerprint and the noise estimate are the same, $\rho_{peak}$ can be replaced directly with $\rho(1,1)$. If $\mathrm{PCE}(\mathbf{\rho})$ is above a decision threshold $\mathbf{\tau}$, the null hypothesis ($H_{0}$) is rejected and the query image with the noise estimate $\mathbf{W}$ is assumed to be acquired with the same camera of the fingerprint $\mathbf{F}$.

3 The H.264/AVC Video Compression

This section presents key aspects of the H.264/AVC (Advanced Video Compression) video compression standard and shows how operations involved in video compression affect the PRNU noise in frame blocks. The H.264/AVC video compression standard is the world’s leading standard for video compression. Nowadays, it is used by almost all smart-phones and video-sharing platforms (or social media) like YouTube and Facebook. The H.264/AVC standard is managed by the JVT (Joint Video Team). Its first version was released in 2003 and is destined to be replaced by the H.265/HEVC (High-Efficiency Video Coding) standard in the next decade. An exhaustive description of video coding techniques is out of the scope of this paper, and the reader can refer to [15] for a comprehensive description of the H.264/AVC video coding standard and [16] for technical details.

A simplified diagram of an H.264/AVC encoder is given in Fig. 1 [15]. A video encoder also embeds a decoder. Modern video compression standards share several key operations such as block processing, prediction, transform, quantization, entropy coding.

Block processing: The input frame is divided into one or more slices containing Macro-blocks of size 16$\times$16. These Macro-blocks are divided into blocks of different sizes (16$\times$16, 8$\times$16, 8$\times$8, 4$\times$8, 4$\times$4 …) according to the type of prediction used to encode them. Subsequent operations such as prediction, transform, and quantization are performed on these sub-blocks.

• 2.

Prediction: Prediction is a process in which a current block’s pixels are predicted from pixels of a previously encoded block(s) within the current frame (intra-frame coding) and/or previous or future encoded frames (inter-frame coding). After prediction, the prediction residue (the difference between the current block and the predicted block) is computed.

• 3.

Transform: A transform operation is applied on block prediction residue and aims at reducing the statistical correlation between its samples such that most of the information it contains can be concentrated into a small number of encoded samples. The H.264/AVC standard uses (integer) Discrete Cosine Transform (DCT) with integer transform cores of size 4x4 or 8x8 (used exclusively in High profile encoders).

• 4.

Quantization: Quantization consists of reducing the precision used to represent sample values. It aims at reducing the number of bits necessary to represent a set of values. In H.264/AVC, each Macro-block has its quantization parameter which can be a scalar or a quantization matrix like in JPEG (used only in High profile encoder). It is important to note that among all the operations involved in video compression, quantization is the only operation which is non-reversible.

• 5.

Entropy coding: Entropy coding is a process through which discrete-valued symbols are represented in a manner that takes advantage of the relative probability of each source symbol. In H.264/AVC, VLC (Variable length coding) or arithmetic coding (CABAC) can be used for entropy coding.

Prediction has brought the greatest increase in coding efficiency to the H.264/AVC compression standard in comparison to the previous coding standards (like MJPEG). Prediction in video compression exploits the spatial and temporal redundancies highly present in video sequences. Spatial redundancy is exploited in intra-frame prediction. In intra-frame prediction, prediction (reference) blocks and blocks to be predicted are all located in the same frame (neighboring blocks). Intra-frame predicted Macro-blocks are called I Macro-blocks. Temporal redundancy is exploited in inter-frame prediction and is based on motion estimation and motion compensation. In inter-frame prediction, a Macro-block is predicted using Macro-blocks in past and future frames. There are mainly two types of inter-frame predicted Macro-blocks in H.264/AVC: P Macro-blocks (which are predicted using only blocks in past frames) and B Macro-blocks (which are predicted using blocks in past and the future frames). A de-blocking filter is applied to inter-frame predicted blocks to reduce block artifacts due to motion compensation. In an H.264/AVC encoded video, three types of frames can be found: I, P, and B frames. An I frame is made only of I Macro-blocks, a P frame of I and P Macro-blocks and, a B frame of I, P, and B Macro-blocks.

3.1 The impact of video compression on PRNU noise

As we see notice, video compression is by far more complex than still image compression. Thus, the statement made in [11] according to which the PRNU noise survives lossy JPEG compression might not hold for H.264/AVC compression given that operations applied to frame blocks during compression also affect the PRNU noise they contain. Here, we determine the condition necessary for the PRNU noise in an encoded block to survive video compression.

To investigate the effects of video compression on the PRNU noise in encoded frame blocks, let us consider a block which is to be encoded. We note: $\mathbf{B}_{cur}$ as the block which is to be encoded, $\mathbf{\widetilde{B}}_{cur}$ as the decoded current block, $\mathbf{\widetilde{B}}_{ref}$ as the reference (or prediction) block (a previously encoded and decoded block), $\mathbf{B}_{cur\delta}$ as the current block’s prediction residue (see Fig. 2). The operations applied to $\mathbf{B}_{cur}$ during its encoding and decoding processes are given by equations (6) to (8).

The prediction residue is computed as:

[TABLE]

The output of the transform (DCT), scaling, and quantization operations for the residue input can be written as:

[TABLE]

Finally, the block’s decoding equation can be written as:

[TABLE]

Equation 8 shows that the content of a decoded block $\mathbf{\widetilde{B}}_{cur}$ is highly dependent on the inverse DCT transform of the block’s prediction residue. A 2D-DCT transform matrix is composed of two classes of coefficients: a DC coefficient which is the one at the location (0,0) in the matrix, and, AC coefficients which are the remaining. That being said, it is easily noticed that the high-frequency (AC) content of an encoded block is lost if its prediction residue’s DCT-AC coefficients are all zero. In such a case, the high-frequency content of the decoded block is (approximately) equal to the one in its reference block(s). Based on this, we can conclude that the PRNU noise in an encoded block is not (completely) destroyed by video compression if the DCT-AC coefficients of its prediction residue are not all null (because the PRNU noise is by essence a high frequency signal). The strength of the PRNU noise remaining in a decoded block depends on the number of its non-null DCT-AC coefficients and on the scene content. It is shown in [17] that high-frequency content scenes (notably edges) interfere with the PRNU noise estimation. If the DCT-AC coefficients of a given block’s prediction residue are all null, then the PRNU noise it contains is irreversibly lost and replaced by the one in its prediction block(s).

4 Proposed Method: Compression-Aware PRNU Estimation

In this section, we first present the strategies which are currently used for camera fingerprint and video noise estimation from video frames. Secondly, we introduce a new approach called the block-based approach for highly compressed videos taking into account the effects of video compression that have been discussed in the previous section.

4.1 Frame-based approach for video source attribution

As it is mentioned in Section 3, an H.264/AVC compressed video is made of three types of frames: I, P, and B frames. For video source attribution, it is suggested in the literature to use only I frames assuming that they are significantly less compressed than B and P frames because intra-frame coding is solely used in I frames. Thus, intuitively, I frames seem to be the best set of frames to use for PRNU fingerprint estimation. But, is this practice better than using all video frames (I, B, and P frames) for PRNU fingerprint estimation with regard to source attribution accuracy? To answer this question, we test the two cases presented in Table 1 where only I frames are used (case-1) and where all the frames in videos are used (case-2). In these two cases, a given camera’s PRNU fingerprint $\mathbf{F}_{v}$ (estimated from a flat-content video) and a given natural-content video’s SPN $\mathbf{W}_{v}$ are all computed using the Equation (2). We also consider the case where we do not have the suspect camera device, but we want to figure out whether two query videos have the same source or not (device linking). In this case, we cannot have a reference fingerprint and thus we simply match the SPN matrices estimated from each query video using (2).

4.2 Block-based approach for video source attribution

The block-based approach goes further than just selecting the frames to be used for fingerprint or noise estimation. In each frame, we seek particular blocks in which the PRNU noise has not been completely degraded by video compression. As we have discussed earlier, the PRNU noise in a block survives compression if the block’s prediction residue DCT-AC coefficients are not all zero. Thus, to estimate the video PRNU noise, we will only use blocks which have at least one non-null DCT-AC coefficient in I, P, and B frames since they still have (at least partially) some amount of PRNU noise at its correct location/block.

The block-based approach requires to analyze all the DCT-AC coefficients of all the frame-blocks of an input video to determine the appropriate blocks for PRNU noise estimation. For each frame of an input video, the DCT-AC coefficients of blocks’ prediction residue are read and checked whether they are all zero or not. This task is carried out by a modified version of the H.264/AVC reference decoder jm16.1 [18]. We assign labels (1 or 0) to all frame-pixels to indicate that the frame-block they are positioned in has some PRNU noise at its correct location (label:1) or not (label:0). As a result, we obtain a binary matrix (frame mask $\mathbf{M}$) for each frame which indicates the appropriate pixels/blocks to be used in the frame-wise PRNU noise estimation. The zeros in the frame mask indicate the location of pixels/blocks where PRNU noise estimation is not feasible. An element of the frame mask $\mathbf{M}$ of any $k$ th frame at pixel location $(r,c)$ is computed according to (9). The frame masks $\mathbf{M}_{k}$ are of the same dimensions with the investigated video’s resolution. As we have mentioned earlier, the H.264/AVC standard uses 4$\times$4 or 8$\times$8 (in High profile encoder) sized integer-DCT core transforms. A flag in Macro-blocks’ header indicates the size of the core used.

[TABLE]

If there is at least one non-zero DCT-AC coefficient in a particular frame-block, we set $M_{k}(r,c)=1$ and use the PRNU noise of that block in video noise estimation. Otherwise, we set $M_{k}(r,c)=0$ which discards the block during video noise estimation. If we set the number of non-zero DCT-AC coefficients threshold to values higher than one (2,3,…), we would discard some frame-blocks that have some amount of proper PRNU noise during video noise estimation. As a result, the accuracy of the source device attribution could decrease.

Fig. 3 shows some examples of I frames from highly compressed (YouTube) videos and their associated frame masks $\mathbf{M}_{k}$. Only white regions (having a mask value of 1) in the frame mask will be used for fingerprint/video noise estimation. The figure shows that, due to intra-frame coding, most of the PRNU noise in uniform regions is replaced by the one in their prediction pixels.

In the block-based approach, the camera fingerprint $\mathbf{F}_{v}$ and the video noise $\mathbf{W}_{v}$ are computed with a modified version of the maximum likelihood PRNU fingerprint estimator (2) as provided in (10).

[TABLE]

In (10), $l$ is the number of frames of the input video, $\mathbf{I}_{k}$ is the decoded image of the $k$ th video frame, $\mathbf{W}_{k}$ is the PRNU noise estimated from $\mathbf{I}_{k}$, and $\mathbf{M}_{k}$ is the frame mask. It should be noted that all the operations in (10) are element-wise. Different from (2), we add a J matrix (an all-one-matrix of the same size with the video resolution) to the equation’s denominator to prevent a division by zero for the case where $M_{k}(r,c)=0\text{ , }\forall k$.

5 Experimental Setup

To evaluate the efficiency of the proposed methods, we used a public dataset (VISION) [9] which comprises native and social media videos. In the dataset, there are 34427 images and 1914 videos acquired with 35 smart-phones from 11 major brands. In this paper, we only used videos acquired with devices which do not feature in-camera digital video stabilization (a subset of 19 cameras). For each device, there are three types of scenes: flat scenes (skies or flat walls), indoor scenes (classrooms, offices, halls, stores, etc.), and outdoor scenes (nature, garden, city, etc.). For each type of scene, three acquisition modes have been used to record videos: still mode, where the user stands still while capturing the video; move mode, where the user walks while capturing the scene; and pan mode, where the user performs a pan and rotation while recording. All videos in the dataset have approximately the same duration of about 1 minute and 15 seconds, which corresponds approximately to 2000 frames per video. Table 2 gives a list of devices used in the experiment (we keep the same IDs used in the dataset) and the number of native videos (with flat and natural content) for each device. By natural content, we mean outdoor or indoor scenes. Each native video has its YouTube version in the dataset. Thus, there are 354 native and 354 YouTube videos of the same scenes (see Table 2).

Tables 3 and 4 present the average video bit-rates and average number of frames (I,B,P) of native and YouTube videos for each device in the VISION dataset, respectively. Videos uploaded to YouTube are re-encoded (re-compressed) to reduce their size. This can be noticed from the video bit-rates of native and YouTube videos. When native videos are uploaded to YouTube, no re-scaling (down-sizing) is performed since YouTube supports 4K resolution, which is high enough for the videos in the VISION dataset. Videos that have $720p$ resolution are re-encoded by YouTube using the H.264 Main profile, while, $1080p$ videos are re-encoded using the High profile which has a bigger coding efficiency than the Main Profile. To achieve a high coding efficiency, the H.264 High profile coarsely quantizes the high-frequency DCT coefficients, whereas the Main profile performs a simple scalar quantization for all frequencies.

All computations in our experiments was performed on a Dell Precision T3610 PC equipped with a 12 cored Xeon processor, 16 GB of RAM, running Ubuntu 16.04 LTS. Video frames were extracted and stored in an uncompressed format using ffmpeg [19]. Video properties (type of frame, bit rate, resolution…) were extracted from videos using ffprobe, included in ffmpeg software.

6 Experimental Results

In this section, we evaluate the efficiency of the frame-based video source attribution (which is the one commonly used in the literature) and the block-based approach (the one we propose) under six different scenarios with an increasing level of difficulty as given in Table 5. Scenario 1 and Scenario 3 correspond to cases where the suspect cameras are available at hand. Thus, native flat content videos can be recorded for camera fingerprint estimation. In the remaining scenarios, neither any suspect camera nor native-flat videos are available. In all the scenarios, we want to figure out whether two input videos (query and reference) originate from the same device or not.

For each scenario, all reference videos are matched against all query videos of the same device and other devices having the same resolution with video PRNU fingerprints. Thus, for each device, there are much more non-matching videos than matching videos. It should be noted that the video PRNU fingerprint estimation is performed in the same way both for the reference and the query videos.

6.1 Source device attribution of native videos

Here, we consider the case where we perform source device attribution exclusively on native videos. This corresponds to Scenario 1 and Scenario 2. Table 6 gives the overall source attribution accuracy (considering all the devices) of each approach on 720p and 1080p videos. The accuracy is measured using Area Under the Curve (AUC) of Receiver Operating Characteristic (ROC) computed from PCE distributions of matching and non-matching cases for all cameras listed in Table 2. Table 6 shows that, when performing source device attribution on native videos, an accuracy of almost 100% can be achieved by using only I frames. This conforms with the results obtained in [8]. The same results are obtained when using all video frames or the block-based method. Nevertheless, using only I frames (C1) is the best option if we are concerned with the computation time.

6.2 Source device attribution of YouTube videos with native reference videos

Here, we consider Scenarios 3 and 4, where reference videos are native videos (flat or natural content) meanwhile query videos are YouTube natural-content videos. Table 7 gives the AUC values per device for the 720p and 1080p video sets, Fig. 4 and Fig. 5 give the associated ROC curves. Globally, a high accuracy is achieved by both methods, but the block-based approach slightly surpasses the frame-based approach.

6.3 Source device attribution of YouTube videos with YouTube reference videos

We now consider the case where the reference and the query videos are all YouTube videos (Scenarios 5 and 6). This is the most challenging case since both reference and query videos are heavily compressed. Table 8 gives the global AUC values of these two Scenarios for all the devices in the set. Corresponding ROC curves are depicted in Fig. 6 and Fig. 7. These results show the effectiveness of the block-based approach that is capable to link YouTube videos with a higher accuracy than the frame-based methods.

6.4 The impact of video motion on source device attribution

To evaluate the impact of video motion on the accuracy of source device attribution, we consider the Scenario-6 (YouTube-natural vs. YouTube natural videos matching) with two video classes: still videos, and move videos. The still-class refers to videos recorded without significant camera movement and content change. The move-class refers to videos recorded with camera motion. This experiment was done using all the cameras in Table 2 which corresponds to 33 still 720p videos, 27 still 1080p videos, 33 move 720p videos, and 27 move 1080p videos. Table 9 gives AUC values of the frame-based and the block-based methods on these two video classes. The table shows that, when the block-based method is used, the highest source attribution accuracy is always obtained when the query videos have significant amount of motion compared to the still case. The same remark holds for the videos encoded with the H.264 High profile (1080p YouTube videos) for both the frame-based and the block-based methods. This is due to the fact that, when a video has lot of motion (scene change), the encoder has to keep more non-zero DCT-AC coefficients to cope with the changes in the scene. We also notice from the table that I frames of 720p videos acquired with the still mode have more PRNU noise than their move counterpart.

7 Discussion and Conclusion

In this study, we investigate the problem of source verification of any two videos (query and reference) to determine whether they originate from the same camera device or not. The proposed scheme takes into account the effect of H.264/AVC video encoding on the PRNU noise in video frames. We first determine a necessary condition for the PRNU noise to survive H.264/AVC compression; then, propose a modified maximum likelihood estimator of video PRNU noise/fingerprint.

The efficacy of the proposed method (called block-based method) is evaluated on non-stabilized videos in the VISION database with Receiver Operating Characteristic (ROC) plots and Area-Under-the-Curve (AUC) measurements. The proposed method is compared with existing video PRNU fingerprint estimation methods using all frames (I, B, P) and solely I-frames under different scenarios based on video content (flat/natural) and video encoding (native/YouTube).

From experimental results, we can infer that an accurate source identification of YouTube videos can be achieved using frame-based methods if we have a fingerprint estimated from I frames of a flat content native video (Table 7). Oppositely to what has been stated in previous studies in the literature, using all frames (I, B, and P frames) yields better source attribution accuracy than using only I frames even when the investigated videos have been re-compressed by YouTube. This means that, despite of compression, there is still enough valid PRNU noise in P and B frames to improve the I-frames estimated PRNU fingerprint. Figures 4 to 7 show that, when the frame-based methods are used, the best attribution accuracy is obtained when reference PRNU fingerprints are estimated from flat-content videos. Also, we notice from these figures that, when the frame-based approach is used, the attribution accuracy of 720p videos is always higher than the one of 1080p videos (which is contradictory to what one could have expected since higher resolution implies more PRNU information). This is because YouTube re-compresses 1080p videos using the H.264 High profile (whereas it uses the Main profile for 720p videos) yielding a higher degradation of the PRNU noise in the encoded blocks. On the other hand, no significant accuracy changes are observed for the proposed block-based method between 720p and 1080p videos.

The main advantage of the proposed method is that it uses only frame blocks that have correct PRNU components during PRNU fingerprint estimation. As a result, we obtain a better PRNU fingerprint than the one estimated using the frame-based methods especially when videos are highly compressed (Table 8 and Figures 6,7). The results presented in this paper represent the performance lower bounds of the frame-based and block-based methods for VISION database; better source device attribution could be obtained with videos that have longer duration, specially with the block-based approach. Our future studies will focus on the evaluation of the proposed method with various video resolutions, codecs, and social media applications.