Efficient Privacy Preserving Viola-Jones Type Object Detection via Random Base Image Representation

TL;DR

This paper introduces a novel Random Base Image (RBI) method for privacy-preserving object detection that significantly accelerates Viola-Jones detection in cloud environments while maintaining accuracy and ensuring low information leakage.

Contribution

The work proposes the RBI representation combined with secure protocols to enhance the efficiency of privacy-preserving Viola-Jones detection compared to previous cryptographic approaches.

Findings

Retains detection accuracy comparable to plain vision algorithms.

Achieves significant speedup over traditional blind vision methods.

Provides theoretical low probability of information leakage.

Abstract

A cloud server spent a lot of time, energy and money to train a Viola-Jones type object detector with high accuracy. Clients can upload their photos to the cloud server to find objects. However, the client does not want the leakage of the content of his/her photos. In the meanwhile, the cloud server is also reluctant to leak any parameters of the trained object detectors. 10 years ago, Avidan & Butman introduced Blind Vision, which is a method for securely evaluating a Viola-Jones type object detector. Blind Vision uses standard cryptographic tools and is painfully slow to compute, taking a couple of hours to scan a single image. The purpose of this work is to explore an efficient method that can speed up the process. We propose the Random Base Image (RBI) Representation. The original image is divided into random base images. Only the base images are submitted randomly to the cloud server. Thus, the content of the image can not be leaked. In the meanwhile, a random vector and the secure Millionaire protocol are leveraged to protect the parameters of the trained object detector. The RBI makes the integral-image enable again for the great acceleration. The experimental results reveal that our method can retain the detection accuracy of that of the plain vision algorithm and is significantly faster than the traditional blind vision, with only a very low probability of the information leakage theoretically.

Figures

Tables

Table 1. Table 1 : Average running time comparison with the Viola-Jones method [ 11 ] and the Blind Vision method [ 2 ] on the FDDB [ 12 ] , the Face96 [ 13 ] and the FEI [ 14 ] datasets. In the second and fourth columns, we simulate Alice and Bob on one PC without communication delays. The third and fifth columns report the time costs in a private cloud environment.

Dataset	Our	Our + Comm. Delays	VJ [11]	VJ [11] + Comm. Delays	Blind Vision [2]
FDDB-face	143.852s	380.992s	0.380s	0.843s	A couple of hours [8]
Face96	173.471s	477.635s	0.358s	0.809s
FEI	152.701s	414.522s	0.363s	0.827s
FDDB-nose	113.398s	294.845s	0.372s	0.836s	A couple of hours [8]
FDDB-eye	85.754s	240.111s	0.496s	0.912s
INRIA Person	80.204s	224.571s	0.333s	0.771s

Equations

$$H(\mathbf{x})=\mathrm{sign}(\sum _{n=1}^N{h}_n(\mathbf{x})),$$

$$h_n(\mathbf{x})=\{\begin{array}{ll}& {\alpha }_n{\mathbf{x}}^T{\mathbf{y}}_{\mathbf{n}}>{\theta }_n\\ & {\beta }_n\text{otherwise,}\end{array}$$

$$\mathbf{x}=\sum _{i=0}^{M-1}{w}_i{\mathbf{B}}_i,$$

Full text

**Index Terms— ** Blind Vision, Random Base Image, Privacy Preserving, Object Detection

1 Introduction

Recently, widespread smart phones with cameras enable people to shot images and videos nearly anytime and anywhere. Millions of surveillance cameras including the driving recorders captures images and videos every second. All these conveniences devices are producing the large-scale visual media data, which is considered as the biggest big data.

Due to the limited storage space of these terminal devices, large-scale visual media data is being uploaded and stored in the cloud servers. Not only the storage, but also the processing of large-scale visual media data are being outsourced to the cloud servers.

The cloud servers have some strong algorithms such as face/object detection, face/object recognition, intelligent video surveillance. Nowadays, people can easily find all the faces in their photos stored in the cloud servers using the powerful face detection algorithms maintained by the cloud servers. However, the cloud servers are always third party entities. Thus the privacy of the users’ visual media data may be leaked to the public or unauthorized parties.

In the meanwhile, the powerful cloud services for visual media analysis and processing need a lot of money, data and time from the cloud server producers. The cloud servers are also reluctant to leak any parameters of the trained models or some protected details of their algorithms with copyrights.

Thus, the privacy of both the content of the visual media from the clients and the parameters of the vision algorithms from the cloud servers should be protected. 10 years ago, Avidan & Butman introduced Blind Vision [2], which is a method for securely evaluating a Viola-Jones type face detector. Blind Vision uses standard cryptographic tools and is painfully slow to compute, taking a couple of hours to scan a single image.

After that, rich literatures have been proposed in this field. The cryptographic tools such as secret sharing (SS) [3], security multi-party computation (SMC) [4], homomorphic encryption (HE) [5], garbed circuit (GC) [6], Chaotic System (CS) [7] are heavily used. Plenty of computer vision applications have been modified to the privacy preserving or secure versions such as private face detection [8], face recognition [4], content based image retrieval [9], visual media search on public datasets [10], intelligent video surveillance [3, 5, 6, 7].

However, most of these work rely heavily on cryptographic tools, which are painfully slow to compute or need bit by bit interaction between the clients and the cloud servers. In this paper, we revisit the Blind Vision [2] and attempt to make the blind vision towards cryptographic-free, without losing the security properties. We use randomness and only a little cryptographic operations to protect the visual media data of the clients and the parameters of the trained models in the cloud servers.

A novel image representation called Random Base Image(RBI) representation is proposed. In this work, we also investigate the object detection in the cloud. We apply our RBI to the famous Viola and Jones object detection method and propose a novel blind object detection method. We separate an image into random base images. The weight of each base image is only known by the client. The base images are sent randomly to the cloud server. The cloud server cannot recover anything from the random base images. A random vector and the secure Millionaire protocol [2] are leveraged to protect the parameters of the trained object detector. The RBI makes the integral-image enable again for the great acceleration. The experimental results reveal that our method is significantly faster than the traditional blind vision, with only a very low probability of the information leakage theoretically.

2 Secure Object Detection

In this section we develop a secure object detector with the random base image representation.

2.1 Notations

Our scenario and the notations are the same as that of traditional Blind Vision [2], as show in Figure 1. Denote some $L$ dimensions finite field $F$ that is large enough to represent all the intermediate results. Denote by $X$ the image that Alice owns. A particular detection window within the image $X$ will be denoted by $x\in F^{L}$ and $x$ will be treated in vector form. Bob owns a strong classifier of the form

[TABLE]

where $h_{n}(\mathbf{x})$ is a threshold function of the form

[TABLE]

and $y_{n}\in F^{L}$ is the hyperplane of the threshold function $h_{n}(\mathbf{x})$. The parameters $\alpha_{n}\in F,\beta_{n}\in F$ and $\theta_{n}\in F$ of $h_{n}(\mathbf{x})$ are determined during training; $N$ is the number of weak classifiers used.

2.2 The Random Base Image Representation

The core idea of our RBI is to separate the original image into some random base images with fixed weights. The original image can be recovered by all the base images. The sparse representation can be considered as the one has such ability. However, they need another image dataset for learning the base images. Further more, there could be reconstruction error. Thus, we fix the weights and randomize the base images themselves.

The detection window $\mathbf{x}$ can be represented as:

[TABLE]

where $\mathbf{B}_{i}$ is the base image with weight $w_{i}$. As is shown in Figure 2, each base image has a fixed weight. The base image itself is randomly determined. The number of the base image is set to $M=256$. Thus, each base image can be a binary image, which is easy for network transfer and fast to compute. In addition, there are $256!$ permutation of the base image which is not easy to guess. The process of the RBI generation is described in Algorithm 1.

2.3 Secure Object Detection with RBI

2.3.1 Secure Object Classifier Protocol

The core of our method is the secure object classifier protocol as is described in Algorithm 2 and Figure 3. For secure object detection, Alice first divides the test image $\mathbf{X}$ into $Q$ detection windows $\{\mathbf{x}_{1},\mathbf{x}_{2},...\mathbf{x}_{Q}\}$. Then the detection windows are randomly sent to Bob as the inputs of the secure face classifier protocol one by one. Using the Algorithm 2, Alice and Bob know which detection windows are the target objects. Because the detection windows are randomly sent to Bob, only Alice learns the location of all the detected faces in the original image. Bob does not learn the contents including where the faces are in the image of Alice. Alice learns nothing about the parameters of the face detector of Bob.

The body of Algorithm 2 is described as follows:

• •

(1): Alice factorizes the detection window $x$ into $M$ random base images $\mathbf{B}=\{\mathbf{B}_{0},\mathbf{B}_{1},...,\mathbf{B}_{M-1}\}$ with weight $w=\{w_{0},w_{1},...,w_{M-1}\}=\{0,1,...,M-1\}$ through Algorithm 1.

• •

(2): Alice randomly shuffles the weight $w$ to $w^{\prime}$. The random base images $\mathbf{B}$ are permuted with the same order of that of $w^{\prime}=\{w^{\prime}_{0},w^{\prime}_{1},...,w^{\prime}_{M-1}\}$ to $\mathbf{B}^{\prime}=\{\mathbf{B}^{\prime}_{0},\mathbf{B}^{\prime}_{1},...,\mathbf{B}^{\prime}_{M-1}\}$, which is sent to Bob.

• •

(3): In one cascade, Bob has $N$ weak classifiers with parameter vectors $\mathbf{y}=\{\mathbf{y}_{0},\mathbf{y}_{1},...\mathbf{y}_{N-1}\}$. Bob randomly add $K$ fake weak classifiers and set their parameters $\alpha$ and $\beta$ to zero. Bob randomly shuffles the $N+K$ true and fake weak classifiers to form $\mathbf{y}^{\prime}=\{\mathbf{y}^{\prime}_{0},\mathbf{y}^{\prime}_{1},...\mathbf{y}^{\prime}_{N+K-1}\}$. Then, Bob generates $N+K$ random positive numbers $s=\{s_{0},s_{2},...,s_{N+K-1}\}$. For each parameter vector $\mathbf{y}^{\prime}_{n}\in\mathbf{y}^{\prime}$. Bob and Alice repeat the following 3 steps.

• –

(3.1): Bob computes the feature responses for all the base image $\mathbf{B}^{\prime}_{m}$ in $\mathbf{B}^{\prime}$ by $F_{m}(n)={\mathbf{B}}_{m}^{{}^{\prime}T}\mathbf{y}^{\prime}_{n},m=0,1,...,M-1$. All the $M$ responses of base images $\mathbf{B}^{\prime}$ on each parameter vector $\mathbf{y}^{\prime}_{n}$ are sent back to Alice.

• –

(3.2): Alice computes the feature responses of the detection window $\mathbf{x}$ by $F(n)=\sum_{m=0}^{M-1}F_{m}(n)w^{\prime}_{m}$.

• –

(3.3): Alice and Bob use the secure Millionaire protocol [2] to determine which number is larger: $F(n)$ or $\theta_{n}$. Bob send $\alpha_{n}+s_{n}$ or $\beta_{n}+s_{n}$ to Alice. Alice store it as $c_{n}$.

• •

(4): Alice and Bob use the secure Millionaire protocol [2] to determine which number is larger: $\sum_{n=1}^{N+K}c_{n}$ or $\sum_{n=1}^{N+K}s_{n}$. If Alice has a larger number then x is positively classified, otherwise x is negatively classified.

2.3.2 Security

The protocol protects the security of both parties. The protocol protects the contents of the image from Alice and the parameters of the face detector from Bob. We analyse the security of Algorithm 2 in the following paragraph.

• •

From Alice to Bob

• –

In step 2, Alice send randomly shuffled base images to Bob. Bob only knows the randomly generated base images and do not know the weight of each base image. The probability of guessing out the right permutation is $1/M!$. Even Bob guesses out the right permutation, he does not know the weight of each base image. Thus, it is almost impossible for Bob to recover the detection window of Alice.

• –

In the 3th sub-step of step 3 and the step 4. Alice and Bob engage in secure Millionaire protocol [2]. so Bob can learn nothing about Alice’s data.

• •

From Bob to Alice

• –

In the 1st sub-step of step 3, Alice can not learn the number of the weak classifiers $N$ or the true filters from the received feature responses. The true filters are obfuscated by the fake filters.

• –

In the 3rd sub-step of step 3, Alice and Bob engage in a secure Millionaire protocol so Alice only learns if $F(n)>\theta_{n}$. She can not learn anything about the parameter $\theta_{n}$. Moreover, at the end of the Millionaire protocol Alice learns either $\alpha_{n}+s_{n}$ or $\beta_{n}+s_{n}$. In both cases, the real parameter ($\alpha_{n}$ or $\beta_{n}$) is obfuscated by the random number $s_{n}$.

• –

In step 4, Alice and Bob use the secure Millionaire protocol to determine which number is larger: $\sum_{n=1}^{N}c_{n}$ or $\sum_{n=1}^{N}s_{n}$. If Alice has a larger number then $x$ is positively classified, otherwise x is negatively classified.

• •

Multiple Cloud Servers

• –

The $M$ random base images can be also sent to multiple cloud server with the same object detector to increase security.

2.3.3 Complexity and Efficiency

The complexity of the protocol is $O(M(N+K)L)$, where $M$ is the number of the base images. $N$ and $K$ are the numbers of the true and fake weak classifiers, respectively. $L$ is the dimensionality of the detection window $x$.

Unlike the traditional Blind Vision [2], in which the OT operation is used extensively, the proposed method only use OT operation to compare 2 numbers. In the secure dot-product protocol, each pixel of each detection window uses a ${OT}_{1}^{256}$ operation, which needs $1$ RSA encryption and $256$ RSA decryption with 128-bit long encryption keys. We leverage our $M$ random images, whose computation is much faster than the RSA encryption and decryption operations.

In addition, in the traditional Blind Vision [2], they convert the integral-image representation to regular dot-product operation, a step that clearly slows down their implementation as they no longer take advantage of the integral-image representation. In our RBI based protocol, the integral-image representation is enabled again, which accelerates the computation obviously.

3 Experiments

We convert the Viola-Jones type object detector [1, 11] to our secure object detector. We implement our RBI based object detector using Microsoft Visual Studio 2012 and OpenCV 2.4.3/10. 111http://opencv.org/ package for computer vision in a 64 bits Windows 7 operating system. The hardware configuration is 3.5GHz AMD A10 Pro-7800 R7 CPU with 12 compute Cores and 8GB Memory.

The face detector is from the OpenCV 2.4.3 package and consists of a cascade of 22 rejectors, where each rejector is of the form presented in Eq. 1. The first rejector consists of 3 weak classifiers. The most complicated rejector consists of 213 weak classifiers. There is a total of 2135 weak classifiers. We also test the nose detector, the eye detector and the full body detector from OpenCV 2.4.10.

3.1 The Detection Accuracy

We test our secure face detector in 3 face detection datasets: The Face Detection Dataset (FDDB) [12], The Face96 Dataset [13], and The FEI Face Database [14].

We randomly select 100 face images from each of the 3 datasets. The detection accuracy (88.46%) of our secure face detector is the same as that of the OpenCV 2.4.3 face detector (88.46%).

The nose and the eye detectors are tested on the FDDB dataset [12]. The full body detector is tested on the INRIA Person dataset [15]. We randomly select 100 images from each of the 2 datasets. The detection accuracy of our secure object detectors is the same as that of the OpenCV 2.4.10 nose, eye and full body detectors.

3.2 Comparison with Other Methods

We compare our method with the Viola-Jones method implemented by the OpenCV package and the method of the traditional Blind Vision [2]. 50 test images with size of $100\times 100$ are randomly selected from each of the 3 datasets. The average running time is shown in Table 1. All the methods are running in client and server mode. For the Viola-Jones, Alice send the original image to Bob. Then, Bob runs the Viola-Jones method and return the detected windows to Alice. Our method is slower than the Viola-Jones method, which is running on plain images without protecting any privacy. According to the traditional Blind Vision method [2], the time-consuming OT operation is heavily used and the integral-image representation is disabled. Thus, they have to take a couple of hours to scan a single image, which is painfully slow. Although in our method, the only information that Bob learns is that how many faces are in the image of Alice, our cryptographic-free method is significantly faster than the previous work towards practical usage of blind vision applications.

In addition, we compare our method with the Viola-Jones method implemented by the OpenCV package and the method of the traditional Blind Vision [2]. 50 test images with size of $100\times 100$ are randomly selected from each of the FDDB and the INRIA Person datasets. The average running times are shown in the last 3 rows of Table 1. All the methods are running in client and server mode.

4 Conclusions and Discussions

We propose a novel random base image representation (RBI) for efficient object detection applications. The traditional blind vision method applies secure multi-party techniques to vision algorithm. Their method reveals no information to either party at the expanse of heavy computation load. Our method is an attempt towards cryptographic-free. Alice learns nothing about the parameters of the face detector of Bob. Bob does not know the contents of the image of Alice. The only information may be leaked is that Bob have a probability $1/M!$ to guess out the right permutation of the base images. This is just a theoretical event. Even Bob guesses out the right permutation, he does not know the weight of each base image. Thus it is almost impossible for Bob to learn the information of the detection window of Alice. Because the heaviest cost of OT operation in the secure dot-production of [2] is avoided by our RBI based dot-production, the Millionaire version protocol of ours need much less time than the traditional blind vision protocol does.

There are several extensions to this work. First is the need to accelerate the secure blind vision to practical use, i.e. to reduce the time cost to near that of the vision algorithm without security consideration. Second is to make both the training and the test blind. This will make the client users to upload more visual data to the cloud without worrying about the privacy leakage.