Deep Learning in the Automotive Industry: Applications and Tools

TL;DR

This paper explores deep learning applications in the automotive industry, focusing on computer vision, tools, datasets, and an end-to-end system for vehicle property recognition, demonstrating real-world manufacturing effectiveness.

Contribution

It introduces a new automotive dataset and an end-to-end deep learning application with mobile data collection and cloud training infrastructure.

Findings

Deep learning effectively automates vehicle property recognition.

Cloud and GPU infrastructures reduce training times and improve accuracy.

The trained classifier performs well in real-world manufacturing settings.

Abstract

Deep Learning refers to a set of machine learning techniques that utilize neural networks with many hidden layers for tasks, such as image classification, speech recognition, language understanding. Deep learning has been proven to be very effective in these domains and is pervasively used by many Internet services. In this paper, we describe different automotive uses cases for deep learning in particular in the domain of computer vision. We surveys the current state-of-the-art in libraries, tools and infrastructures (e.\,g.\ GPUs and clouds) for implementing, training and deploying deep neural networks. We particularly focus on convolutional neural networks and computer vision use cases, such as the visual inspection process in manufacturing plants and the analysis of social media data. To train neural networks, curated and labeled datasets are essential. In particular, both the availability and scope of such datasets is typically very limited. A main contribution of this paper is the creation of an automotive dataset, that allows us to learn and automatically recognize different vehicle properties. We describe an end-to-end deep learning application utilizing a mobile app for data collection and process support, and an Amazon-based cloud backend for storage and training. For training we evaluate the use of cloud and on-premises infrastructures (including multiple GPUs) in conjunction with different neural network architectures and frameworks. We assess both the training times as well as the accuracy of the classifier. Finally, we demonstrate the effectiveness of the trained classifier in a real world setting during manufacturing process.

Figures

Tables

Table 1. TABLE I: Distributed Deep Learning

	CaffeOnSpark	SparkNet	Tensorflow	CNTK
Base Framework	Caffe	Caffe	Tensorflow	CNTK
Model Distribution	replicated	central (spark master)	central (parameter server)	replicated/ partitioned
Model Update	synchronous	synchronous/ asynchronous	synchronous	synchronous/ asynchronous (1 Bit SGD)
Communication	MPI	Spark	gRPC	MPI

Table 2. TABLE II: Cloud Services for Data Analytics

	Amazon	Microsoft	Google
PaaS APIs	Amazon Rekognition	Project Oxford	Prediction API, Google Vision API, Speech API, Natural Language
Advanced Analytics	Amazon Machine Learning	Azure ML (incl. Jupyter Notebooks)	Cloud Machine Learning (with GPUs), DataLab (Jupyter Notebooks)
Deep Learning Framework	DSSTNE, MXNet	CTNK	Tensorflow
Data Platform as a Service	Elastic MapReduce	HDInsight, Data Lake Storage/Analytics	Google Dataproc, Cloud Dataflow
Data Storage	S3, Redshift	Azure Storage, SQL Datawarehouse	Cloud Big Table
Compute Nodes	EC2 (with GPU)	Azure Compute (GPU announced)	Google Compute Engine (no GPU)

Table 3. TABLE III: Object Detection Datasets

	Categories	Number Images	Size
Visual Inspection	100	82,011	9 GB (LMDB)
Cars [64]	196	16,185	1.87 GB (LMDB)
ImageNet 2012 [11]	1000	1,281,167	130 GB (LMDB)
Traffic Signs [61]	43	1,200	54.MB (LMDB)
Places [62]	205	2.5 mio	38.2 GB (LMDB)

Table 4. TABLE IV: Convolutional Neural Network Models

Network	Number Parameters	Number Layers	ImageNet Top 5 Error
AlexNet (2012) [7]	60 mio.	8 (5 convolutional, 3 fully connected)	15.3 %
GoogLeNet (2014) [66, 67]	5 mio.	22	6.7 %
VGG (2014) [68]	$\sim$140 mio.	19 (16 convolutional, 3 fully connected)	7.3 %
Inception v3 (2015) [69]	25 mio.	42	3.58 %
Deep Residual Learning (2015) [12]	$\sim$60 mio.	152	3.57 %

Full text

I Introduction

Machine learning and deep learning has many potential applications in the automotive domain both inside the vehicle, e. g. advanced driving assistance systems (ADAS), autonomous driving, and outside the vehicle, e. g. during development, manufacturing and sales & aftersales processes. Machine learning is an essential component for use cases, such as predictive maintenance of vehicles, personalized infotainment and location-based services, business process automation, supply chain and price optimization. A common challenge of these applications is the need for storage and processing of large volumes of data as well as the necessity to deal with unstructured data (videos, images, text), e. g. from camera-based sensors on the vehicle or machines in the manufacturing process. To effectively utilize this kind of data, new methods, such as deep learning, are required. Deep learning [1, 2] refers to a set of machine learning algorithms that utilize large neural networks with many hidden layers (also referred to as Deep Neural Networks (DNNs) for feature generation, learning, classification and prediction.

Deep learning is extensively used by many online and mobile services, such as the voice recognition and dialog systems of Siri, the Google Assistant, Amazon’s Alexa and Microsoft Cortana, as well as the image classification systems in Google Photo and Facebook. We believe that deep learning has many applications within the automotive industry, such as computer vision for autonomous driving and robotics, optimizations in the manufacturing process (e. g. monitoring for quality issues), and connected vehicle and infotainment services (e. g. voice recognition systems).

The landscape of infrastructure and tools for training and deploying deep neural networks is evolving rapidly. In our previous work, we focused on scalable Hadoop infrastructures for automotive applications supporting workloads, such as ETL, SQL and machine learning algorithms for regression and clustering analysis (e. g. KMeans, SVM and logistic regression) [3]. While deep learning applications are similar to traditional big data systems, training and scaling of DNNs is challenging due to the large data and model sizes involved. In contrast to simpler models, deep learning involves millions, instead of hundreds, of parameters and larger datasets, e. g. video, image or text data, for training. Training these models requires scalable storage (e. g. HDFS), distributed processing, compute capabilities (e. g. Spark), and accelerators (e. g. GPUs, FPGAs). Also, the deployment of these models is a challenging task – for deployment on mobile devices the number of parameters and thus, the required amount of new input data needs to be as small as possible. Modern convolutional neural networks often require billions of operations for a single inference.

This paper makes the following contributions: (i) It provides an understanding of automotive deep learning applications and their requirements, (ii) it surveys existing frameworks, tools and infrastructure for training DNNs and provides a conceptual framework for understanding these, (iii) it provides an understanding of the various trade-offs involved when designing, training and deploying deep learning systems in different environments. In this paper, we demonstrate the usage of deep learning in two use cases implemented on cloud and on-premise infrastructure, using different frameworks (Tensorflow, Caffe, and Torch) and network architectures (AlexNet, GoogLeNet and Inception). We show how to overcome various integration challenges to provide an end-to-end deep learning enabled application: from data collection and labeling, network training and model deployment in a mobile application. We demonstrate the effectiveness of the classifier by analyzing the classification performance of the mobile application during an extended test period.

This paper is structured as follows: in section II, we give an overview of automotive use cases. We evaluate the current tools available for deep learning in section III. We evaluate different deep learning use cases and models in conjunction with different public and proprietary datasets in section IV.

II Automotive Use Cases

Deep Learning techniques can be applied to many use cases in the automotive industry. For example, computer vision is an area in which deep learning systems have recently dramatically improved. Ng et al. [4] utilized convolutional neural networks for vehicle and lane detection enabling the replacement of expensive sensors (e. g. LIDAR) with cameras. Pomerleau [5] used neural networks to automatically train a vehicle to drive by observing the input from a camera, a laser rangefinder and a real driver. In this section we describe a set of automotive use cases for deep learning.

Visual Inspection in Manufacturing: The increased deployment of mobile devices and IoT sensors, has led to a deluge of image and video data that is often manually maintained using spreadsheets and folders. Deep learning can help to organize this data and improve the data collection process.

Social Media Analytics: Applications of computer vision can extend to social media analytics. Consumer-produced image data of vehicles made publicly available through social media can provide valuable information. Deep learning can assist and improve data collection and analysis.

Autonomous Driving: Different aspects of autonomous driving require machine learning technologies, e. g the processing of the immense amounts of sensor data (camera-based sensors, Lidar) and the learning of driving situations and driver behavior.

Robots and Smart Machines: Robotics requires sophisticated computer vision sub-systems. Deep learning performs well for recognizing features in camera images and other kinds of sensors needed to control the machine. While object detection using DNN is well understood, a more challenging task in this domain is object tracking. Further, deep learning enables self-learning robots that become more intelligent over their lifetime.

Conversational User Interfaces: Our connected vehicle already is the platform for a large number of services. Voice dialog systems will become more natural and interactive with deep learning allowing a hands-free interaction with the vehicle.

In the following, we focus on the visual inspection application as an example to understand the trade-offs between different datasets, model architectures, training and scoring performance. Further, we analyze a use case in marketing analytics to discuss performance in a real-world scenario.

III Background, Tools and Infrastructure

In this section, we provide some background on deep learning and survey the landscape of tools for training neural networks.

III-A Background

Neural networks are modeled after the human brain using multiple layers of neurons – each taking multiple inputs and generating an output – to fit the input to the output. The use of multiple layers of neurons allow the model to learn complex, non-linear functions. These Deep Neural Networks (DNNs) are particularly advantageous for unstructured data (which the majority of data is) and complex, non-linear separable feature spaces. Schmidhuber [6] provides an extensive survey of deep neural networks.

DNNs have shown superior results when compared to existing techniques for image classifications [7], language understanding, translation, speech recognition [8], and autonomous robots. Specialized neural networks have emerged for different use cases, e. g. convolutional neural networks (CNN), which pre-process and tile image regions for improved image recognition. Conversely, recurrent neural networks add a hidden layer that is connected with itself for better speech recognition. Promising advances have been made in automatically learning features (also referred to as representation learning), through auto-encoders, sparse coders and other techniques (see [9, 10]). This is particularly important as labeled data is difficult to obtain and the costs for feature engineering are high.

There have been great advances in deep learning observable in the rapid improvements of image classification accuracy in the ImageNet competition [11]. The ImageNet competition comprises a classification of a 1,000 category dataset of $\sim$1.2 mio images. In 2015, the top 5 error rate achieved by a convolutional neural networks (3.57 % for Microsoft’s Residual Nets approach [12]) was better than that of a human (5.1 %). Another example is the recent success of AlphaGo [13] in mastering the Go Game. Go is particularly challenging as the search tree that needs to be mastered by the machine is very large: there are about 200 possibilities per move and a game consists of 150 moves leading to a search tree with a size of about $200^{150}$. AlphaGo uses an ensemble of techniques, such a Monte-Carlo Tree search combined with a set of deep neural networks.

III-B Deep Learning Libraries

Neural networks – in particular deep networks with many hidden layers – are challenging to scale. Also, the application/scoring against the model is more compute intensive than other models. Figure 1 illustrates the different layers of a deep learning system. GPUs have been proven to scale neural networks particularly well, but have their limitations for larger image sizes. Several libraries rely on GPUs for optimizing the training of neural networks [14]. Both NVIDIA’s cuDNN [15] and Intel’s MKL [16] optimize critical deep learning operations, e. g., convolutions. On top of these several high-level frameworks emerged - some of which provide integrated support for distributed training, while others rely on other distributed runtime engines for this purpose.

Several higher-level deep learning libraries for different languages emerged: Python/scikit-learn [17], Python/Pylearn2/Theano [18], Python/Dato [19], Java/DL4J [20], R/neuralnet [21], Caffe [22], Tensorflow [23], Microsoft CNTK [24], Amazon DSSTNE [25], MXNet [26], Lua/Torch [27] and Baidu’s PaddlePaddle [28]. The ability to customize training and model parameters differs; while some tools (e. g., DIGITS [29], Pylearn) focus on a high-level, easy-to-use abstractions for deep learning, frameworks such as Theano and Tensorflow customizable low-level primitives. Further, several high-level frameworks emerged: Keras [30] provides a unified abstraction for specifying deep learning networks agnostic of the backend. Currently, two backends: Theano and Tensorflow are supported. Lasagne [31] is another example for a Theano-based library.

III-C Distributed Deep Learning

The ability to scale neural networks – i. e. to utilize networks with many hidden layers and the ability to it train large datasets – is critical in order to train networks on large datasets in short amounts of time (important to ensure fast research cycles). Neural networks utilizing millions of parameters are generally more compute-intensive than other learning techniques. The deeper the network, the higher the number of parameters and thus, the larger the size of the model. In distributed approaches this model needs to be synchronized across all nodes. To scale neural networks, the usage of GPUs [15], FPGAs [32], multicore machines and distributed clusters (e. g. DistBelief [33], Baidu [34]) have been proposed. In the following, we particularly focus on approaches for supporting distributed GPU clusters.

Training large datasets on large deep learning models requires distributed training, i. e. the usage of a cluster comprising of multiple compute nodes n. Distributed machine learning requires the careful management of computation and communication phases as well as distributed coordination. In general, there are two types of parallelism to exploit: (i) data parallelism and (ii) model parallelism (see Xing et al. [35] for a overview). Data parallelism is generally well-understood and easier to implement ; model parallelism requires the careful consideration of dependencies between the model parameters.

Most distributed deep learning libraries provide a distributed implementation of gradient descent optimized for parallel learning. Implementing data parallelism for gradient descent is well-understood: the data is partitioned among all workers, which each computes parameter updates for its partition. After each iteration parameters are globally aggregated and the model is updated. Systems typically differ in the way the model is stored and updated, and on how coordination between the workers is carried out. Some systems store the model centrally using a central master node, a set of nodes or dedicated parameter servers node(s), while others replicate/partition the model across the worker nodes. Model updates can be done synchronously or asynchronously (Hogwild [36]).

Hadoop [37] and Spark [38] emerged as de-facto-standard for data-parallel applications [3]. However, support for deep neural networks is still in its infancy. Spark provides a good platform for data pre-processing, hyper-parameter tuning, and for distributed communication and coordination. There is ongoing work to implement artificial neural networks in Spark [39] as part of its MLlib machine learning library [40]. In addition, various approaches for integrating Spark with frameworks, such as Caffe and Tensorflow emerged (see table I).

CaffeOnSpark [41] provides several integration points with Spark: it provides Hadoop InputFormats for existing Caffe formats, e. g. LMDB datasets, and allows the integration of Caffe learning/training stages into Spark-based data pipelines. CaffeOnSpark implements a distributed gradient descent. Gradient updates are exchanged using a MPI AllReduce across all machines.

SparkNet [42] utilizes mini-batch parallelization to compute the gradient on RDD-local data on worker-level. In each iteration, the Spark master collects all computed gradients, averages them and broadcasts the new model parameters to all workers. Similarly, TensorSpark [43] utilizes a parameter server approach to implement a “DownpourSGD” (see DistBelief [33]).

Both Tensorflow [23] and CNTK [12] provide different distributed optimizer implementations. Tensorflow offers a relatively low-level API to implement data- and model parallelism using a parameter server with synchronous respectively asynchronous model updates. Communication is implemented using gRPC. CNTK offers several parallel SGD implementations, which can be configured for training a network. The 1-bit SGD [44] reduces the amount of data for model updates significantly by quantizing the gradients to 1-bit. Communication in CTNK is carried out using MPI.

In addition to the frameworks described above, several other systems exist: FireCaffe [45] is another framework built on top of Caffe; [46] and [47] provide alternative distributed Tensorflow implementations.

III-D Cloud Services

Cloud computing becomes increasingly a viable platform for implementing end-to-end deep learning application providing comprehensive services for data storage, processing as well as backend services for applications. In the following we focus on data-related cloud services. Figure 2 categorizes services into three layers: data storage, Platform-as-a-Services (PaaS) for Data and higher-level Software-as-a-Service (SaaS).

An increasing number of infrastructure-as-a-service (IaaS) offerings with GPU support exists: Amazon Web Services (AWS) provide the hardware necessary for deep training and exploration while removing the necessity of obtaining a physical system for computation. All services such as GPU computing and data storage utilize the cloud and can therefore be managed accordingly. Amazon Web Services Elastic Compute Cloud (EC2) is a service that provides cloud computing with resizable compute capabilities including up to four K520 Grid GPUs [48]. Similar capabilities have been announced by Microsoft. While Google does not provide GPU as part of its Google Compute Engine Service, it provides a managed PaaS environment for Tensorflow, which offers GPU support [49].

Every cloud provider provides a managed Hadoop/Spark environment. There are minor differences in the feature: Amazon Elastic MapReduce [50] relies on his own Hadoop distribution and also supports Presto and Mapr, Microsoft’s HDInsight [51] is based on Hortonworks, Google’s Dataproc [52] also utilizes his own distribution. Typically, these Hadoop environments can read data from Blob storage and provide a HDFS cluster. They provide core nodes, which offer important services a such as the Namenode and YARN, and worker nodes, which can be scaled with demand.

Further, there are various cloud products related to search and streaming data. Azure provides a native search engine: Azure Search that can easily index Azure storage. Both Amazon and Microsoft provide a managed ElasticSearch environment. Increasingly, there is the need to react on incoming data streaming using various streaming tools and platforms. Topically, streaming systems consists of a broker engine (e. g. Kafka) and processing tools on various levels (e. g., Storm and Spark Streaming). Azure offers support for Streaming via the Azure Event Hub and Storm at the moment.

In addition several higher-level machine learning emerged. Google’s Prediction API [53] was one of the first services offering machine learning classifications and predictions in the cloud. Microsoft’s Azure ML [54] and Amazon Machine Learning [55] offer similar services. These services allow simple and fast access to machine learning capabilities. Models are easily deployed and published for further usage. In particular, Google and Amazon often provide black-box models with limited abilities for calibration of the model. Microsoft allows the creation of more general data pipelines supporting custom R and Python code.

A lot of shrink-wrapped solutions that offer deep learning capabilities behind a high-level cloud API (Platform as a Service), e. g. for advanced machine learning tasks, such as facial recognition, computer vision and machine translation, are often based on deep learning. Examples are Microsoft’s Project Oxford [56], Google’s Vision API [57] and Natural Language API [58], and IBM’s Watson developer cloud (AlchemyVision API) [59]. The core of these services relies on deep learning technologies. However, these services are constrained by the number of categories they support – Project Oxford’s Image API supports only 86 categories. Also, training on custom categories and data, via transfer learning, is often not possible.

IV Implementation and Evaluation

In this section, we evaluate different convolutional neural networks for object detection on two different datasets (i) images collected at a manufacturing facility and (ii) a hand-curated social media datasets. Further, we evaluate different deep learning frameworks to understand training and inference performance.

IV-A Experiments and Evaluation

In the following, we evaluate different frameworks for training the deep neural networks. For experiments, we use a machine with 2 CPUs, a total of 8 cores, 128 GB memory and a TITAN X GPU. Further, we utilize Amazon Web Services GPU nodes (g2.8xlarge), which provides 32 cores, 60 GB memory and 4 K520 GPUs [48]. For training the Caffe and Torch models, we use DIGITS [29] and the models provided with it. For Tensorflow, we adapted the provided AlexNet implementation [60].

IV-B Datasets

We identified a set of datasets relevant for the automotive industry (see Table III). ImageNet is one of the largest publicly available datasets. The usage of ImageNet and transfer learning is particularly suited for social media analytics and other forms of web data analysis. For enterprise use cases it is required to curate custom datasets. In particular for advanced applications, such as autonomous driving, it is essential to create suitable datasets, as datasets like Traffic Signs [61], Places [62] and Kitti [63], are designed for benchmarking primarily. Real-world applications require more data.

Further, we created a new dataset using data created during the visual inspection process. This dataset contains images from 4 vehicle types and 25 camera perspectives, i. e. a total of 100 categories, that were captured using the mobile application described below. It currently consists of 82,011 images.

IV-C Visual Inspection for Manufacturing

To support the visual inspection process during manufacturing and to aid data collection, we built an iPad application. The application is used by associates to document a subset of produced vehicles using approximately 20 walk-around pictures. Figure 3 shows the architecture of the application and the deep learning backend. The iPad automatically uploads taken images to Amazon S3; The metadata is stored in a relational database backend. Both data movement and storage are encrypted. For data-processing, we utilize a combination of Hadoop/Spark and GPU-based deep learning frameworks deployed both on-premise and in the cloud. For data pre-processing and structured queries, we rely on Hadoop and Spark [65]; for deep learning we rely on some GPU nodes.

The trained network is integrated into the iPad application to validate new images taken by the associate. For this purpose, we compiled Caffe for iOS and used the trained model files.

IV-C1 Models Training

We investigate different convolutional network architectures. Table IV gives an overview of the different model architectures investigated. In the following, we compare the AlexNet and GoogLeNet architectures implemented on top of Tensorflow, Caffe and Torch.

Figure 4 illustrates the training times observed for 30 epochs of the data with different frameworks. There is an improvement in the training times between Caffe 2 and 3 as well as TensorFlow 0.6 and 0.7.1. This can be attributed to the usage of newer versions of cuDNN (v4). We achieved the best training time with Tensorflow 0.7.1. TensorFlow 0.9.0 is also evaluated as the breaking edge version of the software. In our experiment, the training time is slightly slower than with previous Tensorflow, which can be attributed to a single factor; inconsistent training times per iteration. With TensorFlow 0.7.1, each iteration has a standard deviation over all 30 epochs less than 2 seconds. Conversely, TensorFlow 0.9.0, while mostly consistent, has a few iterations which cause the standard deviation to be much larger. This can be seen in figure 4 as the error bar for TensorFlow 0.7.1 is small in comparison to its counterpart for TensorFlow 0.9.0. This inconsistency with some iteration times results in a longer overall training time.

We also compare performance using TensorFlow 0.9.0 on a local machine versus a machine utilizing cloud services. Figure 4 illustrates a performance comparison of the EC2 web service and a local machine containing a TitanX GPU. The local system utilizing TensorFlow provides a quicker training time for the dataset provided, however, AWS EC2 would be a great option if a physical machine with dedicated hardward is unavailable as the training time is  1.5x longer than that of the local machine with the TitanX. The GPU used in AWS EC2 provides the same amount of compute cores as the TitanX, however, the clock speed is slower, allowing faster computation to occur on the TitanX. Also, the K520 GPU provides 8 GB of device memory, while the TitanX provides 12 GB allowing for larger models or larger batch sizes to be used for computation.

Further, a software comparison is made between cuDNN v4 and v5.1 on the TitanX. The update in software directly leads to decreased training time on the same hardware from  9,750 seconds to  7,380 (a decrease of  25 %). For larger datasets and larger networks, this update greatly improves training time allowing for faster production of models.

Figure 5 compares the training times for AlexNet and GoogLeNet using Caffe. Training GoogLeNet is 70 % slower than AlexNet mainly due to the higher complexity of the networks (more deep layers). Inception overshadows both AlexNet and GoogLeNet due to the complexity and deep nature of the network.

Our investigation also included a comparison of the peak accuracies achieved from training our models on different frameworks as well as the time in epochs it took to reach them. Figure 6 shows this comparison for the AlexNet model. There are no changes in peak accuracy performance between versions of Caffe or Tensorflow. This is expected behavior since only the underlying implementation of the frameworks, and not the algorithm of the model, have been changed between versions. The best peak accuracy we recorded was 94 % with all versions of Tensorflow.

Lastly, we compared the number of epochs required by each framework to achieve its peak accuracy: TF shows the quickest convergences with 17 epochs in average, followed by Torch with 23 epochs and Caffe with 28 epochs. Fewer epochs directly translate to a shorter training time.

IV-C2 Multiple GPU Training

The ability to train CNNs on large datasets of images for recognition and detection is critical. In the following we analyze the training time for the Visual Inspection and ImageNet datasets in conjunction with multiple GPUs. We utilize the ImageNet 2012 dataset consisting of 1,281,167 images and 1,000 classes, which is significantly larger than the Visual Inspection datasets with  82,011 images. For training, we use the Caffe framework with GoogLeNet and AlexNet for the Visual Inspection dataset.

We are able to achieve similar accuracies for multiple GPUs training as for single GPU, e. g., for ImageNet a top-5 accuracy of 87 % was obtained. Figure 7 illustrates the execution time, speedup, and efficiency for up to 4 GPUs. As expected the training time decreases with the number of GPUs. The efficiency, however, decreases nonlinearly pointing out that even though the execution time is decreasing, the addition of GPUs is causing an inefficiency. The speedup of using 2 GPUs is 1.5 which corresponds to an efficiency of 0.8, while training using 4 GPUs shows a speedup of 1.8, corresponding to an efficiency of 0.45. For the significantly smaller Visual Inspection dataset a maximum speedup of 1.6 corresponding to an efficiency of 0.4 was observed with 4 GPUs. This shows that the use of more GPUs is not always advantageous as the efficiency drops quickly if the GPUs are not utilized fully. Another interesting observation is the behaviour of GoogLeNet vs. AlexNet: while the training time for GoogLeNet is slightly higher, the scaling efficiency of GoogLeNet is slightly better than for AlexNet.

IV-C3 Model Deployment

For deployment of deep learning models in particular in mobile and embedded environments, the performance is essential. The more complex the network, the more compute-intensive the scoring process. There are two options for deploying the model: (i) on the mobile device and (ii) in the backend system. An important concern in particular for mobile deployment is the model size, which depends on the number of parameters in the model. The trained GoogLeNet model is about 43 MB in size, while the AlexNet model is 230 MB.

In Figure 8 we compare the inference time on different platforms. Not surprisingly, the best performance is achieved on GPUs (TitanX). The performance penalty on mobile devices is acceptable. The inference time on a iPad Air 2 with an A8X custom chips is on average only 22 % slower than on a server side CPU. The performance of Apple’s newest mobile CPU (A9) is only 3.7 % worse than the server side performance. In particular, the mobile deployment performance of GoogLeNet is slightly better than that of AlexNet.

As the performance on the mobile platform is acceptable and the object recognition tasks has a static nature, we integrated the model into the iPad application to give the user the opportunity to quickly verify the taken image. In the future, we explore approaches for further optimizing networks for mobile and embedded deployments, e. g., using compressing techniques [70].

The application was successfully deployed in production. Figure 9 shows the average classification performance computed using a sample of 204,883 classifications collected over a period of multiple weeks. As previously described the classification is done within the mobile application after the image has been taken, i. e. the CNN has not seen the data before. In contrast to the training set, the data was not carefully prepared and pre-processed. The application utilizes a reduced set of 21 categories. As shown in the figure, the accuracy varies between 44 % in category 6 to 98 % in category 1. In average we were able to achieve an accuracy of 81 % on data scored in real-time within the mobile application. In the future, we will utilize the new data to improve the accuracy in the low-performing categories.

IV-D Social Media Analytics

In the following with utilize a CNN for recognition of vehicle models in social media data collected from Twitter. A Python application was developed to display the currently streaming image with its top five classifications predicted by the neural network. Further experiments were conducted using focus regions within the image to improve classification accuracy. More details are discussed in the following sections.

The Cars dataset released by the Stanford AI Lab [71] consists of 16,185 images grouped into 196 categories of the form: Make, Model, Year. We decreased the granularity of the classes into 49 separate car brands as we were primarily concerned with detecting different brands. We used a pre-trained ImageNet GoogLeNet model from the Berkeley Vision and Learning Center (BVLC) [72]. We then applied transfer learning techniques to further train our model on a car models dataset.

To process social media data, we implemented a two version: (i) the standard version processes the image is processed in its original form, (ii) the region-search version adds an additional pre-processing step: First, we conduct a selective search [73] on the image to isolate object regions within the image. Next, these regions are passed to an ILSVRC13 detection network provided by BVLC [74] in order to extract object regions containing cars. Then, these extracted car regions are passed to our model for inferencing. Finally, the top 5 most confident class predictions over all car regions are selected for classification of the input image.

We used a sample of 106 images from the Twitter feed to measure our model’s performance in five categories: classification accuracy, precision, recall, F1 score, and processing speed per image. Figures 10 and 11 show a comparison of the performance metrics between our the standard (i) and region-search version (ii).

Figure 10 compare both models in terms of their classification performance for the top-5 predicted classes. For the standard workflow, we observed a top-5 accuracy of 81.1 % and F1 score of 85.9 %. With the region-search version (ii), the top-5 accuracy improved to 82.1 % and the F1 score to 87.2 %. This is only a very modest, statically insignificant increase of $\sim$1 % . However, we also measured our region-based workflow against only images which our standard version failed to predict correctly, which lead to an improvement in the top-5 accuracy of 53.1 %.

Figure 11 compares both models in terms of processing speed in seconds per image. We found that our standard workflow processed each image on average 0.002 seconds. The standard version significantly outperforms the region-search version, which took an average of 0.13 seconds/image. This outcome is expected due to the extra image preprocessing steps involved in the region-search version.

Overall, we found that both workflows performed the same over the sampled images. However, the region-based workflow showed significant improvement in images where the standard workflow failed, specifically in images where the car being analyzed did not encompass the bulk of the image. Our region-based approach was able to better identify a focus region in the image to pass to our classifier, resulting in more accurate predictions on such images.

V Conclusion and On-Going Research

Deep learning enables computers to learn objects and representations, it is however, associated with several challenges: it requires massive amounts of data, new tools and infrastructures for computation and data. We showed that existing model architectures and transfer learning can be applied to solve computer vision problems in the automotive domain. In this paper, we showed the successful deployment of deep learning for visual inspection and social media analytics. We successfully showed the trade-offs when training and deploying deep neural networks on a diverse set of environments (on-premise, cloud). We showed the effectiveness of the training classifier achieving an accuracy of 85 % during real-world use.

Several challenges for a broader deployment of deep learning remain: The availability of labeled data is critical for development and refinement of deep learning systems. Unfortunately, the datasets publicly available (other than ImageNet) are not sufficient for advanced systems, e. g. for autonomous driving. Curating training data beyond existing public datasets is a tedious task and requires significant effort. To improve the speed of innovation, the training time needs to be further improved.

In the future, we will: (i) investigate distributed deep learning systems to improve training times for more complex networks and larger data sets, (ii) assess and curate available datasets for computer vision use cases in the domain of autonomous driving and (iii) evaluate natural understanding deep learning models (e. g., sequence-to-sequence learning).

Acknowledgements

We thank Ken Kennedy and Colan Biemer for proof-reading. We acknowledge Darius Cepulis for his early work on deep learning benchmarks.