Training Object Detectors With Noisy Data

TL;DR

This paper investigates how label noise affects object detector training and proposes an improved co-teaching method to mitigate noise effects, demonstrating effectiveness on simulated and real-world noisy datasets.

Contribution

It introduces an enhanced co-teaching approach tailored for object detection to handle noisy labels, extending prior classification-focused methods.

Findings

Co-teaching improves detection accuracy with noisy labels.

Simulated noise experiments validate the method's robustness.

Real-world vehicle detection benefits from the approach.

Abstract

The availability of a large quantity of labelled training data is crucial for the training of modern object detectors. Hand labelling training data is time consuming and expensive while automatic labelling methods inevitably add unwanted noise to the labels. We examine the effect of different types of label noise on the performance of an object detector. We then show how co-teaching, a method developed for handling noisy labels and previously demonstrated on a classification problem, can be improved to mitigate the effects of label noise in an object detection setting. We illustrate our results using simulated noise on the KITTI dataset and on a vehicle detection task using automatically labelled data.

Tables

Table 1. TABLE I: Performance on KITTI dataset (Average precision on "Car, Moderate") with varying levels of noise of different types

	Noise Probability
Noise Type	0.0	0.25	0.5
None	0.629	-	-
Whole image label noise (1)	-	0.506	0.385
Per-object bounding box noise (2)	-	0.577	0.502
Additional boxes (3)	-	0.560	0.587
Missing boxes (4)	-	0.593	0.518
Combined noise (5)	-	0.457	0.317

Table 2. TABLE II: Comparison of standard and per-object co-teaching on the KITTI dataset (Average precision on "Car, Moderate")

		Co-teaching type
Noise type	Noise probability	None	Standard	Per-object
Whole image (1)	0.25	0.506	0.617	0.613
Whole image (1)	0.5	0.385	0.342	0.456
Combined (5)	0.25	0.457	0.527	0.666
Combined (5)	0.5	0.317	0.368	0.605

Table 3. TABLE III: Performance when trained using automatically labelled data. Evaluated against both automatic and hand-labelled test sets

Co-teaching type	Test set	AP	Max F1
None	Automatic	0.528	0.572
Standard	Automatic	0.517	0.566
Per-object	Automatic	0.555	0.601
None	Hand subset	0.302	0.399
Standard	Hand subset	0.321	0.414
Per-object	Hand subset	0.327	0.417

Equations

$$\mathbb{E}[n]=\sum _{k=0}^N\max (0,k-pk)\left(\genfrac{}{}{0ex}{}{N}{k}\right)p^k(1-p{)}^{N-k}$$

Full text

I Introduction

Modern object detectors based on convolutional neural networks (CNNs) such as Faster RCNN [1], RetinaNet [2] and SSD [3] require a large quantity of labelled training data to achieve high performance. Manually labelling training data is expensive and time consuming, so assembling a dataset for a new task can be a significant hurdle to implementation. One approach that is often taken is to pre-train with one of a number of freely available established datasets (such as ImageNet [4]) before fine-tuning using a small manually labelled dataset specific to the new task. However, this may not be possible if other sensor modalities are to be used or labelling a small fine-tuning dataset is not practical.

An alternative approach to the labelling problem is to use an automated labelling method. By using information not available at test time (such as expensive sensor modalities [5]) automated (or semi-automated) labelling can provide labels of sufficient quality for training. However, automatically generated labels inevitably include noise which reduces the performance of models trained using the data.

A number of methods have been proposed that aim to reduce the impact of noisy labels when training CNNs (for example [6]). However, the focus of these works is often on classification tasks, with their efficacy being evaluated using small scale datasets such as MNIST and CIFAR. Some methods attempt to estimate the noise transition matrix which makes them hard to use in settings with other forms of noise (such as the size of a bounding box label in an object detection task).

The recently proposed co-teaching algorithm [7] is designed for training CNNs from noisy data. It has been shown to be effective in the presence of large quantities of label noise in a classification setting but the technique is agnostic to the specifics of the problem. This makes it suitable for deployment on object detection tasks and we show how the specifics of the task can be leveraged to modify the algorithm to further improve performance.

In this work we make the following contributions:

• •

We examine the effect of different types of label noise on a typical CNN object detector.

• •

We document the change in performance as a result of applying co-teaching and modify the framework for better results.

• •

We apply these techniques to a real-world object detection problem for which a dataset including multiple sensor modalities has been collected.

II Related Work

Reducing the need to hand-label training data by taking advantage of large quantities of unlabelled, weakly labelled or automatically labelled data has been a topic of research for a long time.

One option is to attempt to label the unlabelled data during training. The well known co-training algorithm [8] iteratively increases the pool of labelled data by labelling unlabelled examples using a pair of classifiers trained on different views of the data in the previous pool. Another method [9] that, like co-training, pre-dates the deep learning era, iteratively adds unlabelled data to the labelled set by scoring detections based on a detector independent metric.

More recently, methods such as [10] and [11] exploit temporal consistency in videos to automatically mine new examples using an existing weak detector.

[12] demonstrates how sharing weights between an auto-encoder-like ladder network on unlabelled data with the feature extractor used for labelled data can leverage the unlabelled data during training to increase performance.

An alternative approach that has been gaining popularity is the use of synthetic training data. The effectiveness of this approach is demonstrated in [13] where the use of synthetic data achieves good results, however it also shows the value of a small quantity of real data for fine-tuning.

If the data already has labels but those labels are noisy there are a number of approaches that have been proposed to reduce the influence of the noise on the training process. Bootstrapping [14], for example, incorporates the current state of the model into the loss term resulting in down-weighting of the influence of the label for examples for which the model is confident.

[15] introduces two methods for estimating the noise transition matrix in classification problems. The S-model [16], also attempts to learn the noise transition matrix.

Countering the view that noise needs to be carefully modelled, [17] shows that a large quantity of noisy data can be effectively used to increase performance on a fine-grained classification task. From a similar perspective, [18] attempts to find the limits of weakly supervised data by pre-training with 3.5 billion images for classification and object detection tasks.

III Co-teaching

Co-teaching [7] is based on the memorisation characteristics of neural networks [19]. This effect means that simple patterns are learnt before examples are memorised. Provided that fewer than half of the labels are noisy (i.e. the labels don’t form their own more dominant pattern), those examples with noisy labels won’t fit the prevailing pattern and so will be more difficult to learn. Consequently, after a period of training, harder (and therefore more likely to be noisy) examples will have higher loss values.

To exploit this effect, co-teaching uses a pair of networks trained in parallel. The same batch is passed through both networks with the gradient updates of only those examples that are classified as low loss by the other network being applied. The proportion of examples in the batch that are excluded is set according to a schedule, increasing from zero at the start of training to a proportion corresponding to the estimated proportion of noisy labels. A schematic of the algorithm is shown in Fig. 2.

The two key hyper-parameters that need choosing are the estimated proportion of noisy labels and a time constant, the number of epochs over which to reduce the proportion of selected examples — if this constant is too large, the network will learn noisy examples before they can be effectively excluded, too small and the network has not had time to learn patterns before the loss values start being used to choose which examples to exclude.

III-A Types of label noise

The original co-teaching work demonstrates the technique’s efficacy in a classification setting with two types of label noise: symmetry noise, where a fixed proportion of labels are replaced by any of the alternative labels with equal probability and pair noise, where a fixed proportion of labels are consistently replaced with the next label. Another type of label noise that is not considered but which is of particular relevance when using automated labelling method, is systematic noise under which the labels of classes with similar appearance are swapped (for example 8 with 3 in the case of MNIST).

In an object detection setting each label comprises a bounding box and an associated class label. In addition to the types of noise already discussed that may affect the class label, there are additional types of label noise that may occur: spurious bounding boxes with random (or systematic) class labels may be present, bounding boxes may be incorrectly sized or positioned or may be missing entirely.

III-B Co-teaching for object detection

In the standard co-teaching formulation, gradient updates are applied for some images in a batch while the gradient updates from the highest loss images are excluded. This framework works effectively in a classification setting where the label applies to the whole image, so if the label is noisy it is reasonable to discard all updates related to that image. In an object detection setting, each image can contain multiple labels only some of which may be noisy. Consequently, by excluding the whole image, potentially useful gradient updates are being discarded.

We therefore introduce a modified formulation where gradient updates are selected on an object-by-object basis rather than an image-by-image basis. The simplest solution would be to rank total losses per object (by object, in this context we are referring to any anchor box that has been selected to contribute to the loss either by being matched to a label or selected as a negative). However, this straightforward modification is not well-posed in the context of training an SSD-style object detector and results in unstable training. This is because the SSD loss is formed of three parts — cross-entropy losses for positives, cross-entropy losses for hard-negatives and bounding box losses for positives. In the naive object-by-object formulation, the two networks suffer no penalty for constantly predicting all negatives and entirely excluding all positive examples on the basis that their high losses are a result of noisy labels. The exclusion of positive losses is exacerbated by the SSD loss in which the number of sampled hard-negatives is three times that of the number of positives (subject to a hyperparameter). To overcome this we construct our per-object formulation by selecting subsets of low loss examples independently for each component of the loss. This ensures that there are always losses associated with positives and hard-negatives contributing to each update step. It also means that each of the loss components can have a different estimated noise fraction and therefore each can have a low loss subset that constitutes a different fraction of the available losses. For example, if the prevailing noise is predominantly in bounding box size and position, the fraction of bounding box losses left out can be larger than the fractions for positive or hard-negative cross entropy. An illustration of the difference between the standard formulation and our proposed method is shown in Fig. 3. To gain more insight into the behaviour of our method the losses that are being removed can be plotted, as shown in Fig. 4.

Our experimental results in section VI demonstrate that this per-object formulation not only improves performance in the presence of label noise but also when compared to employing the standard per-image co-teaching formulation.

IV Automatic Label Generation

A motivating force behind this work is to enable the use of automatically labelled datasets. Being able to use automatically labelled data reduces the barrier to deployment of supervised learning systems in situations where existing hand-labelled datasets are not suitable. This might occur when operating in a domain that is not well served by existing datasets (such as at night) or when additional sensor modalities are required. Unfortunately, due to imperfections in labelling systems, automatically labelled datasets inevitably contain labelling errors hence the need for the method we are proposing.

To validate our approach we recorded and automatically labelled a new vehicle object detection dataset (full details on this dataset are given in [20]). On a series of drives we recorded stereo wide-angle RGB images, monocular zoom lens images as well as radar target data from a cruise control radar.

To perform the labelling we ran an existing object detector (based on a YOLO [21] architecture) over both the wide-angle and zoom lens images. We then combined the detections from the left wide-angle camera and the zoom lens camera by exploiting the fact that the cameras were physically located as close as possible to each other. This allowed us to transfer the long range detections from the zoom lens into the wide-angle images, improving the quality of the labels in the overlapping image region (which largely corresponds to the area of the image where most long range objects occur). We also use a visual odometry system [22] to generate ego-motion estimates from the stereo pair which are used to subtract the ego-motion from the radar target range rates (radial velocities). Some example labelling from the dataset is shown in Fig. 5.

The resulting dataset contains 17553, 2508 and 5015 frames each for training, validation and testing respectively. Each frame consists of the left wide-angle image, the ego-motion adjusted radar data and the generated class labels and bounding boxes.

V Network setup

To demonstrate the effect of noisy labels and co-teaching we conduct experiments using a standard object detector framework based on SSD that we train from scratch for each experiment. One of the downsides of co-teaching is that two networks are trained simultaneously meaning that twice as much memory is required compared to training a normal network111As an upside the process produces two fully trained networks. In this work we only make use of the results from the first network although they could potentially be combined.. As a result we use a relatively small ResNet18 [23] backbone network. For more details of the network configuration see [20].

Standard image augmentations (flips, crops and colour shifts) and L2 weight decay of 1e-3 are used during training. We train using the ADAM optimiser with an initial learning rate of 1e-4. The learning rate is reduced by a factor of 10 after 45k and 55k iterations.

VI Experimental results

We conduct experiments on two datasets, the KITTI 2D object detection dataset [24] and our own dataset described in Section IV.

The KITTI dataset includes images, labels and evaluation code for a 2D object detection benchmark. By adding controlled quantities of noise to this hand labelled dataset we can conduct a set of experiments to demonstrate the effect of label noise when training object detectors. As the labels for the test portion of the dataset are not available we conduct the experiments using splits of the training set with 5985, 374 and 1122 images for training, validation and testing respectively. For the KITTI experiments, all images are resized to 1024 x 320, training runs for 200 epochs using a co-teaching epoch constant of 50 and we evaluate using the KITTI evaluation code.

In all experiments we evaluate using average precision (AP).

VI-A Effect of label noise on KITTI

We first examine the effect of different label noise types on object detector performance. We experiment with five types of noise:

Whole image label noise. This is an approximation of the classification case such that with a certain probability per image, all the labels for ground truth objects in that image are corrupted using pair noise. 2. 2.

Per object bounding box noise. For each ground truth object labelled in an image, with a certain probability the bounding box of that label is shifted and scaled by amounts sampled from a normal distribution. 3. 3.

Spurious additional bounding boxes. For each image, with a certain probability, an extra bounding box of random position and dimensions is added to the image with a label randomly selected from the list of classes. 4. 4.

Missing bounding boxes. With a certain probability each ground truth bounding box may be removed from the set of labels. 5. 5.

Combined bounding box noise. This applies noise types (2), (3) and (4) simultaneously.

The results (Table I) show that object detection performance is significantly affected by label noise. While it is hard to make comparisons across noise types given that the probability is applied in different ways (e.g. extra boxes are per-image while missing boxes are per-box), it is interesting to note that even with 50% of the boxes missing performance only drops from 0.629 to 0.518 (this is in line with a result presented by [25]).

VI-B Co-teaching on KITTI

We next compare the performance of the standard and per-object co-teaching algorithms using the KITTI dataset. We compare using two sorts of noise, whole image noise (1) and combined noise (5) as this is a reasonable approximation of the sort of noise present in our own dataset (see Section IV and Fig. 5).

The results are shown in Table II. It can be seen that our proposed per-object co-teaching formulation improves performance over plain network training in every case and also outperforms standard co-teaching by a large margin in all but the simplest scenario. While it is initially interesting that one of the per-object co-teaching scores exceeds that of the no-noise case in Table I, this result is not replicated in wider metrics. Fig. 6 shows the KITTI evaluation curve for cars with combined noise at a probability of 0.5 using per-object co-teaching.

VI-C Effect of batch size

One likely contributory factor in the outperformance of our per-object formulation over standard co-teaching for object detection relates to batch size. The basic operation in co-teaching is that a batch is sampled at random and then a certain fraction of the updates from the batch are discarded as noisy based on the ranked loss values. However, as the batch is randomly sampled and the fraction discarded is not adjusted based on the properties of the batch, there is a non-zero probability that some noisy updates remain in the pruned batch. If it is assumed that it is possible to perfectly identify noisy and non-noisy items, in the standard formulation the expected number of noisy images $\mathbb{E}[n]$ that remain can be expressed as

[TABLE]

where the probability $p=p_{i}$ the noise probability for each image and $N=N_{i}$ the number of images in the batch. As shown in Fig. 7, by using a larger batch size the proportion of the remaining batch ($\mathbb{E}[n]/N$) that is expected to be noisy is decreased. If a per-object rather than per-image formulation is used then $p=p_{O}$ the noise probability for each object and $N=\phi N_{i}$ where $\phi$ is the mean number of objects per image. As as result, assuming $\phi>1$, the effective batch size is larger and the expected fraction of the batch that is noisy is reduced.

VI-D Co-teaching performance on automatically labelled data

Having proven the effectiveness of our method when noise is artificially added, we employ our method on our automatically labelled dataset (see IV).

We employ the same network as for the KITTI tests with some minor adjustments. Due to the number of very small objects in the dataset we employ a technique described in [26] to increase the number of default boxes used in each detection head. As the dataset is larger than KITTI we train for 70 epochs and the co-teaching epoch constant is set to 20.

As in [20] we conduct two forms of testing. We first conduct tests using the entire test split of the automatically labelled dataset. However, as the labels in the test set may also be noisy (and with potentially the same systematic biases as the training set) we also conduct tests using a small subset of the test set that has been hand-labelled.

For evaluation, as the KITTI evaluation ignores objects below a size threshold, again we follow [20]. This uses an IOU threshold of 0.5 and keeps all objects regardless of size. It also uses the more accurate AP integration method from [27]. The maximum F1 score attainable at an operating point along the precision-recall curve is also calculated.

The results of testing against both the full automatically labelled test set and the hand-labelled subset are shown in Table III. It can be seen that our per-object co-teaching method consistently provides the best performance. The overall average precision curves for all three methods are shown in Fig. 8. It is worth noting that one of the reasons for the lower levels of performance shown by all methods on the hand-labelled subset is that this subset, by virtue of the labeller having access to the zoom lens images, contains a large number of very small objects. This effect can also be seen in Fig. 9 where the "All" curve largely mirrors the shape of the "Small" curve.

VII Conclusions and further work

Learning from datasets with noisy labels is an important avenue of research as it makes the use of supervised machine learning models more viable in many applications. In this work we have shown how noise can affect the training of a CNN object detector and demonstrated how a modified version of co-teaching can be used to mitigate the effects of noisy labels. It is fair to say that this is not a perfect approach: the need to train two networks simultaneously places constraints on the types of models that can be easily used. In addition, there are a number of hyperparameters that have to be tuned to achieve any improvement in performance. At present the only methods for establishing which hyperparameters to use are manual inspection (for an initial estimate of noise parameters) and cross validation. For future work we plan to investigate ways of estimating the noise parameters and, by extension, estimates of the hyperparameters. One possible method for this is through inspection of the distribution of the various loss values during training.

Another interesting direction for future work would be to investigate the effect of dataset size. It is possible that the benefits of co-teaching diminish with larger datasets.

Acknowledgements

The authors would like to thank Martin Engelcke and colleagues for the object detector implementation used to generate the dataset. We also gratefully acknowledge the JADE-HPC facility for providing the GPUs used in this work. Paul Newman is funded by the EPSRC Programme Grant EP/M019918/1.