Frame-to-Frame Aggregation of Active Regions in Web Videos for Weakly Supervised Semantic Segmentation

TL;DR

This paper introduces a video-based approach that leverages temporal information and optical flow to improve weakly supervised semantic segmentation, achieving state-of-the-art results on PASCAL VOC 2012.

Contribution

It proposes a novel frame-to-frame aggregation method using web videos and optical flow to enhance object localization in weakly supervised segmentation.

Findings

Achieves 65.0% mIoU with VGG-16 backbone.

Achieves 67.4% mIoU with ResNet-101 backbone.

Outperforms existing methods under the same supervision level.

Abstract

When a deep neural network is trained on data with only image-level labeling, the regions activated in each image tend to identify only a small region of the target object. We propose a method of using videos automatically harvested from the web to identify a larger region of the target object by using temporal information, which is not present in the static image. The temporal variations in a video allow different regions of the target object to be activated. We obtain an activated region in each frame of a video, and then aggregate the regions from successive frames into a single image, using a warping technique based on optical flow. The resulting localization maps cover more of the target object, and can then be used as proxy ground-truth to train a segmentation network. This simple approach outperforms existing methods under the same level of supervision, and even approaches relying on extra annotations. Based on VGG-16 and ResNet 101 backbones, our method achieves the mIoU of 65.0 and 67.4, respectively, on PASCAL VOC 2012 test images, which represents a new state-of-the-art.

Tables

Table 1. Table 1: Comparison of the performance of weakly supervised segmentation methods using VGG16-based segmentation model on VOC 2012 validation and test sets.

Methods	val	test
Supervision: Image-level and additional annotations
${\text{MIL-seg}}_{\text{ CVPR ’15}}$ [30]	42.0	40.6
${\text{STC}}_{\text{ TPAMI ’17}}$ [45]	49.8	51.2
${\text{TransferNet}}_{\text{ CVPR ’16}}$ [11]	52.1	51.2
${\text{AISI}}_{\text{ ECCV ’18}}$ [16]	61.3	62.1
Supervision: Image-level annotations only
${\text{SEC}}_{\text{ ECCV ’16}}$ [23]	50.7	51.1
${\text{CBTS-cues}}_{\text{ CVPR ’17}}$ [33]	52.8	53.7
${\text{TPL}}_{\text{ ICCV ’17}}$ [22]	53.1	53.8
${\text{AE\_PSL}}_{\text{ CVPR ’17}}$ [44]	55.0	55.7
${\text{DCSP}}_{\text{ BMVC ’17}}$ [2]	58.6	59.2
${\text{MEFF}}_{\text{ CVPR ’18}}$ [9]	-	55.6
${\text{GAIN}}_{\text{ CVPR ’18}}$ [26]	55.3	56.8
${\text{MCOF}}_{\text{ CVPR ’18}}$ [43]	56.2	57.6
${\text{AffinityNet}}_{\text{ CVPR ’18}}$ [1]	58.4	60.5
${\text{DSRG}}_{\text{ CVPR ’18}}$ [17]	59.0	60.4
${\text{MDC}}_{\text{ CVPR ’18}}$ [46]	60.4	60.8
${\text{SeeNet}}_{\text{ NIPS ’18}}$ [15]	61.1	60.7
${\text{FickleNet}}_{\text{ CVPR ’19}}$ [24]	61.2	61.9
Ours	63.9	65.0

Equations

$$\widehat{{\mathcal{M}}_2^c}={\mathcal{M}}_2^c\cup \mathbb{W}({\mathcal{M}}_1^c,{\mathcal{P}}_{1\to 2}).$$

Full text

1 Introduction

Semantic segmentation is one of the most important tasks in computer vision and one that has made tremendous progress with fully annotated pixel-level labels [52, 3]. However, real applications of semantic image segmentation require a large variety of object classes and a great deal of labeled data for each class, and labeling pixel-level annotations is laborious. This problem can be addressed by weakly supervised methods that use more easily obtainable annotations such as scribbles, bounding boxes, or image-level tags.

Weakly supervised semantic segmentation methods have evolved rapidly. Scribble supervision [39] can now achieve 97% of the performance of manually supervised semantic segmentation with the same backbone network, and even the weakest image-level supervision [24] can produce 90% of that performance. However, the rate at which segmentation techniques relying on weak annotations are improving is declining rapidly. For instance, under image-level supervision, a 7.9% improvement on PASCAL VOC 2012 validation images [7] was achieved between 2016 and 2017 [23, 2], 1.8% between 2017 and 2018 [46], but only 0.8% since then [24].

Most methods of weakly supervised segmentation use localization maps activated by a classifier (e.g. a CAM [53]) as a proxy ground truth to train their segmentation network. However, because such localization maps are coarse and therefore activate only part of a target object, they provide a poor version of proxy ground truth. Some researchers have tried to address this problem by erasing discriminative part of the target object [44, 15] or introducing different sizes of receptive fields [46, 24, 25], but these developments encounter limitations because of the finite information that can be obtained from the image-level annotations, resulting in a coarse proxy ground truth. Of course, stronger methods of supervision, such as instance-level saliency [16], scribbles [40, 41], or bounding boxes [4] can be used, but the extra supervision required makes it harder to expand object classes or move to a new domain.

One way of reducing the effort required for strong supervision is to use so-called webly supervised segmentation, which makes use of the vast amount of image and video data on the web, which can be used in combination with other existing weakly annotated data. However, the data obtained from the web may be of low quality or may not depict any objects corresponding to the search term. Attempts have been made to improve the quality of such data by filtering method using knowledge learned from images with accurate image-level annotations (e.g. the PASCAL VOC 2012 dataset [7]), or by using existing image segmentation techniques such as GrabCut [32]. Despite labeling issues, some of the sophisticated webly supervised methods have shown remarkable performance [35, 12, 14, 21]. However, additional usage of web images brings the same level of coarseness as existing weakly annotated data, so the performance improvements using web images are merely due to the effects of data augmentation. On the other hand, the temporal variations in video allow a classifier to activate different regions of the target object, so that video offers the possibility of obtaining better pixel-level annotations than static images.

We propose a method of collecting regions activated in different frames, using a warping technique based on optical flow. Optical flow provides pixel-level displacements between two successive frames, making it possible to deduce which parts of the frames correspond. Warping the activated areas in the first frame into the second frame determines which areas of the second frame should be activated. These warped areas can be aggregated with the parts of the second frame which are activated on their own merits. We repeat this step so that the areas activated in several frames become available in a single frame. Figure 1(a) shows how this works on an example, and Figure 1(b) demonstrates the resulting increase in the area of the activated regions, and how this corresponds more clearly to the ground truth of the original image. In addition, the availability of multiple frames in a video allows more effective filtering to refine inaccurate labels (see Section 3.2).

Existing methods of webly supervised segmentation depend on off-the-shelf segmentation techniques [12, 35] such as GrabCut [32], complicated optimization of energy function [12, 42], or heuristic constraints [35] to generate a proxy ground truth of the data obtained from the web. None of these are needed by our technique.

The main contributions of this paper can be summarized as follows:

• $\bullet$

We propose simple data filtering and incremental warping techniques which allow web videos to be used as an additional data source in weakly supervised semantic image segmentation.

• $\bullet$

We empirically demonstrate that our method of processing web video data improves the performance of several methods of weakly supervised segmentation.

• $\bullet$

Our technique significantly outperforms other state-of-the-art methods on the Pascal VOC 2012 benchmark in both weakly supervised and webly supervised settings.

2 Related Work

Over the past four years, improvements of over 20$\%$ in the PASCAL VOC 2012 benchmark have been achieved by fully supervised semantic segmentation [27, 48]. However, because of the difficulty of obtaining pixel-level annotations of all the types of image and classes of object encountered in real applications, it is difficult to apply semantic segmentation widely. Weakly supervised semantic segmentation methods have been proposed to address this problem, and they have achieved promising performance (Section 2.1). Webly supervised semantic segmentation methods using images or videos obtained by web crawlers have been introduced as a way of closing the gap between the performance of weakly and fully supervised methods (Section 2.2).

2.1 Weakly supervised semantic segmentation

The goal of weakly supervised semantic segmentation is to train a image segmentation network with relatively imprecise delineations of the objects to be recognized. Weak supervision can take the form of scribbles [41, 40], bounding boxes [4], or image-level tags, which is the approach on which will now focus.

Most methods of image-level annotation are based on a class activation map (CAM) [53]. However, it is widely known that a CAM only identifies small discriminative parts of a target object [49, 17, 44]. Thus, the localization maps obtained by CAM are insufficiently complete to be used as a proxy ground truth to train a segmentation network. Several techniques have been proposed to expand these activated regions to the whole target object. Erasing methods [22, 44, 15] prevent a classifier from focusing solely on the discriminative parts of objects by removing those regions. Other methods construct CAMs that embody the multi-scale context of the target object, by means of different sizes of receptive fields. MDC [46] computes CAMs from features which have different receptive fields, realized by several convolutional blocks dilated at different rates. Pyramid Grad-CAM [25] collects features from each of several densely connected layers and merges the resulting localization maps. FickleNet [24] selects features stochastically by using modified dropout technique: this not only prevents the classifier from concentrating solely on the discriminative part, but also uses a different receptive field for every inference.

Region growing methods try to expand regions of the target object, starting from the initial CAM as a seed. AffinityNet [1] and CIAN [8] consider pixel-level semantic affinities, which identify relationship of pixels, and grow regions based on those affinities. SEC [23] and DSRG [17] progressively refine initial localization maps during the training of their segmentation network by means of resulting segmentation maps during training time, which are refined using a conditional random field (CRF).

2.2 Webly supervised semantic segmentation

Along with the growth of weakly supervised semantic segmentation, some researchers have attempted to improve the performance of weakly supervised methods using additional images or videos obtained from the web. WebS-i2 [21] collects two types of web data: images showing objects of the target class against a white backgrounds and images containing common backgrounds without any objects of interest class. It then trains the segmentation network by means of an iterative refinement process on realistic images whose weak annotations are accurately labelled. Web-Crawl [12] selects videos from YouTube by examining thumbnails, and segments them by spatio-temporal graph-based optimization. Bootstrap-Web [35] finds images which are expected to be easy to segment using heuristic constraints such as the sizes of objects, and exchanges knowledge between two networks; one is trained by filtered easy-to-segment web images and the other is trained by realistic hard-to-segment images. All these methods improve segmentation performance to some extent, but they are largely dependent on complicated optimization methods, on heuristic constraints, or on off-the-shelf segmentation methods such as GrabCut [32]

3 Proposed Method

Our goal is to train a segmentation network with weakly annotated image data $\mathcal{I}$ and web video data $\mathcal{V}$. While the image-level labels of each image in $\mathcal{I}$ have been manually annotated, $\mathcal{V}$ is annotated with noisy video-level tags because it has been collected from the web by searching with the name of each object class as the search term. Our training procedure has the following steps: A deep neural network is trained on $\mathcal{I}$ to identify classes of object (Section 3.1). After it has been trained, the network processes the videos in $\mathcal{V}$, and the classification results can be used to filter out the irrelevant frames or update the labels of the videos $\mathcal{V}$ (Section 3.2). The proxy ground truths for selected sequences of frames are then generated using an incremental warping method (Section 3.3). Finally, the proxy ground truth is used to train a segmentation network (Section 3.4). The overall procedure is shown as Algorithm 1.

3.1 Learning with precise labels

We train a deep convolutional neural network using $\mathcal{I}$, which has precise image-level multi-class labels. In order to obtain class activation maps (CAM) (Section 3.3), we modify the VGG-16 network [36] to be fully convolutional, by removing all the fully connected layers and adding additional convolutional layers so that the number of channels of the final output feature corresponds to the number of classes of interest. We then apply global average pooling (GAP) and a sigmoid function to the feature output by the network, so as to obtain a score for each class. We use these results to update the parameters of the classifier by means of a sigmoid cross-entropy loss function, of a sort widely used for multi-label classification.

3.2 Data filtering for noisy dataset

Images or videos which are expected to depict a certain class of object can be obtained from the web by searching, using the name of that class as the search term. That name is then used to label the images or videos that are acquired. But not all the resulting images or videos will actually show an object of that class, and many of them will contain objects of classes other than that corresponding to the search term. For example, a video obtained by searching for "horse" may show a person riding a horse; but it will just be labeled "horse", which is the search term.

Most webly supervised segmentation methods use the knowledge obtained from precisely annotated image data to eliminate web images or videos which do not depicit any objects corresponding to the search term. For example, Bootstrap-Web [35] uses SEC [23], which is trained by precisely annotated data, to obtain pixel-level class masks from web images. It discards images which have too few or too many pixels assigned to an object of the search-term class in the corresponding mask. Web-Crawl [12] deals specifically with videos, and rejects a video if it has fewer than 5 frames with classification scores for its search term that achieve a threshold. These methods work on the assumption that there is only a single class of object in an image or video. But automatically collected images or videos can be expected to contain objects of many classes. In PASCAL VOC 2012 data, more than 36% of training images are annotated with more than one class.

We therefore introduce an incremental multi-class filtering and label refining method which considers several successive frames of a video. We can eliminate videos which show no objects of interest and correct inaccurate labels much more effectively than these processes can be performed on single images. When the same class of object is found in consecutive frames of a video, it becomes increasing likely that an object of that class is actually depicted. This means that labels can be assigned with more confidence to a video than to an image, and multi-class labelling becomes more feasible.

A classifier trained in Section 3.1 processes the videos in $\mathcal{V}$, and infers the object classes present in each frame, which are taken to be those with a score larger than a threshold $\tau$. The set $\mathcal{C}$ of labels of these classes is then attached to each frame. We add a sequence of $K$ frames to the filtered video set $\hat{\mathcal{V}}$ if those $K$ frames all show objects from the same set of classes $\mathcal{C}$, and one of those classes corresponds to the search term. We eliminate frames that do not satisfy the above conditions.

3.3 Generating proxy ground truth

We now have a set of images $\mathcal{I}$, with accurate image-level labels, and a set of web videos $\hat{\mathcal{V}}$ with labels which are more accurate than those originally obtained from the search terms. We now describe the creation of a proxy ground truth for the frames in $\hat{\mathcal{V}}$, and its use to train a segmentation network.

3.3.1 Inference localization maps

We use a CAM [53] to obtain a localization map for each class of object in the images. Zhang et al. [49] showed empirically, and also proved mathematically that the $c-$th channel of the last feature becomes the CAM for the $c-$th class, within a fully convolutional network of the type described in Section 3.1.

We obtain CAMs from images which are scaled or flipped horizontally. After applying inverse transformations to the CAMs, we select the maximum value at each pixel over all the maps. Figure 2 shows some examples of the effect of flipping and scaling on CAMs. The CAM from a small image provides a coarse localization, and a CAM from a larger image identifies more detail of the object. Additionally, different regions are activated by flipped image.

3.3.2 Incremental warping of localization maps

It is widely known that a CAM only identifies the small discriminative part of a target object [49, 17, 44]. Our method obtains information about larger regions of a target object by merging information from successive frames of a video. The masks that indicate different regions of the target object are obtained from successive frames, and then aggregated into a single mask by considering optical flow. An analysis of optical flow between two successive frames provides a warp, which encodes the displacements which relate the pixels in one frame to those in the other. By warping the mask obtained from a frame to the following frame, we can transfer the activated region of the first frame to the second frame, allowing the union of the activated regions of both frames to be considered in a single image.

Let $\mathcal{X}\in\hat{\mathcal{V}}$ be a video containing $K$ frames $\{x_{i}\}_{i=1}^{K}$. We compute masks $\{\mathcal{M}_{i}^{c}\}_{i=1}^{K}$ for each class $c$ in $\mathcal{C}$ by thresholding the localization maps of the $\{x_{i}\}_{i=1}^{K}$ with the threshold $\theta_{f}$. We obtain optical flows $\{\mathcal{P}_{i\rightarrow i+1}\}_{i=1}^{K-1}$ between each pair of successive frames.

We will now consider a single aggregation step, which combines the masks $\mathcal{M}_{1}^{c}$ and $\mathcal{M}_{2}^{c}$ obtained from successive frames $x_{1}$ and $x_{2}$, for a single class $c$. Let $\mathbb{W}(I,f)$ be the function which warps an image $I$ following the flow field $f$, using bilinear interpolation. If $\mathcal{P}_{1\rightarrow 2}$ is the flow field between $x_{1}$ and $x_{2}$, then we can use this function to warp $\mathcal{M}_{1}^{c}$ to the space of $x_{2}$. The warped mask $\mathbb{W}(\mathcal{M}_{1}^{c},\mathcal{P}_{1\rightarrow 2})$ expresses regions of $x_{2}$ which correspond to the activated regions of $x_{1}$. An aggregated mask $\hat{\mathcal{M}}_{2}^{c}$ can then be obtained as the union of the warped mask from $x_{1}$ and the mask $\mathcal{M}_{2}^{c}$ obtained from $x_{2}$.

[TABLE]

We repeat this procedure for the remaining annotated object classes in $\mathcal{C}$, and call the result $\hat{\mathcal{M}}_{2}$. This procedure is illustrated in Figure 3(a), and Figure 3(b) shows that a unioned mask map $\hat{\mathcal{M}}_{2}$ contains the activated regions from both frame $x_{1}$ and $x_{2}$. $\hat{\mathcal{M}}_{2}$ can then be warped using the optical flow $\mathcal{P}_{2\rightarrow 3}$ and aggregated with $\mathcal{M}_{3}$, producing $\hat{\mathcal{M}}_{3}$. By repeating this procedure until the $K-$th frame, we can obtain $\hat{\mathcal{M}}_{K}$, which contains the union of all the activated regions from all $K$ frames. Aggregated masks from all the videos in $\hat{\mathcal{V}}$ can then be used as a proxy ground truth to train a segmentation network.

3.4 Segmentation Network

Many weakly supervised segmentation methods are built upon existing weakly supervised segmentation networks. For example, MDC [46] is based on a slightly modified version of the AE-PSL [44], and GAIN [26], TPL [22], and Boost-Web [35] are based on SEC [23]. We used FickleNet [24], but we also experimented with two other popular weakly supervised semantic segmentation networks: SEC [23] and DSRG [17]. All these segmentation networks are trained using the proxy ground truth of $\hat{\mathcal{V}}$ generated by the method described in Section 3.3. The background of $\hat{\mathcal{V}}$ is identified by saliency detection  [13] and includes all pixels with saliency values lower than $\theta_{b}$. When each segmentation network is trained, we use the proxy ground truth of $\mathcal{I}$ provided by the authors of each segmentation method.

For each iteration in training time, we create a batch; half of the elements in the batch come from $\mathcal{I}$, and the other half from $\hat{\mathcal{V}}$. We obtain the segmentation losses $L_{\mathcal{I}}$ and $L_{\mathcal{\hat{V}}}$ from the data from $\mathcal{I}$ and $\hat{\mathcal{V}}$ respectively. We then update the segmentation network by $L_{\mathcal{I}}+L_{\mathcal{\hat{V}}}$. Since $\mathcal{I}$ and $\hat{\mathcal{V}}$ may have different data distributions, we perform the domain adaptation according to [12]: the segmentation network is trained using both $\mathcal{I}$ and $\hat{\mathcal{V}}$ to predict segmentation masks for the images from $\mathcal{I}$, and those maps are used as a proxy ground truth to fine-tune the network.

4 Experiments

4.1 Experimental Setup

Image Dataset: We conducted experiments on the PASCAL VOC 2012 image segmentation benchmark [7], which contains 20 foreground object classes and one background class. Using the same protocol as other work on weakly supervised semantic segmentation, we trained our network using augmented 10,582 training images with image-level annotations. We determined mean intersection-over-union (mIoU) values for 1,449 validation images and 1,456 test images. The results for the test images were obtained from the official PASCAL VOC evaluation server.

Web Video Dataset: Starting with the Web-Crawl [12] dataset, we filtered out irrelevant frames and refine inaccurate labels with a threshold $\tau=0.9$, and aggregated masks from $K=5$ frames into a single mask, yielding 15,000 final samples for training a segmentation network. If several sets of frames can be selected from each video in Section 3.2, we chose only one to avoid similarity of samples. We obtained optical flows using PWC-Net [38]. The foreground and background thresholds, $\theta_{f}$ and $\theta_{b}$, were set to 0.2 and 0.12 respectively.

Classification Network: Our classifier is based on the VGG-16 network [36], pre-trained using the Imagenet [5] dataset. The VGG-16 network was modified by removing all the fully connected layers and the last pooling layer, and we replaced the convolutional layers of the last block with convolutions dilated with a rate of 2. We added two convolutional layers with 1024 channels and a kernel size of 3 with 2D dropout [37].

Segmentation Network: As already stated, we experimented with FickleNet [24], SEC [23], and DSRG [17]. We followed the settings recommended by the authors of those methods, except for the optional domain adaptation process, during which the learning rate was reduced to 0.01 from the default learning rate.

Reproducibility: PyTorch [29] was used for training the classifier, extracting CAMs, and obtaining optical flow [38], and we used the Caffe deep learning framework [20] in the segmentation step.

4.2 Experimental Results

4.2.1 Results on Image Segmentation

Weakly supervised segmentation: Table 1 shows comparison of recently introduced weakly supervised semantic segmentation methods with various levels of supervision. These methods all use a segmentation model based on VGG-16 [36]. Our method achieves mIoU values of 63.9 and 65.0 for PASCAL VOC 2012 validation and test images respectively, which is 94.4% of that of DeepLab [3], trained with fully annotated data, which achieved an mIoU of 67.6 on validation images. Our method is 3.1% better on test images than the best method which uses only image-level annotations for supervision. Our method also significantly outperformed several methods which have additional, as well as image-level annotations. These methods include TransferNet [11] which is trained on pixel-level annotations of 60 classes not included in the PASCAL VOC. AISI [16] has a salient instance detector, which is trained on well-annotated instance-level saliency maps, which are one of the most difficult forms of annotation to obtain. Our method also thoroughly outperformed existing methods based on the ResNet backbone [10] as shown in Table 2.

Figure A1 shows some examples of predicted segmentation masks produced by Bootstrap-Web [35], FickleNet [24], and our system, using both VGG-16 and ResNet-based segmentation models. In general, our proxy ground truth covers larger regions of the target object than those produced by other methods, so that the segmentation masks produced by our method tend to be more accurate.

Webly supervised segmentation: Table 3 shows mIoU values achieved on the PASCAL VOC 2012 dataset of webly supervised segmentation methods and the total number of training samples of each method. Our method showed the best performance despite being trained on a relatively small amount of data: 10.5k PASCAL VOC images and 15k frames of web video, whereas Web-Crawl [12] and Boot-Web [35] used 971k and 87.3k samples for training, respectively.

4.2.2 Results on Video Segmentation

In Table 4, we assessed the segmentation results produced by our system on the YouTube-Object dataset [31], and compared with state-of-the-art video segmentation methods with various degrees of supervision. We used segmentation masks annotated by Jain et al. [18] for ground-truth of evaluation. We also report the mIoU of DSRG [17] and FickleNet [24] trained on $\mathcal{I}$ alone as baselines. Our method showed better performance than the existing methods and even surpassed the methods which use stronger supervision, such as bounding boxes. A few examples of predicted segmentation masks for the YouTube-Object dataset are shown in Figure 6.

4.3 Ablation Study

Number of frames aggregated: Figure 5(a) shows mIoU scores for the PASCAL VOC 2012 validation images with different number of aggregated frames $K$. Using a single frame ($K=1$) results in only slight performance improvement on the score without any web videos ($K=0$). Increasing the number of successive frames across which maps are aggregated improves performance, which we expect, as larger regions of the target are represented by the final mask. But we find that aggregation above $K=5$ is not beneficial. This can be attributed to the approximations of the optical flow involved in frame-to-frame warping. Errors can be expected to build up as regions are warped across more frames, undermining the accuracy of the proxy ground truth (see Appendix). The examples of aggregated mask at each incremental warping step are shown in Figure 7.

Number of web samples: The effect of the number of web samples is shown in Figure 5(b). Without any web videos, FickleNet [24] is trained with the PASCAL VOC data alone, and the mIoU is 61.2. The mIoU value increases monotonically up to 15,000 samples. More samples produce little change in the performance.

Other weakly supervised segmentation networks: In addition to FickleNet [24], we experimented with SEC [23] and DSRG [17] with our method. Table 5 shows the performance of those three segmentation networks with image data $\mathcal{I}$ alone, with an additional video dataset $\mathcal{V}$, and also with domain adaptation $\mathcal{DA}$. SEC [23] does not have a retraining process, so we add a retraining step, before domain adaptation, along the line of that in DSRG [17]. The results in Table 5 show that our method works effectively with the three weakly supervised semantic segmentation networks.

The results for SEC [23] offer the possibility of a fairer comarison with Bootstrap-Web [35], which is based on SEC. For the PASCAL VOC validation images, our method achieved mIoU values of 61.1, while Bootstrap-Web [35] achieved 58.8.

5 Conclusions

We have proposed a method to use videos automatically obtained from the web as additional data in weakly supervised semantic segmentation. We obtain activated regions from each frame of a video and aggregate them on a single image, so that our proxy ground truth covers large regions of the target object. This method does not require additional supervision, it can be realized without complicated optimization processes or off-the-shelf segmentation methods, and it requires relative few samples, because a lot of information extracted from many frames can be aggregated into a single frame. We have demonstrated that our method produces better results than those from other state-of-the-art weakly and webly supervised approaches. We have also demonstrated that our method works effectively with several weakly supervised semantic segmentation networks.

Acknowledgements: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [2018R1A2B3001628], AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund, Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01367 ), Samsung Electronics (DS and Foundry), and the Brain Korea 21 Plus Project in 2019.

A Appendix