Cross-Modal Weakly Supervised RGB-D Salient Object Detection with a Focus on Filamentary Structures

TL;DR

This paper introduces a new framework for detecting salient objects in RGB-D images using weak supervision, especially improving detection of filamentary structures.

Contribution

A novel cross-modal weakly supervised SOD framework with pseudo-label generation and asymmetric feature extraction for better filamentary object detection.

Findings

The proposed framework outperforms existing methods in detecting salient objects with complex and filamentary structures.

The use of cross-modal pseudo-labels and asymmetric networks improves boundary accuracy in RGB-D saliency detection.

An edge constraint module enhances the sharpness of predicted salient regions.

Abstract

Current weakly supervised salient object detection (SOD) methods for RGB-D images mostly rely on image-level labels and sparse annotations, which makes it difficult to completely contour object boundaries in complex scenes, especially when detecting objects with filamentary structures. To address the aforementioned issues, we propose a novel cross-modal weakly supervised SOD framework. The framework can adequately exploit the advantages of cross-modal weak labels to generate high-quality pseudo-labels, and it can fully couple the multi-scale features of RGB and depth images for precise saliency prediction. The framework mainly consists of a cross-modal pseudo-label generation network (CPGN) and an asymmetric salient-region prediction network (ASPN). Among them, the CPGN is proposed to sufficiently leverage the precise pixel-level guidance provided by point labels and the enhanced semantic supervision provided by text labels to generate high-quality pseudo-labels, which are used to supervise the subsequent training of the ASPN. To better capture the contextual information and geometric features from RGB and depth images, the ASPN, an asymmetrically progressive network, is proposed to gradually extract multi-scale features from RGB and depth images by using the Swin-Transformer and CNN encoders, respectively. This significantly enhances the model’s ability to perceive detailed structures. Additionally, an edge constraint module (ECM) is designed to sharpen the edges of the predicted salient regions. The experimental results demonstrate that the method shows better performance in depicting salient objects, especially the filamentary structures, than other weakly supervised SOD methods.

Figures

Full text

1. Introduction

Salient object detection (SOD) aims to emulate human visual attention by capturing the most salient objects and regions in an image, which offers crucial cues for higher-level visual tasks such as image retrieval [1,2,3], object tracking [4], image quality assessment [5], scene analysis [6,7], object classification [8], and autonomous driving [9]. Fully supervised SOD tasks require extensive pixel-level annotation, which is time-consuming and labor-intensive [10,11,12,13,14]. The emergence of weakly supervised learning methods effectively addresses the data labeling issue and enhances the transferability of the model. However, weakly supervised methods [15,16,17,18,19] usually rely on image-level labels or limited annotation information, and it is difficult for weakly supervised methods to provide sufficient annotation information as guidance during the model’s training.

To enrich the information of annotation, researchers propose multi-source weakly supervised methods. However, obtaining high-quality pseudo-labels is still difficult. Low-quality pseudo-labels are inadequate in enabling the model to recognize complete edges, particularly filamentary structures. Since semantic information provides feature representations of the object and visual information provides its positional information, this work selects complementary cross-modal weak labels to generate supervisory information that is both rich and accurate. Moreover, as depth images provide rich geometric information [20,21,22,23,24], an increasing number of studies have utilized depth images as a crucial 3D spatial cue for SOD tasks. Depth images help the model distinguish salient objects from the background by perceiving the depth structure of the scene, which is particularly useful in complex scenes. However, there are significant differences among different modalities. For example, RGB images contain rich contextual information, while depth images primarily provide geometric information. This difference makes it difficult for traditional fusion methods such as channel concatenation or element-wise summation to effectively utilize the complementary information of the two modalities. Therefore, when fusing different modalities [8,25,26,27], researchers must design complex network structures and loss functions. At the same time, the number of existing RGB-D datasets is limited, and some datasets have low annotation quality, which results in the inability to meet the training requirements of models. Therefore, in the context of weakly supervised learning, it remains a challenge to utilize multi-modal and multi-dimensional data to address the issue of missing salient regions in complex scenarios, especially the detection of filamentary structures.

To address the aforementioned issue, this approach explores the complementarity of cross-modal weak labels by combining the semantic information of text labels with the visual information of point labels, thereby enriching the dimensionality of the supervisory information. Meanwhile, it enhances the utilization of the advantageous properties of RGB and depth images, providing the saliency detection model with a more comprehensive understanding of the scene. Specifically, a cross-modal pseudo-label generation network (CPGN) is designed, which leverages the semantic information contained in text labels as prior knowledge and integrates the powerful image–text matching capability of CLIP to generate initial pseudo-labels. The annotations and positional information provided by point labels are then employed to hierarchically refine the initial pseudo-labels, providing high-quality pseudo-labels aligned with human perception for salient object detection. To fully leverage the contextual and geometric features, an asymmetric salient-region prediction network (ASPN) is proposed to capture the advantageous features of different images. Various attention mechanisms are embedded to explore the correlation between RGB and depth features, enabling efficient coupling of multi-dimensional information. Additionally, an edge constraint module (ECM) is integrated into the ASPN to sharpen the edges of the saliency results. By efficiently combining multi-modal and multi-dimensional information, the proposed approach achieves salient object detection in complex backgrounds under a weakly supervised learning framework and demonstrates outstanding performance in filamentary structure detection.

The main contributions of this study are as follows:

1. We propose a cross-modal weakly supervised method for the RGB-D SOD task, in which the text and point annotations are used to provide rich semantic information and position information, respectively. By fully integrating the advantages of weak labels from different modalities, the method effectively enhances the detection performance of salient objects in complex scenes, especially in highlighting filamentary structures.
2. A cross-modal pseudo-label generation network (CPGN) is designed to provide stronger supervision for model training, which first leverages the multimodal alignment capability of CLIP to activate semantic priors from text labels, obtaining coarse annotations, and then point labels progressively enhance the richness and accuracy of these annotations to generate high-quality pseudo-labels.
3. An asymmetric salient-region prediction network (ASPN) is proposed to extract and integrate the contextual information of RGB images and the geometric information of depth images. In ASPN, an edge constraint module is introduced to sharpen the edges of salient objects. Meanwhile, we construct a cross-modal weakly supervised saliency detection dataset (CWS) for the weakly supervised RGB-D SOD task.

2. Related Work

2.1. Weakly Supervised Salient Object Detection

Salient object detection (SOD) has garnered increasing attention, and although current methods have achieved excellent results, they rely on time-consuming and labor-intensive pixel-level manual annotations. As a result, weakly supervised strategies have become mainstream. These strategies use image-level labels [18,28,29], scribbles [30,31,32], bounding boxes [33], and points [34] to supervise model training. For example, Li et al. [28] and Yu et al. [31] perform self-training by using predicted saliency maps as part of the pseudo-labels. Zeng et al. [35] employs multi-task learning, including multi-label classification and caption generation, to assist in generating pseudo-labels. Liu et al. [33] iteratively refine pseudo-labels by learning a multi-task mapping refinement network with salient bounding boxes. Point-based methods, such as the PSOD proposed by Piao et al. [34], integrate point labels and edges to refine pseudo-labels.

Our research focuses on leveraging the powerful capabilities of CLIP, using it to convert image–text similarity into pixel-text matching for precise segmentation, while also utilizing the strong foreground–background differentiation ability of point labels to create complete pseudo-labels.

2.2. Weakly Supervised RGB-D Salient Object Detection

With the advancement of depth imaging technology, depth maps have been introduced as a crucial 3D cue in the SOD task. Depth maps enhance the ability of SOD models to detect salient objects in complex scenes, leading to the rapid development of RGB-D salient object detection. Meanwhile, in order to reduce the cost of pixel-level annotations, weakly supervised RGB-D SOD solutions need to be further explored. Xu et al. [17] addressed the sparsity issue of scribbles through a secondary labeling method. Li et al. [15] modeled the pixels in RGB images to supplement the missing pixel-level annotations in scribbles.

The method leverages the attribute advantages of cross-modal weak labels. Meanwhile, it utilizes the Swin-Transformer and a CNN network to create a dual-branch architecture for feature extraction, and it proposes an edge constraint module to refine the edge details of the salient object.

2.3. Contrastive Language–Image Pre-Training

CLIP (Contrastive Language–Image Pre-Training) is a visual–language pre-training model introduced by OpenAI [36,37]. Its core idea is to establish a correlation between images and text, enabling better mutual understanding through measuring the similarity between images and text. CLIP is implemented by combining and matching a large number of images with their corresponding textual information to build a large-scale dataset. CLIP can pair input images with text and compute similarity scores between them. In practice, for an input image or text, the model outputs similar images or text, facilitating more intelligent information processing.

Most existing weakly supervised SOD methods use CAMs as pseudo-labels. However, standard CAMs are often noisy, which can significantly degrade model performance. This method uses CLIP to generate class activation maps (CAMs), delving into the relationship between text and objects in images to create a robust foundation for obtaining results that meet our requirements.

3. Proposed Method

3.1. Overview

We solve some existing issues in the weakly supervised salient object detection (SOD) task for RGB-D images, e.g., the insufficient supervision ability of weak labels, the incomplete prediction of object regions and the failure to detect filamentary structures. The overall framework we proposed is shown in Figure 1. Among them, it can be divided into a cross-modal pseudo-label generation network (CPGN) and an asymmetric salient-region prediction network (ASPN). Firstly, we designed the CPGN, which takes as input $[eqn]$ , where i is the index of dataset samples and $[eqn]$ represents the paired RGB image ( $[eqn]$ ), text labels ( $[eqn]$ ), and point labels ( $[eqn]$ ). The network outputs high-quality pseudo-labels ( $[eqn]$ ) to guide the ASPN to predict results with complete regions, especially for the detection of filamentary structures. Secondly, in view of the attribute differences between RGB images and depth images, the ASPN employs an asymmetric encoder structure based on the Swin-Transformer and a CNN to progressively extract the features of RGB-D images. The structure is designed to enable the ASPN to fully extract contextual information from RGB images while making the ASPN more sensitive to the spatial structure information in depth maps. Finally, we designed an edge constraint module (ECM), which utilizes shallow features and deep features to generate edge features to sharpen the edges of the salient objects by the ASPN. In conclusion, this method capitalizes on the synergy of the two networks to tackle the issues of insufficient supervision of weak labels, incomplete salient regions, and the absence of filamentous structures in current weakly supervised SOD tasks for RGB-D images.

3.2. Cross-Modal Pseudo-Label Generation Network

3.2.1. CLIP-Based Grad-CAM

Class activation mapping (CAM) has been widely used in weakly supervised SOD tasks. However, the pseudo-labels generated only by using CAM often contain noise, and the separation between the foreground and the background is not satisfactory, which makes it difficult to directly use them to guide the model training. To achieve better results, we divide the text label activation task into two stages: pseudo-label generation and refinement.

The text labels are obtained by calibrating the Caps dataset in [38]. The text labels consist of 122 foreground object descriptions (such as cat, dog, and people riding on bicycles) and 23 common background classes (such as sky, ground, and beach). As shown in Figure 1, the RGB image and the text labels (foreground label and background set) of the image are fed into the CPGN simultaneously.

We need to match the information from the text labels with the object regions in the image, and the CLIP model is well suited for this task due to its strong zero-shot capability. Therefore, our text labels include both foreground information of the object and a common background set. We use the softmax function to ensure mutual exclusivity between different classes, thereby suppressing the background and achieving good segmentation performance in the presentation of the matching results. Specifically, the scores after applying softmax are calculated as follows:

$$[eqn]$$

where $[eqn]$ represents the score for the $[eqn]$ object text, C is the total number of object texts, and $[eqn]$ is the cosine similarity score for the $[eqn]$ object.

We use Grad-CAM as the visualization method for the matching results. Grad-CAM is a visualization tool for deep learning models that allows us to intuitively understand the regions of the image that CLIP focuses on when given an input object text. It then visualizes this result and generates CAMs $[eqn]$ . The specific formula is as follows:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ is the weight of the $[eqn]$ channel in the last feature layer $[eqn]$ of the CLIP model for the $[eqn]$ class, Z is the total number of pixels, $[eqn]$ is the pixel coordinate $[eqn]$ in the $[eqn]$ channel of the last feature layer $[eqn]$ of the CLIP model, and $[eqn]$ is the gradient information of class c backpropagated through the $[eqn]$ channel of the last feature layer $[eqn]$ in the CLIP model.

Finally, the result refinement is achieved by combining the CAMs $[eqn]$ of each class c with the attention weights $[eqn]$ from the CLIP image encoder. Specifically, to make $[eqn]$ pay more attention to the object, we threshold the CAMs to obtain masks and then cover the regions related to category c in the masks with boxes to obtain class masks $[eqn]$ . After that, we perform sinkhorn normalization and transpose sum processing on $[eqn]$ , respectively, to enhance the object weights. By multiplying $[eqn]$ by $[eqn]$ , we obtain the class attention map. We use the semantic similarity among pixels in the class attention map to refine the CAM, thereby covering more incomplete areas of the object in $[eqn]$ . The generation of our initial pseudo-labels $[eqn]$ is shown in the following equation:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ is the class mask, $[eqn]$ is the sinkhorn normalization, and $[eqn]$ denotes matrix vectorization. Since the class activation map is essentially the confidence of each pixel in the image regarding class c, a hyperparameter $[eqn]$ (i.e., the filtering parameter) is set to ignore pixels with low confidence, completing the text label activation process, as represented by:

$$[eqn]$$

where $[eqn]$ is the initial pseudo-label and $[eqn]$ represents the coordinates of a pixel in the class activation map $[eqn]$ .

3.2.2. Point Label Activation and Pseudo-Label Improvement

Due to the semantic ambiguity of text labels, the masks generated from them contain a lot of noise, which causes the model to learn incorrect information. To address this issue, we introduce point labels into the pseudo-label generation process. The method leverages the advantages of multiple modalities, refining the initial pseudo-labels from both the foreground and background to obtain the final pseudo-label with sufficient supervisory information. We fully leverage the advantages of point labels to improve the quality of pseudo-labels. We divide point labels into foreground points $[eqn]$ and background points $[eqn]$ . The flood fill algorithm is a method used to fill a connected region with a specified color or value, starting from a given seed point. Using the flood fill algorithm, we fill the edge $[eqn]$ detected by the edge detector [39] with the foreground point labels to obtain the foreground activation map $[eqn]$ . In this method, the foreground point is set as the seed point, with a radius of one-fifth of the given image size, and the Low Difference and High Difference are set to 20 and 50, respectively. The specific operation is defined as:

$$[eqn]$$

where $[eqn]$ represents the flood fill algorithm. Then, we add the foreground activation map pixel-wise with the initial pseudo-labels to complete the foreground refinement process, followed by using the background points as seed points to flood fill the entire image to remove non-object noise and obtain the final pseudo-label $[eqn]$ . The specific operation is defined as:

$$[eqn]$$

We use $[eqn]$ to guide the training of the ASPN.

3.3. Asymmetric Salient-Region Prediction Network

3.3.1. Asymmetric Encoder

Weakly supervised SOD tasks for RGB-D images typically employ encoder structures based on ResNet, VGG, or Transformer backbones. These fundamental encoders are commonly pre-trained on RGB images to initialize the model. However, recent studies have found attribute differences between RGB and depth images. The extraction of depth image features using these encoders may introduce interference in the network. Depth images contain rich spatial positional information, and CNN-based encoders are more sensitive to spatial structures. Therefore, we use CNN-based encoders to extract features from depth images. Moreover, encoders based on the Swin-Transformer are better equipped to capture contextual information. Therefore, we use the Swin-Transformer encoder to extract features from RGB images.

We extract hierarchical features from the RGB features. Specifically, the Swin-Transformer first divides the input RGB image $[eqn]$ into non-overlapping $[eqn]$ patches, which are then flattened and represented as:

$$[eqn]$$

Here, $[eqn]$ refers to dividing the RGB image into $[eqn]$ patches, and $[eqn]$ is the operation that transforms each patch into a higher-dimensional embedding feature space. The Swin-Transformer updates features layer by layer through hierarchical feature extraction. Each layer $[eqn]$ consists of two core operations:

• (a)Performing multi-head self-attention operations on each patch:

$$[eqn]$$

Here, $[eqn]$ refers to the window-based multi-head self-attention operation, $[eqn]$ is the input feature at layer $[eqn]$ , and $[eqn]$ is the feature after the window-based multi-head self-attention operation.

• (b)To better capture the contextual relationships, multi-head self-attention operations are performed between windows:

$$[eqn]$$

Here, $[eqn]$ refers to the cross-window multi-head self-attention operation. By stacking multiple layers of such operations, contextual features are gradually extracted. The final output is the contextual features $[eqn]$ after hierarchical feature extraction. Depth features are typically used to supplement the spatial positional information of the RGB features. Networks trained from scratch tend to achieve better performance in tasks related to depth maps. Therefore, we adopted a CNN-based backbone to extract depth features in the simplest way possible. As shown in Figure 2, the depth map is converted into a higher-dimensional embedding feature space through convolution layers with different receptive fields. Then, the learned depth features are transformed into multi-scale features with three different resolutions via three identical CNN-based blocks.

Finally, for the fourth stage, we use a 1 × 1 convolution layer to obtain the final output. In this way, we acquire the depth features, represented as $[eqn]$ . We use a serial structure in the depth branch, which is capable of extracting deeper information and helps reduce structural distortion during the feature extraction process of depth maps.

3.3.2. Mixed Attention Module (MAM)

As shown in Figure 3, we feed the RGB features $[eqn]$ and the depth features $[eqn]$ extracted from the asymmetric encoder into the Transformer basic blocks. The cross-modal information is output by mixing their respective attention weights in the mixed self-attention layer. Specifically, before the mixed self-attention layer, we extract the query $[eqn]$ , key $[eqn]$ , and value $[eqn]$ from both $[eqn]$ and $[eqn]$ . as $[eqn]$ and $[eqn]$ , respectively. We use $[eqn]$ along with $[eqn]$ and $[eqn]$ as the head in the RGB feature stream, while $[eqn]$ is used together with $[eqn]$ and $[eqn]$ as the head in the depth feature stream. This process is defined as:

$$[eqn]$$ $$[eqn]$$

After mixing the two through mixed self-attention, the output information is summed to obtain the attention weights:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ and $[eqn]$ are linear projections. After obtaining the feature maps $[eqn]$ $[eqn]$ mixed through the Transformer basic blocks, we further utilize spatial attention and channel attention to better fuse the features of different modalities. This helps generate the final multi-scale features at each scale, $[eqn]$ , with the following operations:

$$[eqn]$$

Here, $[eqn]$ and $[eqn]$ represent channel attention and spatial attention, respectively.

3.3.3. Edge Constraint Module (ECM)

Complete edges can enhance the performance of saliency detection models. Therefore, we propose an edge constraint module (ECM), which progressively refines the output of edge features. Low-level features contain rich texture information, while high-level features contain rich semantic information. The ECM combines multi-scale features to sharpen the edges of the salient object. Figure 4 illustrates the schematic diagram of our edge constraint module. The module is divided into two parts: the generation of edge features $[eqn]$ and the generation of the learning target for edge features $[eqn]$ . Specifically, $[eqn]$ and $[eqn]$ are fed into two separate 1 × 1 convolution layers and upsampled. Then, the obtained features are concatenated to generate a new feature. Next, the feature is fed into two consecutive $[eqn]$ and $[eqn]$ layers, and the edge features are output through a sigmoid function for learning the object boundaries, which can be expressed as follows:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ refers to a 1 × 1 convolution layer followed by an upsampling operation, $[eqn]$ refers to a 3 × 3 convolution layer followed by an upsampling operation, and $[eqn]$ denotes the sigmoid activation function. In the generation phase of $[eqn]$ , we obtain the edge map of the salient object in the final output of each iteration. This edge map guides the ECM in learning the edges of the prediction results. This process can be expressed as:

$$[eqn]$$

where $[eqn]$ represents the prediction results of the ASPN in each iteration and $[eqn]$ denotes the edge extraction using the sobel operator.

3.3.4. Progressive Decoder

To obtain the prediction result $[eqn]$ , we use a progressive decoder. We first feed $[eqn]$ and $[eqn]$ into the TB layer. The features from the TB layer are then summed element-wise to generate enhanced features. These refined features are subsequently split into two branches: one branch performs upsampling before being fed back into the TB layer, while the other branch combines with $[eqn]$ in the same manner as the previous step. After incorporating $[eqn]$ into the decoding process, the edge features are added by element-wise multiplication during the decoding process, as shown in Figure 5. The decoder equation is as follows:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ represents the intermediate features, $[eqn]$ is a 3 × 3 convolution layer followed by BN and ReLU layers, TB consists of three $[eqn]$ blocks stacked together, $[eqn]$ denotes the $[eqn]$ level feature, $[eqn]$ represents the upsampling operation, and Sig $[eqn]$ is the sigmoid activation function.

Finally, a 1 × 1 convolution layer is used as the projection head for the salient features, outputting the salient object image $[eqn]$ , expressed as:

$$[eqn]$$

Here, $[eqn]$ is the predicted result and $[eqn]$ represents the convolution operation with a 1 × 1 convolution kernel. We take $[eqn]$ as the final saliency map.

3.4. Hybrid Loss Function

Since the pseudo-labels generated by the CPGN provide rich supervisory information and suppress the noise introduced by weak labels, we employ a hybrid loss function that includes pixel-level contrast loss $[eqn]$ , enhanced hybrid loss $[eqn]$ , and edge loss $[eqn]$ to train the ASPN. $[eqn]$ treats the pseudo-labels and the pixels with the same predictions as positive sample pairs, while the rest are considered negative sample pairs. It trains the ASPN by increasing the similarity of positive pairs and decreasing the similarity of negative pairs. The total loss function is given by the following expression:

$$[eqn]$$

We fix the value of $[eqn]$ to $[eqn]$ in the order of 0.9, 0.1, 1, in all our experiments, where $[eqn]$ is defined as:

$$[eqn]$$

where $[eqn]$ is the output before the projection head of the final layer of the progressive decoder. $[eqn]$ is defined as:

$$[eqn]$$

$[eqn]$ is proposed in [40], where $[eqn]$ is the output after the projection head of the final layer of the progressive decoder. $[eqn]$ optimizes the predicted results from both the edge and region perspectives. We fix the value of $[eqn]$ to $[eqn]$ in the order of 1, 0.5, 0.5, in all our experiments.

Since the foreground in the edge image contains only a small number of pixels, it may be dominated by the abundance of background pixels. Therefore, we use $[eqn]$ and $[eqn]$ to optimize the edge features. $[eqn]$ is robust to the imbalance between positive and negative samples. It focuses more on the overlapping part of the two sets in the predicted results and pseudo-labels, rather than simply considering the number of elements in each set. This allows $[eqn]$ to better balance the contributions of foreground and background pixels when the pixel distribution is uneven (e.g., when foreground pixels are far fewer than background pixels). $[eqn]$ is adjusted by weighting to balance, and $[eqn]$ is calculated based on the ratio of foreground pixels to background pixels in $[eqn]$ . We use a combination of $[eqn]$ and $[eqn]$ to optimize the edge features. $[eqn]$ is defined as:

$$[eqn]$$

4. Experiment and Result Analysis

To maximize the effectiveness of our proposed model, we followed the traditional training setup and performed re-annotation. We selected 1485 images from the NLPR [41] dataset and 700 images from the NJU2K [42] dataset to construct a new cross-modal dataset, CWS. Furthermore, we tested our model on the public datasets LFSD [43], DES [44], SIP [45], DUT-RGBD [46], NLPR [41], NJU2K [42], and STERE [47].

4.1. Evaluation Metrics

We evaluate the performance of the models on four golden evaluation metrics, i.e., S-measure, F-measure, E-measure, and mean absolute error (MAE).

4.2. Implementation Details

During the construction of the training set, we utilized the CLIP pre-trained ViT-B-16 [48] model for CAM generation. The feature map used to generate the CAM was extracted from the final feature map before the last self-attention layer of ViT. Our model was trained on a single NVIDIA GTX 3090 GPU with 24 GB of memory. The input RGB and depth images were resized to 256 × 256. To avoid overfitting during training, all images were augmented with data augmentation techniques such as random flipping, random cropping, random rotation, and color enhancement. With the backbone network for the RGB stream employed by the Swin-Transformer model pre-trained on ImageNet, we used the AdamW optimizer with $[eqn]$ , $[eqn]$ , and a weight decay of $[eqn]$ to optimize our model. For the proposed PCL, we set the temperature hyperparameter $[eqn]$ . The initial learning rate was set to $[eqn]$ , and we applied a polynomial decay strategy with the formula: $[eqn]$ where $[eqn]$ . The model was trained for 300 epochs with a batch size of 6.

4.3. Performance Comparison with the State of the Art

We compare our proposed method with both weakly supervised and fully supervised models, including MIRV [15], SSSD [32], DENet [17], and SCWS [31], which are based on scribble supervision; PSOD [34], which is based on point supervision; and MSOD [49], JSM [38], and MSW [35], which are based on multiple weak labels. For pixel-level supervision, we compare CPNet [50], HFIL [51], TPCL [28], CATNet [11], and C2DFNet [52]. To ensure a fair comparison, we either used the salient object maps provided by the authors or ran the models released by the authors to predict on the RGB-D datasets. For models designed for RGB images, we only input the RGB images from the RGB-D datasets for comparison.

4.3.1. Quantitative Evaluation

Our method was compared against state-of-the-art single-label weakly supervised, multiple-label weakly supervised, and fully supervised SOD models for RGB-D images. We evaluated the performance across all datasets listed in Table 1 using the F-measure ( $[eqn]$ ), S-measure ( $[eqn]$ ), E-measure ( $[eqn]$ ), and MAE ( $[eqn]$ ). Among the compared methods, MSOD, JSM, and MSW employ two or more weak labels as supervision sources, similar to our approach, while MIRV, Denet, SCWS, and POSD rely on a single supervision source. CPNet, HFIL, TPCL, CATNet, and C2DFNet, on the other hand, are fully supervised models using pixel-level labels.

As shown by the results, our model achieves state-of-the-art performance on datasets with rich filamentary structures (SIP) and those with complex scenes (DUT-RGBD). Although our method lags behind on the DES dataset, further investigation revealed that this is due to the dataset’s limited size (only 135 images) and high homogeneity, with simple structures that diminish the advantages of our method in handling complex structural information.

4.3.2. Qualitative Evaluation

To further demonstrate the effectiveness of our method, Figure 6 shows a comparison with state-of-the-art methods in various scenes. It can be seen that compared to single weakly supervised models (such as MIRV, SSSD, DENet, SCWS, and PSOD), our model exhibits a stronger ability to localize salient objects and segment complete regions. When compared to multiple weakly supervised models (such as MSOD, JSM, and MSW), our model produces smoother salient object edges. Notably, our model shows an overwhelming advantage in representing fine filamentary structures (e.g., rows 1, 2, 4, 6, and 7), even demonstrating competitive performance when compared to fully supervised tasks. For occluded objects (e.g., row 3), our model is also able to accurately identify and segment salient objects, exhibiting a superior understanding of complex spatial structures—surpassing even fully supervised models like CATNet and C2DFNet, which aligns with the original intent of the SOD task for RGB-D images. Furthermore, our model excels in recognizing multiple objects (e.g., row 5), where it accurately segments the structures of multiple targets without introducing background noise. Overall, our model surpasses the limitations of weak labels in object completeness recognition and remains competitive even among fully supervised models.

4.3.3. Complexity Comparisons

We calculate the FLOPs and parameters for weakly supervised methods based on RGB-D images, and the results are shown in Table 2 below. The FLOPs and parameters of our model are relatively high. However, compared with other methods, our method achieves higher detection performance for filamentary structures. Additionally, it is worth noting that the primary computational cost of our model lies in the Swin-Transformer backbone, which accounts for 87 M parameters.

4.4. Ablation Studies

We conducted various ablation studies to validate the effectiveness of each key component of the model on the RGB-D dataset. This includes an examination of the effectiveness of weakly supervised pseudo-labels in relation to the model architecture. We quantitatively present the benefits of each component in a table. Our model achieved the best performance when all components were utilized.

We selected four datasets as the test datasets for the ablation experiments. The NJU2K and NLPR are two representative datasets for the SOD task on RGB-D images. The depth maps have better quality in DUT-RGBD, while DES represents a case where our method performs relatively poorly. The performance on these four datasets is sufficient to represent the effectiveness of our method.

4.4.1. Effectiveness of Cross-Modal Weak Labels

As shown in Table 3, we disassembled our cross-modal weak labels to train the model with the pseudo-labels formed by the text labels and the point labels, respectively. We found that the performance of the model was significantly reduced when only single weak labels were used, which was sufficient to prove that a single source was not enough to meet the requirements of SOD tasks for RGB-D images. Figure 7 shows the comparison of prediction results using cross-modal weak labels and using single-modal weak labels. It can be seen from the figure that when we only use text labels, the background is noisy, and when only point labels are used, the predicted results are incomplete. These are consistent with the defects of single-modal weak labels, respectively, while the prediction results obtained by the combination of cross-modal weak labels proposed by us have abundant filamentous structures and a clean background.

4.4.2. Effectiveness of Asymmetric Encoder and ECM

We added two sets of comparative experiments using symmetric encoders to verify the effectiveness of the asymmetric architecture. Moreover, we validated the effectiveness of the ECM in each set of experiments. The qualitative evaluation is shown in Figure 8, where it can be clearly observed that our asymmetric structure better captures the filamentary structure of the salient object. The ECM also sharpens the edges of the salient region. The experimental results are shown in Table 4. From the quantitative results, we observed that the asymmetric structures based on the Swin-Transformer and CNN encoders achieved better performance in the experiments without the ECM. After incorporating the ECM into the network, the performance improved significantly, which strongly demonstrates the effectiveness of the proposed asymmetric encoder combined with the ECM.

5. Discussion

Weakly supervised salient object detection holds great potential for applications in medical imaging, defect detection, and autonomous driving. The reduced annotation cost and accurate localization of salient objects provide a solid technical foundation for practical applications. The introduction of depth information can enhance the ability of the predictive model to recognize objects in low-contrast or complex background scenarios. In this study, we propose a cross-modal weak supervision framework that integrates text and point labels to provide both semantic and pixel-level guidance. This approach helps the model to learn a more complete object contour, especially in capturing filamentary structures. It also introduces depth information as guidance to overcome adverse environmental factors and to clearly delineate the geometric information of the object. These advantages contribute to the competitiveness of our approach in real-world applications. Our method significantly improves the detection of complete salient objects by aggregating multi-modal and multi-dimensional information. In future work, how to use a small amount of supervision information to complete the detection of complex structures is expected to become a development direction for weakly supervised salient object detection, and this is also the main issue we need to explore.

6. Conclusions

We propose a high-performance cross-modal weakly supervised salient object detection (SOD) framework for RGB-D images that consists of two main components: pseudo-label generation and salient object detection. Since the issue of weak labels is insufficient supervision, we propose the CPGN, which can overcome the limitations of different weak labels by combining the advantages of text labels and point labels, thus generating high-quality pseudo-labels to train the ASPN and providing sufficient and accurate saliency information. Additionally, considering the attribute differences between RGB and depth images, we utilize an asymmetric encoder structure, where RGB-D image features are extracted using a Swin-Transformer and CNN-based encoders, respectively. To further enhance the model’s perception of edge information, we propose the ECM, which constrains the edges of salient objects through multi-scale feature integration. Based on experimental results from seven datasets, our proposed method outperforms other weakly supervised approaches and demonstrates competitive performance even when compared to fully supervised methods in fine filamentary structure segmentation.