Novel cross-dimensional coarse-fine-grained complementary network for image-text matching
Meizhen Liu, Anis Salwa Mohd Khairuddin, Khairunnisa Hasikin, Weitong Liu
TL;DR
This paper introduces a new network for better image-text matching by combining fine and coarse-grained analysis across dimensions.
Contribution
The novel CDGCN network addresses cross-dimensional semantic consistency in image-text matching.
Findings
CDGCN improves image-text matching by aligning fine-grained image regions with text words.
The CGDSA module enhances global semantic consistency by aggregating features across dimensions.
CDGCN outperforms existing methods by 7.7–16% on Flickr30K and MS-COCO datasets.
Abstract
The fundamental aspects of multimodal applications such as image-text matching, and cross-modal heterogeneity gap between images and texts have always been challenging and complex. Researchers strive to overcome the challenges by proposing numerous significant efforts directed toward narrowing the semantic gap between visual and textual modalities. However, existing methods are usually limited to computing the similarity between images (image regions) and text (text words), ignoring the semantic consistency between fine-grained matching of word regions and coarse-grained overall matching of image and text. Additionally, these methods often ignore the semantic differences across different feature dimensions. Such limitations may result in an overemphasis on specific details at the expense of holistic understanding during image-text matching. To tackle this challenge, this article…
Genes, proteins, chemicals, diseases, species, mutations and cell lines named across the full text — each resolved to its canonical identifier and authoritative record.
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques
Introduction
As the volume of digital information increases rapidly, research on images and texts has become a prominent focus within the multimodal applications (Yadav & Vishwakarma, 2024; Zhang et al., 2024a), attracting more scholars to work on cross-modal studies involving vision and language. This includes key areas such as cross-modal retrieval (Hu et al., 2023; Ma et al., 2024; Sun et al., 2023; Zeng et al., 2022), image captioning (Li et al., 2024a; Ma et al., 2023; Yan et al., 2021; Yu et al., 2019) and visual question answering (Dong et al., 2021; Li et al., 2022b; Liu, Li & Lin, 2023; Qian et al., 2024) multimodal tasks. These works focused on the core challenge of matching images and text, which is a fundamental task in cross-modal retrieval. It involves finding the most pertinent sentences (or images) in a dataset based on a query image (or sentence). The image-text matching task has found widespread application in various fields, including information retrieval, social media analysis, and intelligent recommendation systems (Chhabra & Vishwakarma, 2023; Hu & Huang, 2021; Moro, Salvatori & Frisoni, 2023).
In recent times, the image-text matching task has remained a focal point of extensive research, owing to the considerable heterogeneity gap between visual and textual data. Despite proposing numerous attempts to tackle this problem with various methods (Jeong et al., 2024; Wang et al., 2023a; Wu et al., 2024; Yao et al., 2024; Zhao et al., 2023), several challenges still persist. Firstly, existing methods often rely too heavily on a single type of pooling method, failing to effectively explore and utilize modal dimensional information, which lead to the loss of some essential discriminative information. Secondly, in pursuit of high accuracy, existing methods overly focus on capturing detailed information in images and texts, resulting in misalignment between global and local information matching of the image and text.
To address these challenges, our work introduces a cross-dimensional coarse-fine-grained complementary network for image-text matching. This approach provides a novel perspective on image-text alignment and serves as a valuable reference for other pertinent research endeavors in the realm of multimodal learning (such as video-text matching and audio-text matching). Additionally, the potential real-world applications of this work encompasses product recommendations, image search engines and social media content analysis. Thus, by improving the accuracy of information retrieval, this method allows users to access relevant content with improved accuracy and efficiency.
Specifically, existing methods can generally be categorized into two main groups: coarse-grained global matching and fine-grained local matching. The coarse-grained global method indicated in Fig. 1A aligns the entire image-text pair by mapping the visual and textual data to a shared space, unifying the semantic representation of the image and sentence. The advantages of coarse-grained global matching methods such as average pooling (Li et al., 2019), maximum pooling, and Generalized Pooling Operator (GPO) (Chen et al., 2021) are flexibility, simplicity, and short runtime. However, those pooling methods ignore that different dimensions of a feature could carry distinct interpretations, and the significance of various dimensions with the same feature might differ, hence, may further reduce discriminative information. For instance, Zhu et al. (2023) pointed out in the external spatial attention aggregation (ESA) model that existing pooling methods often use vector weighting to obtain global information from local features, unable to focus on dimensions at a fine-grained level, which may discard crucial detail information, resulting in poor retrieval performance. Wu et al. (2023) discovered that different dimensions of a feature may have different meanings, and the importance of different dimensions within the same feature should vary when aggregating features. Zhang et al. (2023) proved that dimensions are not independent and better feature extraction can be achieved by jointly representing latent semantics through intrinsic dependencies between dimensions. These studies suggest that existing coarse-grained pooling methods may unintentionally lose important discriminative information. Meanwhile, the fine-grained local method depicted in Fig. 1B emphasizes aligning relevant image-text pairs by utilizing the correspondence between local information (i.e., visual regions and text words). This method has high matching accuracy and is currently the most popular image-text matching method. Still, if the fine-grained local matching method is used alone, it will lead to excessive attention to the relationship between regions and texts while ignoring the semantic consistency problem between global and local information.
The general frameworks of image text matching.(A) The coarse-grained global method by aligning the entire image text pair, (B) the fine-grained local method by aligning relevant image text pairs.
To address the shortcomings of relying solely on either global matching or local matching methods, mining richer cross-dimensional features from images and texts, this work proposes the CDGCN method as shown in Fig. 2, which uses fine-grained cross-dimensional semantic dependency branch to obtain the local similarity of region-words and then semantic matching, and coarse-grained cross-dimensional semantic aggregation branch to receive the global semantic matching of images and texts, the global semantic matching is used as a supplement to the local semantic alignment to achieve global and local semantic information consistency. This approach distinguishes information elements at the fine-grained similarity dimension level and aggregates features at the coarse-grained dimension level, thus increasing the accuracy of cross-modal retrieval and significantly improving the application potential of multimodal learning.
The proposed CDGCN logic framework of image-text matching.
The primary contributions of this article are outlined as follows:
- (1)A Cross-Dimensional Coarse-Fine-Grained complementary network (CDGCN) is proposed, integrating coarse-grained cross-dimensional global semantic matching across dimensions with fine-grained local region word cross-dimensional semantic alignment.
- (2)A plug-and-play Coarse-Grained Cross-Dimensional Semantic Aggregation Strategy (CGDSA) is introduced. This strategy aggregates local features into global features by employing cross-dimensional vector aggregation weights across different dimensions and similar dimension.
- (3)Extensive experiments and comparisons were performed on cross-validated datasets including Flick30k and MSCOCO, to demonstrate the advantages of the proposed CDGCN in cross-modal image-text matching.
The structure of the paper is as follows: “Related Work” provides an overview of pertinent research in cross-modal image-text retrieval. “Methodology” elaborates on the CDGCN method, detailing its components and functionality. “Results and Analysis” presents experimental findings, encompassing comparative analyses, ablation studies, hyper-parameter investigations, visualization and analysis. Finally, “Conclusion” offers a comprehensive summary and future scope of our work.
Related work
Regarding the progressive work on image-text matching and the extraordinary performance exhibited by recent vision-language pre-trained models, this subsection focuses on three aspects: coarse-grained global matching, fine-grained local matching, and vision-language pre-trained models matching. Finally, it compares studies similar to our work.
Coarse-grained global matching
The process involves independently encoding images and texts into a unified embedding space using two separate branches, obtaining coarse-grained global representations through specific aggregators, and performing similarity-matching calculations. Existing work typically utilizes GCN (Li et al., 2019, 2024b; Wang et al., 2020b) or transformer (Wang et al., 2020a; Wen, Gu & Cheng, 2020; Wu et al., 2019) to enhance the contextual semantics of local features. For instance, Li et al. (2019) introduced a simple semantic reasoning network (VSRN) to learn local image features with key scene concepts. Wang et al. (2020a) introduced a consensus-aware module (CVSE) that incorporates common sense knowledge into local features of two patterns. An important research direction in coarse-grained global matching focuses on the work of clustering pooling strategies (Chen et al., 2021; Li et al., 2019; Zeng et al., 2021; Zhu et al., 2023). For instance, Li et al. (2019) leveraged gates and memory mechanisms to conduct global semantic inference on features, selectively identify discriminative information, and derive representations of the entire scene. Chen et al. (2021) introduced the optimal pooling method (GPO) that automatically adapts to different characteristics without manual tuning, down sampling intra-modal features into global features. Zhu et al. (2023) introduced a flexible ESA strategy, facilitating element-level fusion of image and text features in specific spatial dimensions. Kim, Kim & Kwak (2023) proposed the idea of utilizing a set that transforms an image or text sample into a collection of unique embedding vectors, designed to encompass diverse semantic elements of the samples. However, previous works still follow the existing paradigm, this work considers that the same feature may mean different things in different dimensions and have different importance in different dimensions. Aggregating vector weights in both different and same dimensions can maximize the acquisition of potentially rich dimensional semantics.
Fine-grained local matching
The goal is to identify all potential alignments between textual words and image regions, thereby inferring the comprehensive correlation between the image and text, better performance can usually be achieved (Chen et al., 2020; Diao et al., 2021; Fu et al., 2023; Lee et al., 2018; Liu et al., 2020; Wang et al., 2023b; Zhang et al., 2022a, 2022b, 2024b). Most work in this field is based on attention mechanisms. Lee et al. (2018) introduced the SCAN technique as a prominent approach for dynamically targeting the common semantics across different modalities in response to each query content. On this basis, many variants were proposed. The focus of a sequence of works is to devise precise and complex matches (Chen et al., 2020; Diao et al., 2021; Lee et al., 2018; Liu et al., 2020; Zhang et al., 2022b). For instance, Chen et al. (2020) introduced Recurrent Cross Attention, which iteratively extracts and enhances shared semantics across multiple levels. Zhang et al. (2022b) proposed measuring the degree of similarity and dissimilarity through two matches (NAAF). Pan, Wu & Zhang (2023) proposed a new encoding strategy (CHAN) to find the most relevant region-word pairs to address redundant alignment and cross-attention mechanisms. Diao et al. (2021) first used the vectorised representation method (SGRAF), which is able to calculate the similarity and inference between two modal at a finer granularity. Simultaneously, various techniques concentrates on leveraging external and other information to construct relationships within the same modal and between different modal of sentences and images (Fu et al., 2023; Zhang et al., 2022a). For example, Zhang et al. (2022a) proposed the region position (CMCAN) to reflect the surrounding visual scene, while Fu et al. (2023) emphasized and enriched the interaction dynamics between visual and texts through semantic relationships between instances. Nevertheless, the emphasis of these methods remains on purely calculating the localized similarity between image segments and textual terms to gather the overall image-text correlation, which can easily lead to excessive attention to the the connections between image areas and textual terms, thereby causing a discrepancy between the global and local information. our proposed method focuses on using coarse-grained global matching as a complement to fine-grained image segments and textual terms matching to solve the problem of semantic inconsistency between global and local information, and adopting cross-dimensional semantic dependency in the process of fine-grained image segments and textual terms matching to mine richer image and text features.
Vision-language pre-trained model matching
Early Vision-Language Pre-trained (VLP) models followed a detector-based roadmap (Chen et al., 2019). The CLIP model (Radford et al., 2021), ALIGN (Jia et al., 2021), and unicoder-VL (Li et al., 2020) respectively utilized 400 million image-text pairs, 1.8 billion noisy image-text pairs, and 3.8 million web image-text data for contrastive training. These models achieved efficient cross-modal retrieval and cross-modal understanding and generation tasks by integrating multimodal information. Although this approach demonstrated strong performance, it also posed challenges of expensive computation and unstable detector training. Subsequently, methods based on Vision Transformers (ViTs) adopted a pure Transformer architecture for image encoding, eliminating the need for object detectors and enabling an end-to-end VLP framework. For instance, Huang et al. (2021) proposed an “out-of-the-box” solution that does not rely on complex preprocessing steps; Kim, Son & Kim (2021) utilized the Transformer architecture to directly take unprocessed image segments and textual word embeddings as inputs, avoiding complex visual feature extraction processes. Li et al. (2021) introduced momentum distillation techniques, using a momentum model to enhance visual and language feature alignment, resulting in significant performance improvements across multiple vision-language tasks. However, these methods had limitations in fine-grained cross-modal alignment, increased computational costs, and introduced noisy visual information for cross-modal fusion. More recent works have made further advancements in enhancing the effectiveness of the pre-trained process for vision-language models. For instance, Fu et al. (2024) introduced a patch fusion method designed to obtain a compact representation of visual token sequences and strengthen the integration across different modes. Additionally, the BLIP-2 model series (Li et al., 2023) adopted a guided language-image pre-trained method utilizing pre-fixed image encoders and substantial language models, achieving effective fusion and precise matching of visual and language information.
Recent studies have produced works similar to this research. For instance, Xu et al. (2020) introduced the CASC joint framework that performs general local alignment and global semantic consistency multi-label prediction. Similarly, Li et al. (2024b) proposed the memory networks to achieve complementarity between independent and interactive embedding approaches, thereby improving retrieval efficiency. However, these methods do not consider address fine-grained cross-dimensional semantic matching, nor did it consider the relationship between different dimensions of the same feature and different features of the same dimension in coarse-grained matching. Zhang et al. (2023) conducted precise mining and modeling of dimensional semantic dependencies (x-dim), but still only performed fine-grained cross-dimensional semantic matching. In contrast, the approach presented in this work explicitly constructs a coarse-grained complementary cross-dimensional semantic aggregation network, performs fine-grained cross-dimensional semantic dependency matching, and further aggregates cross-dimensional image and text semantic to ensure consistency between global and local information.
Methodology
This work offers an in-depth elucidation of the proposed CDGCN, as depicted in Fig. 3. This work comprises four distinct components: (1) Feature extraction. (2) Fine-grained cross-dimensional semantic dependency branch learns interdependencies between different dimensions. (3) Coarse-grained cross-dimensional semantic aggregation module aggregates vector weights in different and same dimensions. (4) Global semantic matching is used as a supplement to local semantic alignment for coarse-fine-grained image-text matching.
Overview of the proposed CDGCN framework.The proposed CDGCN framework combines coarse-grained cross-dimensional global semantic consistency with fine-grained local region word cross dimensional semantic alignment to quantify the correlation between sentences and images. CDGCN framework consists of two parts: (A) fine-grained cross-dimensional semantics dependency branch: Searching for semantic dependencies between different dimensions of vectors in a similarity shared space for fine-grained word and image region matching. (B) Coarse-grained cross-dimensional semantic aggregation (CGDSA) module: consider the varying significance of different dimensions within the same feature, based on the interaction between different vectors in the same dimension, cross-dimensional aggregation weights are used to aggregate local fine-grained features into global features.
Feature extraction
Image representation. For an image v, this work uses a bottom-up attention network (Anderson et al., 2018) to extract salient regions, which employs ResNet-101 (He et al., 2016) as the backbone and is implemented by Faster-RCNN (Ren et al., 2015). Then, this work adds a fully connected (FC) layer to map each region feature to a d-dimensional local feature, represented as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm V} = \left{ {{{\rm v}1},...{{\rm v}{\rm m}}} \right} \in {{\rm R}^{{\rm m}, \times ,{\rm d}}}\end{document} , which is the visual fragment and local feature of image v, and the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm m}\end{document} is the number of image region features.
Text representation. For a sentence u, this work uses a sequence model, bidirectional gated recursive unit (BiGRU) (Schuster & Paliwal, 1997), or pre-trained BERT model (Devlin et al., 2018) to extract the word feature set. This work also adds a fully connected (FC) layer to maintain the same dimension as the image, represented as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm U} = \left{ {{{\rm u}1},...{{\rm u}{\rm n}}} \right} \in {{\rm R}^{{\rm n}, \times, {\rm d}}}\end{document} , which is the text fragment and local feature of the text u, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm n}\end{document} is the number of word features.
Fine-grained cross-dimensional semantic dependency branch
For the semantic similarity between any word feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm u} = \left{ {{{\rm u}{\rm i}}} \right}{{\rm i} = 1}^{\rm d}\end{document} and region feature \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm ; v} = \left{ {{{\rm v}{\rm i}}} \right}{{\rm i} = 1}^{\rm d}\end{document} , according to Fig. 1B, typical approaches often employ implicit independent aggregation to reflect it for all dimension correspondences, i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \mathop \sum \nolimits_{{\rm i} = 1}^{\rm d} {{\rm s}{\rm i}}\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm s}{\rm i}}\end{document} can be determined by the product of scalars \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm u}{\rm i}}\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm v}{\rm i}}\end{document} in internal product operations. That is to say, The aggregated process of cross-modal correspondence vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm s}{\rm i}} = \left{ {{{\rm s}{\rm i}}} \right}_{{\rm i} = 1}^{\rm d}\end{document} in the similarity sharing-space is based on the unconscious assumption that each dimension is perceived as a distinct isolated component.
Inspired by Zhang et al. (2023), not all dimensions in a similarity shared-space are independent, and some local dimensions have potential relationships between them, defined as cross-dimensional semantic dependencies. This work introduces the conditional probability of each dimension to represent joint semantics. It selects dimensions with significant correlation for sparse correlation probability learning based on adaptive adjustment of learned weights to sparsity. Firstly, as shown in the dimension-dependent perception aggregation in Eq. (1):
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\rm Ag}{{\rm g}_{\rm w}}\left( {\rm s} \right){\rm \; } = \mathop \sum \limits_{{\rm i} = 1}^{\rm d} \left \vert \left \vert{\left[ {\matrix{ {{{\rm w}_{11}}} & \cdots & {{{\rm w}_{1{\rm d}}}} \cr \vdots & \ddots & \vdots \cr {{{\rm w}_{{\rm d}1}}} & \cdots & {{{\rm w}_{{\rm dd}}}} \cr } } \right]_{\rm W}} \times {\left[ {{{\rm s}_1},...,{{\rm s}_{\rm i}},....} \right]^{\rm T}}\right \vert \right \vert _{{{\rm l}_2}}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm W} \in {{\rm R}^{{\rm d} \times {\rm d}}}\end{document} represents a learning weight matrix and each row of which represents a semantic dependency between various dimensions. Specifically, for row i of the weight matrix, The formula for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \widehat {{{\rm s}_{\rm i}}}\end{document} is as follows:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{{\rm \widehat s}_{\rm i}}} = {{\rm w}_{{\rm i}1}}.{{\rm s}_1} + ... + {{\rm w}_{{\rm ij}}}.{{\rm s}_{\rm j}} + ... + {{\rm w}_{{\rm id}}}.{{\rm s}_{\rm d}}, \,{\rm j} \in \left[ {1,{\rm d}} \right].$$\end{document}It models the dependency relationships among all dimensions within the shared representation space of the i-th joint representation, collectively representing latent semantics. To explicitly quantify, the conditional probabilities for each dimension are introduced as follows:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\rm p}({\rm \widehat s}{_{\rm i}}|{{\rm s}_{\rm j}}) = {\rm Sigmoid}\left( {{{\rm w}_{{\rm ij}}}} \right), \,{\rm j} \in \left[ {1,{\rm d}} \right],$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm p}(\widehat {{{\rm s}{\rm i}}}|{{\rm s}{\rm j}}){\rm } \in \left[ {0,1} \right]\end{document} reflects the extent of dependency of the j-th dimension for the jointly represented \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} { \rm \widehat s}{\rm i}\end{document} . The likelihood that this dimension in the shared space participates in the joint representation is positively correlated with the value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm p}( {{{\rm \widehat s}{\rm i}}}|{{\rm s}{\rm j}})\end{document} . The learned conditional probability \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} { {\rm p}( {{{\rm \widehat s}{\rm i}}}|{{\rm s}_{\rm j}})} {{\rm j} = 1}^{\rm d}\end{document} represents the dependence of all dimensions on latent semantics \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{{\rm \widehat s}{\rm i}}}\end{document} , Zhang et al. (2023) proved in the article that the probability density is approximately a normal distribution. Drawing from the statistical characteristics of conditional probability, this work initially allows the model to independently acquire a soft threshold for distinguishing if the dimensions exhibit joint dependencies. The significance is as follows:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\rm t}_{\rm i}} = {{\rm u}_{\rm i}} + {{\rm \alpha }_{\rm i}} \cdot {{\rm \sigma }_{\rm i}}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm ; }{{\rm u}{\rm i}}\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm \sigma }{\rm i}}\end{document} denote the mean and standard deviation of the sampling probability values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} { {\rm p}( {{{\rm \widehat s}{\rm i}}}|{{\rm s}{\rm j}})} {{\rm j} = 1}^{\rm d}\end{document} . \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm \alpha }{\rm i}}\end{document} is a learnable parameter for adaptively adjusting the soft threshold to regulate the ratio of selected dimensions. Further scaling operations obtain the modified sparse probability as:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\hat {\rm p}( {{{\rm \widehat s}_{\rm i}}}|{{\rm s}_{\rm j}}) = {\rm \delta }\left( {{{ l} ^{{\rm \gamma }({\rm p}(\widehat {{{\rm s}_{\rm i}}}|{{\rm s}_{\rm j}}) - {{\rm t}_{\rm i}})}}} \right)$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm \gamma }\end{document} represents a trainable scaling parameter, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm \delta }\left( \cdot \right)\end{document} is the activation function.
In other words, based on the learning of conditional probability \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm p}( {{{\rm \widehat s}{\rm i}}}|{{\rm s}{\rm j}})\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm t}{\rm i}}\end{document} as a critical point, dimensions with dependency less than the threshold \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm t}{\rm i}}\end{document} will be discarded, while dimensions with dependency greater than the threshold \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm t}_{\rm i}}\end{document} will continue to be used. With the operations mentioned above, Eq. (1) will be modified to Eq. (6):
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\widehat {{{\rm s}_{\rm i}}} = \mathop \sum \limits_{{\rm i} = 1}^{\rm d} \hat {\rm p}(\widehat {{{\rm s}_{\rm i}}}|{{\rm s}_{\rm j}})\cdot {{\rm w}_{{\rm ij}}}\cdot {{\rm s}_{\rm j}},{\rm \; \; \; j} \in \left[ {1,{\rm d}} \right].$$\end{document}Finally, adaptive dimensional semantic dependency perception can be formulated as:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\rm Ag}{{\rm g}_{\rm w}}\left( {\rm s} \right){\rm \; } = \mathop \sum \limits_{{\rm i} = 1}^{\rm d} ||\left( {\hat {\rm P}\odot {\rm W}} \right) \times {{\rm s}^{\rm T}}||_{{{\rm l}_2}}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \hat {\rm P } = \hat {\rm p}({{{\rm s}{\rm i}}}|,{{\rm \widehat s}{\rm j}})\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm i},{\rm j} \in \left[ {1,{\rm d}} \right]\end{document} can be regarded as an adaptive regularizer to achieve fine-grained dimensional dependency modeling.
Coarse-grained cross-dimensional semantic aggregation module
The fine-grained cross-dimensional semantic dependency branch has completed the fine-grained cross-dimensional region-word similarity computation, and in order to avoid excessive focus on the local image region and text word information to maintain the consistency of global and local semantic information, coarse-grained global image text information needs to be introduced as an auxiliary balance. However, current global features generated by aggregation of commonly used pooling operations, such as average pooling and maximum pooling, for example, contain a lack of semantic features. Inspired by Wu et al. (2023), his work found that different dimensions may have different meanings for the same feature. Taking the term “match” as an illustration, certain dimensions may connote “light” while others represent “competition”. Similarly, some dimensions of “doctor” may represent “physician” while others represent “doctor of philosophy”. It means that when consolidating global features, the significance of various dimensions within the same feature vector should vary.
Based on previous research, this work introduces a novel aggregation module called CGDSA. Unlike common pooling operations, as shown in Fig. 4, CGDSA module considers the relationships between different vectors on the same dimension and the differences of the same feature between different dimensions. Specifically, taking an image as an example, this work first enhances the obtained features:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\rm F\; } = {{\rm W}_2}\left( {{\rm ReLu}\left( {{{\rm W}_1}{\rm U} + {{\rm b}_1}} \right)} \right) + {{\rm b}_2}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm W}_1}\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm b}_1}\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm W}_2}\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm b}_2}\end{document} are the parameters of the two fully connected layers, the fully connected layer transforms u into the d-dimensional common space to standardize the representation dimension, ReLu activation function for nonlinear mapping to obtain higher semantic information.
The structural partitioning of coarse-grained cross-dimensional semantic aggregation (CGDSA) module.(1) Same-Dimension Attention Aggregation (SDAA). (2) Different-Dimension Aggregation (DDA).
Then, this work draws on an additional external d * d space (denoted by M) to keep the image and text mapped within a fixed dimensional attention space, which allows for relationships between different vectors in the same dimension. The expression formula for this matrix is as follows:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\rm F}_1} = {\rm softmax}\left( {{\rm F*M}} \right).$$\end{document}Meanwhile, under the influence of the combination of vertical vectors, discriminative information in the horizontal dimension cannot be flexibly integrated in a fine-grained manner. Therefore, this work aggregates features from the direction of column elements to obtain aggregation weights to obtain richer features, and the specific formula is as follows:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\rm F}_2} = {\rm softmax}\left( {\rm F} \right).$$\end{document}Note that the softmax function is executed separately in the region dimension column direction. Column-wise softmax involves applying the softmax function to each column individually, ensuring that the sum of N elements in each column equals 1.
Additionally, this work introduces vectors as aggregation weights, which can achieve finer granularity aggregation compared to scalars as aggregation weights. Lastly, this work amalgamates features of the identical dimension mapped to the external space and features of diverse dimensions to derive the ultimate aggregated feature. The computation formula is shown as Eq. (11):
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\hat {\rm u} = \mathop \sum \limits_{{\rm i} = 1}^{\rm N} {{\rm F}_1}\odot \left( {{{\rm u}_{\rm i}}} \right)+\mathop \sum \limits_{{\rm i} = 1}^{\rm N} {{\rm F}_2}\odot \left( {{{\rm u}_{\rm i}}} \right).$$\end{document}Note that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm F}_1}\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm F}2}\end{document} are the vectors, where each element stands for the combined weight of each dimension of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm u}{\rm i}}\end{document} , of course, the aggregation of textual features can also be accomplished by the operation of this module, notated as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \widehat {{\rm v; }}\end{document} .
Coarse-fine-grained semantic matching
The goal of the CDGCN method is the combination of fine-grained cross-dimensional semantic dependency match branch and coarse-grained cross-dimensional semantic aggregation match branch. The CDGCN is included into the cross attention inference to establish cross-modal fine-grained matching. Taken out an image-text pair ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm U},{\rm V})\end{document} from the dataset which comprises n text-words \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left{ {{{\rm u}{\rm q}}} \right}{{\rm q} = 1}^{\rm n}\end{document} and m salient image-regions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left{ {{{\rm v}{\rm p}}} \right}{{\rm p} = 1}^{\rm m}\end{document} , the semantic similarity of all text-word image-region pairs is obtained as follows (Lee et al., 2018):
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\rm r}_{{\rm qp}}} = {\rm Agg}\left( {{{\rm s}_{{\rm qp}}}} \right),\, \rm s.t.\;{{\rm s}_{{\rm qp}}} = {{\rm u}_{\rm q}}\odot {{\rm v}_{\rm p}}.$$\end{document}For each text word query \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm u}_{\rm q}}\end{document} , its relevant area can be represented as:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{{\rm \widehat v}_{\rm q}}} = \mathop \sum \limits_{{\rm p} = 1}^{\rm m} {{\rm \beta }_{{\rm qp}}}{{\rm v}_{\rm p}},\quad \,{\rm s}.{\rm t}.{\rm \; \; }{{\rm \beta }_{{\rm qp}}} = \displaystyle{{{\rm exp}\left( {{\rm \lambda }\odot {\rm \sigma }\left( {{{\rm r}_{{\rm qp}}}} \right)} \right)} \over {\mathop \sum \nolimits_{{\rm p} = 1}^{\rm m} {\rm exp}\left( {{\rm \lambda }\odot {\rm \sigma }\left( {{{\rm r}_{{\rm qp}}}} \right)} \right)}}{\rm \; }$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm \beta }{{\rm qp}}}\end{document} is weight, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm \lambda }\end{document} denotes a trainable scaling parameter. Following the same logic, this work can obtain the correlation score between the q-th text-word and the image as: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{{\rm \widehat r}{\rm q}}} = {\rm Agg}\left( {{{\rm s}{\rm q}}} \right)\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{{\rm \widehat s}{\rm q}}} = {{\rm u}{\rm q}}\odot {{{\rm \widehat v}{\rm q}}}\end{document} . Finally, the overall correlation between image and text is inferred as:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\rm S}{\left( {{\rm U},{\rm V}} \right)_{{\rm fine}}} = \displaystyle{1 \over {\rm n}}\mathop \sum \limits_{{\rm q} = 1}^{\rm n} {{\widehat {\rm r}_{\rm q}}}.$$\end{document}Similarly, this work maps the aggregated image features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \hat {\rm v}\end{document} and text features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \hat {\rm u}\end{document} into a shared space to compute cosine similarity for coarse-grained global matching: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm s}{\left( {\hat {\rm v},\hat {\rm u}} \right)_{{\rm coarse}}} = \displaystyle{{{{\hat {\rm v}}^{\rm \top }}\cdot \hat {\rm u}} \over {||\hat {\rm v}||\cdot ||\hat {\rm u}||}} \end{document} .
This work trains and computes bi-directional ranking losses (Faghri et al., 2017) for matching image-text pairs:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\rm {\scr L}}\left( {{\rm U},{\rm V}} \right){\rm \; } = {\left[ {{\rm \gamma } - {\rm S}\left( {{\rm U},{\rm V}} \right) + {\rm S}\left( {{\rm U},{{\rm V}^ - }} \right)} \right]^ + } + {\left[ {{\rm \gamma } - {\rm S}\left( {{\rm U},{\rm V}} \right) + {\rm S}\left( {{{\rm U}^ - },{\rm V}} \right)} \right]^ + }$$\end{document}The final objective function of the CDGCN method derived is:
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\rm {\scr L}}{\rm \; } = {\rm \; }{\rm {\scr L}}\left( {{\rm U},{\rm V}} \right){{\rm \; }_{{\rm fine}}} + {\rm \alpha }{\rm {\scr L}}{\left( {{\rm U},{\rm V}} \right)_{{\rm coarse}}}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm V}^ - }\end{document} denotes hardest negative image and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\rm U}^ - }\end{document} denotes hardest negative text, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\rm \alpha }\end{document} is a hyper-parameter that balances the loss functions between fine-grained cross-dimensional semantic matching and coarse-grained cross-dimensional semantic aggregation matching.
Results and analysis
Datasets and evaluation metrics
Datasets: This work uses Flickr30K dataset (Young et al., 2014) and the MS-COCO dataset (Lin et al., 2014), where each image is paired associated with five texts descriptions. The Flickr30k dataset comprises 29,000 training images, 1,000 testing images and 1,014 validation images. The MS-COCO contains 82,738 training images, 5,000 testing images, and 5,000 validation images. Results for MS-COCO were evaluated by averaging outcomes across five runs for the 1 K test images and for the entire set of 5K test images. The Flickr30K dataset can be accessed at: https://shannon.cs.illinois.edu/DenotationGraph/; and the MSCOCO dataset is accessed through URL: https://cocodataset.org/#download. Codes are accessible at https://github.com/meizhenliu/CDGCN-code.
Evaluation metrics: The proposed method’s effectiveness is assessed using metrics R @ K (where K = 1, 5, 10) and rSum (Chen et al., 2021). Specifically, R@K quantifies the proportion of accurately retrieved true results among the top K items returned, whereas rSum represents the sum of all R@K scores. Particularly, R@K indicates the percentage of correct ground truth results retrieved within the first K search items, and rSum signifies the total of all R@K values.
Implementation details
All experiments have been conducted using an NVIDIA GeForce RTX 3090 GPU and the proposed CDGCN is implemented using PyTorch. In the visual branch, this work uses the Bottom-Up and Top-Down Attention (BUTD) network (Anderson et al., 2018; Li et al., 2019) pre-trained on the Visual Genome (VG) dataset to extract N = 36 region features from images, each with 2,048 dimensions. These features are then sent to a linear fully connected layer to map them into a shared space of 1,024 dimensions. For text representation, this work employs BiGRU model with a word embedding size of 300, along with the default pre-trained BERT (Devlin et al., 2018) with 768 dimensions. When using the BiGRU text encoder, this work initialize the word vectors with 300-dimensional global vectors (GLoVe) embeddings (Pennington, Socher & Manning, 2014), applying a dropout rate of 0.1 during training to enhance the generalization of text representations. In the case of using the BERT text encoder, our work makes use of the standard pre-trained BERT model, consisting of 12 transformer layers, each equipped with 12 attention heads and a hidden layer dimension of 768 units. This work also uniformly applys text augmentation strategies to train BERT. Additionally, after encoding both the images and sentences, this work incorporates L2 normalization to process the features, ensuring the stability and consistency of the representations.
During the training process, the visual encoder’s parameters remain unchanged, whereas the parameters of BERT undergo fine-tuning. This work uses the Adam optimizer, with an initial learning rate of 0.0002 for the Flickr30k dataset and 0.0005 for the MSCOCO dataset. The training process spans 30 epochs across both datasets, with a batch size of 128 for each dataset. The learning rate is reduced by 10% every 15 epochs. The margin parameter λ is set to 0.2. The hyperparameter α, which balances the coarse-grained cross-dimensional semantic aggregation matching branch and the fine-grained cross-dimensional semantic dependency matching branch in the loss function, is set to 0.06 for the Flickr30k dataset and 0.1 for the MSCOCO dataset.
During testing, this work evaluates the single model obtained from training, denoted as-single. For model ensembling, this work averages the similarity matrices of any two models obtained from multiple training runs, the ensembled results are denoted as-ensembled*.
Benchmark with state-of-the-art methods
The performance of the CDGCN is compared with state-of-the-art models using individual and integrated models. In this section, the CDGCN which is a relatively low-cost small model is compared with previous low-cost small models. In addition, this work also presents the retrieval results of the state-of-the-art vision-language pre-training large models (Li et al. 2023, 2021) in Table 1. It can be seen that the vision-language pre-training large models (such as BLIP-2) demonstrate superior ability with significant improvement over existing methods. To ensure an unbiased comparison of experimental results, this work categorizes existing methods (focusing only for the low-cost small models) into two groups based on their distinct approaches to extracting visual and textual backbone features. Tables 1 and 2 respectively showcase the quantitative findings of the CDGCN and existing methods on the Flickr30K and MSCOCO 1 K datasets. Even when employing a standalone model rather than an integrated one, the proposed CGDSN has shown significant improvement compared to the state-of-the-art methods (Fu et al., 2023; Pan, Wu & Zhang, 2023; Yao et al., 2024; Zhang et al., 2023; Zhu et al., 2023). The proposed CDGCN has impressive advantages in R @ 1,5,10 and rSum. Notably, compared to the existing SOTA X-dim method (Zhang et al., 2023), the proposed CDGCN achieved a rSum improvement rate of 4.6–7.7% relative to the two datasets. For larger MSCOCO 5K test sets, the proposed method surpasses the SOTA model in all evaluation metrics and has a significant performance gap in Table 3. The proposed CDGCN achieved a rSum improvement rate of 5.0–8.8% for the two datasets, which more convincingly demonstrates the superiority of the proposed CDGCN.
Table 1: The comparisons with state-of-the-art methods on Flickr30K test set, utilizing comprehensive assessment indicators such as recall at K (R@K, where K is 1, 5, 10, representing the recall rate of the top K results relative to the query) and rSum, which is the sum of recall-K values at K = 1, 5, 10 for both image retrieval based on text and text retrieval based on image tasks.The term “Region” indicates the use of Faster-RCNN to extract features from image regions. “BiGRU” and “BERT” signify their use in extracting word features from texts. “Single” refers to the result obtained from a single model, while “ensemble” indicates the combined results of two models.*
Table 2: Comparisons with state-of-the-art methods on MSCOCO 1K test set, utilizing comprehensive assessment indicators such as recall at K (R@K, where K is 1, 5, 10, representing the recall rate of the top K results relative to the query) and rSum, which is the sum of recall-K values at K = 1, 5, 10 for both image retrieval based on text and text retrieval based on image tasks.The term “Region” indicates the use of Faster-RCNN to extract features from image regions. “BiGRU” and “BERT” signify their use in extracting word features from texts. “single” refers to the result obtained from a single model, while “ensemble” indicates the combined results of two models.*
Table 3: Comparisons with state-of-the-art methods on MSCOCO 5K test set, utilizing comprehensive assessment indicators such as recall at K (R@K, where K is 1, 5, 10, representing the recall rate of the top K results relative to the query) and rSum, which is the sum of recall-K values at K = 1, 5, 10 for both image retrieval based on text and text retrieval based on image tasks.The term “Region” indicates the use of Faster-RCNN to extract features from image regions. “BiGRU” and “BERT” signify their use in extracting word features from texts. “Single” refers to the result obtained from a single model, while “ensemble” indicates the combined results of two models.*
This research achievement substantially advances the acquisition and understanding of image and text features, enhancing the robustness and the accuracy of image-text matching. It is well suited for application across various multimodal domains, including e-commerce, intelligent assistants, medical image analysis, and social media. This technology enables more accurate image retrieval and recommendation systems, improves user interaction experiences, optimizes content generation and editing tools, and supports intelligent diagnosis and public opinion analysis systems. Hence, this work improves the operational efficiency and user satisfaction of various industries and promotes innovation and development of multimodal data processing technology.
Ablation studies
As shown in Tables 4 and 5, this work conducts comprehensive ablation studies on the Flick30k dataset to validate the efficacy of the proposed CDGCN components. By default, experiments are performed on the fusion mechanism framework using Region+BERT.
Table 4: The ablation study of coarse-grained cross-dimension aggregation branches and fine-grained cross-dimension dependency branches on Flickr30K, using comprehensive evaluation metrics including recall at K (R@K, K = 1, 5, 10, defined as the recall rate at the top K results to the query) and rSum, which sums the recall-K values at K = 1, 5, 10 for both image-to-text and text-to-image retrieval tasks.Using Faster-RCNN to extract region features for images and use BERT to extract word features for texts. “Single” denotes the result of one model,“ensemble” denotes the ensemble results of two models.*
Table 5: The ablation study of Flickr30K coarse-grained cross-dimensional aggregation branch, using comprehensive evaluation metrics including recall at K (R@K, K = 1, 5, 10, defined as the recall rate at the top K results to the query) and rSum, which sums the recall-K values at K = 1, 5, 10 for both image-to-text and text-to-image retrieval tasks.Using Faster-RCNN to extract region features for images and use BERT to extract word features for texts. “Single” denotes the result of one model, “ensemble” denotes the ensemble results of two models.*
Firstly, this work conducts ablation studies to explore the efficacy of two branches of the CDGCN. The without (W/O) fine-grained branch represents the removal of fine-grained cross-dimensional semantic dependency matching branches, the W/O fine-grained cross-dimension branch represents the retention of fine-grained semantic matching branches without performing similarity cross-dimensional dependency calculations, and the W/O coarse-grained cross-dimensional semantic matching branch represents the removal of coarse-grained cross-dimensional semantic matching branches. The comparative outcomes are depicted in Table 4. The performance of rSum without fine-grained cross-dimension module decreased by 4.8%, indicating the importance of using fine-grained cross-dimension semantic matching branches. In addition, the performance of rSum W/O fine-grained cross-dimension shows a significant decrease, which also indirectly verifies the powerful effect of similarity cross-dimension dependency. The performance of the rSum W/O coarse-grained branch module decreased by 1.9%, indicating that using coarse-grained cross-dimensional semantic matching branches can effectively obtain rich image text features while maintaining consistency between overall and local features, thereby enhancing the performance of image-text retrieval.
Secondly, to verify the effectiveness of the CGDSA module, this work conducts ablation experiments on each component in the image-text retrieval task. It uses three standard pooling methods to aggregate local features and Table 5 presents the comparative outcomes. This work shows that only same dimension attention aggregation (OSDAA) represents only external spatial attention aggregation in the same dimension, meanwhile, only different dimension attention aggregation (ODDA) represents only cross-dimensional aggregation in different dimensions with scalar representing scalar aggregation weight, and row-wise softmax representing row element direction aggregation feature. As the data displayed in Table 5, it is clear that the importance of each component in CGDSA module is significant. If only the DDA module is used, without the SDAA module and cross-dimensional aggregation features of different dimensions, the performance of rSum decreases by 4.8%. If only the SDAA module is used, without the DDA module, and without features that interact with different vectors of the same dimension, the performance of rSum decreases by 2.5%. Using scalar aggregation weights instead of vectors cannot obtain richer element-level aggregation, significantly reducing rSum’s performance. Furthermore, the CGDSA method performs better than other methods. For example, using mean pooling resulted in a 12.5% and 11.4% decrease in R @ 1 for text retrieval and image retrieval, individually. GPO resulted in a 2.1% and 1.8% decrease in R @ 1 for text retrieval and image retrieval, particularly, resulting in a 6.4% decrease in rSum performance. Different from simple pooling methods, this work emphasizes the differences in feature dimensions and the mapping of identical dimensions and vectorizes the aggregation weights, thereby achieving a finer-grained aggregation of features across identical and distinct dimensions.
In addition, as a plug-and-play CGDSA module, the CGDSA method can also be directly used in visual semantic embedding (VSE) tasks. Compared with other pooling strategies, all indicators of the proposed CGDSA method have been significantly improved. Table 6 illustrates that the CGDSA module enhances the effectiveness of visual semantic embedding. The CGDSA method improves R @ 1 by 12.5% and 11.4% respectively when compared to average pooling text retrieval and image retrieval. In addition, the improvement of R@1 using the CGDSA method has improved by 2.1% and 1.8%, particularly, resulting in a 6.4% performance improvement when compared to the GPO method. In addition, this work also compared the fusion methods of only SDAA and only DDA, and the results showed that the sum of the elements in the SDAA and DDA had the best effect. The CGDSA (concatenation) indicates that the work performed subsequent operations on elements; meanwhile, CGDSA (element added) suggests that the work directly added elements to operate. The CGDSA (element added) obtains the most encouraging results compared to other methods since the proposed method managed to extract richer information.
Table 6: Ablation study on visual semantic embedding using different pooling strategies, using comprehensive evaluation metrics including recall at K (R@K, K = 1, 5, 10, defined as the recall rate at the top K results to the query) and rSum, which sums the recall-K values at K = 1, 5, 10 for both image-to-text and text-to-image retrieval tasks.Using Faster-RCNN to extract region features for images and use BERT to extract word features for texts.
Sensitivity to parameters
Hyperparameter α: After computing the coarse-grained cross-dimensional semantic aggregation matching branch and the fine-grained cross-dimensional semantic dependency matching branch, it is imperative to combine the loss functions of these two branches to ensure the consistency of local and global semantic information. The hyperparameter α serves as a weighting factor to balance these loss functions. Specifically, α adjusts the relative contribution of the two branches to the overall loss, as represented in Eq. (16). Typically α is a scalar value, ranging between 0 and 1. Within the CDGCN framework, α plays a crucial role in balancing the loss of the coarse and fine-grained matching branches. By defining α and dynamically adjusting it during the training process, the optimal parameter value that maximizes the model’s performance can be discovered. Other parameters of the CDGCN remain unchanged during the adjustment of α.
The findings of the ablation experiments are presented in Tables 7, 8, and Fig. 5. When α is set to 0, it indicates that the coarse-grained cross-dimensional global matching branch is not engaged in the calculation, resulting in relatively low performance of CDGCN. Notably, setting α to 0.1 on Flick30k and 0.05 on MSCOCO yields the optimal performance of CDGCN. Conversely, as α rises, the retrieval performance of CDGCN progressively deteriorates. The performance degradation is attributed to an excessive emphasis on global semantic information from images and text, which undermines the effective matching of detailed features in images regions and text words. This imbalance affects the performance of the model and compromises the model’s efficacy. Hence, selecting an appropriate α is crucial for CDGCN. Ultimately, when all parameter values are varied within appropriate ranges, the performance remains relatively stable. This demonstrates the robustness of the CDGCN and its insensitivity to the selection of hyper-parameters.
Table 7: The value of hyper-parameter α on Flick30k, using comprehensive evaluation metrics including recall at K (R@K, K = 1, 5, 10) and rSum, which sums the recall-K values at K = 1, 5, 10 for both image-to-text and text-to-image retrieval tasks.Using Faster-RCNN to extract region features for images and use BERT to extract word features for texts.
Table 8: The value of hyper-parameter α on MSCOCO, using comprehensive evaluation metrics including recall at K (R@K, K = 1, 5, 10) and rSum, which sums the recall-K values at K = 1, 5, 10 for both image-to-text and text-to-image retrieval tasks.Using Faster-RCNN to extract region features for images and use BERT to extract word features for texts.
Impact of hyper-parameters. as α changes, the changes of metrics R @ K and rSum.(A) on Flick30k, (B) on MSCOCO 1k, (C) on MSCOCO 5 k.
Generalization study
In cross-modal image-text matching tasks, evaluating a model’s generalization capabilities is critical for ensuring its practicality in real-world applications. The significance of cross-dataset generalization research lies in its potential to enhance the robustness of models across disparate datasets, thereby improving the efficiency of real-world deployments. Assessing model’s performance across multiple datasets offers significant insights into the model’s adaptability to varying environmental conditions–a key factor in scenarios where data diversity is unavoidable. Strengthening this robustness minimizes dependence on specific datasets and significantly enhances the model’s generalization ability, ensuring reliable performance on unseen data and increasing its practical applicability.
This work conducts cross-validation experiments to verify the proposed CDGCN’s generalization ability for overall coarse-fine-grained image-text matching. The experiment involves training baseline models and CDGCN on the MSCOCO dataset, followed by cross-validation on the Flickr30K dataset to test the CDGCN’s zero-shot ability. The detailed outcomes are depicted in Table 9, whether it’s the “Region+BiGRU” version or the “Region+Bert” version, the CDGCN model exhibits stronger generalization capabilities than the baseline model, demonstrating more robust performance. The findings validate the efficacy of the proposed CDGCN in cross-dimensional semantic aggregation and coarse-grained semantic consistency.
Table 9: Comparison of model generalization capabilities, using comprehensive evaluation metrics including recall at K(R@K, K = 1, 5, 10) and rSum, which sums the recall-K values at K = 1, 5, 10 for both image-to-text and text-to-image retrieval tasks, Region represents using Faster-RCNN to extract region features for images.BiGRU and BERT represent using them to extract word features for texts. Baseline denotes the basic image-text matching model.
The implications of such research extend to numerous practical applications, with models exhibiting robust cross-dataset generalization capabilities being more likely to achieve successful implementation. These models demonstrate consistent efficacy across diverse data environments, mitigating the challenges posed by the variability of data characteristics and thereby accelerating the processes of deployment and optimization. For instance, integrating the CDGCN model into e-commerce recommendation systems allows for user-uploaded images and textual descriptions that may originate from disparate data distributions. By harnessing the insights garnered from cross-dataset generalization research, we can enhance and modify the model according to specific needs, improving its performance in recommendation tasks and enriching the user experience. Furthermore, the pursuit of advancements in cross-dataset generalization has catalyzed innovations in technology and methodology.
Visualization and qualitative result analysis
Visualization of visual and textual features: Utilizing a sample of 1,000 images from the Flicker30K test set, this work utilizes t-SNE (Van der Maaten & Hinton, 2008) to transform the acquired final visual features into a two-dimensional space. Following this, the K-means clustering technique is employed to group these features into 40 clusters. Each cluster is distinguished by a unique color, and the centroid of each cluster is annotated with its respective corresponding group number. The outcomes of visualization are illustrated in Fig. 6A, where this work highlights examples from certain clusters. As demonstrated in Fig. 6A, it is evident that images from different clusters exhibit distinct semantic signs, whereas those from the same cluster display consistent semantic symbols, such as young boys, adorable dogs, baseball and soccer, basketball sports, skiing, hats, all visually similar. For Group 20, featuring adorable dogs, and Group 37, focusing on skiing, there is a significant disparity in the two-dimensional feature map, indicating substantial differences in the features extracted by the proposed model for these two groups. Furthermore, despite Group 2, encompassing baseball and soccer, and Group 10, representing basketball and extreme sports, falling under the category of sports, they are still segregated into two classes due to minor distinctions, shown to be relatively close in the two-dimensional feature map. These visualizations underscore the proposed CDGCN model’s efficacy in effectively extracting global and distinctive local visual features, demonstrating strong global semantic coherence and highlighting its effectiveness.
(A) Visualization of learned visual-cross-modal representations via t-SNE on Flickr30K. (B) Two-dimensional visualizations of encoded cross-dimensional visual and textual representation vectors from the Flickr30K test set.Detailed images with numbered divisions in (A) available at https://doi.org/10.5281/zenodo.14930469.
Concurrently, this work projects the 1,000 images and their corresponding top-ranking sentences into a shared two-dimensional space, as illustrated in Fig. 6B. The distribution of the dimensionality-reduced image vectors and text vectors appears largely similar. However, the visualization also reveals that a minority of image and text features are not mapped to the same distribution positions, reflecting potential matching errors during the model’s retrieval process.
Qualitative retrieval results visualization and analysis: This work conducted qualitative visualization and analysis of retrieval results on the Flickr30K dataset (detailed qualitative visualization and analysis figures available at https://doi.org/10.5281/zenodo.14930469). Due to the requirement of the journal that the retrieval visualization of the Flickr30K dataset must be accompanied by the permission signature of the person in the image, the author of the Flickr30k dataset is also unable to achieve it. Therefore, based on the Flickr30K dataset, the authors used their own portraits to create images (similar text descriptions) that were similar to the scene in the dataset, presenting the qualitative visualization of the model. CGDSA represents the Coarse-Grained Cross-Dimensional Semantic Dependency branch, FGDSD represents the Fine-Grained Cross-dimensional Semantic Aggregation branch, and CDGCN represents the Cross-Dimensional Coarse-Fine-Grained Complementary Network.
Qualitative retrieval results: Fig. 7 illustrates the top-5 text results retrieved by CGDSA, FGDSD, and CDGCN models for three query images, ranked by similarity scores. The proposed CDGCN model has retrieved almost all matching texts correctly (e.g., two out of three images matching perfectly, with only one text mismatch in the last image). Even in cases of incorrect matches, the semantic content of the text (e.g., words like “girls,” “lawn,” and “playing” in the incorrectly matched text for the third image) aligns well with the visual semantics of the image. In comparison, CGDSA and FGDSD models exhibit a higher incidence of mismatched texts. FGDSD method has incorrect matches in the second and three query images (e.g., failing to capture the image information “men and women” and incorrectly matching it with the text “several girls” in the second image). CGDSA method has mismatches in the first and third query images (e.g., not capturing the color of the hat and whether there are earrings in the first image, leading to incorrect matches with texts “brown and white hat” and “beige hat”). These findings indicate that the sentences retrieved by the CDGCN method provide detailed descriptions of visual scenes, suggesting that the image representations obtained by introduced model effectively capture rich dimensional feature information, avoiding mismatching issue by the complementary of coarse-fine-grained information.
The top 5 retrieval results of providing an image to retrieve sentences using the CGDSA, FGDSD, and CDGCN methods based on Flickr30k datasets using our own portrait pictures.The retrieved five sentences are arranged in descending order of relevance. The sentences that match the given image are displayed in black font, while mismatched sentences are indicated in red font. Additionally, any missing matches are highlighted in green font. CGDSA represents the coarse-grained cross-dimensional semantic dependency branch, FGDSD represents the fine-grained cross-dimensional semantic aggregation branch, and CDGCN represents the cross-dimensional coarse-fine-grained complementary network.
As illustrated in Fig. 8, three sets of query texts are provided, each of which retrieves the top three images through CGDSA, FGDSD, and CDGCN models. These images are displayed from left to right by their degree of matching. The images enclosed in red boxes signify the correct retrievals derived from the given texts. This illustration demonstrates that the CDGCN model can largely retrieve the correct images (correctly retrieving two out of three texts, with one incorrect retrieval). For straightforward text-to-image retrieval tasks, all three models perform satisfactorily. However, for images that strongly emphasize the overall layout and comprehensive information (such as the first query text), both CGDSA and CDGCN adeptly match the correct images. In contrast, FGDSD overemphasizes minute details like “cake hat” and “candles are inserted into the cake,” thereby overlooking other contextual elements, leading to erroneous matches. In cases with extensive, intricate details (the second query text), CDGCN and FGDSD exhibit their strengths by accurately matching these details. Although CGDSA can also capture cross-dimensional feature information for broad-scale matching, it fails to achieve precise matching for certain detailed elements, resulting in failed matches. The analysis above underscores that the CDGCN method achieves a high matching rate in text-to-image retrieval, adeptly synthesizing local and global information to complement each other, thereby retrieving the ground truth images with greater accuracy.
The top 3 retrieval results obtained by inputting text for image retrieval using the CGDSA, FGDSD, and CDGCN methods based on Flickr30k datasets using our own portrait pictures.The three retrieved images are arranged in descending order of relevance. The image that closely aligns with the provided text is accentuated within a red bounding box, whereas those that do not correspond are demarcated with black boxes. CGDSA represents the coarse-grained cross-dimensional semantic dependency branch, FGDSD represents the fine-grained cross-dimensional semantic aggregation branch, and CDGCN represents the cross-dimensional coarse-fine-grained complementary network.
Error analysis of retrieval results: As depicted in Fig. 7, the third given image resulted in errors across all three models: CGDSA, FGDSD, and CDGCN. Specifically, the text retrieved by the CDGCN method (Sentence 3) was mismatched, failing to align with the provided image. Despite this, the words “girls,” “lawn,” and “playing” within the sentence retained a high semantic relevance to the image content, carrying substantial semantic information closely related to the image. However, the actual content did not match due to discrepancies in minute details. The image showed three girls are lying on the lawn, while the retrieved text, though containing “girls,” “lawn,” and “playing,” omitted the crucial detail “setting.” For the missing correct text, while it included semantically relevant terms like “girls” and “lawn,” it failed to accurately capture verbs such as “huddled together” and “listening”, resulting in an inability to match the image. Similarly, as shown in Fig. 8 for text-to-image retrieval, given Text 3, all three models—CGDSA, FGDSD, and CDGCN—produced erroneous results. Specifically, the first two images retrieved by the CDGCN method are mismatched, while the third image is correctly matched. All three images were visually highly related, depicting a girl riding a bike, with different dressing details, movements, and categories of bikes. Figure 9 visualized examples of text-to-image retrieval errors, where words such as “girl,” “bike,” and “road” aligned well with corresponding visual regions in the images (though interpretations of “bike” varied). However, as seen in Fig. 9, third image (c), the verb “pedaling” lacked detailed semantic capture in the image, reducing the text-to-image match accuracy. Comparing the corresponding texts of the first two images (Text in black font in Fig. 9) with the query text revealed that the query sentence is relatively simple, encompassing basic semantic concepts found in all three images. The key verbs or actions in the first two images, “stepped,” “braking,” and “exploring,” “cruising,” are crucial semantic details missing from the query sentence, which the CDGCN failed to identify accurately.
Visualization of erroneous examples in text-to-image retrieval.Words from the given sentence are highlightedaboutthe most relevant regions in the retrieved image, marked by red bounding boxes. The correct sentence that originally matched the retrieval image is indicated in black font, with crucial information differing from the given sentence encircled in yellow.
Based on the above analysis, the errors in retrieval results may stem from the CDGCN model’s issues in capturing verbs, actions, and anthropomorphized vocabulary in both images and texts, overlooking minor details that diminish matching accuracy. Enhancing feature extraction performance, strengthening the relationships between words and verbs or prepositions to capture more local features, and improving the model’s architecture to enhance global contextual understanding (including anthropomorphized sentences) could potentially elevate the accuracy of image-text retrieval.
Conclusion
This work proposes an image-text matching method called Cross-Dimensional Coarse-Fine-Grained Complementary Network. The framework integrates a coarse-grained overall cross-dimensional semantic aggregation branch with a fine-grained local cross-dimensional semantic dependency branch to form establish a holistic approach that ensures consistency between global and local semantic information. A plug-and-play coarse-grained cross-dimensional semantic aggregation strategy is also developed to deeply analyze the importance and relationship between different dimensions of the same feature and various features in the fixed dimensional attention space. The proposed CDGCN has been validated on two benchmark datasets to extract richer features from image text, avoiding the problem of mismatch between overall and local features, and enhancing the efficacy of image-text matching techniques. Despite the progress achieved by the proposed model, certain limitations remain apparent. Firstly, our reliance on traditional methodologies for model construction may not constitute the most optimal solution. In the future, the integration of currently popular pre-trained vision-language large models to establish a “multi-modal model fusion network” could potentially extract more representative image and text features, enhancing retrieval accuracy and efficiency. Furthermore, our current focus exclusively on cross-modal image-text matching presents an opportunity to expand towards multi-modal retrieval, optimizing the CDGCN model to undertake cross-modal video-text retrieval tasks. In addition, based on error analysis, the CDGCN model will be optimized and planned to be applied to an intelligent search system embedded in smart education for actual campus projects.
Supplemental Information
10.7717/peerj-cs.2725/supp-1Supplemental Information 1Code.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Anderson P He X Buehler C Teney D Johnson M Gould S Zhang L Bottom-up and top-down attention for image captioning and visual question answering 2018 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Piscataway IEEE 60776086
- 2Chen H Ding G Liu X Lin Z Liu J Han J IMRAM: iterative matching with recurrent attention memory for cross-modal image-text retrieval 2020 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Piscataway IEEE 1265512663
- 3Chen J Hu H Wu H Jiang Y Wang C Learning the best pooling strategy for visual semantic embedding 2021 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Piscataway IEEE 1578915798
- 4Chen Y-C Li L Yu L El Kholy A Ahmed F Gan Z Cheng Y Liu J Uniter: learning universal image-text representations 2019 Ar Xiv 10.48550/ar Xiv.1909.11740 · doi ↗
- 5Chhabra A Vishwakarma DK Multimodal hate speech detection via multi-scale visual kernels and knowledge distillation architecture Engineering Applications of Artificial Intelligence 202312610699110.1016/j.engappai.2023.106991 · doi ↗
- 6Devlin J Chang M-W Lee K Toutanova K BERT: pre-training of deep bidirectional transformers for language understanding 2018 Ar Xiv preprint 10.48550/ar Xiv.1810.04805 · doi ↗
- 7Diao H Zhang Y Ma L Lu H Similarity reasoning and filtration for image-text matching 2021 Proceedings of the AAAI Conference on Artificial Intelligence 12181226
- 8Dong X Liu L Zhu L Nie L Zhang H Adversarial graph convolutional network for cross-modal retrieval IEEE Transactions on Circuits and Systems for Video Technology 20213231634164510.1109/TCSVT.2021.3075242 · doi ↗