GATI-RS model using Bi-LSTM and multi-head attention mechanism to enhance online shopping experience for the elderly with accurate click-through rate prediction

TL;DR

This paper introduces a new recommendation system to improve online shopping for elderly users by accurately predicting their click behavior.

Contribution

A novel GATI-RS model combining Bi-LSTM and multi-head attention for enhanced CTR prediction in elderly user recommendations.

Findings

The GATI-RS model improved CTR prediction accuracy by 40% compared to LSTM.

The model's loss function decreased rapidly and remained stable during training.

For elderly users specifically, the model outperformed the baseline by 42% in accuracy.

Abstract

With the rapid development of e-commerce and the increasing aging population, more elderly people are engaging in online shopping. However, challenges they face during this process are becoming more apparent. This article proposes a recommendation system based on click-through rate (CTR) prediction, aiming to enhance the online shopping experience for elderly users. By analyzing user characteristics, product features, and their interactions, we constructed a model combining bidirectional long short-term memory (Bi-LSTM) and multi-head self-attention mechanism to predict the item click behavior of elderly users in the recommendation section. Experimental results demonstrated that the model excels in CTR prediction, effectively improving the relevance of recommended content. Compared to the baseline model long short-term memory (LSTM), the GATI-RS framework improved CTR prediction accuracy by 40%, and its loss function rapidly decreased and remained stable during training. Additionally, the GATI-RS framework showed significant performance improvement when considering only elderly users, with accuracy surpassing the baseline model by 42%. These results indicate that the GATI-RS framework, through optimized algorithms, significantly enhances the model’s global information integration and complex pattern recognition capabilities, providing strong support for developing recommendation systems for elderly online shoppers. This research not only offers new insights for e-commerce platforms to optimize services but also contributes to improving the quality of life and well-being of the elderly.

Figures

Prediction results of item clicks in the recommendation section for elderly online shopping users.

Click prediction for elderly users in online shopping: analyzing behavior patterns with Bi-LSTM.

Prediction results of item clicks in the recommendation section for elderly online shopping users.

Performance evaluation of GATI-RS framework against baseline models in online shopping recommendation tasks.

AUC comparison of ordinary user group models.

Performance analysis of GATI-RS framework in elderly-specific online shopping recommendations.

AUC comparison for elderly user group.

Full text

Introduction

Background

With the rapid development of e-commerce and the advent of an aging society, more elderly people are starting to shop online. However, they face unique challenges and needs when using e-commerce platforms, as shown in the figure. These challenges include difficulty adapting to complex interfaces, unfamiliarity with new technologies, and decreased ability to recognize information (Gaiha, Lempert & Halpern-Felsher, 2020; Smith et al., 2020; Abbott et al., 2023). According to Lawrence (2020), 75% of adults over 75 use the internet daily, with elderly people now accounting for 28% of all e-commerce purchases. Notably, women over 75 are among the fastest-growing groups of internet users. As the elderly become more familiar with technology, their online activities are increasing (Lawrence, 2020). Capital One Shopping reports that 41% of Baby Boomers and 28% of seniors prefer to purchase products and services online (Capital One Shopping, 2025). The increase in elderly users has highlighted the challenges they face in online shopping, including complex dynamic pricing, fake reviews, and unfamiliarity with technology use (Bondi et al., 2021). Elderly people need simpler forms, larger fonts, and more intuitive user interfaces to improve their online shopping experience (Adchitects, 2025).

Despite efforts by e-commerce platforms to simplify interfaces and functions, they still struggle to fully meet the needs of the elderly (Chintala, Liaukonyte & Yang, 2024; Knutha & Ahrholdtb, 2022). Elderly people are highly sensitive to discounts and promotions but often lack the technical means to fully utilize these offers, impacting their online shopping experience and satisfaction (Smith et al., 2020; Son & Park, 2023). In recent years, researchers and businesses have started focusing on how data analysis and machine learning can enhance the online shopping experience and engagement for the elderly. However, most existing click-through rate (CTR) prediction models have not fully incorporated age-related features or the unique behavioral patterns of elderly users, leading to less effective recommendations and engagement for this demographic. For instance, many CTR prediction models primarily focus on younger, tech-savvy user groups, neglecting the specific needs of elderly users (Abbott et al., 2023; Smith et al., 2020).

By analyzing elderly users’ online shopping behavior and needs, we can provide more personalized and convenient shopping experiences, thereby improving their online shopping satisfaction and loyalty. Specific details are shown in Fig. 1.

Prediction results of item clicks in the recommendation section for elderly online shopping users.

In summary, predicting online shopping click behavior has become an important research direction. By building accurate CTR prediction models, e-commerce platforms can optimize recommendation systems, enhancing the relevance of recommended content and significantly improving the online shopping experience for elderly users. This study proposes a recommendation system (GATI-RS) combining bidirectional long short-term memory (Bi-LSTM) and multi-head self-attention mechanism, focusing on the unique needs of elderly users. Bi-LSTM effectively captures the temporal dependencies of user behavior, better recognizing user dynamics. The multi-head self-attention mechanism excels in handling global feature dependencies, providing more comprehensive feature integration, enabling the model to better serve elderly users. This research not only offers new methods for optimizing e-commerce recommendation systems but also helps elderly users find desired products more efficiently, enhancing their shopping experience.

Literature review

With the rapid development of artificial intelligence (AI) and machine learning (ML), their applications in various fields are becoming increasingly widespread. In analyzing existing recommendation systems, several key challenges emerge: handling temporal dependencies, addressing elderly user behavior patterns, and balancing model complexity with interpretability. Recent studies have explored various approaches but with notable limitations. Rausch, Derra & Wolf (2022) analyzed clickstream data and proposed different ML methods to predict online shopping cart abandonment rates. While their methods achieved reasonable accuracy, they overlooked crucial temporal dependencies in user behavior and did not account for elderly users’ unique shopping patterns, which often involve longer decision-making processes and different browsing behaviors. In the context of visual-based recommendations, Li, Li & Shao (2016) explored computer vision and pattern recognition applications in online recommendation systems, particularly in the fashion industry. Their approach demonstrated strong performance in image-based recommendations but faced challenges in capturing temporal dependencies and elderly user shopping behavior, which often requires more than visual cues alone. Addressing physical retail contexts, Alfian et al. (2023) employed radio frequency identification (RFID) technology and ML models to analyze in-store customer browsing activities. While their approach effectively captured physical browsing patterns, it did not translate well to online shopping scenarios where temporal characteristics play a crucial role. Similarly, Beltzung et al. (2020) focused on detecting fake stores using ML based on source code structure similarity, but their work did not address the temporal aspects of online shopping behavior. Wang et al. (2022) and Zhou et al. (2022) made significant contributions to online and manufacturing scheduling optimization. However, their methods’ computational complexity limited their applicability to real-time recommendation scenarios, particularly for elderly users who may require more immediate responses.

Recent advances in preference modeling include Kim & Zhang’s (2023) attribute-based mixed-membership model for consumer preference structures. While innovative, their focus on the fashion market limits its generalizability, particularly for elderly users who may have different preference patterns across various product categories. Wei et al. (2023) proposed a dynamic Bayesian networks (DBN) based recommendation method that showed promising results. However, their model’s complex architecture and high computational costs pose challenges for real-world deployment, especially in scenarios requiring quick responses for elderly users who may be less tolerant of system delays. Studies focusing on user behavior include Beyari & Garamoun’s (2022) investigation of AI tools’ impact on online purchasing decisions. Their findings, while valuable, were limited to Saudi Arabian consumers and lacked validation across different cultural and age demographics, particularly the elderly population. Damaševičius & Zailskaitė-Jakštė (2022) proposed a Human-Centric Cyber Security model incorporating deep learning methods for click-through rate prediction, which achieved superior performance (AUC of 0.8062) on the Avazu dataset. While their work provides valuable insights into user behavior prediction and security, it primarily focuses on general online behavior patterns without specific consideration for age-specific user groups. In the domain of review analysis, Bilal & Almazroi (2023) and Olmedilla, Rocio Martinez-Torres & Toral (2022) proposed advanced methods using BERT and CNN respectively for review usefulness classification. While achieving high accuracy, these approaches primarily focused on general user populations without considering the specific needs and behaviors of elderly users.

More and more researchers are beginning to focus on these challenges and explore innovative solutions. However, there are still many shortcomings in previous studies, as shown in Table 1.

Table 1: Literature review summary.

Our contributions

• Proposed a model combining Bi-LSTM and multi-head self-attention mechanism to analyze user characteristics, product features, and their interactions, accurately predicting item click behavior of elderly users in the recommendation section. The Bi-LSTM effectively captures the temporal dependencies in user behavior, particularly for elderly users, who exhibit different shopping patterns. The multi-head self-attention mechanism addresses global feature dependencies, allowing the model to integrate various factors.
• Introduced a multi-head self-attention mechanism to handle global feature dependencies, enhancing the recommendation system’s global information integration and complex pattern recognition capabilities. This improvement is especially beneficial for elderly users, whose purchasing decisions are influenced by multiple factors, such as special offers, technology unfamiliarity, and cognitive overload. The mechanism allows the model to provide more relevant, personalized, and diverse recommendations, significantly improving the relevance of item clicks for elderly users in the recommendation section.
• Through experimental validation on real-world datasets, our model improved CTR prediction accuracy by 40%. Experimental results showed that the GATI-RS framework performed excellently in elderly user recommendation scenarios, with the loss function rapidly decreasing and remaining stable during training. These results highlight the model’s ability to better adapt to elderly users’ unique behavior patterns and improve their overall shopping experience.

Our approach

Problem description

We frame item click prediction as a classification task, predicting whether users will click on recommended items based on their behavior. Each user’s behavior is captured in a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} d\end{document} -dimensional feature vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{x}}_i}\end{document} , which includes browsing patterns, purchase history, and search queries.

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\bf{x}}_i} = [{x_{i1}},{x_{i2}}, \ldots ,{x_{id}}]\quad {\mathrm{for}}\quad i = 1,2, \ldots ,N$$\end{document}

where N represents the total user count, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {x_{ij}}\end{document} is the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} j\end{document} -th feature for user \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} i\end{document} . We define the click behavior through a binary indicator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {y_i}\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${y_i} = \left\{ {\matrix{ 1 \hfill & {{\mathrm{if}}\;{\mathrm{user}}\;{\mathrm{clicks}}\;{\mathrm{the}}\;{\mathrm{recommended}}\;{\mathrm{item}}} \hfill \cr 0 \hfill & {{\mathrm{otherwise}.}} \hfill \cr } } \right.$$\end{document}

Using logistic regression, we estimate the click probability:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$P({y_i} = 1|{{\bf{x}}_i}) = {1 \over {1 + \exp ( - {{\bf{w}}^ \top }{{\bf{x}}_i})}}$$\end{document}

with model parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{w}}\end{document} . The optimization uses log-loss:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$L({\bf{w}}) = - {1 \over N}\sum\limits_{i = 1}^N {\left[ {{y_i}\log P({y_i} = 1|{{\bf{x}}_i}) + (1 - {y_i})\log (1 - P({y_i} = 1|{{\bf{x}}_i}))} \right]}.$$\end{document}

We optimize parameters through gradient descent:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\bf{w}}^{(t + 1)}} = {{\bf{w}}^{(t)}} - \eta \nabla L({{\bf{w}}^{(t)}})$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \eta\end{document} denotes the learning rate.

Problem 1 Our core objective is to find optimal parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{w}}^*}\end{document} that maximize prediction accuracy:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\bf{w}}^*} = \arg {\max _{\bf{w}}}{1 \over N}\sum\limits_{i = 1}^N {\left[ {\delta \left( {{y_i}, {\mathbb I}\left( {P({y_i} = 1|{{\bf{x}}_i}) \; > \; 0.5} \right)} \right)} \right]}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\mathbb I}( \cdot )\end{document} indicates the indicator function, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \delta (a,b)\end{document} represents the Kronecker delta function, evaluating to 1 when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} a = b\end{document} and 0 otherwise.

Bi-LSTM: understanding user interest changes and behavior patterns

Capturing temporal dependencies: user interest and behavior analysis with Bi-LSTM model

• Current recommendation models face challenges in accurately tracking user behavior patterns, particularly in identifying dynamic interest shifts (Wang et al., 2023). These limitations result in decreased prediction accuracy and increased false recommendations (Chao et al., 2023; Ni et al., 2023).
• By implementing bidirectional data processing, our Bi-LSTM model effectively captures temporal patterns in user behavior sequences, enabling precise detection of interest evolution over time.

Figure 2 depicts the architecture of our Bi-LSTM model for click prediction. The model processes online shopping user characteristics through two parallel LSTM layers: a forward layer (shown in pink) and a backward layer (shown in blue). The forward layer analyzes the sequential progression of user behaviors, while the backward layer captures temporal dependencies in reverse order. Critical temporal features (marked as X1 to Xn) are processed bidirectionally to capture complex behavioral patterns. This dual-directional analysis is particularly effective for distinguishing between elderly and young user shopping patterns, as illustrated by the separate feature sequences on the right. The bidirectional structure enables more accurate temporal dependency modeling, ultimately enhancing the recommendation system’s ability to predict click behaviors across different user demographics.

Click prediction for elderly users in online shopping: analyzing behavior patterns with Bi-LSTM.

Capturing temporal dependencies with Bi-LSTM model

We employ the Bi-LSTM model to analyze temporal patterns in user behavior sequences. The user behavior at time step \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} t\end{document} is represented as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\bf{x}}_t} = [{x_{t1}},{x_{t2}}, \ldots ,{x_{td}}]\quad {\mathrm{for}}\quad t = 1,2, \ldots ,T$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} d\end{document} is the feature dimension and T denotes time steps. In the forward LSTM, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{W}}\end{document} represents weight matrices, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{b}}\end{document} represents bias vectors, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sigma\end{document} is the sigmoid function:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign{ & {\overrightarrow{ {\bf{i}}_t}} = \sigma \left( {{{\bf{W}}_{xi}}{{\bf{x}}_t} + {{\bf{W}}_{hi}}{{\overrightarrow {{\bf{h}} }_{t - 1}}} + {{\bf{b}}_i}} \right) \\ & {\overrightarrow {{\bf{f}}_t}} = \sigma \left( {{{\bf{W}}_{xf}}{{\bf{x}}_t} + {{\bf{W}}_{hf}}{{\overrightarrow {{\bf{h}} }_{t - 1}}} + {{\bf{b}}_f}} \right)\\ & {\overrightarrow {{\bf{c}}_t}} = {\overrightarrow {{\bf{f}}_t}} \odot {\overrightarrow {{\bf{c}}_{t - 1}}} + {\overrightarrow {{\bf{i}}_t}} \odot \tanh \left( {{{\bf{W}}_{xc}}{{\bf{x}}_t} + {{\bf{W}}_{hc}}{{\overrightarrow {{\bf{h}} }_{t - 1}}} + {{\bf{b}}_c}} \right)\\ & {\overrightarrow {{\bf{o}}_t}} = \sigma \left( {{{\bf{W}}_{xo}}{{\bf{x}}_t} + {{\bf{W}}_{ho}}{{\overrightarrow {{\bf{h}} }_{t - 1}}} + {{\bf{b}}_o}} \right)\\ & {\overrightarrow {{\bf{h}}_t}} = {\overrightarrow {{\bf{o}}_t}} \odot \tanh ({\overrightarrow {{\bf{c}}_t}}).}$$\end{document}

Similarly for backward LSTM, with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\overleftarrow {\bf{h}}! _t}\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\overleftarrow {\bf{c}}! _t}\end{document} representing hidden and cell states:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign { & {\overleftarrow {{\bf{i}}_t}} = \sigma \left( {{{\bf{W}}_{xi}}{{\bf{x}}_t} + {{\bf{W}}_{hi}}{{\overleftarrow {{\bf{h}} }_{t + 1}}} + {{\bf{b}}_i}} \right)\\ & {\overleftarrow {{\bf{f}}_t}} = \sigma \left( {{{\bf{W}}_{xf}}{{\bf{x}}_t} + {{\bf{W}}_{hf}}{{\overleftarrow {{\bf{h}}}_{t + 1}}} + {{\bf{b}}_f}} \right)\\ &{\overleftarrow {{\bf{c}}_t}} = {\overleftarrow {{\bf{f}}_t}} \odot {\overleftarrow {{\bf{c}}_{t + 1}}} + {\overleftarrow {{\bf{i}}_t}} \odot \tanh \left( {{{\bf{W}}_{xc}}{{\bf{x}}_t} + {{\bf{W}}_{hc}}{{\overleftarrow {{\bf{h}} }_{t + 1}}} + {{\bf{b}}_c}} \right)\\ & {\overleftarrow {{\bf{o}}_t}} = \sigma \left( {{{\bf{W}}_{xo}}{{\bf{x}}_t} + {{\bf{W}}_{ho}}{{\overleftarrow {{\bf{h}}}_{t + 1}}} + {{\bf{b}}_o}} \right)\\ & {\overleftarrow {{\bf{h}}_t}} = {\overleftarrow {{\bf{o}}_t}} \odot \tanh ({\overleftarrow {{\bf{c}}_t}}).}$$\end{document}

The Bi-LSTM output combines forward and backward hidden states:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\bf{h}}_t} = \left[ {{{\overrightarrow {\bf{h}} }_t};{{\overleftarrow {\bf{h}} }_t}} \right]. $$\end{document}

The final prediction uses weight matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_o}\end{document} and bias vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{b}}_o}\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{\bf{o}}_t} = {{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }\!_t} \oplus {{\overleftarrow {\bf{h}}\! }_t}} \right) + {{\bf{b}}_o} = {{\bf{W}}_o}\left(\left [ {\matrix{ {{{\overrightarrow {\bf{h}} }\!_t}} \hfill \cr {{{\overleftarrow {\bf{h}}\! }_t}} \hfill \cr } } \right]\right) + {{\bf{b}}_o}.$$\end{document}

The loss function with weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {w_t}\end{document} at time step \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} t\end{document} is defined as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$L = - {1 \over T}\sum\limits_{t = 1}^T {\left[ {{w_t}{y_t}\log P({y_t} = 1|{{\bf{h}}_t}) + (1 - {w_t})(1 - {y_t})\log P({y_t} = 0|{{\bf{h}}_t})} \right]}.$$\end{document}

The click and non-click probabilities are calculated as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$P({y_t} = 1|{{\bf{h}}_t}) = {1 \over {1 + \exp \left( { - \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }\!_t} \oplus {{\overleftarrow {\bf{h}} }\!_t}} \right) + {{\bf{b}}_o}} \right)} \right)}}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$P({y_t} = 0|{{\bf{h}}_t}) = 1 - {1 \over {1 + \exp \left( { - \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }\!_t} \oplus {{\overleftarrow {\bf{h}} }\!_t}} \right) + {{\bf{b}}_o}} \right)} \right)}}.$$\end{document}

The optimization objective for parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_o}\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{b}}_o}\end{document} is:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign{ & {\left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)^*} = \arg {\min _{{{\bf{W}}_o},{{\bf{b}}_o}}}\left\{ - {1 \over T}\sum\limits_{t = 1}^T \left[ {w_t}{y_t}\log \left( {{1 \over {1 + \exp \left( { - \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }\!_t} \oplus {{\overleftarrow {\bf{h}} }\!_t}} \right) + {{\bf{b}}_o}} \right)} \right)}}} \right){\mathrm{}}\right.\right.\\ & \left.\left. + (1 - {w_t})(1 - {y_t})\log \left( {1 - {1 \over {1 + \exp \left( { - \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }\!_t} \oplus {{\overleftarrow {\bf{h}} }\!_t}} \right) + {{\bf{b}}_o}} \right)} \right)}}} \right)\right]\right\}.}$$\end{document}

Theorem 1 The Bi-LSTM model converges to a unique solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)^*}\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sigma ( \cdot )\end{document} is the sigmoid function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \lambda\end{document} is the regularization parameter, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left| {|{{\bf{W}}_o}} \right||_F^2\end{document} is the Frobenius norm:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign{ {\scr L}({{\bf{W}}_o},{{\bf{b}}_o},{\bf{H}}) = \;& {1 \over T}\sum\limits_{t = 1}^T {\left[ {{y_t}\log \sigma \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }\!_t} \oplus {{\overleftarrow {\bf{h}} }\!_t}} \right) + {{\bf{b}}_o}} \right)} \right.} \\ & \left. { + \;(1 - {y_t})\log \left( {1 - \sigma \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }\!_t} \oplus {{\overleftarrow {\bf{h}} }\!_t}} \right) + {{\bf{b}}_o}} \right)} \right)} \right] + \lambda \left| {|{{\bf{W}}_o}} \right||_F^2.}$$\end{document}

Corollary 1 *The optimized parameters enhance attention mechanism performance, where * \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_Q}\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_K}\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_V}\end{document} are learnable parameter matrices:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{Q}},{\bf{K}},{\bf{V}} = {\bf{H}}{{\bf{W}}_Q},{\bf{H}}{{\bf{W}}_K},{\bf{H}}{{\bf{W}}_V}\quad {{where}}\quad {{\bf{W}}_Q},{{\bf{W}}_K},{{\bf{W}}_V} \in {\mathbb R} {^{d \times d}}.$$\end{document}

The attention weights are computed as follows, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {d_k}\end{document} is the dimension of the key vector:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${{Attention}}({\bf{Q}},{\bf{K}},{\bf{V}}) = {\bf{H}} \cdot {{softmax}}\left( {{{{{\bf{W}}_Q}{\bf{W}}_K^ \top {{\bf{H}}^ \top }{\bf{H}}} \over {\sqrt {{d_k}} }}} \right){{\bf{W}}_V.}$$\end{document}

For detailed evidence, see the Appendix “Mathematical Theorems and Corollary Proofs”.

Multi-head self-attention mechanism

Capturing global dependencies: multimodal feature fusion and analysis with multi-head self-attention mechanism

• Static feature analysis in conventional models overlooks crucial interactions between different feature types (Salachoris et al., 2024), leading to accuracy degradation and increased maintenance overhead (Van Thieu & Mirjalili, 2023; Rahman et al., 2023).
• The multi-head self-attention mechanism enables parallel processing of multimodal features, facilitating dynamic importance adjustment and comprehensive feature fusion. This approach particularly enhances recommendation precision for elderly users by capturing their unique shopping patterns.

Figure 3 illustrates the architecture of our multi-head attention mechanism for personalized recommendation. The model integrates scaled dot-product attention with multi-head structure to capture global feature dependencies, specifically designed to enhance recommendation accuracy for elderly users by processing different user characteristic sequences in parallel.

Prediction results of item clicks in the recommendation section for elderly online shopping users.

Capturing global dependencies with multi-head self-attention mechanism

For elderly-oriented shopping recommendations, we employ multi-head self-attention to analyze multimodal user behaviors. The mechanism integrates three key components: user features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{U}}\end{document} , item features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{P}}\end{document} , and temporal behavior patterns \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{H}} = { {{\bf{h}}_t}} _{t = 1}^T\end{document} from Bi-LSTM:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{X}} = \left[ {\matrix{ {\bf{U}} \hfill \cr {\bf{P}} \hfill \cr {\bf{H}} \hfill \cr } } \right]$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{U}} \in {\mathbb R }{^{{d_u}}}\end{document} captures user preferences, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{P}} \in {\mathbb R }{^{{d_p}}}\end{document} represents item characteristics, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{H}} \in {\mathbb R} {^{T \times {d_h}}}\end{document} encodes temporal patterns, and the combined feature matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{X}} \in {\mathbb R}{^{({d_u} + {d_p} + T \cdot {d_h}) \times d}}\end{document} . The feature transformation for attention computation is:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{Q}} = {\bf{X}}{{\bf{W}}_Q},\quad {\bf{K}} = {\bf{X}}{{\bf{W}}_K},\quad {\bf{V}} = {\bf{X}}{{\bf{W}}_V}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_Q},{{\bf{W}}_K},{{\bf{W}}_V} \in {\mathbb R} {^{d \times d}}\end{document} are learnable weight matrices for query, key, and value transformations respectively, adapting to elderly shopping patterns. The attention mechanism computes behavioral relevance through:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\mathrm{Attention}}({\bf{Q}},{\bf{K}},{\bf{V}}) = {\mathrm{softmax}}\left( {{{{\bf{Q}}{{\bf{K}}^ \top }} \over {\sqrt {{d_k}} }}} \right){\bf{V}}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {d_k}\end{document} is the dimension of the key vector, and the scaling factor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sqrt {{d_k}}\end{document} prevents attention scores from becoming too large in magnitude. To capture diverse shopping preferences of elderly users, we employ multiple attention heads:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\mathrm{MultiHead}}({\bf{Q}},{\bf{K}},{\bf{V}}) = {\mathrm{Concat}}({\mathrm{hea}}{{\mathrm{d}}_1},{\mathrm{hea}}{{\mathrm{d}}_2}, \ldots ,{\mathrm{hea}}{{\mathrm{d}}_h}){{\bf{W}}_O}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} h\end{document} is the number of attention heads and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_O}\end{document} is the output projection matrix. Each attention head processes specific behavioral aspects:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\mathrm{hea}}{{\mathrm{d}}_i} = {\mathrm{Attention}}({{\bf{Q}}_i},{{\bf{K}}_i},{{\bf{V}}_i})\; {\mathrm{where}}\;\; {{\bf{Q}}_i} = {\bf{XW}}_Q^i,{{\bf{K}}_i} = {\bf{XW}}_K^i,{{\bf{V}}_i} = {\bf{XW}}_V^i$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{W}}_Q^i,{\bf{W}}_K^i,{\bf{W}}_V^i \in {\mathbb R}{^{d \times {d_h}}}\end{document} are the weight matrices for the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} i\end{document} -th head. The integrated output combines all behavioral patterns:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{O}} = {\mathrm{Concat}}({\mathrm{hea}}{{\mathrm{d}}_1},{\mathrm{hea}}{{\mathrm{d}}_2}, \ldots ,{\mathrm{hea}}{{\mathrm{d}}_h}){{\bf{W}}_O}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_O} \in {\mathbb R} {^{(h \cdot {d_h}) \times d}}\end{document} is the output weight matrix. For optimizing elderly user recommendations, we define the loss function as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign{ L = \; & - {1 \over N}\sum\limits_{i = 1}^N \left [ {y_i}\log \left( {{1 \over {1 + \exp \left( { - \left( {{{\bf{W}}_O}\left( {{\mathrm{Concat}}({\mathrm{Attention}}({{\bf{Q}}_1},{{\bf{K}}_1},{{\bf{V}}_1}), \ldots ,{\mathrm{Attention}}({{\bf{Q}}_h},{{\bf{K}}_h},{{\bf{V}}_h}))} \right)} \right)} \right)}}} \right)\\ \right.\\ & \left. + \;(1 - {y_i})\log \left( {1 - {1 \over {1 + \exp \left( { - \left( {{{\bf{W}}_O}\left( {{\mathrm{Concat}}({\mathrm{Attention}}({{\bf{Q}}_1},{{\bf{K}}_1},{{\bf{V}}_1}), \ldots ,{\mathrm{Attention}}({{\bf{Q}}_h},{{\bf{K}}_h},{{\bf{V}}_h}))} \right)} \right)} \right)}}} \right)\right]}$$\end{document}

where N is the number of samples and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {y_i}\end{document} represents the actual click behavior. The click probabilities for recommendation items are computed as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$P({y_i} = 1|{{\bf{O}}_i}) = {1 \over {1 + \exp \left( { - {{\bf{O}}_i}} \right)}}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{O}}_i}\end{document} is the model output for the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} i\end{document} -th sample. The optimal parameters are found through:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\theta ^*} = \arg {\min _\theta }L(\theta )$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \theta\end{document} represents all model parameters, including \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_Q},{{\bf{W}}_K},{{\bf{W}}_V}\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_O}\end{document} .

Theorem 2 For elderly-oriented recommendations, the model converges to optimal parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\theta ^*}\end{document} that minimize:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign{ {\scr{L}}(\theta) = \; & \frac{1}{N} \sum_{i=1}^{N} [ y_i \log \sigma\left(\mathbf{O}_i \right) + (1 - y_i) \log \left(1 - \sigma\left(\mathbf{O}_i \right)\right)] \\ & + \lambda \sum_{i=1}^{h} \left( \left\| \mathbf{W}_Q^i \right\|_F^2 + \left\| \mathbf{W}_K^i \right\|_F^2 + \left\| \mathbf{W}_V^i \right\|_F^2 \right) }$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sigma ( \cdot )\end{document} denotes the sigmoid function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \lambda\end{document} is the regularization parameter, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left| {| \cdot } \right||_F^2\end{document} is the Frobenius norm.

Corollary 2. The attention weights adapt dynamically to elderly users’ behavior patterns through: [

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{A}} = {{softmax}}\left( {{1 \over {\sqrt {{d_k}} }}\left( {\left[ {\matrix{ {{{\bf{U}}_1},{{\bf{U}}_2}, \ldots ,{{\bf{U}}_{{d_u}}}} \hfill \cr {{{\bf{P}}_1},{{\bf{P}}_2}, \ldots ,{{\bf{P}}_{{d_p}}}} \hfill \cr {{{\bf{H}}_{11}},{{\bf{H}}_{12}}, \ldots ,{{\bf{H}}_{1{d_h}}}} \hfill \cr {{{\bf{H}}_{21}},{{\bf{H}}_{22}}, \ldots ,{{\bf{H}}_{2{d_h}}}} \hfill \cr \vdots \hfill \cr {{{\bf{H}}_{T1}},{{\bf{H}}_{T2}}, \ldots ,{{\bf{H}}_{T{d_h}}}} \hfill \cr } } \right]\left[ {\matrix{ {{{\bf{w}}_{Q1}},{{\bf{w}}_{Q2}}, \ldots ,{{\bf{w}}_{Qd}}} \hfill \cr } } \right]} \right){{\left( {\left[ {\matrix{ {{{\bf{U}}_1},{{\bf{U}}_2}, \ldots ,{{\bf{U}}_{{d_u}}}} \hfill \cr {{{\bf{P}}_1},{{\bf{P}}_2}, \ldots ,{{\bf{P}}_{{d_p}}}} \hfill \cr {{{\bf{H}}_{11}},{{\bf{H}}_{12}}, \ldots ,{{\bf{H}}_{1{d_h}}}} \hfill \cr {{{\bf{H}}_{21}},{{\bf{H}}_{22}}, \ldots ,{{\bf{H}}_{2{d_h}}}} \hfill \cr \vdots \hfill \cr {{{\bf{H}}_{T1}},{{\bf{H}}_{T2}}, \ldots ,{{\bf{H}}_{T{d_h}}}} \hfill \cr } } \right]\left[ {\matrix{ {{{\bf{w}}_{K1}},{{\bf{w}}_{K2}}, \ldots ,{{\bf{w}}_{Kd}}} \hfill \cr } } \right]} \right)}^ \top }} \right)$$\end{document}

] where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{A}}\end{document} represents the attention weight matrix that captures feature interactions, with components \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{U}}_i}\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{P}}i}\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{H}}{ij}}\end{document} representing user features, product features, and temporal behavior features respectively.

For detailed evidence, see the Appendix “Mathematical Theorems and Corollary Proofs”.

GATI-RS model

The space complexity of the algorithm is mainly: the input feature vector dimension is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} d\end{document} , and the hidden layer dimension is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} h\end{document} . The total number of parameters for each LSTM is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} 4(dh + hh + h)\end{document} . Therefore, the total number of parameters for Bi-LSTM is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} 2 \times 4(dh + hh + h)\end{document} . The attention mechanism parameters include the query, key, and value weight matrices, each with dimensions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} d \times d\end{document} . There are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} h\end{document} heads in the multi-head attention mechanism, each with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} 3 \times d \times d\end{document} weight matrix parameters. Thus, the total number of parameters is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} h \times 3 \times d \times d\end{document} . The number of parameters in the output layer depends on the dimension of the output vector and the hidden state. Assuming the dimension of the output vector is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} o\end{document} , the number of parameters in the output layer is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} hd \times o\end{document} . In summary, the algorithm’s space complexity is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(8d{h^2} + 3h{d^2} + hdo)\end{document} . The time complexity of the algorithm is mainly: the computation complexity for each time step is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(dh + {h^2})\end{document} , and the total number of time steps is T. Therefore, the time complexity of Bi-LSTM is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(T(dh + {h^2}))\end{document} . The attention mechanism computes the query, key, and value matrices, as well as the attention weights and output. The computation complexity for the query, key, and value matrices is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O({d^2})\end{document} , the attention weight computation complexity is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O({d^2} + d)\end{document} , and the attention output computation complexity is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O({d^2})\end{document} . The total time complexity is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(h \times ({d^2} + {d^2} + d)) = O(h{d^2})\end{document} . The computation complexity of the output layer is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(hdo)\end{document} . In summary, the algorithm’s time complexity is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(T(dh + {h^2}) + h{d^2} + hdo)\end{document} .

As shown in Table 2, the time and space complexities of different models vary significantly, each with its own strengths and trade-offs. The proposed GATI-RS model demonstrates a relatively balanced approach, with time complexity \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(T(dh + {h^2}) + h{d^2} + hdo)\end{document} , which is well-suited for large-scale and long-sequence data processing. However, this complexity might raise concerns for very large datasets in certain scenarios. In comparison, the Complex Network Model’s time complexity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(N \cdot \log (N))\end{document} is relatively more efficient in terms of computation for large-scale data but could lead to performance issues when data becomes too extensive.

Table 2: Comparison of time and space complexity of recommendation system models.

Regarding space complexity, the GATI-RS model, with a space complexity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O(8d{h^2} + 3h{d^2} + hdo)\end{document} , is optimized for handling multi-head self-attention mechanisms efficiently, which are crucial in recommendation tasks involving high-dimensional data. In contrast, the Complex Network Model’s space complexity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} O({N^2})\end{document} may cause memory bottlenecks when dealing with large-scale network data. While models like DURation, DiffRec, and NSRM excel in specific scenarios, they do not offer the same balance of time and space complexity as GATI-RS. Overall, the GATI-RS model offers a more adaptable and efficient solution when considering both time and space complexities for complex recommendation tasks.

Algorithm 1: Pseudo-code for item click prediction model.

Experiments and results

Dataset and experimental parameters

Dataset description (https://zenodo.org/records/14198398): In this study, we employ a comprehensive e-commerce dataset from Alibaba Cloud Tianchi platform, which is publicly available under the CC-BY-SA-NC 4.0 license. The dataset exhibits significant scale, as shown in Table 3, containing 9,526,571 interaction samples from 5,752,528 unique users and 31,431,181 items in the training set, and 3,555,419 samples from 2,242,720 users and 17,877,169 items in the test set. The data captures user interactions through a paginated interface where multiple items are displayed simultaneously, recording both exposure and click events in each session. The dataset features encrypted user-item interaction logs, including user demographics, item characteristics, and behavioral sequences. Notably, the item positions in the dataset are non-continuous due to the interspersion of other business content, reflecting authentic e-commerce scenarios.

Table 3: Comprehensive dataset description.

Computing infrastructure: The experiments were conducted on a system running Windows OS, with hardware configurations including an Intel Core i7-9700K CPU, 16 GB RAM, and an NVIDIA GeForce RTX 2070 GPU. We used Python 3.8 and TensorFlow 2.x for development and experiments.

Data preprocessing: Prior to model training, we conducted comprehensive data preprocessing. Numerical features were standardized using z-score normalization to address varying scales and improve model convergence. Categorical variables underwent one-hot encoding to prevent artificial ordinal relationships. Missing values were handled using different strategies based on feature characteristics: median imputation for skewed numerical distributions, mean imputation for normal distributions, and mode imputation with missing indicators for categorical features. This hybrid approach was chosen to preserve the underlying patterns in elderly user behavior while maintaining data quality. Additionally, we engineered temporal features by extracting cyclical components and creating time-window based aggregations to better capture usage patterns specific to elderly users.

The specific parameters used in the experiment are shown in Table 4. Through extensive experimentation and ablation studies, key hyperparameters were optimized: a learning rate of 0.001 was selected via grid search, and a batch size of 256 was chosen for optimal training efficiency. The model employs a three-layer neural network (128/256/128 neurons) with ReLU activation. To prevent overfitting, we implemented L2 regularization (0.0005), dropout (0.5), and early stopping after five epochs without improvement. The attention mechanism uses eight heads with dimension 64, while the Bi-LSTM component consists of two layers with 128 hidden units, effectively capturing temporal dependencies in the data.

Table 4: Detailed experimental parameter settings.

The sample size of the test dataset in this study is 2,242,720 users, and 17,877,169 items, with a total of 9,526,571 interactions in the training set. To analyze the importance of users in different age groups in online shopping, users are divided into young, middle-aged, and elderly stages. As shown in Table 5, elderly users account for 23.34% of our dataset, which is close to one-fourth. Although middle-aged users represent the largest proportion at 60.73%, elderly users remain a significant group for online shopping. This group often demonstrates higher purchasing power and loyalty, especially in areas such as health products, household items, and healthcare products. Therefore, optimizing for the preferences and needs of elderly users can significantly enhance user satisfaction and sales on e-commerce platforms.

Table 5: Proportion of users in different age groups.

Framework experimental results

To fully validate our model design and demonstrate the necessity of each component, we conducted comprehensive ablation studies. We compared three model variants: (1) the complete GATI-RS framework with multi-head self-attention, (2) GATI-RS without the attention mechanism, and (3) a baseline LSTM model. These experiments were conducted on both general users and elderly users to verify the effectiveness of our design choices.

Click prediction for online shopping users

As shown in Fig. 4A, we conducted control experiments with different optimization schemes to analyze the model performance. Using LSTM as the baseline model enables objective evaluation and clearly demonstrates the improvements achieved by the GATI-RS framework under various optimization algorithms. The experimental results confirm that the GATI-RS framework achieves superior performance across different optimization configurations. As shown in Fig. 4, we conducted control experiments with different optimization schemes to analyze model performance. Using LSTM as the baseline enables clear demonstration of performance improvements achieved by the GATI-RS framework under various optimization algorithms.

Performance evaluation of GATI-RS framework against baseline models in online shopping recommendation tasks.

In the experimental results (Fig. 4A), when excluding the multi-head self-attention mechanism, prediction accuracy decreases by approximately 10% compared to the complete GATI-RS framework. Using LSTM alone results in a 40% lower accuracy. While both optimization algorithms show improvements in click-through rate prediction for online shopping recommendations, the baseline LSTM model performs notably worse, with accuracy approximately 30% below the GATI-RS framework.

The loss function changes during training provide additional insights into model performance (Fig. 4B). The GATI-RS framework demonstrates superior optimization, with its loss function steadily decreasing and stabilizing. Models without multi-head self-attention show rapid initial loss decrease but stabilize at a higher value than GATI-RS, indicating limitations in capturing global dependencies. The LSTM baseline exhibits the slowest convergence and highest final loss, suggesting insufficient capability in complex pattern recognition and global information learning.

Figure 5 presents the Area Under the Curve (AUC) results for each model configuration. The GATI-RS framework, incorporating the multi-head self-attention mechanism, achieved the highest AUC score of approximately 0.92. This result demonstrates the model’s strong ability to distinguish between positive and negative classes, indicating superior predictive power and feature integration capabilities. In contrast, the GATI-RS framework without the multi-head self-attention mechanism scored an AUC of approximately 0.82, highlighting the importance of the self-attention mechanism in capturing global feature dependencies. The LSTM baseline model showed the lowest performance with an AUC of around 0.62, underscoring its limited capacity to handle complex patterns and global information. These results confirm that the GATI-RS framework with the multi-head self-attention mechanism outperforms other configurations in the normal user group, providing more accurate and relevant recommendations.

AUC comparison of ordinary user group models.

Click prediction for elderly online shopping users

As shown in Fig. 6, we conducted detailed comparative analyses focusing specifically on elderly online shopping users. From Fig. 6A, the model without multi-head self-attention achieves relatively high accuracy but remains approximately 7% below the GATI-RS framework. While this simplified model offers reduced computational complexity, its limited global information integration capability results in inferior pattern recognition performance. The baseline LSTM model shows the lowest accuracy, approximately 40% below GATI-RS, despite its theoretical advantages in temporal data processing. This significant performance gap likely stems from LSTM’s limitations in capturing global dependencies within complex patterns.

Performance analysis of GATI-RS framework in elderly-specific online shopping recommendations.

The loss function analysis (Fig. 6B) further validates these findings. The GATI-RS framework demonstrates rapid initial convergence and stable optimization. While the model without multi-head self-attention shows similar early-stage behavior, it stabilizes at a higher loss value, indicating reduced optimization capability. The LSTM baseline exhibits the slowest convergence and highest final loss, further evidencing its limitations in complex pattern recognition. These comparative analyses demonstrate that the integration of multi-head self-attention significantly enhances the GATI-RS framework’s capability in global information integration and pattern recognition, particularly for elderly users’ shopping behavior patterns. The framework establishes more comprehensive user and item representations, leading to superior click-through rate prediction accuracy in elderly-oriented recommendation systems.

Figure 7 displays the AUC results for the elderly user group across three model configurations: the GATI-RS framework with multi-head self-attention, the GATI-RS framework without the self-attention mechanism, and the LSTM baseline model. The GATI-RS framework achieved an AUC of approximately 0.89 for the elderly user group, showing that the model effectively captures complex behavioral patterns specific to elderly users. The model without the multi-head self-attention mechanism reached an AUC of 0.82, slightly lower than the complete GATI-RS framework. This result suggests that the self-attention mechanism enhances the model’s ability to integrate global dependencies, which is particularly beneficial for elderly users who may have unique and multifactorial behavior patterns. The LSTM baseline model demonstrated the lowest AUC of around 0.59, indicating its limitations in accurately predicting click behavior for elderly users. These results validate the advantage of the GATI-RS framework, especially with the inclusion of multi-head self-attention, in providing tailored and accurate recommendations for the elderly demographic.

AUC comparison for elderly user group.

Statistical validation and error analysis

To establish the statistical significance of our findings, we employed a systematic evaluation protocol. Each model configuration was evaluated through stratified k-fold cross-validation (k = 10) to ensure unbiased performance estimation. For statistical testing, we utilized paired-sample t-tests between model variants, with Bonferroni correction for multiple comparisons. Additionally, bootstrap resampling was performed to estimate robust confidence intervals for all performance metrics. The statistical results presented in Table 6 demonstrate several key findings.

Table 6: Statistical analysis of model performance.

The complete GATI-RS framework achieves the highest performance with 84.75% \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \pm\end{document} 1.23% accuracy and 0.92 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \pm\end{document} 0.01 AUC score. The relatively small confidence intervals indicate stable model performance across different initializations. When removing the multi-head self-attention mechanism, we observe a statistically significant performance drop (p < 0.005) to 74.75% \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \pm\end{document} 1.45% accuracy and 0.82 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \pm\end{document} 0.02 AUC. The wider confidence intervals in the attention-free variant suggest that the attention mechanism not only improves performance but also contributes to model stability. The LSTM baseline shows substantially lower performance with 44.75% \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \pm\end{document} 1.82% accuracy and 0.62 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \pm\end{document} 0.03 AUC, with the difference being highly significant (p < 0.001). The notably larger confidence intervals in the baseline model indicate higher sensitivity to initialization conditions, demonstrating the architectural advantages of our proposed framework in achieving both superior and more stable performance.

Comparison with recent research

To systematically evaluate the contribution of each model component, we conducted comprehensive ablation studies by removing key components and comparing performance metrics. Table 7 summarizes the experimental results across different model variants for both general and elderly user groups.

Table 7: Ablation study results.

The ablation study results in Table 7 clearly demonstrate the importance of each model component. For general users, the complete GATI-RS achieves the highest performance with an AUC of 0.92. When the multi-head self-attention mechanism is removed, the model’s AUC decreases to 0.82, highlighting the importance of attention in capturing global dependencies. The baseline LSTM model shows significantly lower performance with an AUC of 0.62. Similar trends are observed in the elderly user group, where the complete model achieves an AUC of 0.89, while the version without attention mechanism shows reduced performance with an AUC of 0.82. The baseline LSTM model demonstrates the lowest performance with an AUC of 0.59. This comprehensive comparison validates our architectural choices, particularly highlighting the crucial role of the attention mechanism in capturing complex user preferences and interaction patterns.

These ablation results provide strong empirical support for our model design decisions. To further validate the effectiveness of GATI-RS framework, we conducted comprehensive comparisons with several state-of-the-art recommendation system models, focusing on their performance in predicting item clicks across different user scenarios.

As shown in Table 8, we compare the performance of various recommendation system models. While these models were evaluated on different datasets, the comparisons remain meaningful as all models were tested on real-world, large-scale datasets within recommendation contexts. Each model addresses similar core challenges: capturing user preferences, item characteristics, and interaction patterns. The underlying recommendation problem remains consistent across studies, allowing us to examine performance improvements and methodological innovations across approaches.

Table 8: Accuracy comparison of recommendation system models.

The GATI-RS model achieves an accuracy of 84.75% on our dataset. The Multidimensional model (Yuan, Chen & Yang, 2023) reported 81.20% accuracy by integrating information entropy and classification features. The Exploiting Textual Reviews model (Oudah & Hussein, 2022) reached 83.00% accuracy using textual review information. The Lightweight Method (LBD) (Knyazev & Oosterhuis, 2023) achieved 82.50% accuracy with confidence measurements. The Multi-criteria Rating and Review model (Hasan, Ding & Cuzzocrea, 2022) obtained 79.00% accuracy using multi-dimensional ratings and reviews.

Discussion

With the rapid aging of society and the development of e-commerce, more elderly people are engaging in online shopping. However, they face increasing challenges during this process. This study analyzes the online shopping behavior of elderly users, revealing the unique problems they face and exploring methods to improve their shopping experience. While our GATI-RS model shows promise in addressing these challenges, several limitations and considerations should be noted.

• Elderly users face significant obstacles due to their unfamiliarity with complex interfaces and new technologies. Many elderly users feel confused and helpless when faced with cumbersome operational procedures and complex interface designs. Additionally, their declining ability to recognize information often leads to frustration during online shopping, reducing their shopping frequency and satisfaction. Our model’s current implementation, while effective, requires further optimization to reduce computational complexity and improve real-time performance in high-traffic scenarios.
• Elderly users are highly sensitive to discounts and promotions but lack the technical means to fully utilize these benefits. This often necessitates the help of younger people, increasing the complexity and dependence of their online shopping experience.
• Studies show that elderly users also face issues such as dynamic pricing and fake reviews during online shopping, further complicating their shopping experience. These social phenomena indicate that current e-commerce platforms’ design and functionality still fall short of fully meeting the needs of elderly users. Future work should focus on expanding the model’s capabilities to address these challenges while maintaining user privacy and ensuring robust performance across diverse elderly user groups.

Conclusion

This study examines the online shopping experience of elderly users on e-commerce platforms and proposes a click-through rate prediction-based recommendation system aimed at enhancing their shopping satisfaction and convenience. To address these challenges, we propose the GATI-RS model, which combines Bi-LSTM and multi-head self-attention mechanisms to capture the complex relationships between user behavior and item features, providing more personalized and accurate product recommendations for elderly users. Experimental results show that this model performs excellently in improving click-through rate prediction accuracy, not only enhancing the relevance of recommended content but also helping elderly users find the products they need more efficiently, significantly improving their online shopping experience. The GATI-RS model not only offers new ideas and methods for enhancing the online shopping experience of the elderly but also provides strong support for e-commerce platforms to optimize services and meet diverse user needs. Future work should focus on testing the model’s generalizability across different elderly user populations and exploring alternative feature integration methods to further enhance recommendation performance.

Appendix: mathematical theorems and corollary proofs

Theorem 1 Under the conditions of the Bi-LSTM model, we can find a unique solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)^*}\end{document} using gradient descent and backpropagation algorithms, such that the loss function L reaches a minimum, thereby maximizing the accuracy of click behavior prediction. Specifically, there exists a unique set of parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)\end{document} that satisfies:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign{ {\scr L}({{\bf{W}}_o},{{\bf{b}}_o},{\bf{H}}) = \; & {1 \over T}\sum\limits_{t = 1}^T {\left[ {{y_t}\log \sigma \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }_t} \oplus {{\overleftarrow {\bf{h}} }_t}} \right) + {{\bf{b}}_o}} \right)} \right.} \\ & {\mathrm{}}\left. { + \;(1 - {y_t})\log \left( {1 - \sigma \left( {{{\bf{W}}_o}\left( {{{\overrightarrow {\bf{h}} }_t} \oplus {{\overleftarrow {\bf{h}} }_t}} \right) + {{\bf{b}}_o}} \right)} \right)} \right] + \lambda \left| {|{{\bf{W}}_o}} \right||_F^2 } $$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sigma ( \cdot )\end{document} denotes the sigmoid function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \lambda\end{document} is the regularization parameter, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left| {|{{\bf{W}}_o}} \right||_F^2\end{document} is the Frobenius norm.

Proof 1 To prove the theorem, we first define the form of the loss function:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $L = - {1 \over T}\sum\limits_{t = 1}^T {\left[ {{y_t}\log \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}) + (1 - {y_t})\log (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}))} \right]} + \lambda \left| {|{{\bf{W}}_o}} \right||_F^2$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{h}}_t} = {\overrightarrow {\bf{h}} _t} \oplus {\overleftarrow {\bf{h}} _t}\end{document} is the output of the Bi-LSTM model.

Next, we need to compute the gradient of the loss function with respect to the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_o}\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{b}}_o}\end{document} . First, compute the gradient with respect to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_o}\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {{\bf{W}}_o}}} = - {1 \over T}\sum\limits_{t = 1}^T {\left[ {{y_t}{{\partial \log \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})} \over {\partial {{\bf{W}}_o}}} + (1 - {y_t}){{\partial \log (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}))} \over {\partial {{\bf{W}}_o}}}} \right]} + 2\lambda {{\bf{W}}_o}$\end{document}

Using the chain rule, this can be further decomposed as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})} \over {\partial {{\bf{W}}_o}}} = {1 \over {\sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})}} \cdot \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}) \cdot (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})) \cdot {\bf{h}}_t^T$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}))} \over {\partial {{\bf{W}}_o}}} = - {1 \over {1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})}} \cdot \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}) \cdot (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})) \cdot {\bf{h}}_t^T$\end{document}

Combining these two terms:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {{\bf{W}}_o}}} = - {1 \over T}\sum\limits_{t = 1}^T {\left[ {{y_t}(1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})) - (1 - {y_t})\sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})} \right]} {\bf{h}}_t^T + 2\lambda {{\bf{W}}_o}$\end{document}

Similarly, compute the gradient with respect to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{b}}_o}\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {{\bf{b}}_o}}} = - {1 \over T}\sum\limits_{t = 1}^T {\left[ {{y_t}{{\partial \log \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})} \over {\partial {{\bf{b}}_o}}} + (1 - {y_t}){{\partial \log (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}))} \over {\partial {{\bf{b}}_o}}}} \right]}$\end{document}

Again using the chain rule:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})} \over {\partial {{\bf{b}}_o}}} = {1 \over {\sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})}} \cdot \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}) \cdot (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}))$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}))} \over {\partial {{\bf{b}}_o}}} = - {1 \over {1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})}} \cdot \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}) \cdot (1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o}))$\end{document}

Combining these two terms:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {{\bf{b}}_o}}} = - {1 \over T}\sum\limits_{t = 1}^T {\left[ {{y_t}(1 - \sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})) - (1 - {y_t})\sigma ({{\bf{W}}_o}{{\bf{h}}_t} + {{\bf{b}}_o})} \right]}$\end{document}

These gradients are used for parameter updates through gradient descent:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{W}}_o^{(k + 1)} = {\bf{W}}_o^{(k)} - \eta {{\partial L} \over {\partial {{\bf{W}}_o}}}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{b}}_o^{(k + 1)} = {\bf{b}}_o^{(k)} - \eta {{\partial L} \over {\partial {{\bf{b}}_o}}}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \eta\end{document} is the learning rate.

In each iteration, gradient descent continuously adjusts the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_o}\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{b}}_o}\end{document} until the loss function L converges to a minimum. Therefore, we can eventually find a unique solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)^*}\end{document} that minimizes the loss functionL, thereby maximizing the accuracy of click behavior prediction.

Corollary 1 The optimized Bi-LSTM model parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)^*}\end{document} can serve as the input for the multi-head self-attention mechanism to further enhance the model’s ability to capture user behavior patterns and interest changes. Specifically, let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{H}} = { {{\bf{h}}_t}} _{t = 1}^T\end{document} be the output sequence of the Bi-LSTM, which can then be used as the input for the multi-head self-attention mechanism, expressed as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{Q}},{\bf{K}},{\bf{V}} = {\bf{H}}{{\bf{W}}_Q},{\bf{H}}{{\bf{W}}_K},{\bf{H}}{{\bf{W}}_V}\quad {{where}}\quad {{\bf{W}}_Q},{{\bf{W}}_K},{{\bf{W}}_V} \in {\mathbb R}{^{d \times d}}$$\end{document}

The attention weights and outputs are calculated through the self-attention mechanism:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign { {{Attention}}({\bf{Q}},{\bf{K}},{\bf{V}}) & = {{softmax}}\left( {{{{\bf{Q}}{{\bf{K}}^ \top }} \over {\sqrt {{d_k}} }}} \right){\bf{V}}\\ & {\mathrm{ }} = {{\ softmax}}\left( {{{{\bf{H}}{{\bf{W}}_Q}{{({\bf{H}}{{\bf{W}}_K})}^ \top }} \over {\sqrt {{d_k}} }}} \right){\bf{H}}{{\bf{W}}_V}\\ & {\mathrm{ }} = {{ softmax}}\left( {{{{\bf{H}}{{\bf{W}}_Q}{\bf{W}}_K^ \top {{\bf{H}}^ \top }} \over {\sqrt {{d_k}} }}} \right){\bf{H}}{{\bf{W}}_V}\\ & {\mathrm{ }} = {\bf{H}} \cdot {{softmax}}\left( {{{{{\bf{W}}_Q}{\bf{W}}_K^ \top {{\bf{H}}^ \top }{\bf{H}}} \over {\sqrt {{d_k}} }}} \right){{\bf{W}}_V}}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {d_k}\end{document} is the dimension of the key vector. Through the multi-head self-attention mechanism, the model can more effectively capture the global dependencies in user behavior, thereby further improving the accuracy of click prediction.

Proof 2 First, based on the optimized Bi-LSTM model parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)^*}\end{document} obtained in the theorem, we obtain the Bi-LSTM output sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{H}} = { {{\bf{h}}_t}} _{t = 1}^T\end{document} . To further enhance the model’s ability to capture user behavior patterns and interest changes, we input the Bi-LSTM output sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{H}}\end{document} into the multi-head self-attention mechanism.

Define the query matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{Q}}\end{document} , key matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{K}}\end{document} , and value matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{V}}\end{document} as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $${\bf{Q}} = {\bf{H}}{{\bf{W}}_Q},\quad {\bf{K}} = {\bf{H}}{{\bf{W}}_K},\quad {\bf{V}} = {\bf{H}}{{\bf{W}}_V}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_Q},{{\bf{W}}_K},{{\bf{W}}_V} \in {\mathbb R} {^{d \times d}}\end{document} are trainable weight matrices.

Next, compute the attention weights and outputs using the self-attention mechanism. First, calculate the dot product of the query matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{Q}}\end{document} and key matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{K}}\end{document} and scale by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sqrt {{d_k}}\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{{\bf{Q}}{{\bf{K}}^ \top }} \over {\sqrt {{d_k}} }} = {{{\bf{H}}{{\bf{W}}_Q}{{({\bf{H}}{{\bf{W}}_K})}^ \top }} \over {\sqrt {{d_k}} }}$\end{document}

Apply the softmax function to the scaled dot product results to obtain the attention weights:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{softmax}}\left( {{{{\bf{Q}}{{\bf{K}}^ \top }} \over {\sqrt {{d_k}} }}} \right) = {{softmax}}\left( {{{{\bf{H}}{{\bf{W}}_Q}{{({\bf{H}}{{\bf{W}}_K})}^ \top }} \over {\sqrt {{d_k}} }}} \right)$\end{document}

Multiply the attention weights by the value matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{V}}\end{document} to get the output of the self-attention mechanism:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{Attention}}({\bf{Q}},{\bf{K}},{\bf{V}}) = {{softmax}}\left( {{{{\bf{Q}}{{\bf{K}}^ \top }} \over {\sqrt {{d_k}} }}} \right){\bf{V}}$\end{document}

Expanding this further:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $\eqalign {{{Attention}}({\bf{Q}},{\bf{K}},{\bf{V}}) & = {{ softmax}}\left( {{{{\bf{H}}{{\bf{W}}_Q}{{({\bf{H}}{{\bf{W}}_K})}^ \top }} \over {\sqrt {{d_k}} }}} \right){\bf{H}}{{\bf{W}}_V} \\ & {\mathrm{ }} = {{ softmax}}\left( {{{{\bf{H}}{{\bf{W}}_Q}{\bf{W}}_K^ \top {{\bf{H}}^ \top }} \over {\sqrt {{d_k}} }}} \right){\bf{H}}{{\bf{W}}_V} \\ & {\mathrm{ }} = {\bf{H}} \cdot {{softmax}}\left( {{{{{\bf{W}}_Q}{\bf{W}}_K^ \top {{\bf{H}}^ \top }{\bf{H}}} \over {\sqrt {{d_k}} }}} \right){{\bf{W}}_V}}$\end{document}

Through the multi-head self-attention mechanism, we can capture the global dependencies in user behavior. This is because the self-attention mechanism can calculate the influence of other positions on each position in the sequence, thereby capturing richer temporal dependencies and behavior patterns.

Therefore, by combining the optimized Bi-LSTM model parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\left( {{{\bf{W}}_o},{{\bf{b}}_o}} \right)^*}\end{document} with the multi-head self-attention mechanism, we can further enhance the model’s ability to capture user behavior patterns and interest changes, thereby improving click prediction accuracy.

Theorem 2 Under the conditions of the multi-head self-attention mechanism, we can find a unique solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\theta ^*}\end{document} using gradient descent and backpropagation algorithms, such that the loss function L reaches a minimum, thereby maximizing the accuracy of click behavior prediction. Specifically, there exists a unique set of parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \theta\end{document} that satisfies:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\eqalign { {\mathrm{ }}{\scr L}(\theta ) = \; & {1 \over N}\sum\limits_{i = 1}^N {\left[ {{y_i}\log \sigma \left( {{{\bf{O}}_i}} \right)} \right.} {\mathrm{ }}\left. { \;+\; (1 - {y_i})\log \left( {1 - \sigma \left( {{{\bf{O}}_i}} \right)} \right)} \right] \\ & + \lambda \sum\limits_{i = 1}^h {\left( {\left| {|{\bf{W}}_Q^i} \right||_F^2 + \left| {|{\bf{W}}_K^i} \right||_F^2 + \left| {|{\bf{W}}_V^i} \right||_F^2} \right)}}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sigma ( \cdot )\end{document} denotes the sigmoid function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \lambda\end{document} is the regularization parameter, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left| {| \cdot } \right||_F^2\end{document} is the Frobenius norm.

Proof 3 To prove the theorem, we first define the form of the loss function:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $\eqalign{ L = \; & - {1 \over N}\sum\limits_{i = 1}^N {\left[ {{y_i}\log \sigma ({{\bf{O}}_i}) + (1 - {y_i})\log (1 - \sigma ({{\bf{O}}_i}))} \right]} \\ &+ \lambda \sum\limits_{i = 1}^h {\left( {\left| {|{\bf{W}}_Q^i} \right||_F^2 + \left| {|{\bf{W}}_K^i} \right||_F^2 + \left| {|{\bf{W}}_V^i} \right||_F^2} \right)} }$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{O}} = {{Concat}}({{hea}}{{{d}}_1},{{hea}}{{{d}}_2}, \ldots ,{{hea}}{{{d}}_h}){{\bf{W}}_O}\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \sigma ( \cdot )\end{document} denotes the sigmoid function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \lambda\end{document} is the regularization parameter, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \left| {| \cdot } \right||_F^2\end{document} is the Frobenius norm.

Next, we need to compute the gradient of the loss function with respect to the parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \theta\end{document} . First, compute the gradient with respect to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {{\bf{W}}_O}\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {{\bf{W}}_O}}} = - {1 \over N}\sum\limits_{i = 1}^N {\left[ {{y_i}{{\partial \log \sigma ({{\bf{O}}_i})} \over {\partial {{\bf{W}}_O}}} + (1 - {y_i}){{\partial \log (1 - \sigma ({{\bf{O}}_i}))} \over {\partial {{\bf{W}}_O}}}} \right]}$\end{document}

Using the chain rule, this can be further decomposed as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log \sigma ({{\bf{O}}_i})} \over {\partial {{\bf{W}}_O}}} = {1 \over {\sigma ({{\bf{O}}_i})}} \cdot \sigma ({{\bf{O}}_i}) \cdot (1 - \sigma ({{\bf{O}}_i})) \cdot {{Concat}}({{hea}}{{{d}}_1},{{hea}}{{{d}}_2}, \ldots ,{{hea}}{{{d}}_h})_i^T$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log (1 - \sigma ({{\bf{O}}_i}))} \over {\partial {{\bf{W}}_O}}} = - {1 \over {1 - \sigma ({{\bf{O}}_i})}} \cdot \sigma ({{\bf{O}}_i}) \cdot (1 - \sigma ({{\bf{O}}_i})) \cdot {{Concat}}({{hea}}{{{d}}_1},{{hea}}{{{d}}_2}, \ldots ,{{hea}}{{{d}}_h})_i^T$\end{document}

Combining these two terms:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {{\bf{W}}_O}}} = - {1 \over N}\sum\limits_{i = 1}^N {\left[ {{y_i}(1 - \sigma ({{\bf{O}}_i})) - (1 - {y_i})\sigma ({{\bf{O}}_i})} \right]} {{Concat}}({{hea}}{{{d}}_1},{{hea}}{{{d}}_2}, \ldots ,{{hea}}{{{d}}_h})_i^T$\end{document}

Similarly, compute the gradient with respect to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{W}}_Q^i,{\bf{W}}_K^i,{\bf{W}}_V^i\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {\bf{W}}_Q^i}} = - {1 \over N}\sum\limits_{i = 1}^N {\left[ {{y_i}{{\partial \log \sigma ({{\bf{O}}_i})} \over {\partial {\bf{W}}_Q^i}} + (1 - {y_i}){{\partial \log (1 - \sigma ({{\bf{O}}_i}))} \over {\partial {\bf{W}}_Q^i}}} \right]} + 2\lambda {\bf{W}}_Q^i$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log \sigma ({{\bf{O}}_i})} \over {\partial {\bf{W}}_Q^i}} = {1 \over {\sigma ({{\bf{O}}_i})}} \cdot \sigma ({{\bf{O}}_i}) \cdot (1 - \sigma ({{\bf{O}}_i})) \cdot {\bf{X}}_i^T$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial \log (1 - \sigma ({{\bf{O}}_i}))} \over {\partial {\bf{W}}_Q^i}} = - {1 \over {1 - \sigma ({{\bf{O}}_i})}} \cdot \sigma ({{\bf{O}}_i}) \cdot (1 - \sigma ({{\bf{O}}_i})) \cdot {\bf{X}}_i^T$\end{document}

Combining these two terms:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${{\partial L} \over {\partial {\bf{W}}_Q^i}} = - {1 \over N}\sum\limits_{i = 1}^N {\left[ {{y_i}(1 - \sigma ({{\bf{O}}_i})) - (1 - {y_i})\sigma ({{\bf{O}}_i})} \right]} {\bf{X}}_i^T + 2\lambda {\bf{W}}_Q^i$\end{document}

The gradient calculations for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\bf{W}}_K^i,{\bf{W}}_V^i\end{document} are similar and omitted here.

Finally, we update the model parameters through gradient descent:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${\bf{W}}_O^{(k + 1)} = {\bf{W}}_O^{(k)} - \eta {{\partial L} \over {\partial {{\bf{W}}_O}}}$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} ${\bf{W}}_Q^{i(k + 1)} = {\bf{W}}_Q^{i(k)} - \eta {{\partial L} \over {\partial {\bf{W}}_Q^i}},\quad {\bf{W}}_K^{i(k + 1)} = {\bf{W}}_K^{i(k)} - \eta {{\partial L} \over {\partial {\bf{W}}_K^i}},\quad {\bf{W}}_V^{i(k + 1)} = {\bf{W}}_V^{i(k)} - \eta {{\partial L} \over {\partial {\bf{W}}_V^i}}$\end{document}

In each iteration, gradient descent continuously adjusts the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} \theta\end{document} until the loss function L converges to a minimum. Therefore, we can eventually find a unique solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} {\theta ^*}\end{document} that minimizes the loss function L, thereby maximizing the accuracy of click behavior prediction.

Supplemental Information

10.7717/peerj-cs.2707/supp-1Supplemental Information 1Result.

10.7717/peerj-cs.2707/supp-2Supplemental Information 2Supplementary Information.

10.7717/peerj-cs.2707/supp-3Supplemental Information 3Readme.