# TransNeural: An Enhanced-Transformer-Based Performance Pre-Validation Model for Split Learning Tasks

## TL;DR

This paper introduces TransNeural, a transformer-based model that improves performance estimation in split learning tasks by accounting for data distribution and resource variations.

## Contribution

The novelty lies in using transformers to model device data similarities and automatically learn relationships between latency, convergence, and system variables in split learning.

## Key findings

- TransNeural improves latency estimation accuracy by 9.3% compared to traditional methods.
- Convergence estimation accuracy is improved by 22.4% using the proposed algorithm.
- The model accounts for deviations in user-reported data during estimation.

## Abstract

While digital twin networks (DTNs) can potentially estimate network strategy performance in pre-validation environments, they are still in their infancy for split learning (SL) tasks, facing challenges like unknown non-i.i.d. data distributions, inaccurate channel states, and misreported resource availability across devices. To address these challenges, this paper proposes a TransNeural algorithm for DTN pre-validation environment to estimate SL latency and convergence. First, the TransNeural algorithm integrates transformers to efficiently model data similarities between different devices, considering different data distributions and device participate sequence greatly influence SL training convergence. Second, it leverages neural network to automatically establish the complex relationships between SL latency and convergence with data distributions, wireless and computing resources, dataset sizes, and training iterations. Deviations in user reports are also accounted for in the estimation process. Simulations show that the TransNeural algorithm improves latency estimation accuracy by 9.3% and convergence estimation accuracy by 22.4% compared to traditional equation-based methods.