Modified swarm-based metaheuristics enhance Gradient Descent initialization performance: Application for EEG spatial filtering

TL;DR

This paper introduces modified swarm-based metaheuristics to improve Gradient Descent initialization, significantly enhancing EEG spatial filtering and outperforming traditional optimizers in complex, multi-scale convex landscapes.

Contribution

The paper proposes a new optimization framework that modifies swarm-based algorithms like ICA and PSO to better initialize Gradient Descent in challenging multi-scale convex problems.

Findings

Improved EEG classification accuracy using the proposed method.

Enhanced EEG loss function fitness compared to baseline optimizers.

Outperforms baseline optimizers in CEC 2014 benchmark functions.

Abstract

Gradient Descent (GD) approximators often fail in the solution space with multiple scales of convexities, i.e., in subspace learning and neural network scenarios. To handle that, one solution is to run GD multiple times from different randomized initial states and select the best solution over all experiments. However, this idea is proved impractical in plenty of cases. Even Swarm-based optimizers like Particle Swarm Optimization (PSO) or Imperialistic Competitive Algorithm (ICA), as commonly used GD initializers, have failed to find optimal solutions in some applications. In this paper, Swarm-based optimizers like ICA and PSO are modified by a new optimization framework to improve GD optimization performance. This improvement is for applications with high number of convex localities in multiple scales. Performance of the proposed method is analyzed in a nonlinear subspace filtering objective function over EEG data. The proposed metaheuristic outperforms commonly used baseline optimizers as GD initializers in both the EEG classification accuracy and EEG loss function fitness. The optimizers have been also compared to each other in some of CEC 2014 benchmark functions, where again our method outperforms other algorithms.