On Achilles Heel of Some Optical Network Designs and Performance Comparisons

TL;DR

This paper uncovers potential vulnerabilities in optical network design comparisons, highlighting that heuristic solutions validated on small instances may not generalize, risking inaccurate performance assessments in large-scale scenarios.

Contribution

It identifies a critical flaw in current evaluation methods of optical network designs, emphasizing the need for solution quality verification beyond small-scale tests.

Findings

Heuristic methods may not scale well to realistic scenarios.

Performance comparisons based solely on heuristics can be misleading.

The paper demonstrates the impact of overfitting in network design evaluations.

Abstract

This non-conventional paper represents the first attempt to uncover a possible vulnerability in some proposals for optical network designs and performance comparisons. While optical network designs and planning lie at the heart of achieving fiber capacity efficiency and/or operational efficiency, its combinatorial nature makes it computationally hard to reach optimal solutions for realistic scenarios. Therefore, the well-established way that have been taken for granted by not-so-small number of research papers is that an optimization model based on mixed integer linear programming (MILP) is first proposed and then due to the intractability of such combinatorial model, an heuristic algorithm is offered as an approximation. The solution-quality comparison between the MILP and heuristic is then carried out on small-scale instances including topologies and traffic tests to verify the efficacy of the proposed heuristic and the next step is to use such allegedly verified heuristic for optical network designs of realistic scenarios. This approach may nevertheless leave a critical vulnerability as there is no guarantee that one performs well in small tests will generalize adequately for large-scale cases, a common pitfall widely referred as the peril of extrapolation and/or overfitting. Besides, it is not uncommon that in some research works, for benchmarking purpose, the comparison between a new design proposal whose performance is obtained from on one heuristic and a reference design based on another heuristic is carried out. As the result of missing solution quality check, such performance comparison relied merely on heuristic solutions may be equally vulnerable as its results can be distorted and thus, be far from the possibly achieved zones. In this work, we pinpoint those issues and provide a realistic case study to highlight and demonstrate the impact of such vulnerabilities.