Rethinking the Harmonic Loss via Non-Euclidean Distance Layers

TL;DR

This paper extends harmonic loss by exploring various distance metrics beyond Euclidean, demonstrating improved model performance, interpretability, and sustainability across vision and language tasks.

Contribution

It systematically evaluates non-Euclidean distance metrics in harmonic loss, revealing benefits in accuracy, interpretability, and environmental impact.

Findings

Cosine distances improve accuracy and reduce emissions in vision tasks.

Bray-Curtis and Mahalanobis enhance interpretability with varying efficiency.

Cosine-based harmonic losses stabilize training and improve representations in language models.

Abstract

Cross-entropy loss has long been the standard choice for training deep neural networks, yet it suffers from interpretability limitations, unbounded weight growth, and inefficiencies that can contribute to costly training dynamics. The harmonic loss is a distance-based alternative grounded in Euclidean geometry that improves interpretability and mitigates phenomena such as grokking, or delayed generalization on the test set. However, the study of harmonic loss remains narrow: only Euclidean distance is explored, and no systematic evaluation of computational efficiency or sustainability was conducted. We extend harmonic loss by systematically investigating a broad spectrum of distance metrics as replacements for the Euclidean distance. We comprehensively evaluate distance-tailored harmonic losses on both vision backbones and large language models. Our analysis is framed around a three-way evaluation of model performance, interpretability, and sustainability. On vision tasks, cosine distances provide the most favorable trade-off, consistently improving accuracy while lowering carbon emissions, whereas Bray-Curtis and Mahalanobis further enhance interpretability at varying efficiency costs. On language models, cosine-based harmonic losses improve gradient and learning stability, strengthen representation structure, and reduce emissions relative to cross-entropy and Euclidean heads. Our code is available at: https://anonymous.4open.science/r/rethinking-harmonic-loss-5BAB/.