OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWeb

TL;DR

OrthoGeoLoRA introduces a geometric constraint to Low-Rank Adaptation, improving parameter-efficient fine-tuning of language models for social science concept retrieval with better performance and efficiency.

Contribution

It proposes OrthoGeoLoRA, a novel geometric reparameterization of LoRA that enforces orthogonality, enhancing fine-tuning effectiveness and stability in social science retrieval tasks.

Findings

Outperforms standard LoRA on social science concept retrieval benchmarks.

Enables more stable and efficient fine-tuning with fewer parameters.

Demonstrates improved ranking metrics in multilingual social science datasets.

Abstract

Large language models and text encoders increasingly power web-based information systems in the social sciences, including digital libraries, data catalogues, and search interfaces used by researchers, policymakers, and civil society. Full fine-tuning is often computationally and energy intensive, which can be prohibitive for smaller institutions and non-profit organizations in the Web4Good ecosystem. Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), reduces this cost by updating only a small number of parameters. We show that the standard LoRA update $\Delta W = BA^\top$ has geometric drawbacks: gauge freedom, scale ambiguity, and a tendency toward rank collapse. We introduce OrthoGeoLoRA, which enforces an SVD-like form $\Delta W = B\Sigma A^\top$ by constraining the low-rank factors to be orthogonal (Stiefel manifold). A geometric reparameterization implements this constraint while remaining compatible with standard optimizers such as Adam and existing fine-tuning pipelines. We also propose a benchmark for hierarchical concept retrieval over the European Language Social Science Thesaurus (ELSST), widely used to organize social science resources in digital repositories. Experiments with a multilingual sentence encoder show that OrthoGeoLoRA outperforms standard LoRA and several strong PEFT variants on ranking metrics under the same low-rank budget, offering a more compute- and parameter-efficient path to adapt foundation models in resource-constrained settings.