J6: Jacobian-Driven Role Attribution for Multi-Objective Prompt Optimization in LLMs

TL;DR

This paper introduces J6, a Jacobian-based method for multi-objective prompt optimization in large language models, enabling interpretable, conflict-aware updates that improve adaptation by considering geometric interactions.

Contribution

J6 provides a novel Jacobian decomposition for multi-objective prompt tuning, allowing both interpretability and dynamic conflict-aware optimization strategies.

Findings

Enables both hard and soft update strategies based on Jacobian decomposition.

Provides insights into parameter attribution and task interference.

Demonstrates improved adaptation in multi-objective LLM tuning.

Abstract

In large language model (LLM) adaptation, balancing multiple optimization objectives such as improving factuality (heat) and increasing confidence (via low entropy) poses a fundamental challenge, especially when prompt parameters (e.g., hidden-layer insertions h and embedding modifications w) interact in non-trivial ways. Existing multi-objective optimization strategies often rely on scalar gradient aggregation, ignoring the deeper geometric structure between objectives and parameters. We propose J6, a structured Jacobian-based method that decomposes the gradient interaction matrix into six interpretable components. This decomposition enables both hard decision-making (e.g., choosing the dominant update direction via argmax) and soft strategies (e.g., attention-style weighting via softmax over J6), forming a dynamic update framework that adapts to local conflict and synergy. Moreover, the interpretable structure of J6 provides insight into parameter attribution, task interference, and geometry-aligned adaptation. Our work introduces a principled and extensible mechanism for conflict-aware prompt optimization, and opens a new avenue for incorporating structured Jacobian reasoning into multi-objective neural tuning.