MedSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering

TL;DR

MedSteer introduces a training-free activation steering method for endoscopic image synthesis, enabling precise counterfactual generation with high fidelity and structural preservation, improving data augmentation for medical imaging tasks.

Contribution

It proposes a novel, training-free activation steering framework that generates counterfactual endoscopic images while maintaining structural integrity, outperforming existing inversion-based methods.

Findings

Achieves high concept flip rates of 0.800, 0.925, and 0.950 across experiments.

Significantly improves dye removal to 75% compared to baseline methods.

Enhances downstream polyp detection with higher AUC of 0.9755.

Abstract

Generative diffusion models are increasingly used for medical imaging data augmentation, but text prompting cannot produce causal training data. Re-prompting rerolls the entire generation trajectory, altering anatomy, texture, and background. Inversion-based editing methods introduce reconstruction error that causes structural drift. We propose MedSteer, a training-free activation-steering framework for endoscopic synthesis. MedSteer identifies a pathology vector for each contrastive prompt pair in the cross-attention layers of a diffusion transformer. At inference time, it steers image activations along this vector, generating counterfactual pairs from scratch where the only difference is the steered concept. All other structure is preserved by construction. We evaluate MedSteer across three experiments on Kvasir v3 and HyperKvasir. On counterfactual generation across three clinical concept pairs, MedSteer achieves flip rates of 0.800, 0.925, and 0.950, outperforming the best inversion-based baseline in both concept flip rate and structural preservation. On dye disentanglement, MedSteer achieves 75% dye removal against 20% (PnP) and 10% (h-Edit). On downstream polyp detection, augmenting with MedSteer counterfactual pairs achieves ViT AUC of 0.9755 versus 0.9083 for quantity-matched re-prompting, confirming that counterfactual structure drives the gain. Code is at link https://github.com/phamtrongthang123/medsteer