Assessing and Improving Syntactic Adversarial Robustness of Pre-trained Models for Code Translation

TL;DR

This paper introduces CoTR, a novel approach to evaluate and enhance the syntactic adversarial robustness of pre-trained models in code translation, demonstrating improved robustness through adversarial training and data augmentation.

Contribution

The study proposes CoTR, a new method combining adversarial example generation and robustness enhancement techniques specifically for code translation models.

Findings

CoTR-A significantly reduces existing PTMs' performance.

CoTR-D improves the robustness and generalization of PTMs.

Evaluation on Java to Python datasets confirms effectiveness.

Abstract

Context: Pre-trained models (PTMs) have demonstrated significant potential in automatic code translation. However, the vulnerability of these models in translation tasks, particularly in terms of syntax, has not been extensively investigated. Objective: To fill this gap, our study aims to propose a novel approach CoTR to assess and improve the syntactic adversarial robustness of PTMs in code translation. Method: CoTR consists of two components: CoTR-A and CoTR-D. CoTR-A generates adversarial examples by transforming programs, while CoTR-D proposes a semantic distance-based sampling data augmentation method and adversarial training method to improve the model's robustness and generalization capabilities. The Pass@1 metric is used by CoTR to assess the performance of PTMs, which is more suitable for code translation tasks and offers a more precise evaluation in real world scenarios. Results: The effectiveness of CoTR is evaluated through experiments on real world Java to Python datasets. The results demonstrate that CoTR-A can significantly reduce the performance of existing PTMs, while CoTR-D effectively improves the robustness of PTMs. Conclusion: Our study identifies the limitations of current PTMs, including large language models, in code translation tasks. It highlights the potential of CoTR as an effective solution to enhance the robustness of PTMs for code translation tasks.