Task-oriented Memory-efficient Pruning-Adapter

TL;DR

This paper introduces a task-oriented pruning-adapter method for large language models that balances training and inference efficiency without sacrificing accuracy on GLUE tasks.

Contribution

It proposes a novel pruning-adapter approach that improves memory efficiency and speeds up training while maintaining high accuracy, addressing limitations of existing adapters and pruning methods.

Findings

Achieves high memory efficiency during training and inference.

Speeds up training time significantly.

Maintains comparable accuracy on GLUE tasks.

Abstract

The Outstanding performance and growing size of Large Language Models has led to increased attention in parameter efficient learning. The two predominant approaches are Adapters and Pruning. Adapters are to freeze the model and give it a new weight matrix on the side, which can significantly reduce the time and memory of training, but the cost is that the evaluation and testing will increase the time and memory consumption. Pruning is to cut off some weight and re-distribute the remaining weight, which sacrifices the complexity of training at the cost of extremely high memory and training time, making the cost of evaluation and testing relatively low. So efficiency of training and inference can't be obtained in the same time. In this work, we propose a task-oriented Pruning-Adapter method that achieve a high memory efficiency of training and memory, and speeds up training time and ensures no significant decrease in accuracy in GLUE tasks, achieving training and inference efficiency at the same time.