Class-Imbalanced-Aware Adaptive Dataset Distillation for Scalable Pretrained Model on Credit Scoring

TL;DR

This paper introduces a novel framework combining dataset distillation and pretrained models to improve credit scoring on tabular data, addressing class imbalance issues for better scalability and performance.

Contribution

It proposes a new method that integrates class imbalance-aware dataset distillation with pretrained models, enhancing scalability and effectiveness in financial tabular data analysis.

Findings

Achieved a 2.5% improvement in AUC on financial datasets.

Demonstrated the effectiveness of imbalance-aware techniques in dataset distillation.

Enabled large pretrained models to better handle tabular credit scoring data.

Abstract

The advent of artificial intelligence has significantly enhanced credit scoring technologies. Despite the remarkable efficacy of advanced deep learning models, mainstream adoption continues to favor tree-structured models due to their robust predictive performance on tabular data. Although pretrained models have seen considerable development, their application within the financial realm predominantly revolves around question-answering tasks and the use of such models for tabular-structured credit scoring datasets remains largely unexplored. Tabular-oriented large models, such as TabPFN, has made the application of large models in credit scoring feasible, albeit can only processing with limited sample sizes. This paper provides a novel framework to combine tabular-tailored dataset distillation technique with the pretrained model, empowers the scalability for TabPFN. Furthermore, though class imbalance distribution is the common nature in financial datasets, its influence during dataset distillation has not been explored. We thus integrate the imbalance-aware techniques during dataset distillation, resulting in improved performance in financial datasets (e.g., a 2.5% enhancement in AUC). This study presents a novel framework for scaling up the application of large pretrained models on financial tabular datasets and offers a comparative analysis of the influence of class imbalance on the dataset distillation process. We believe this approach can broaden the applications and downstream tasks of large models in the financial domain.