Vision Transformer for femur fracture classification

TL;DR

This study demonstrates that Vision Transformers significantly improve femur fracture classification accuracy over CNNs, especially when combined with specialist diagnosis, using the largest annotated dataset of proximal femur fractures.

Contribution

The paper introduces a modified Vision Transformer approach for femur fracture classification, outperforming CNNs and demonstrating its effectiveness with the largest dataset and expert collaboration.

Findings

ViT correctly predicted 83% of test images

Average diagnostic improvement of 29% with ViT assistance

First successful application of ViT in sub-fracture classification

Abstract

In recent years, the scientific community has focused on the development of CAD tools that could improve bone fractures' classification, mostly based on Convolutional Neural Network (CNN). However, the discerning accuracy of fractures' subtypes was far from optimal. This paper proposes a modified version of a very recent and powerful deep learning technique, the Vision Transformer (ViT), outperforming CNNs based approaches and consequently increasing specialists' diagnosis accuracy. 4207 manually annotated images were used and distributed, by following the AO/OTA classification, in different fracture types, the largest labeled dataset of proximal femur fractures used in literature. The ViT architecture was used and compared with a classic CNN and a multistage architecture composed of successive CNNs in cascade. To demonstrate the reliability of this approach, 1) the attention maps were used to visualize the most relevant areas of the images, 2) the performance of a generic CNN and ViT was compared through unsupervised learning techniques, and 3) 11 specialists were asked to evaluate and classify 150 proximal femur fractures' images with and without the help of the ViT, then results were compared for potential improvement. The ViT was able to correctly predict 83% of the test images. Precision, recall and F1-score were 0.77 (CI 0.64-0.90), 0.76 (CI 0.62-0.91) and 0.77 (CI 0.64-0.89), respectively. The average specialists' diagnostic improvement was 29% when supported by ViT's predictions, outperforming the algorithm alone. This paper showed the potential of Vision Transformers in bone fracture classification. For the first time, good results were obtained in sub-fractures classification, with the largest and richest dataset ever. Accordingly, the assisted diagnosis yielded the best results, proving once again the effectiveness of a coordinated work between neural networks and specialists.