The Master Key Filters Hypothesis: Deep Filters Are General

TL;DR

This paper shows that deep filters in depthwise separable CNNs remain general and transferable across different datasets and domains, challenging the idea that filters become more specialized in deeper layers.

Contribution

It provides evidence that deep filters in DS-CNNs are general, not specialized, and demonstrates their transferability across datasets and architectures.

Findings

Deep filters maintain generality across layers and domains.

Frozen filters from different datasets perform well in transfer learning.

Larger datasets improve the performance of transferred filters.

Abstract

This paper challenges the prevailing view that convolutional neural network (CNN) filters become increasingly specialized in deeper layers. Motivated by recent observations of clusterable repeating patterns in depthwise separable CNNs (DS-CNNs) trained on ImageNet, we extend this investigation across various domains and datasets. Our analysis of DS-CNNs reveals that deep filters maintain generality, contradicting the expected transition to class-specific filters. We demonstrate the generalizability of these filters through transfer learning experiments, showing that frozen filters from models trained on different datasets perform well and can be further improved when sourced from larger datasets. Our findings indicate that spatial features learned by depthwise separable convolutions remain generic across all layers, domains, and architectures. This research provides new insights into the nature of generalization in neural networks, particularly in DS-CNNs, and has significant implications for transfer learning and model design.