Recently, Tencent Youtu Lab has made an innovative breakthrough in the task of model compression, and proposed a Stripe-Wise Pruning algorithm (SWP) based on filter skeleton, refreshing the SOTA effect of filter pruning. Relevant papers (Pruning Filter in Filter) have been included in the Conference on Neural Information Processing Systems(NeurIPS 2020), a top international conference in the field of machine learning.
Figure 1 Differences between Stripe-Wise Pruning and several mainstream Pruning methods
Neural network has two attributes, structure and parameter, both of which are of great significance. This paper points out that the filter of neural network has a shape attribute in addition to the usual parameter attribute. Shape properties have previously been implicit in the parameters by training the parameters of each filter to obtain a different shape. The shape property of filter is of great significance. Filters with proper shape can have better performance even if the parameters are random.
Therefore, this paper uses a module named Filter Skeleton (FS) to explicitly learn the shape of filters (① in the figure). When the training is over, we can multiply FS back onto the parameters, so no additional parameters are introduced (② in Figure).
FIG. 2 Schematic diagram of PFF method flow
For parameters that are not on the skeleton, the whole stripe (stripe, 1*1 filter) is clipped by stripe method.
Specifically, the Filter can be equivalently transformed from Filter wise to stripe wise by calculating the sequence transformation through convolution (3). It can then be clipped using the normal filter pruning method (④ in Figure).
The innovations of this method include:
(1) In addition to the parameter attributes, the filter also has shape attributes, and shape attributes are of great significance.
(2) A filter skeleton module is proposed to learn the shape of the filter and guide the pruning of the model.
(3) By transforming ordinary Convolution into Stripe-Wise Convolution, the model after pruning is realized in a structured way.
The sequential pruning algorithm achieves the SOTA effect on CIFAR10 and ImageNet data sets.
