Abstract: Shift operation is an efficient alternative over depthwise separableconvolution. However, it is still bottlenecked by its implementation manner,namely memory movement. To put this direction forward, a new and novel basiccomponent named Sparse Shift Layer (SSL) is introduced in this paper toconstruct efficient convolutional neural networks. In this family ofarchitectures, the basic block is only composed by 1x1 convolutional layerswith only a few shift operations applied to the intermediate feature maps. Tomake this idea feasible, we introduce shift operation penalty duringoptimization and further propose a quantization-aware shift learning method toimpose the learned displacement more friendly for inference. Extensive ablationstudies indicate that only a few shift operations are sufficient to providespatial information communication. Furthermore, to maximize the role of SSL, weredesign an improved network architecture to Fully Exploit the limited capacityof neural Network (FE-Net). Equipped with SSL, this network can achieve 75.0%top-1 accuracy on ImageNet with only 563M M-Adds. It surpasses othercounterparts constructed by depthwise separable convolution and the networkssearched by NAS in terms of accuracy and practical speed.

References & Citations