In this work, we aim to build a high efficient deep network to remove artifact of compressed image. The degeneracy and small receptive field problem might be caused by reducing the computational cost of convolutional network via general approaches, such as pooling features to low resolution space, reducing the width of network and using small size convolutional kernel. For the reasons, we propose a multi-scale reshuffling network to efficiently reduce the compression artifact of compressed images without degeneracy. We firstly present a reshuffling network which includes a downscaling and a upscaling reshuffling operation. The downscaling reshuffling periodically rearranges high resolution to low resolution space without any information loss. The upscaling reshuffling is the reverse transformation of downscaling reshuffling, which allows us reconstructing high resolution image from the low resolution features. A densely connected structure is applied to efficiently extract features without degeneracy. The low resolution representations is gradually recovered to the higher resolution spaces which leads to a multi-scale structure. Results show the effectiveness of the proposed method.