Optical character recognition (OCR) of shopping receipts plays an important role in smart business and personal financial management. Many challenging issues remain in current OCR systems for text recognition of shopping receipts captured by mobile phones. This research constructs a multi-task model by integrating saliency object detection as a branch task, which enables us to filter out irrelevant text instances by detecting the outline of a shopping receipt. Moreover, the developed model utilized a deformable convolution so as to learning visual information more effectively. On the other hand, to deal with attention drift of text recognition, we propose a transformer-based decoupled attention network, which is able to decouple the attention and prediction processes in attention mechanism. This mechanism can not only increase prediction accuracy, but also increase the inference speed. Extensive experimental results on a large-scale real-life dataset exhibit the effectiveness of our proposed method.