In recent years, sketches have attracted much attention as a form of free media, and at the same time, the automatic coloring task of hand-drawn sketches has become one of the research hotspots in the field of vision. Sketch coloring is a challenging task, mainly due to: a) lack of inherent color information; b) the result of coloring is prone to cross-color and overflow; c) poor controllability of the coloring process. Based on this, we design a unique deep network that combines self-attention mechanism to alleviate this problem in this paper. Specifically, we first use the encoding and decoding structure of U-net to capture the delicate details in the sketch; a unique self-attention module is then integrated to automatically capture key areas of the input sketch, thereby improving color distribution modeling and contextual awareness, enhancing the results of automatic coloring. Experiments on a large-scale anime sketch dataset show that the proposed method exhibits significant performance improvement, capturing delicate details in sketches while maintaining faithful color restoration. In addition, our model also supports guided coloring based on provided clues, which providing users more controllability.