Transformer-based methods have achieved some effectiveness in image deraining because they can use self-attention mechanisms to capture the global information of the image. In this paper, we find that most existing transformers usually overlook important semantic information in local regions obscured by rain when dealing with rain removal. Without considering the consistency between the quality and semantics of raindrop removal under different backgrounds, most existing methods can easily deviate from the details of a region. To address this issue, we propose an effective network, Semantic-Aware Transformer for Deraining (SADformer), which can assist a deraining model in learning rich and diverse priors encapsulated in a semantic segmentation model. We focus on incorporating semantic knowledge from three key aspects: First, we design a multiscale semantic information extraction module to extract different scale features from the semantic maps and integrate them into the network to address the challenges posed by variations in semantic complexity and raindrop scales across images. Second, we introduce a semantic information attention module that incorporates the semantic maps into the model, allowing it to take into account the semantic features of the image and capture local details that may be missed by transformers. Finally, to better visualise the potential images processed by the front-end network, we introduce a multi-scale convolution module. These convolution modules operate at different scales, facilitating the recovery of spatial details and texture information in the image. Extensive experimental results show that the proposed method achieves favourable performance.