Recent years have witnessed unprecedented success achieved by deep learning models in the field of computer vision. However, their vulnerability towards carefully-crafted adversarial examples has also attracted the increasing attention of researchers. Motivated by the observation that adversarial examples are due to the non-robust feature learned from the original dataset by models, we propose the concepts of salient feature (SF) and trivial feature (TF). The former represents the class-related feature, while the latter is usually adopted to mislead the model. We extract these two features with coupled generative adversarial network model and put forward a novel detection and defense method named salient feature extractor (SFE) to defend against adversarial attacks. Concretely, detection is realized by separating and comparing the difference between SF and TF of the input. At the same time, correct labels are obtained by re-identifying SF. Extensive experiments are carried out on MNIST, CIFAR-10, and ImageNet datasets where SFE shows superior results in effectiveness and efficiency compared with state-of-the-art baselines. Furthermore, we provide an interpretable understanding of the defense and detection process. The code of SFE could be downloaded from ( https://github.com/haibinzheng/SFE).