Semantic communication systems (SCS) have gained extensive attention with the advancement of Artificial Intelligence (AI), which transmits the data feature instead of the raw bits, whereby the communication efficiency can be substantially enhanced, e.g., via a neural network or encoder to convert from the massive user data to corresponding light-weight feature map. However, SCS can be vulnerable to adversarial noise when transmitting the feature data, which may mislead the downstream tasks at the receiver side, e.g., leading to misclassification due to the disturbed receiving information. In this paper, we investigate the overshadowing-based attacks by perturbing the physical signal with artificial adversarial noise during the semantic feature transmission. Specifically, we directly attack the waveform after the modulation of the feature bits, and conduct both the white-box and black-box attacks to evaluate the vulnerability. In our attack methods, we use the local transfer model to acquire the gradient details and provide the gradient-based strategy for generating the perturbation. The experiment results demonstrate that both white-box and black-box attacks can be a critical threat for SCS and significantly degrade the performance of downstream tasks.