Violence behavior detection has played an important role in computer vision, its widely used in unmanned security monitoring systems, Internet video filtration, etc. However, automatically detecting violence behavior from surveillance cameras has long been a challenging issue due to the real-time and detection accuracy. In this brief, a novel multi-scale spatio-temporal network termed as MSTN is proposed to detect violence behavior from video stream. To begin with, the spatio-temporal feature extraction module (STM) is developed to extract the key features between foreground and background of the original video. Then, temporal pooling and cross channel pooling are designed to obtain short frame rate and long frame rate from STM, respectively. Furthermore, short-time building (STB) branch and long-time building (LTB) branch are presented to extract the violence features from different spatio-temporal scales, where STB module is used to capture the spatial feature and LTB module is used to extract useful temporal feature for video recognition. Finally, a Trans module is presented to fuse the features of STB and LTB through lateral connection operation, where LTB feature is compressed into STB to improve the accuracy. Experimental results show the effectiveness and superiority of the proposed method on computational efficiency and detection accuracy.