Attention Deficit/Hyperactivity Disorder (ADHD) is one kind of neurodevelopmental disorders common in children. Due to the complexity of the pathological mechanism, there is a lack of objective diagnostic methods up to now. This paper aimed to propose automatic ADHD diagnostic method using resting state functional magnetic resonance imaging (rs-fMRI) data with the spatio-temporal deep learning models. Unlike traditional methods, this paper constructed a deep learning method called 4-D CNN based on granular computing which were trained based on derivative changes in entropy, and can calculate granularity at a coarse level by stacking layers. Considering the structure of rs-fMRI as time-series 3-D frames, several models of spatial and temporal granular computing and fusion were proposed, including feature pooling, long short-term memory (LSTM) and spatio-temporal convolution. This paper introduced an approach to augment dataset which can sample one subject's rs-fMRI frames into several relatively short term pieces with a fixed stride. The public dataset of ADHD-200 Consortium was used to train and validate our method. And the results of evaluations showed that our method outperformed traditional methods on the dataset (accuracy: 71.3%, AUC: 0.80). Therefore, our 4-D CNN method can be used to build more accurate automatic assistant diagnosis tool of ADHD.