Multiple object tracking suffers from many challenges including huge computation work, crowd scenes. In order to solve these problems, we proposed a novel online multiple object tracking algorithm based on recurrent neural networks (RNNs) and appearance model. Compared to traditional algorithms, the RNNs can handle the motion state of the target well because it is trained with a quantity of data extracted from real world scenes. In addition, RNNs is helpful to improve tracking speed because it predicts the trajectories of objects without complex appearance calculations. The appearance feature is significant for tracking, especially in crowed scenes. The appearance model is extracted by convolutional neural networks trained with MARS dataset which is more targeted for the multi object tracking. In order to balance the speed and accuracy of tracking, a novel simple decision method was proposed to decide which features should be used. Otherwise, the cascade matching is integrated into the data association to solve a lot of subproblems in tracking. The experimental evaluation shows our algorithm is fast and accurate.