Because of the huge amount of traffic generated today, classifying different applications through traffic classification has become a difficult task. However, categorization of network traffic using machine learning suffers from a shortage of tagged traffic data and difficulty in feature selection, as is the case in many applications in the real world. Aiming at solving this problem, this paper proposes a method based on particle swarm optimization (PSO) to do feature selection, combined with machine learning for classification, and the accuracy of classification is used as the fitness of PSO. This experiment uses the ISCXVPN2016 public dataset for experimental study. In addition to the experimental accuracy of 97.71% when the PSO algorithm is used in combination with decision trees, the accuracy of the traditional support vector machines (SVM) algorithm increases by 27.96% when combined with PSO.