The long-term consistent visual tracking of large-scale moving swarms of animals or autonomous moving robots (AMR) is extremely challenging when the three factors are involved: 1) similar appearance of animals or AMR, 2) frequent and unpredictable occlusions, and 3) non-linear maneuvers. When facing such difficulties, existing multiple object tracking (MOT) algorithms are prone to identity switches and suffer severe performance degredation. This letter addresses this issue by exploiting the group motion pattern behind the order parameter in collective motion studies and incorporating it into the MOT recursion. We present a rigorous mathematical proof that the order parameter bounds the affine transformation estimation error of inter-frame group maneuver. This allows us to extract and model local consistency in agent motions with guaranteed accuracy. To utilize detected consistent motion patterns, an outlier correction technique is developed to improve the robustness against data association errors. Finally, we perform a simulation case study on the swarm of 45 agents and an experimental study with 13 ground AMR. The comparison results show the effectiveness of the proposed method and its advantage over four existing MOT algorithms.