Color imaging in the hot rolled condition provides the better reaction of heavy rail on surface defects. Multiple linear CCD array were used for the multi-angle shot imaging of the thermal heavy rail. Uneven illumination and complex surface morphology are worsened by the difficulties in image stitching algorithm, making the whole surface of the heavy rail with distortion, interference and a growing number of suspected regions. An adaptive vector median filter for color images was proposed to the enable effective follow-up processing on the image. In RGB space, color image was decomposed, where the de-differentiated image pixel variance was introduced to extract a complete defect or the suspected areas. This was followed by the use of a variety of characteristic parameters in the suspected region for defect recognition based on SVM. Experiments show that this algorithm for the identification of the heavy rail surface defects is proven to be more rapid in testing the inclusions, cracks and oxide skin defects with a good promotional value.