Auto-focus technology is an important way to improve the accuracy of indentation diameter measurement system, intelligent and automate. This article describes the use of image processing method to achieve the auto-focus technologies of indentation diameter measurement system, at its core is to choose an appropriate image clarity evaluation function. After studying a number of image clarity evaluation functions is proposed the image clarity evaluation function which based on the vector mode and improved DCT transform. Experiments show that the proposed algorithm has a good unimodality, accuracy, stability, reliability and rapidity. Finally using COM Component Technology-based Matlab and VB mixed programming to ensure that algorithms and software can be design.