The overall qualities of chestnuts are greatly affected by worm-eaten chestnuts, as they lead to a reduction of profits. Hence a fast, accurate and non-destructive method for sorting chestnuts is in great demand. In this study, the technology of machine vision was employed to grade chestnuts. A recognition method to identify worm-eaten chestnuts is presented based on the edge image of the wormhole. First, by applying a Sobel operator, binary images were gained through extracting the edges of the gray images, which were preprocessed with the denoising method of a Wiener filter. The binary image contained both the edge of the contour and the wormhole. The wormhole edges were obtained through separating the wormhole edge in light of the character that the gray degree of pixels in the neighborhood of the wormhole edge is lower than the threshold value set. Second, through morphological dilating and eroding, the denoised wormhole edge images were obtained. The connected component of the binary images of the wormhole edge were labeled, and the first three longest components were considered as feature values of the worm channel, which were then normalized. Third, the normalized data were input to a back-propagation (BP) neural network for training, where the hidden layer was 7. And only three steps were needed for iteration. When the model was utilized for prediction, the recognition rate was as high as 100%. The results showed that the proposed worm-eaten chestnut recognition method is accurate and fast, and it can provide a basis for on-line detection. Since the gray degree of the wormhole region is close to the normal region, this study used an enhanced boundary detection method to extract the edge of the worm channel solely, rather than the normally used region segmentation.