The neighborhood structure can significantly impact the effectiveness of image search, and fulfilling the reversibility of neighborhood may improve the image search quality. This paper proposes an effective and efficient scheme for reconstructing the symmetry relationship of k-nearest neighborhood (KNN). In particular, we design a verifying function to learn the prior knowledge of neighborhood reversibility among images. By exploiting the prior knowledge, the image search system will give higher rank to those images that satisfy the reversibility of KNN relationship with the query. In addition, we systematically investigate the sensitivity of neighborhood size on image search quality and propose an adaptive selection scheme for improving robustness of neighborhood reversibility learning methods. The extensive experimental results show that the proposed scheme remarkably improves the image search quality and give a comparable but more stable performance to the state-of-the-art method for various image datasets.