This paper proposes a design of a system for creating image similarity datasets which are necessary for testing the quality of supervised ranking algorithms. In particular, the main goal is to facilitate the creation of similar images rankings for given a imaginary dataset. The system was designed in a manner that involves user feedback in the process of creating the rankings. In each iteration of ranking construction, the query image and twelve candidates are presented to the user, who is intended to select the most similar one. Moreover, in order to accelerate the method convergence the approach based on simulated annealing is adapted. It initially chooses the images randomly from a dataset and in the later stages the images with rank rate above zero are chosen with certain probability.