Predicting user preferences

The many metrics employed for the evaluation of search engine results have
not themselves been conclusively evaluated. We propose a new measure for a
metric’s ability to identify user preference of result lists. Using this measure,
we evaluate the metrics Discounted Cumulated Gain, Mean Average Precision
and classical precision, finding that the former performs best. We also
show that considering more results for a given query can impair rather than
improve a metric’s ability to predict user preferences.