Using multiple query aspects to build test collections without human relevance judgments

M Efron - European Conference on Information Retrieval, 2009 - Springer
European Conference on Information Retrieval, 2009Springer
Collecting relevance judgments (qrels) is an especially challenging part of building an
information retrieval test collection. This paper presents a novel method for creating test
collections by offering a substitute for relevance judgments. Our method is based on an old
idea in IR: a single information need can be represented by many query articulations. We
call different articulations of a particular need query aspects. By combining the top k
documents retrieved by a single system for multiple query aspects, we build judgment-free …
Abstract
Collecting relevance judgments (qrels) is an especially challenging part of building an information retrieval test collection. This paper presents a novel method for creating test collections by offering a substitute for relevance judgments. Our method is based on an old idea in IR: a single information need can be represented by many query articulations. We call different articulations of a particular need query aspects. By combining the top k documents retrieved by a single system for multiple query aspects, we build judgment-free qrels whose rank ordering of IR systems correlates highly with rankings based on human relevance judgments.
Springer
Showing the best result for this search. See all results