Putting Google Scholar to the test: a preliminary study
Program: electronic library and information systems
ISSN: 0033-0337
Article publication date: 20 February 2007
Abstract
Purpose
To describe a small‐scale quantitative evaluation of the scholarly information search engine, Google Scholar.
Design/methodology/approach
Google Scholar's ability to retrieve scholarly information was compared to that of three popular search engines: Ask.com, Google and Yahoo! Test queries were presented to all four search engines and the following measures were used to compare them: precision; Vaughan's Quality of Result Ranking; relative recall; and Vaughan's Ability to Retrieve Top Ranked Pages.
Findings
Significant differences were found in the ability to retrieve top ranked pages between Ask.com and Google and between Ask.com and Google Scholar for scientific queries. No other significant differences were found between the search engines. This may be due to the relatively small sample size of eight queries. Results suggest that, for scientific queries, Google Scholar has the highest precision, relative recall and Ability to Retrieve Top Ranked Pages. However, it achieved the lowest score for these three measures for non‐scientific queries. The best overall score for all four measures was achieved by Google. Vaughan's Quality of Result Ranking found a significant correlation between Google and scientific queries.
Research limitations/implications
As with any search engine evaluation, the results pertain only to performance at the time of the study and must be considered in light of any subsequent changes in the search engine's configuration or functioning. Also, the relatively small sample size limits the scope of the study's findings.
Practical implications
These results suggest that, although Google Scholar may prove useful to those in scientific disciplines, further development is necessary if it is to be useful to the scholarly community in general.
Originality/value
This is a preliminary study in applying the accepted performance measures of precision and recall to Google Scholar. It provides information specialists and users with an objective evaluation of Google Scholar's abilities across both scientific and non‐scientific disciplines and paves the way for a larger study.
Keywords
Citation
Robinson, M.L. and Wusteman, J. (2007), "Putting Google Scholar to the test: a preliminary study", Program: electronic library and information systems, Vol. 41 No. 1, pp. 71-80. https://doi.org/10.1108/00330330710724908
Publisher
:Emerald Group Publishing Limited
Copyright © 2007, Emerald Group Publishing Limited