Quantitative deconstruction of citation impact indicators: Waxing field impact but waning journal impact

1.1 Background

In recent years, we have seen in several studies that worldwide impact levels in a number of subfields appeared to be on the increase (e.g. Adair and Vohra, 2003; Nederhof et al., 1999). This phenomenon, although by no means general, could be witnessed most clearly in the average field citation scores (FCSm) figures in several of those studies (e.g. Nederhof et al., 1999).

The producer of the Science Citation Indices, Thomson ISI, has categorized journals in Subject Categories, such as “Physics, Particles and Fields” or “Economics”. The average citation rate of all articles in these journals can be computed, and is called “mean field citation score”, or “FCSm” (Moed et al., 1995; see ). The FCSm is used as a reference level against which research performance is benchmarked (e.g. Moed et al., 1995; Nederhof and Van Wijk, 1999; Van Raan, 1996). Then, a higher average impact of journal articles in a subfield (a higher FCSm score) means that research groups operating in that field who produce work with a constant citation impact are seen as declining when their stable impact score is compared with the higher FCSm score. Even research groups who actually improved their average number of citations per paper (CPP) may then be perceived as being in decline. In most older studies using the FCSm indicator (and many recent studies as well), FCSm scores tended to be more or less stable over time (e.g. Moed et al., 1995). Usually, only when groups switched between research fields, or witnessed a change in their production of document types (that tend to have different impact levels), appreciable shifts in FCSm scores did appear.

An increase in the average field citation score (FCSm) is by no means an impossible or even an unlikely phenomenon. For example, if reference lists of articles increase only slightly on average combined with a stable percentage of recent references, or the percentage of recent articles in references increases somewhat, the impact of (recent) articles increases on average as a result. It is well‐known that subfields differ in impact levels, and this is even one of the principal reasons behind the normalization of impact scores by comparison with a citation average based on journals or subfields (e.g. Moed et al., 1995; Zitt et al., 2003). Nevertheless, until recently, large shifts in average citation rates were not frequently observed on the subfield level, or were mainly limited to subfields with already high impact levels.

Of course, it is theoretically possible that an observed increase in the impact of subfields is artifactual, for example due to a database effect or an error in computation. This would affect the normalized impact scores of evaluated research units. However, previous tentative investigations of this phenomenon in several subfields suggested that the increase in FCSm in these subfields is autonomous, and not an artifact of ISI Citation Index coverage of journals (Nederhof et al., 1999; see Adair and Vohra, 2003). Until now, a full and thorough investigation, in which citation impact indicators are disassembled, has not been conducted. The present study intends to fill this gap. We study the increase of FCSm as visible in the bibliometric results of two academic institutes active in veterinary and animal sciences (VAS) (see below).

1.2 Science indicators

In CWTS research performance studies, two international reference values are computed. A first value represents the mean citation rate of the journals in which the research unit has published (the mean journal citation score (JCSm)), taking into account both the type of paper (e.g. normal article, review, and so on), as well as the specific years in which the research unit's papers were published. To give an example, the number of citations received during the period 1995‐1998 by a letter published by a research unit in 1995 in journal X is compared to the average number of citations received during the same period (1995‐1998) by all letters published in the same journal (X) in the same year (1995). The latter score is the journal citation score (JCS) of the letter. Generally, a research unit publishes its papers in several journals rather than one. Therefore, we calculated a weighted average JCS indicated as JCSm, with the weights determined by the number of papers published in each journal. Self‐citations are excluded from the computation of JCSm. A unit U that has published two articles in journal Y in 1996 (JCS=3) and one letter in journal X in 1995 (JCS=0.3) obtains a JCSm of Σ(JCS)/ ΣN =(3+3+0.3)/(1+1+1) (where N designates the number of papers)or 6.3/3 is 2.1 citations per publication.

The second reference value presents the mean citation rate of the subfields (journal categories) in which the research unit is active (FCSm). Our definition of subfields is based on a classification of scientific journals into categories developed by ISI. In calculating FCSm, we used the same procedure as the one we applied in the calculation of JCSm, with journals replaced by subfields. In most cases, a research unit is active in more than one subfield (i.e. journal category). In those cases, we calculate a weighted average value, the weights being determined by the total number of papers the research unit has published in each subfield. Suppose that journal X belongs to subfield Z, and that all 1995 articles in subfield Z are cited 1.5 times on average in 1995‐1998, while journal Y belongs to subfield A where all 1996 letters are cited 0.6 times on average in 1996‐1998. Then, the unit U mentioned before obtains an FCSm score of Σ(FCS)/ΣN=1.5+1.5+0.6)/(1+1+1) or 1.2 citations per publication (where N designates the number of papers). If a paper appears in a journal that is classified in more than one subject category, the paper (and its citations) is distributed over the subject categories. Thus, a paper with seven citations published in a journal categorized in three subject categories is counted as a 0.33 publication (with 2.33 citations) in each subject category. As a reference value, the FCSm values of the three subfields are used, similarly divided by three.

Two important indicators compare the average number of citations to the oeuvre of a research unit (CPP) to the two international reference values, namely the corresponding journal and field mean citation scores (JCSm and FCSm, respectively), by calculating the ratio for both. Self‐citations are excluded in the calculation of the ratios CPP/FCSm and CPP/JCSm, to prevent that ratios are affected by divergent self‐citation behavior. To maintain a balanced result, self‐citations are removed from both the numerator and the denominator.

If the ratio CPP/FCSm is above 1.0, this means that the oeuvre of the research unit is cited more frequently than an “average” publication in the subfield(s) in which the research unit is active. FCSm constitutes a world subfield average in a specific (combination of) subfield(s). In this way, one may obtain an indication of the international position of a research unit, in terms of its impact compared to a “world” average. This world average is calculated for the total population of articles published in CI journals assigned to a particular subfield or journal category. If the ratio CPP/JCSm is above 1.0, the mean impact of a research unit's papers exceeds the mean impact of all articles published in the journals in which the particular research unit has published its papers (the research unit's journal set). We apply a statistical test to establish whether the average impact of a research unit's publication oeuvre (CPP) differs significantly from the average impact of all papers in the research unit's journal set (JCSm) or from the world subfield average (FCSm) in the subfield(s) in which the research unit is active The test is based on the negative binomial distribution (see Schubert and Glänzel, 1983). If a research unit has a citation per publication ratio (CPP) significantly above (below) the average field (FCSm) or journal citation score (JCSm), this is indicated in the tables by means of a “+” (“−”) symbol directly after the numerical value of the indicators CPP/FCSmand CPP/JCSm.

So‐called “block indicators” were computed, with equal sets of publication years and citation years (see Schubert et al., 1989). For example, for the publications appearing in 1994‐1998, citations were counted also in 1994‐1998.

4.1 Changes in FCSm within subfields

If the increase in FCSm is not to a large extent due to changes in preference between subfields, it may be related to changes in impact within subfields. An increase in FCSm might be either obtained by the addition of high impact journals to the subfield, or by the elimination of low impact journals (or by some combination of both). Research groups publishing in a stable set of journals containing neither the new high impact journals nor many of the eliminated low impact journals may then have a stable JCSm score, whereas their FCSm score increases. Of course, parallel results would be expected for both institutes across subfields.

To examine this hypothesis, we first identified the main subfields (in terms of number of publications) that showed the overall pattern at the level of institutes: for these, FCSm increased considerably, but JCSm to a much lesser extent, or even declined, while the JCSm/FCSm ratio declined substantially. Here, as before, only articles were considered. As Table IV shows, for WU this pattern of JCSm and FCSm changes was observed in three of their 15 main subfields, but these do not include their four most important subfields in terms of publications (Agriculture, Diary and Animal products, Veterinary Science, Genetics and; Heredity, and Nutrition and Dietetics).

For UU, a relatively minor change occurred in their main subfield Veterinary Science, where FCSm increased a little, but JCSm dropped, resulting in a JCSm/FCSm decline (see Table IV). In four other subfields, JCSm increased, but less steeply than FCSm. For both institutes, it is notable that six out of eight of the largest increases in FCSm occur in fields with an already high impact level, thus providing extra leverage on overall FCSm scores.

If we compare the results in subfields of both institutes, few parallel developments are visible. In the four main subfields listed in Table IV that UU and WU share (Immunology, Genetics and Heredity, Veterinary Science, and Endocrinology and Metabolism) under the listed conditions, only one parallel case is found. It seems that FCSm of articles does not generally increase with a concomitant falling behind of JCSm. It should be taken into account that both institutes differ in focus and publish differently across subfields. Even then, however, one might have expected, as visible for UU, a shift in the second major subfield of WU, Veterinary Science. Here, in contrast to UU findings, the WU results show stable JCSm, FCSm, and JCSm/FCSm figures. Only in one subfield, Immunology, both institutes do show parallel findings in accord with the hypothesis, at least for FCSm and JCSm/FCSm values. Therefore, we decided to examine this subfield in detail. Also, we took a closer look at the Genetics and Heredity subfield where FCSm as well as JCSm strongly increased for UU.

4.2 Changes in coverage of journals and FCSm

To allow a detailed examination of changes in journal coverage in both subfields, journals were classified into four categories according to their status as source journal for the citation indices (CI):

1. 1.

   “Dropped” journals (i.e. after initial CI coverage somewhere in 1989‐1997, the journal is no longer a CI source journal in subsequent years).

2. 2.

   “Whole period” journals (CI source journal each year during 1989‐1998).

3. 3.

   “Interrupted” journals (i.e. irregular CI coverage in 1989‐1998; at least one year not covered in between two periods of coverage).

4. 4.

   “New” journals (i.e. CI coverage of journal starts somewhere in 1990‐1998).

Some “dropped” journals may have reappeared as “new journals” after a change of name or a split‐up (e.g. Biochimica et Biophysica Acta, American Journal of Physiology). As both of these events frequently influence the coverage of topics and make‐up of journals, and thus may have an impact on the citation scores of the journal, we decided not to classify these journals under “whole period”. We computed the worldwide JCSm/FCSm scores for the four types of journals for all publications in each subfield, irrespective of UU and WU publication patterns. In this analysis, a journal is characterized by a single JCSm/FCSm score for each five‐year period (either 1989‐1993 or 1994‐1998), which covers all articles in that journal published during the five‐year period, as well as all citations to these articles received during that five‐year period.

It is clear that in the two subfields, both the dropped journals and the interrupted journals, have an impact below the subfield average, as evident from their JCSm/FCSm score (see Table V). The new journals have the highest average impact of the four categories in 1994‐1998, although in Immunology, the impact level of new journals that were covered already in 1989‐1993 was lower than average, whereas in Genetics and Heredity the new journals started with a very high average impact. By definition, none of the new journals obtained full five‐year impact scores in the first period (otherwise they would not have been “new”), and in many cases, papers of new journals were followed for considerably shorter periods. If the new journals reached their impact peaks only after several years, this may have contributed to a lower initial impact, as observed in one subfield (Immunology) in the first period.

However, in the other subfield (Genetics and Heredity), the initial impact of newly covered journals was higher in the earlier period than in the more recent one. Here, journals may have reached their impact peaks in a shorter period, or may have had top contributions in the first issues, which generated a high citation level from the start. In the two subfields, continuously covered journals declined a bit in impact. In both subfields, neglecting publication in newly covered journals may lead to a declining impact compared to the world subfield average. In Genetics and Heredity, the increase in impact of the “whole period” journals has not matched that of FCSm. Moreover, their article share dropped from 77 percent to 65 percent, whereas the article share of new journals increased from 12 percent to 32 percent. Note, however, that the impact of new journals can not explain the increase of FCSm in Immunology, as the impact of new journals is similar to that of the established, “whole period” journals, which, apparently, have kept up pace with the increased FCSm, and saw their article share decline from 86 percent to 82 percent. Thus, while new journals are important in both subfields, only in one subfield do they contribute substantially to differences in impact development of JCSm and FCSm. The next step is to assess the effects of journal change in the impact figures of the two universities (see Tables VI and VII).

Table VI shows the relative importance of the four types of journals for UU and WU across all their subfields. Also, a combined score for VAS in The Netherlands has been computed. Both UU and WU tend to publish the bulk of their articles in continuously covered journals (see Table VI). Publications in newly covered journals represent 8 percent‐10 percent of their total output, while about 4 percent appears in journals that were dropped by ISI. Publication in irregularly covered journals is of some importance only for UU, who publish frequently in the journal Veterinary Quarterly, which accounts for most of the publications in the “Interrupted” journal category.

To maximize reliability, results for all VAS articles are presented in Table VII. Then it becomes clear that on average, continuously covered ISI source journals (“whole period journals”) drop only slightly in impact between 1989‐1993 and 1994‐1998. The impact of irregularly covered journals (mostly Veterinary QuarterlyUU articles) drops precipitously, while in the most recent period, mostly relatively new journals with a low impact are selected as outlet. The impact of CI journals for which coverage is discontinued remains stable and below average. In 1994‐1998, few VAS papers appear in dropped journals, which effectively reduces the contribution of this category to the overall impact.

In general, impact of VAS journals declines in the major journal categories. Apparently, instead of selecting high impact new journals as outlet, relatively low impact new journals were chosen by VAS researchers. This lowers JCSm scores, and as a result, JCSm/FCSm scores. Following the same procedures as earlier in this article, we can compute that the combined effect of publishing changes between the two periods in dropped, new, and interrupted journals causes the JCSm/FCSm five‐year score of VAS to decrease by 4 percent (p < 0.05). This decline is largely due to the strongly increased number of VAS articles in both interrupted and new journals in 1994‐1998. Although the decline in JCSm/FCSm score due to this factor is by no means negligible, it still does not account for the major part of the observed variance in FCSm and JCSm scores on the level of articles. So, further study is needed. In the following section, results of a case study allow examination on an even more detailed level.

5.1 Overall impact scores

As a case study, we examine the performance of WU and UU in the subfield of Immunology. Here, the performance of both institutes declined significantly compared to the world subfield level (see Table VIII).

It is interesting to note that, even though many factors have been held constant, the JCSm and FCSm values for the two institutes differ considerably. For example, the JCSm for UU for 1994‐1998 is more than twice as high as that of WU in the same period. Although the FCSm value for both institutes is similar, but not equal, in 1989‐1993, the gap between the FCSm values of both institutes widens to almost a full point in 1994‐1998. While the difference in JCSm values may at least be partly due to a simple difference in the impact level of the UU and WU journals sets that are involved, the differences in FCSm values are less easily explained. The figures illustrate strikingly that it is dangerous to interpret the FCSm value, as constructed in the present set of performance indicators, as a simple indicator of the general impact level of a subfield.

The performance figures (CPP/FCSm) are about average for both institutes in 1989‐1993, but significantly below average in 1994‐1998. For both institutes, the FCSm value increases strongly: with 38 percent for WU and with 27 percent for UU. At the first blush, this seems to indicate that worldwide, Immunology articles are cited much more heavily in the recent period than in 1989‐1993. However, alternative explanations should be considered, as we shall see. First, we note that the JCSm value increases only moderately for UU (6 percent), while it even declines with more than a third for WU (−35 percent). In both cases, but strongly so for WU, the JCSm/FCSm indicator shows a considerable decline in the impact level of the journals. As the value for P shows, fractional counting has been applied, in which a paper in a journal that is classified in, for example, three subject categories is counted as 0.33 (P). Here, the FCSm value concerns only immunology citation scores.

5.2 The increase in FCSm scores

Especially the WU shows large and contrasting fluctuations in JCSm and FCSm over time. Therefore, we look at those results in more detail. Table IX presents the WU performance data for each year in the five‐year period (see column A), counting from −4 (1989, 1994 respectively) to 0 (1993, 1998 respectively). Columns B and C contain the number of publications for each year, while columns D and E present the matching FCS values for Immunology articles.

As Table IX shows, the oldest papers in both periods (respectively from 1989 (see column D) and 1993 (see column E) database years; indicated as year −4) are the most heavily cited ones, with an average of, respectively, 10.8 and 11.7 citations received in five years. Not surprisingly, the most recent papers in each period (respectively from 1993 and 1998; indicated as year 0) are hardly cited at all in the year of publication, the only year for which data have been collected.

If we compare the FCS data in columns D and E, an increase in FCS is visible. The FCS in 1994‐1998 is higher in each of the five years than the FCS score in corresponding years in 1989‐1993, although to a negligible extent for the 1991 and 1996 papers (indicated as year −2). On average, the FCS increases with 10.5 percent between 1989‐1993 (total: 26.4) and 1994‐1998 (total: 29.2).

Multiplying the number of publications (P) with the FCS scores renders the figures in columns headed by B*D and C*E. If these numbers are totaled and divided by the total number of articles in the corresponding periods, we obtain the FCSm scores already presented in Table VIII 5.2 for 1989‐1993 and 7.2 for 1994‐1998.

To isolate the effect of the change in FCSm score over time, the papers from one period can be weighted with the corresponding FCSm scores from the other period. If the WU papers from 1989‐1993 are weighted with the 1994‐1998 FCS weights, their average FCSm score would increase from 5.20 to 5.72, which accounts only for about a quarter of the 5.20 to 7.20 hike that was actually found (see column B*E).

Another factor is even more important. This concerns the distribution of papers across the five‐year periods. Older papers are weighted much more heavily on FCSm than more recent ones, as Table IX shows. Indeed, on the one hand, the WU has more papers in the first three years of the 1994‐1998 period (82 percent of its total output) than in the corresponding period of 1989‐1993 (only 63 percent of its total output). On the other hand, the WU counts only 18 percent of its papers in 1994‐1995, whereas it has 37 percent of its papers in the corresponding period of 1992‐1993. Thus, in the 1994‐1998 period, a much higher percentage of papers is published in “old” years with a high FCSm than in the 1989‐1993 period, which has an overrepresentation of papers in “young” years with a low FCSm value. As a result, when the 1994‐1998 papers are weighted with the 1989‐1993 FCS values, the FCSm increases substantially from 5.20 to 6.16 (see column C*D). Thus, there is a large effect of the distribution of articles over the period, accounting for nearly half of the observed FCSm increase.

However, only when we combine the multiplicative effects of increasing FCSm and distribution of articles over the period, the great increase in FCSm from 5.20 to 7.20 results. Similar factors are behind the less pronounced FCSm increase for UU.

The large power of the distributional effect can be illustrated by a hypothetical example. If another group had 20 Immunology papers published in 1993, and 20 also in 1994 (but none in other years), the FCSm for 1989‐1993 would amount to 4.40, whereas the FCSm for 1994‐1998 would have been 234.8, using the figures from Table IX, columns D and E. The resulting more than 53‐fold increase in FCSm would have been due for only 8 percent to an inherent increase in FCSm (p < 0.001), but for 92 percent to the distributional effect (p < 0.001).

5.3 The change in JCSm scores

However, although this does explain the observed increase in FCSm, it does not account for the meager trend results witnessed for the JCSm scores. If similar forces were active there as with FCSm, one would expect an increase of about 38 percent in JCSm for WU and of 27 percent for UU. However, as noted in , the JCSm score actually declines for WU, and it increases only modestly for UU.

Once again, we take a closer look at the most divergent case, the WU findings. Table X provides the distribution of JCSm scores of journals in which the WU published during 1989‐1993 and 1994‐1998. Note that JCSm scores involve whole counts instead of fractional counts, as used with FCSm scores, as publications need not be divided across subfields in JCSm scores. Notwithstanding the inclusion of a relatively large number of older publications, few articles from 1994‐1998 obtain a high JCS score (see Table X). Only one 1994‐1998 article obtains a JCSm higher than 3.5, whereas the earlier period counts seven articles surpassing this threshold. The normalized JCSm/FCSm scores show that nine articles in 1989‐1993 have appeared in journals with close to (five) or above average impact, versus just one in 1994‐1998. The bulk of the 1994‐1998 articles appears in journals with an impact that represents only 20 percent‐40 percent of the average value in the subfield. Apparently, the WU published its papers in 1994‐1998 in Immunology journals with both a much lower absolute and relative impact level than in 1989‐1993, resulting in the decrease witnessed in its JCSm score in 1994‐1998.

The average decrease in JCSm of the journal set in which WU published during 1989‐1998 was 11 percent between the earlier and the later period. This is considerably lower than the 35 percent decline in JCSm that was actually observed for the WU (see ). Therefore, the decrease in JCSm that is observed in Immunology is only for a minor part due to a decline in impact of the journals in which it publishes. The decrease in JCSm of WU is mostly due to a selection of lower impact journals in 1994‐1998 compared to 1989‐1993.