The relationship between altmetric score with received citations in Iranian pediatrics articles

Introduction

Evaluating the impact of scientific publications is carried out using various methods. “Scientific citations” are published for different sources, such as journals, books, articles, conference abstracts, and patents, and have been one of the most important evaluation tools for the impact of scientific publications. However, even though citations have various advantages, relying on the number of scientific citations can have disadvantages and there are various criticisms surrounding them. For example:

Consequently, there are weaknesses and existing deficiencies in methods based on citation only, and also the tendency towards using the online environment has become a means for the appearance of other criteria for scientific evaluation and analysis of the scientific process. Priem et al. states that today methods based on citations are not the only criteria for evaluation because citations only measures visible effects, while nowadays using Web 2 tools, circumstances have been provided for creating new measurement criteria, which make the inconspicuous results of research evident. Priem name it “altmetrics or alternative metrics” (Priem et al., 2012). Altmetric is a means for measuring the effect of scientific publications on the web, especially in social media, which has a more expansive audience. It is an easy access method with better capability for tracing the effects of publications in various forms of media (Piwowar, 2013).

Currently, Almetric.com is considered one of the most authentic databases for calculating the altmetric score of approximately five million articles. Briefly, in order to report the altmetric score, categorize and provide an up-to-date description of articles, this database considers the following sources:

Evaluating the effects of scientific publications through official channels using scientific citations and on the Web using altmetrics each has their advantages and disadvantages. Until now, using methods based on citations and altmetrics in various scientific communities has been discussed to a great extent, and much research has been carried out for evaluating the relationship between these two methods. Since the use of social media has increased during the past few years, article coverage using altmetric scores has also increased during these years, and as a result it is better that scientific citations and altmetric scores complement each other and be evaluated more for recent articles (Costas et al., 2014)

Evaluating the impact of scientific publications with citation- and alt-metric methods has been carried out thus far in various fields (Salajegheh, 2015), (Hassan & Gillani, 2016), (Costas et al., 2014), (Robinson-García et al., 2014). Pediatrics is among the topics of great regard in various countries, since this field, in addition to having an important role in promoting a country’s ranking in the World Health Organization (WHO), assures children’s wellbeing and is part of the country’s development infrastructure. Iranian scientific publication in pediatrics has approximately one percent of the world's pediatrics publications, and also includes more than one percent of all scientific publications in Iran during the past five years in the Web of Science database. According to the WHO statistics, Iran’s significant scientific activities and advances in the field of pediatrics has resulted in impressive development and improvement of health indices in this field (http://apps.who.int/gho/data/node.country.country-IRN).

The present study is an attempt to monitor the impacts of Iranian pediatric articles as one of the leading fields of scientific publication in Iran with citation-metrics and altmetric methods, and hence obtain the relationship between altmetric score and number of received citations. More importantly, by accurately studying the number of times articles are mentioned in sources and media involved with the altmetric score, their relationship with the number of article received citations will be considered, and finally the correlation between the number of article mentioned in these sources with the altmetric score will be evaluated.

Methods

This is a practical study of an analytical descriptive type, and the research methodology is scientometric. The literature included was research articles, review articles, and conference abstracts (due to their importance in this field and scientometric studies) in the field of pediatrics published from Iran during March 2010 to September 2016, as found by the Web of Science database. This time period was used due to the prevalence of social media use during these years.

The sample included 1332 articles, which had been retrieved with the following procedure:

Altmetric data was found using the altmetric.com database, which requires a unique identifier for articles. The following steps were undertaken for finding the altmetric data of all articles:

1.   All 1332 articles were initially added to Endnote and were evaluated using the DOI and PUBMED ID. Then, articles that had none of the aforementioned identifiers were searched through "find reference updates" button on Endnote to complete reference information and obtain their DOI or PMID.

2.   In cases where identifiers were not found by this method, searches were done in Pubmed, Crossref, doi, and Scopus in an attempt to obtain their DOI or PM ID.

3.   From 1332 articles, the aforementioned identifiers were obtained for 1138 articles. All 1138 articles having identifiers were searched using the Webometric Analysis software (v.2 beta student) and also altmetric.com API in order to find the altmetric score and details of the scores. A total of 256 articles had an altmetric score then the citations number of these 256 articles were outcomed using the HistCite software.

Data analysis was performed using SPSS v21 software. Descriptive and inferential statistics (Kolmogorov-Smirnov test and Spearman correlation) were calculated.

Results

Discussion

Articles published from Iran was about 1% of the published articles in the field of pediatrics in the world, and regarding the population of Iran (80 million; about 1% of the world population) it seems to be an adequate publication quantity. Regarding the initial research question, regarding received citations for Iranian pediatrics articles, our findings indicate that 1332 articles received 2879 citations (on average two citations), while over 40% of the articles received no citations, which suggests the unbalanced quality of articles. The findings indicate that among 1332 articles, only 256 had altmetric scores. The low number of articles with altmetric scores may be due to lack of coverage by the Altmetric.com database for many of the articles, such as those that did not have unique identifiers, coverage of only preferred and not all social media, and also difficultly in tracing and monitoring the impacts of the articles, due to the high amount of information found on the Internet.

The present findings indicate that there is a direct correlation between the altmetric score and number of received citations for articles. Therefore, it seems that altmetrics can also be considered as a tool alongside scientific citations for the evaluation of scientific publication impacts. The correlation between citations and altmetric scores has are been previously confirmed in various research, for example by Costas et al. (2014); Hassan & Gillani (2016); Robinson-García et al. (2014), and Salajegheh (2015).

The three top articles with the most number of citations were all written with the cooperation of more than five authors; two articles had international cooperation. Article that obtained half of the total altmetric score (502 from 1024 total score) was written with the cooperation of authors from around the world. Therefore, it seems that cooperative writing, especially international cooperation, can have a significant effect on the impact of scientific publications in the field of pediatrics in Iran. In a study by Erfanmanesh (2015), approximately half of the articles with the highest altmetric score were articles with international cooperation. Among the top ten articles with regard to the number of received citations and the obtained altmetric score, only one article was a duplicate publication.

The Iranian Journal of Pediatrics, Pediatric Nephrology and Pediatric Research had the highest number of publications; however the most citations were obtained by the Iranian Journal of Pediatrics, International Journal of Pediatric Otorhinolaryngology, and Childs Nervous System. JAMA Pediatrics, Iranian Journal of Pediatrics, and International Journal of Pediatric Otorhinolaryngology were also the top journals regarding altmetric score. That the Iranian Journal of Pediatrics and International Journal of Pediatric Otorhinolaryngology were the top best journals in evaluating the number of citations and altmetric score, could be a confirmation of the relationship between citations and altmetric score. On the other hand, with regard to the second and third ranking of journals, none of these were among the five journals receiving the highest number of citations and altmetric score, and it seems that the Iranian authors need to put greater effort in choosing appropriate journals to publish their works in the field of pediatrics.

Calculating the mean number of citations to article in each year shows that the years of 2010, 2011 and 2012 had the highest mean number of citations. Therefore, the highest altmetric scores obtained is approximately related to recent years, including 2016, 2013, and 2015. With regard to the fact that in the evaluated years, earlier years had higher citations and more recent years had higher altmetric scores, it could be stated that contrary to receiving citations, which are a long and time-consuming process, new publications have obtained higher altmetric scores. This issue can be due to the greater use of social media during recent years. On the other hand, it could be stated that new articles are discussed and cited in social media at a faster pace after publication and up-to-date topics have more advocates in social media. This issue was also discussed in the study by Erfanmanesh (2015). Therefore, regardless of citations, which are a better indication of the effectiveness of older articles, evaluation of new scientific publications may be better specified by altmetric score. Regarding the relationship between altmetric score and citations, which has been shown in this and previous studies, obtaining higher altmetrics scores by new publications could be a new and better outlook on the potential for these articles to obtain citations in the future. This would however require further investigation.

Findings indicate that pediatrics articles in Iran have been mentioned in 10 media platforms, including Mendeley, CiteULike, weblogs, mainstream media, Twitter, Reddit, Facebook, Pinterest, F1000 and Google Plus. Among these, the reads of articles in reference manager resources (Mendeley and CiteULike) is over two and half times that of articles in social media. The pediatrics articles from Iran were mentioned 918 times in social media, including Twitter, Facebook, weblogs, mainstream media, Pinterest, Reddit, Google Plus, and F1000. Over all social media platforms, Twitter alone had 76% of the mentions. In previous studies, the number of reads in Mendeley and mentions in Twitter were higher than the other sources (Araújo et al., 2015; Erfanmanesh, 2015; Hassan & Gillani, 2016; Robinson-García et al., 2014) and (Zahedi et al., 2014). The number of mentions to articles in Pinterest, Reddit, Google Plus and F1000 were very low, whereas the evaluation of their relationship with scientific citations was disregarded. In a study by Thelwall et al. (2013), the same instance occurred for Google Plus and Reddit.

Regarding the relationship between data sources used in altmetric score with received citations in Iranian pediatrics articles, findings show that the number of readers in Mendeley and the number of scientific citations obtained had a direct and significant relationship. In studies by Asadi et al. (2014); Bar-Ilan et al. (2012); Erfanmanesh (2015); Haustein et al. (2014); Li et al. (2012), and Zahedi (2014), the relationship between the number of Mendeley reads and citations was confirmed. However, in the present study there was no significant relationship observed between the readers of CiteULike and scientific citations. In studies by Asadi et al. (2014); Bar-Ilan et al. (2012), and Li et al. (2012), the relationship between readers of CiteULike and number of citations was significant.

The altmetric.com database does not involve the number of Mendeley reads in the altmetric score calculation algorithm because this database cannot access user profiles in Mendeley, and user tendency towards specific author, journal, publisher or organization in Mendeley is not clear for this database (https://www.altmetric.com/about-our-data/the-donut-and-score/). In addition, it seems that with regard to the direct correlation between the number of readers of articles in Mendeley and received citations, which was also confirmed in previous studies (Asadi et al., 2014; Bar-Ilan et al., 2012; Erfanmanesh, 2015; Haustein et al., 2014; Li et al. 2012), and (Zahedi, 2014)), in order to show the potential of Mendeley reads to represent the article’s impact, it is better to involve an altmetric score calculation algorithm. It should be considered that with a high number of users and article reads in Mendeley, an individual effect is quite difficult to measure the general trend of an article.

A significant relationship was not observed between the number of mentions in evaluated social media and the number of citations. Finally, for certainty, further investigations show that there is also no meaningful relationship between total mentions of articles in all social media and the number of citations. In a study by Xia et al. (2016), a significant relationship was found between the number of citations and mentions on Twitter. In addition, in a study by Thelwall et al. (2013), there was a significant relationship between citations and articles on Facebook, weblogs, and mainstream media.

In order to further evaluate the correlation, the number of mentions in social media with altmetric scores was also evaluated and it was specified that the number of articles mentioned on Twitter, Facebook, weblogs, mainstream media and Mendeley with altmetric score had a direct and significant relationship. This may confirm the fact that the general algorithm created for the altmetric score in altmetric.com database is accurate. However, this issue definitely requires further investigations at different points in time, due to the extensive area of social media, its increasing growth, and changes in society’s approval of them.

In order to compare the ranking of Iranian pediatrics articles to other articles published by the same journals, the ranking of 0.43 for Iranian articles is indicative that the status of articles published from Iran are in the upper half of articles in pediatrics journals.

Conclusion

Today, with the increasing orientation of the public and researchers to social media, due to easy access, a wide-ranging audience, expansive relationship with other specialists, sharing of scientific publications and related discussions, disregarding the effects of the web domain in the evaluation of scientific publications does not seem to be correct. Based on the current research findings related to the direct relationship between altmetric score and scientific citations, as also seen in previous research, it seems that alongside scientific citations, altmetrics can also assist in better evaluation of the impact of scientific publications, especially for newer publications. Altmetric covers various indices and a wide range of social media and is updated at a higher speed. However, scientific citations are time-consuming and they are affected by the publishing source, thus it seems that evaluation of a new document and also articles that are published in unfamiliar journals, such as journals not indexed in authentic databases, is better performed with altmetrics. On the other hand, the expansive realm of social media makes it much more difficult to manipulate the altmetric score rather than manipulating the number of citations.

In this research and previous research, among readable media, Mendeley and among social media, Twitter and Facebook, have the greatest share of article mentions. According to this research and previous studies, the number of readers, especially in Mendeley, has a direct relationship with the number of citations, and it is better to be involved in the altmetric score calculation algorithm. The relationship between mentions in social media involved in altmetric score and the number of citations also requires more extensive evaluation.

Finally, by considering the extensive realm of Web2, its massive amount of data, the quick growth of social media, appearance of new facilities and new social media, and changes in people’s orientation, presenting an appropriate strategy for calculating the altmetric score requires extensive research and should be kept up-to-date.

Data availability

Dataset 1: CSV file with article titles, authors, years, journals, PMID, DOI, Mendeley readers, CiteULike readers, and Twitter, Facebook, Blog, Wikipedia, Google Plus, Pinterest, and F1000 mentions. doi, 10.5256/f1000research.12020.d168908 (Nemati-Anaraki et al., 2017).