The life cycle of scholarly articles across fields of research

Sebastian Galiani, Ramiro Gálvez 10 June 2017

a

A

Many influential metrics used to assess researcher performance are based on citation counts, and so the better we understand citation behaviour, the better our understanding of scientific output (Bornmann and Daniel 2008). Little effort has been devoted to understanding and quantifying how article citations evolve over the years following an article’s publication, and how this varies across fields of research. Given that several influential citation metrics specify a limit to the number of years since publication, this behaviour should be taken into account when evaluating researchers across fields of research.

In a previous article, we showed that citations of articles published in the top-five economics journals have a clear-cut lifecycle that varies greatly across fields of economic research (applied, applied theory, theory, econometric methods) (Anauati et al. 2016). At that time, we proposed a methodology for identifying these cycles that took into consideration citation inflation (the fact that citations are now more common) and skewness in the distribution of citations. Based on these findings, in our follow-up research we delved further into the lifecycle of academic articles. This time we focused on differences across broader fields of research (Galiani and Gálvez 2017).

We chose to study fields of research in the social sciences, life sciences, physical sciences, mathematical studies, and health studies, with the aim of covering a broad range of patterns in citation trends. We analyse articles from the following disciplines: astronomy and astrophysics, biochemistry, biology, economics, finance, mathematics, medicine, physics, political science, psychology, sociology, and statistics. For each of these 12 fields of research, we chose five prestigious within-the-field general research journals and sampled articles following a strategy that ensured that our final sample would be representative in terms of periods-of-time and success as measured by total citation counts. Additionally, we made sure that our sampling strategy would not be affected by citation inflation.

This created a representative sample of 59,707 research articles published between 1985 and 2000. We then gathered Google citations of each, for every year starting from two years before publication to late 2015, more than 5 million citations.

Raw trends in citation counts

Figure 1 plots the share of total citations received up to two and five years after publication, relative to all citations received up to 15 years after publication, for each field of research. This is because two and five years are the periods usually considered when calculating journal impact factors.

Figure 1 Share of total citations received up to two and five years after publication relative to all citations received up to 15 years after publication

Figure 1 shows that citation dynamics vary greatly between fields of research. Disciplines regarded as successful in terms of citation counts (economics and finance) receive a relatively small proportion of their citations in the first years after publication. This suggests that an important factor in driving the success of articles in those fields is that they are cited for longer, rather than a surge in citations soon after their publication.

To better visualise these differences, Figure 2 plots the number of citations per year after publication for the mean paper (solid line) and the median one (dashed line) for each field of research. This figure illustrates the effect of citation inflation by differentiating estimates for articles published from 1985 to 1989 (1985-1989, in blue) and published 1995 to 1999 (1995-1999, in red).

Figure 2 Estimated annual trends in citations of the mean and median articles, by field of research

Notes: Estimates are calculated for articles published from 1985 up to 1989 and from 1995 up to 1999. Values are smoothed using five-year centred moving averages. Note that the y-axis scales vary across sub-figures.

It is clear from Figure 2 that trends differ. In disciplines such as physics, astronomy and astrophysics, biochemistry, and biology, the number of citations per year declines after a period of time. In other disciplines, such as economics, finance, mathematics, political science, sociology, and statistics, the curve of mean citations per year keeps rising.

Figure 2 shows two features that should be taken into account when analysing the evolution of citation behaviour across time, and across fields of research:

  • Curves for 1995-1999 always lie at higher values than the ones estimated for 1985-1989. This shows citation inflation, and that it has not occurred at the same rate in all fields of research.  
  • Mean citation curves lie at higher values than median citation curves. This implies  a strong positive skewness in the distribution of citations per year. Note that this asymmetry in the distribution of citations is common to all fields of research, and persists as the papers age. The strength of this trend seems to differ across fields, however.

Lifecycle of articles across fields of research

As curves plotted in Figure 2 may be rising because of citation inflation, we used quantile regression to identify the lifecycle of papers across fields of research. The specification meant it was possible to control for secular trends in citations (taking into account citation inflation) and allowed the analysis across different (conditional) levels of success of papers. Figure 3 shows the results of estimating the lifecycle of articles across fields of research. The purple curves correspond to lifecycles estimated using quantile regression. The green curves correspond to lifecycles as estimated using regressions.

Figure 3 Estimated lifecycles of citations of research articles obtained by regression analysis

Notes: Values are smoothed using five-year centred moving averages. Note that the y-axis scales vary across sub-figures.

Controlling for citation inflation, annual citations exhibit a life-cycle pattern across all disciplines, Nevertheless, there are differences between different fields of research:

  • The peak of annual citations is much higher for some disciplines than for others.
  • The peak in citations is not reached at the same time across fields and, after this peak is reached, annual citation values differ in the way that they decline.
  • Differences between quantile regression and OLS estimates also vary greatly across disciplines.

As many bibliometric indices are especially sensitive to highly successful articles, understanding the dynamics of highly cited articles is important in order to understand their potential drawbacks. Figure 4 shows the results of a similar analysis focusing on higher conditional quantiles of the response variable distribution. From this analysis, it is evident that, even for highly successful research articles, a life-cycle pattern is observed in all the disciplines covered by this study. Moreover, evidence again points to strong differences across disciplines.

Figure 4 Estimated life cycles of research articles obtained by regression analysis across fields of research for highly successful articles

Notes: Values are smoothed using five-year centered moving averages. Note that the y-axis scales vary across sub-figures.

Discussion and conclusions

Lifecycle patterns for annual citations exist in a broad range of disciplines, but vary greatly across fields of research. Social sciences such as economics, political science, and sociology have longer life cycles, with annual citations reaching a peak later than in other disciplines, and then declining gradually. There is a similar pattern for mathematics and statistics articles, although the number of citations per year is lower than in the social sciences. In  astronomy and astrophysics, biochemistry, biology, medicine and physics, citations are more concentrated in the first years after publication. As filtering articles by their age is a common practice when calculating evaluative metrics, these differences are not neutral, and should be taken into account when evaluating researchers or institutions.

References

Anauati, V, S Galiani and R H Gálvez (2016), "Quantifying the life cycle of scholarly articles across fields of economic research", Economic Inquiry 54 (2): 1339-1355.

Bornmann, L, and H D Daniel (2008), “What do citation counts measure? A review of studies on citing behaviour”, Journal of Documentation 64 (1): 45-80.

Galiani, Sebastian and Gálvez, Ramiro H. (2017), “The Life Cycle of Scholarly Articles Across Fields of Research.” Available at SSRN: https://ssrn.com/abstract=2964565 or http://dx.doi.org/10.2139/ssrn.2964565

a

A

Topics:  Education Frontiers of economic research

Tags:  citations, evaluation, scholarly research, journals

Professor of Economics, University of Maryland

PhD student in computer science at the Computer Science Department, FCEyN, Universidad de Buenos Aires

Events