As Gibson (2014) states, economics is unusual among academic disciplines in the emphasis it places on publication in a narrow set of top journals. Publishing in these outlets has been shown to have a powerful influence on the direction of research in economics, the career paths of young researchers, the reputation and pay of economics scholars, the decision of funding agencies, and on the rankings of departments and universities (e.g. Hamermesh 2018, Serrano 2018, Gibson et al. 2017). However, the reputation of a publication venue does not necessarily go hand in hand with citation performance. To better understand the relation between them, we quantify and characterise how citation patterns differ across journal tiers: top five, second tier, and top field (Anauati et al. 2018). Do total received citations and their dynamics over time (i.e. life cycles) differ across journal tiers? If differences are to be found, do they vary across articles’ levels of success (as measured by citation counts) or fields of economics research?
We delve into this issue by exploring a dataset containing detailed citation and article characteristics data. We first collect data on yearly received citations of 6,083 research articles published between 1992 and 1996 in the top five economics journals, a sample of second-tier general interest journals, and a sample of top field journals. Specifically, we cover a time span of 23 years of citation counts for each article. Then, we classified each article into one of four fields of economics research (theory, applied research, applied theory, and econometric methods).
- Our main result is that citation patterns vary greatly across journal tiers. Our data suggest eleven clear-cut patterns. First, as expected, total citations received by a typical (as measured by mean and median citations) article in a top five journal tend to be higher than the ones received by non-top five outlets. Concretely, in top five journals, the mean (median) article receives 491 (217) citations, compared with 147 (51) citations for second tier journals, and 136 (51) citations for top field journals.
- Second, as Figure 1 shows, average and median yearly citations received by top five journal papers are higher than the ones second tier and top field journal papers received for every year since publication.
Figure 1 Yearly mean and median citations received across journal tiers and fields of economics research
Note: Mean and median citations are smoothed using five-year centred moving averages. Light orange lines show the trajectory of individual papers’ yearly citations.
- Third, as can be seen in Figure 1, the life cycles of articles across journal tiers also differ. Median articles published in non-top five journals reach a peak of yearly citations that is lower and earlier in time than that of top five journals. This suggests that articles in the top five journals not only receive more citations, but also that the citations are received for longer periods of time – that is, the life cycles of their citations are longer.
- Fourth, no matter which tier, citation patterns tend to be more favourable for applied and applied theory papers than for theory and econometric methods papers (as in Anauati et al. 2016; see also here).
- Fifth, as shown in Figure 2, there is a relative overlap in total citations – the less highly cited articles in the top journals, for instance those in the first decile of the distribution of citations, are easily bettered by median articles in less prestigious outlets. This is in line with the results of Hamermesh (2018), Stern (2013), and Oswald (2007).
Figure 2 Distribution of total citations by publication venue
Note: 1 was added to the total citations of each article to avoid dropping articles with no citations when plotting in logarithmic scale.
- Sixth, when focusing on interactions between articles’ success and fields of economics research, we find, as shown in Figure 3, that the distribution of total citations of applied, applied theory, and theoretical top five publications stochastically dominate (i.e. total citations are higher, no matter the articles’ success) than that of second-tier and top field journal papers. However, for econometric method papers, the distribution of total citations of field journals crosses the distribution of top five at the 99.37thpercentile. From the inset graphs within each panel of Figure 3, note that it can be observed that applied, applied theory, theory, and econometric methods papers in the first decile of the distribution of citations of top five journals have the same number of citations as articles in the 56th, 54th, 37th, and 27th(55th, 53th, 37th, and 28th) percentiles of the distribution of citations of second tier (top field) journals. This confirms the finding mentioned above – that is, less highly cited articles in top five journals are easily outperformed by median articles in non-top five journals.
Figure 3 Empirical quantile functions of articles’ total citations across journal tiers and fields of economics research
Note: 1 was added to the total citations of each article to avoid dropping articles with no citations when plotting in logarithmic scale.
- Seventh, in general, no matter the articles’ success, the distribution of total citations of top field journals is in general slightly above that of second tier journals, and only for highly successful papers does this revert. In contrast, the distribution of econometric method papers in top field journals stochastically dominates that of second-tier journals (see Figure 3).
- Eighth, in general, a non-successful top five article receives little more than four times as many cites as an equivalently non-successful second tier or top field article; while a successful top five article receives around 3.5 times as many cites as an equivalently successful second tier or top field article. This suggests that, at least as measured by received citations, the statement that an article in a top five journal should be valorised at least as much as five good publications outside these venues may be overestimated.
- Ninth, no matter the articles’ success, in relative terms, the overall magnitude of the difference in citations received by articles published in top five journals against the ones received by articles published in second tier or top field journals tend to be larger for applied theory papers and smaller for econometric methods papers. This is clearly seen in Figure 4 that plots percentage differences of total citations’ empirical quantile functions across journal tiers.
Figure 4 Percentage difference of empirical quantile functions of articles total citations across journal tiers
- Tenth, how relative differences vary across articles’ success differs greatly across fields of economics research. For applied articles the gap narrows very mildly as one moves toward more successful articles, whereas this narrowing is sharp for the case of econometric methods papers (even converging at high values) and applied theory papers (which shows a dramatic narrowing, but not strong enough to attain convergence). Notably, theory papers show the opposite pattern – the gap widens as one moves to highly successful papers (see Figure 4).
- Eleventh, in general, top field journals behave relatively similar to second tier journals in terms of citation patterns, econometric methods articles published in top field outlets being an exception (see Figure 4).
Conclusions
We find clear-cut descriptive facts that may help in the understanding of incentives behind important issues in economics (e.g. the direction of research in economics, the career paths of young researchers, the reputation and pay of economics scholars, inter alia). The most important is that citation patterns effectively vary greatly across journal tiers, affecting not only their total received citations but also how citations are distributed as articles grow older. We find that there is a relative overlap in total citations and that, in terms of total received citations, a (non-)successful article in a top five journal should be valorised as around 3.5 (4) times a (non-)successful article in a second tier or top field journal. Finally, we show that the way citation patterns differ across journal tiers is strongly associated with articles’ success and the field of economics research. For example, in all fields of economics research, except for theory, the more successful a top five article is, the relatively less successful it is when compared to an equivalent article published in non-top five venues; for theory papers, the opposite pattern is observed – successful top five articles are even relatively more successful when compared to non-top five ones.
References
Anauati, M V, S Galiani and R H Gálvez (2018), “Differences in citation patterns across journal tiers in economics,” NBER, Working paper 25101.
Anauati, M V, S Galiani and R H Gálvez (2016), “Quantifying the life cycle of scholarly articles across fields of economic research,” Economic Inquiry 54(2): 1339–1355.
Gibson, J, D L Anderson and J Tressler (2014), “Which journal rankings best explain academic salaries? Evidence from the University of California,” Economic Inquiry 52(4): 1322–1340.
Gibson, J, D L Anderson and J Tressler (2017), "Citations or journal quality: Which is rewarded more in the academic labor market?" Economic Inquiry 55(4): 1945–1965.
Hamermesh, D S (2018), “Citations in economics: Measurement, uses, and impacts,” Journal of Economic Literature 56(1): 115–56.
Oswald, A J (2007), “An examination of the reliability of prestigious scholarly journals: Evidence and implications for decision‐makers,” Economica 74(293): 21–31.
Serrano, R (2018), “Top5itis,” Working paper.
Stern, D (2013), “Uncertainty measures for economics journal impact factors,” Journal of Economic Literature 51(1): 173–89.
