VoxEU Column Frontiers of economic research

World-leading economics research in Europe

This column measures the quality of university economics research in Europe and the UK by concentrating on within-journal rankings of influential articles. The UK and Europe are doing relatively well, and top-level research output is not concentrated at half a dozen world-famous institutions. Science-funding policies by European governments should reflect this diversity.

“Quality that is world-leading in terms of originality, significance and rigour…comparable to the best work in the field or sub-field whether conducted in the UK or elsewhere. Such work … has become, or is likely to become, a primary point of reference in its field or sub-field.” - Research Assessment Exercise 2008

Should European science funding be concentrated upon a tiny number of elite universities and departments? And in 2009, echoing the earlier concerns of Dreze and Estevan (2006), is European economics of world-class quality?

My instinct is that as academics we suffer from – and here I include myself – the following biases:

  • We tend to overestimate the importance of our own department;
  • we tend to overestimate the importance of our own sub-field of economics;
  • we tend to be badly informed about the relationship between journals’ identities and articles’ influence (it has become common, in my 30-year professional lifetime, to hear people focus on a tiny number of journal “labels” per se, and even sometimes to speak as though a publication in a place like the American Economic Review is an end in itself rather than important for its content or its contribution to human welfare);
  • we tend to be poorly informed about the latest flow of research and excessively influenced by out-of-date stock variables (such as, for example, the long-standing reputation of another department or a particular economist).

Here I collect data on the world’s most-cited articles over the period of the Research Assessment Exercise just completed in the UK, namely, from 2001-2008.1 I then calculate the proportion of articles from the UK and continental Europe. My aim is to build a measure of quality akin to that described in the quote above.

I use data from the ISI Web of Science produced by Thomson. It is probably the most widely used source of citations data.2 Citations are taken here as a proxy for the objective quality of an article (measured with the benefit of hindsight). I take the journals listed by the Helpman Committee in the recent ESRC Benchmarking Report on Economics in the UK.

For each journal, I searched on articles published between January 2001 and December 2008. Thus I included an extra year of publications after the official end of the RAE period (though this has no material effect on citations numbers). I used the rank-by-citations facility of the Web of Science to order these from the most-cited downwards.

It is perhaps worth emphasising that there is evidence that early citations numbers to an article are a good indicator of long-run citations numbers. See, for example, Adams (2005). In other words, if an article acquires few citations early on it is extremely unusual – of course there are occasional exceptions – for it ever to acquire a high number.
The key data are set out in Table 1 at the end of this column. It tells us the influential recent articles from UK economics and, in particular, where they lie in a world-ranking of influence.

To try to adjust for the fact that some journals attract a particularly large quantity of good articles, I allow different journals to have different numbers of articles in the key table – 50 for the American Economic Review, 10 for the Economic Journal, and so on. These cut-offs were chosen to try to be fair to the different journals. I attempted approximately to equalise the numbers of citations of the marginal excluded article.

The data reveal a substantial achievement for the UK relative to its size. Nevertheless, the UK numbers are far behind those for the (obviously much larger) US. A fuller statement of the findings is in Oswald (2009).

This column proposes a methodology that could be applied in most scientific disciplines. Someone who – unlike me – can identify the top 20 journals in mathematics, or anthropology, or chemistry, and can use the ISI Web of Science to do a citation count, can do this same exercise for those other scholarly disciplines. In my judgment, that comparison would be interesting, and would allow a sharp test of which parts of UK and European research are truly highly-ranked in the world.

Table 2 re-does the exercise for the rest of the European nations. It can be seen that the non-UK part of Europe contributes 56 out of the whole set of 450 world-leading papers.
Interestingly, in both the UK case and the continental European case, we find that large numbers of these unusually influential papers are coming from a wide range of institutions and countries.

Conclusions

This column examined objective data on the world’s most influential economics articles of 2001-2008, the UK Research Assessment Exercise period. My aim has been design a practical way to measure the quality of university research in Europe and the UK, particularly a means of identifying four-star, world-leading work. I would argue that this paper’s bibliometric approach is a way of coping with the serious problem – pointed out in different ways by Starbuck (2005) and Oswald (2007) – for national scientific evaluation that the elite journals publish many ‘poor’ articles, that is, ones that go on to have no impact. The paper does this by concentrating on within-journal rankings of influential articles.
UK economics comes out moderately well on my criterion. It produces 10% of the really important work. Continental Europe produces slightly more than this.

One of the interesting things to come out of the analysis is how widely spread the (truly) top-level research is in Europe. It is not all being produced by half a dozen world-famous institutions. Science-funding policies by European governments should reflect this diversity.

References

Adams, Jonathan. 2005. Early citation counts correlate with accumulated impact. Scientometrics, 63 (3), 567-581.

Dreze, Jacques h. and Estevan, Fernanda. 2006. Research and higher education in economics: Can we deliver the Lisbon objectives? Journal of the European Economic Association, 5 (2), 271-304.

Helpman, Elhanan et al. 2008. ESRC International benchmarking study of UK economics. ESRC. SWINDON.

Oswald, Andrew J. 2007. An examination of the reliability of prestigious scholarly journals: Evidence and implications for decision-makers. Economica, 74, 21-31.

Oswald, Andrew J. 2009. World-leading research and its measurement. Warwick Economics Research Paper, University of Warwick.

Starbuck, William H. 2005. How much better are the most prestigious journals? The statistics of academic publication. Organization Science, 16, 180-200.

Tables

Table 1. Most-cited economics articles by UK-affiliated authors, 2001-2008

Journal [Marginal article’s citation count]

X

UK papers amongst X most-cited articles?

Their rank

UK institutions producing these highly-ranked papers

AER [55]
50
Yes
12th; 32nd; 35th; 38th
Warwick+LSE;
LSE;
Cambridge;
LSE

EJ [41]

10
Yes
5; 10
LSE+Oxford; Nottingham

REStud [35]

20
Yes
9;
11; 12; 13
UCL+Oxford+Cambridge; LBS;
LSE;
Cambridge
Econometrica [32]
50
Yes
11; 17; 19; 29; 36
Oxford;
York;
UCL;
Warwick;
Oxford
IER [29]
10
Yes
1
Warwick+Cardiff+Oxford

REStats [39]

20
Yes
13
Warwick

JEEA [8]

10
No
 
 

JPE [30]

50
Yes
45
Essex

QJE [45]

50
Yes
40
UCL

JEconometrics [39]

20
Yes
1;
6;
10;
14; 15; 19

Cambridge+Edinburgh; Edinburgh+QMW; Kent+StAndrews; Lancaster+Strathclyde; IFS;
Oxford

JPubEcon [35]

10
Yes
1; 5
Warwick; LSE
JDevEcon [30]
10
Yes
3; 7
Nottingham; Oxford

JHealthEcon [36]

10
Yes
2; 9
Sheffield; Leicester

JMonetaryEcon [51]

10
Yes
4; 7
Royal Holloway; Oxford

JIntEcon [27]

10
Yes
4; 5; 10
Warwick; LSE;
LBS

JFinance [51]

50
Yes
41
Oxford

Rand Journal [44]

10
Yes
1; 6; 10
Oxford; LSE;
Oxford

JUrbanEcon [22]

10
Yes
4; 7; 10
UCL+LSE; LSE;
LSE

JOLE [28]

10
No
 
 
JEnvEcon&Man[36]
10
Yes
3; 7
Strathclyde; UEA

JLaw&Econ [22]

10
No
 
 
JET [34]
10
No
 
 

Total # world-leading papers from the UK

   
45
 

Notes to reading this table: Reading the top row, consider the 50 most-cited articles appearing in the American Economic Review during 2001-2008. The UK produced the 12th, 32nd, 35th, and 38th most-cited articles. Those four articles came from, respectively, Warwick and LSE on the first, LSE on the second, Cambridge on the third, and LSE on the fourth.

Different choices of X reflect journals’ instrinsic differences in citation counts. The number in brackets is the marginal article’s citation count – the 50th most-cited paper in the American Economic Review attained 55 cites. The Journal of the European Economic Association has only recently started publishing papers so cannot be compared to others on cites.

The citations totals were collected in December of 2008

Table 2. Most-cited economics articles by European-affiliated authors, 2001-2008

Journal [Marginal article’s citation count]

X

(Non-UK) European papers amongst X most-cited articles?

Their rank

(Non-UK) European nations producing these highly-ranked papers

AER [55]
50
Yes
2nd; 3rd; 5th; 33rd; 34th; 35th; 47th

Switzerland;
Germany;
Germany;
France;
Spain;
Italy;
Spain

EJ [41]

10
Yes
4
Netherlands

REStud [35]

20
Yes
5; 15; 19
France;
Spain;
France
Econometrica [32]
50
Yes
17; 18; 19; 21; 26; 27; 32; 34; 43
France;
Spain;
France+Germany;
Denmark;
France;
Germany;
Netherlands;
Denmark;
France
IER [29]
10
Yes
2  Italy+Belgium

REStats [39]

20
No    

JEEA [8]

10
Yes
1; 2; 7;
9;
10
Spain; Sweden;
Sweden+Ireland; Spain; Switzerland

JPE [30]

50
Yes
10; 24 Sweden+Belgium+Italy;
Italy
QJE [45] 50 Yes 21;
26;
40;
50
France;
Italy;
France;
Italy
JEconometrics [39] 20 Yes 4;
20
Netherlands; France+Belgium

JPubEcon [35]

10
Yes
2;
10
Italy; Switzerland+Denmark
JDevEcon [30]
10
Yes
1; 2;
3
Denmark; Norway;
France

JHealthEcon [36]

10
Yes
8; 10
Sweden; Germany

JMonetaryEcon [51]

10
No
 
 

JIntEcon [27]

10
Yes
1; 4; 7;
9
Ireland; Germany;
Italy+Spain+Germany; Spain

JFinance [51]

50
Yes
3; 10; 19; 25; 41

Netherlands;
France;
Spain;
Finland;
Finland

Rand Journal [44]

10
Yes
9 France

JUrbanEcon [22]

10
No    

JOLE [28]

10
Yes
 1
France
JEnvEcon&Man[36]
10
No
 
 

JLaw&Econ [22]

10
Yes
 2
Austria
JET [34]
10
Yes 3; 5; 6 Italy; France;
Spain

Total # world-leading papers from the UK

    56
 
Footnotes

1 Bruce Charlton of Newcastle University has given me extremely helpful suggestions on this paper. I have also benefited from the comments of Danny Blanchflower, Richard Blundell, Jacques Dreze, Amanda Goodall, Steffen Huck, Joao Jalles, Jim Malcomson, Hashem Pesaran, Neil Shephard, John Van Reenen, Steve Venti, John Vickers, Bruce Weinberg and Tim Worrall. I have deliberately not discussed this paper with the Economics RAE Panel member from Warwick University, Steve Broadberry, and have no way of knowing if the Panel used data like mine.

2 Google Scholar and Scopus are possible alternatives. It would be interesting to check the later calculations on such data, although it would be surprising to me if the conclusions changed.

 

735 Reads