Economics gets out more often: Using extramural citations to assess economic scholarship

Josh Angrist, Pierre Azoulay, Glenn Ellison, Ryan Hill, Susan Feng Lu 17 November 2017

a

A

The 2017 Nobel in Memorial Prize Economic Sciences, awarded to University of Chicago’s Richard Thaler, has given behavioural economics well-deserved recognition. Thaler and colleagues are fascinated by differences between economists’ benchmark model of rational decision-making and the seemingly irrational decisions that psychologists hope to explain. Behavioural economists work in the space between these two social sciences. This intersection is a recent development: as Camerer (1999) observed, “[E]conomists routinely – and proudly – use models that are grossly inconsistent with findings from psychology.”

Some hold the opinion that the distance between economics and other social sciences reflects the insularity and hubris of economists. Fourcade et al. (2015) argued that academic economics is characterised by a self-serving sense of superiority that reflects the guild-like structure of our discipline (two of the three authors are sociologists). Thaler’s prize reminds us that economics is the only social science for which there is a Nobel. Many economists claim the ear of kings and princes, and most have plenty of outside opportunities to make a buck. Following the Great Recession, this has prompted external critics of the discipline (and some from within, like Zingales 2013) to ask whether our mission of pure scholarship has been corrupted or devalued.

A lonely island?

Our recent paper looking at 'extramural citations' attempts to gauge the scientific standing of academic economics research (Angrist et al. 2017a). We find that the polemical claims that economics is an insular social science are out of date. Economics pays less attention to other social sciences than do political science and sociology – but Figure 1 shows that, since around 1990, this insularity has declined. Economists are now more likely than psychologists to cite other social science disciplines (left-hand panel).

Measured by citations to non-social-science disciplines, economics looks even less insular. The centre panel shows that economists are more likely than other social sciences to cite business disciplines (finance, accounting, marketing, and management) and are at least as likely to cite a group of seven other disciplines as three out of four of other social sciences (right-hand panel).

Figure 1 Social science insularity

Note: The left-hand panel of this figure shows citation rates from each social science to all other social sciences as a proportion of the total for each citing discipline. The centre panel shows citation rates from each social science to four business disciplines. The right-hand panel shows citations to seven other disciplines. Plots are smoothed using five-year moving averages. Papers cited were published between 1955 and 2015.
Source: Angrist et al. (2017).

We also find that, since 1990, economics has been especially and increasingly attentive to political science. Behavioural economics is grounded in psychology, but the flow of influence runs both ways. Figure 2 (right-hand panel) shows that psychology’s citation rate of economics papers, while still low in absolute terms, has more than doubled since the early 2000s.

Figure 2 Citations of economics papers in other fields

Notes: Citation rates are weighted by the importance of the citing journal. Plots are smoothed using five-year moving averages.
Source: Researchers’ calculations using data from Web of Science and EconLit.

Citations in computer science and operations research have also grown rapidly (Figure 2, right-hand panel). Our analysis also shows that political science and sociology citations to economics grew rapidly in the 1970s and 1980s. Sociology has been citing economics more than it cites political science or psychology since the late 1970s. Among the social sciences, only anthropology has remained distant.

Change from within

This growth in the extramural influence of economics reflects an important change within. Through the early 1990s, economic theory took pride of place, garnering more intramural citations and appearing more often than empirical work in our top journals. Our study of the evolution of fields and styles in economic research (Angrist et al. 2017b) shows, however, that recent decades have seen the share of economics’ citations to empirical work grow steadily. Cites to empirical work now account for about half of citations in top economics journal publications. We discovered this by using a machine-learning algorithm that reads abstracts, titles, keywords, and references to classify economics into three research styles, empirical, theoretical, and econometric theory. Figure 3, below, documents the empirical shift in economics publications and in intramural citations.

Figure 3 Weighted publications and citations by style

Note: Five-year moving averages of weighted publication and citation shares in each style.
Source: Angrist et al. (2017).

This move towards empiricism probably reflects what Angrist and Pischke (2010) called the “credibility revolution” in economic research. Empirical economics gets more attention now because its quality has increased, and so the work is more credible and convincing, and more likely to be discussed, challenged, and replicated in later work. In short, it has become more scientific.

The shift towards empirical work in economics is widespread. Figure 4 plots empirical shares of citations by economics field (we also used a machine learning algorithm to classify economics papers into fields, like IO, labour, and macro). In the early 1980s, only development and labour economics had a majority of their citations go to empirical work. The weighted empirical share has since grown in all fields. It now exceeds 90% in these two fields. Even macroeconomics, criticised after the great recession for an excess of ivory-tower theorising, has seen its empirical share grow by more than 50%. In most fields, the trends reflect both increasing numbers of empirical papers, and the generally improved journal placement of empirical work.

Figure 4 Weighted fraction empirical by field

Note: Five-year moving averages of weighted publication shares in each style.
Source: Angrist et al. (2017).

Strength in diversity

The rise in the extramural influence of economics scholarship has been accelerated by the diversity of research. Most disciplines cite papers from more than one field of economics, and the fields of primary interest vary across disciplines. Labour economics is the most cited field in sociology, while computer scientists tend to reference micro. Unsurprisingly, perhaps, political science reads political economy. There are almost as many focal fields as disciplines.

Empirical work has been drawing a greater share of attention from most of the disciplines in which economics is important. The left-hand and centre panels of Figure 5 show the strong growth in empirical research citations in social science and business disciplines. In psychology (recall Thaler), public health, medicine, and multidisciplinary science, the rise in the extramural influence of economics coincided with the growing share of citations to empirical work. This is consistent with the idea that empirical work caused increased citation rates.

In other cases, including political science and sociology, the rise in the empirical share of citations came after economics became influential. In these fields, the rise of empirical work may have allowed economics to remain influential, even as interest in theoretical economics waned.

Figure 5 Extramural citation shares to styles

Note: This figure plots aggregated weighted citation shares from social science disciplines (psychology, sociology, political science, anthropology), business disciplines (management, finance, accounting, marketing), and mathematical disciplines (statistics, operations research, computer science, and math) to economics styles. Plots are smoothed with five-year moving averages. Papers cited were published between 1970 and 2015.

The theory–empirical pattern is not universal. Heightened interest in economics in operations research and computer science, which also started around 2000, reflects these fields’ enduring interest in economic theory. Citations to empirical work have increased in the mathematical disciplines we studied (OR, statistics, computer science, and mathematics), but the majority of citations there were still to economic or econometric theory. This suggests again that the diversity of economics is a source of strength.

We find little evidence in citation statistics to support the notion that economics is intellectually isolated. Instead, we find growing links between economics and a wide range of other disciplines. This reinforces our view that economic scholarship has never been more exciting or useful than it is today.

References

Angrist, J, P Azoulay, G Ellison, R Hill and S Feng Lu (2017a), “Inside job or deep impact? Using extramural citations to assess economic scholarship”, NBER, Working paper 23698.

Angrist, J, P Azoulay, G Ellison, R Hill and S Feng Lu (2017b), “Economic research evolves: Fields and styles”, American Economic Review 107(5): 293–297.

Angrist, J, and J Pischke (2010), “The credibility revolution in empirical economics: How better research design is taking the con out of econometrics”, Journal of Economic Perspectives 24(2): 3–30.

Camerer, C (1999), "Behavioral economics: Reunifying psychology and economics", Proceedings of the National Academy of Sciences 96(19): 10575-10577.

Fourcade, M, E Ollion, and Y Algan (2015), “The superiority of economists”, Journal of Economic Perspectives 29(1): 89–114.

Zingales, L (2013), “Preventing economists’ capture”, in D Carpenter and D Moss (eds), Preventing Regulatory Capture: Special Interest Influence and How to Limit It, Cambridge, UK: Cambridge University Press.

a

A

Topics:  Frontiers of economic research

Tags:  citations, research, social science, Behavioural economics, machine learning

Ford Professor of Economics and co-director of the School Effectiveness and Inequality Initiative, MIT

International Programs Professor of Management, MIT Sloan School of Management

Gregory K. Palm Professor of Economics, MIT

PhD candidate, Economics Department, MIT

Assistant Professor of Management, Krannert School of Management, Purdue University

Events