Women are under-represented in economics, and more so the higher the rank. Numerous explanations have been offered for this disparity, including differences in competitiveness and in the allocation of time between work and family, but the concern remains: do stereotype biases or other forms of discrimination lead decision makers to undervalue the contributions of women? On teaching evaluations, female professors get lower grades, despite the fact that students appear to learn as much from women as from men (Boring 2017, Mengel et al. 2019). When it comes to being accepted at important economic conferences, the evidence is mixed (Chari and Goldsmith-Pinkham 2019, Hospido and Sanz 2019). To assess the evaluation of scientific contributions by male and female authors, Rebecca Blank randomly assigned submissions at the American Economic Review to referees, with or without masking the author’s name and affiliation. She found a large but imprecisely estimated positive effect of blinding on the acceptance rate of female-authored papers (Blank 1991). Later studies show that women seem to be held to higher standards in academic promotions (Sarsons 2017) and the publishing process (Hengel 2019). Equally worrisome, a recent study documents a hostile environment for female economists (Wu 2019).
Producing high-quality research and publishing in high-impact journals is the most important task for an academic seeking promotion, summarised as the imperative to ‘publish or perish’. Our study asks: Are referees and editors in economics gender neutral? In this column, we present a comprehensive analysis of the role that gender plays in the editorial process (Card et al. 2019) based on submissions to four leading journals in economics between 2003 and 2013: the Quarterly Journal of Economics, the Review of Economic Studies, the Journal of the European Economic Association and the Review of Economics and Statistics.
Construction of the data set
We obtain our data from information stored in the Editorial Express (EE) system, which is used by all four journals. For confidentiality reasons, we wrote a program that could be run by journal staff to create an anonymised database, combining information in the EE system with gender information from pre-coded lists of author and referee names. Prior to running our extraction program, we obtained the names of all authors and referees from each journal for the relevant years. We developed a protocol for assigning gender to the names on these lists. Overall, we are able to assign gender to about 97% of the names in our data set.
For each of our nearly 30,000 submissions, we observe paper characteristics (e.g. submission date, field of the paper), author gender and prominence, referee gender and prominence, recommendations by the referees, and final ‘revise and resubmit’ (R&R) decisions by the editor. Author and referee prominence is measured by the number of publications in 35 high-impact journals in the five years prior to submission. Of all submissions, roughly half are desk-rejected and the other half are sent to referees. Two thirds of the non-desk-rejected papers are male-authored, 7% are female-authored, and the rest are mixed gender. Among the mixed gender papers, we distinguish between those in which the female co-author is the most prominent (4%) and those in which the most published co-author is male (17%). As a measure of paper quality, we count accumulated google scholar citations collected in mid-2015. In order to give some time for papers to accumulate citations, we restrict the sample to those papers submitted before 2013.
Before receiving the data, and to address concerns over data mining, we wrote a pre-analysis plan, laying out the research design regarding the role of gender in the editorial process. Our pre-analysis plan is registered under AEARCTR-0003048.
Do female and male referees evaluate the same paper in the same way?
One key question in the editorial process concerns referee behaviour. Does the gender of the referee matter for how a paper is evaluated? We first look at how female- and male-authored papers are matched to referees. Controlling for all observable characteristics, including field controls, we find evidence for assortative matching: compared to male-authored papers, female-authored papers are seven percentage points more likely to be assigned to a female referee. The estimated coefficient for mixed-gender papers is five percentage points (when the most senior author is female) and three points (when the most senior author is male). Does this referee-assignment process affect how a paper is evaluated? The data suggest that a referee’s gender does not matter. Following a simple audit-style analysis with paper fixed effects, referee gender does not affect the evaluation of either all-male-authored or all-female-authored papers (nor mixed-gender papers). This is consistent with an earlier study relying on a leading field journal (Abrevaya and Hamermesh 2012). In sum, authors should not care whether their papers are sent to male or female referees.
Are female and male authors held to the same standard?
Although female and male referees evaluate the same paper in similar ways, this does not necessarily imply that referees set the same bar for male- and female-authored papers. It is possible that referees of both genders maintain biases for or against female-authored papers. To find out, we need to make comparisons across papers by different gender groups, taking into account differences in quality, which we measure by accumulated google cites from submission to mid 2015. We then investigate whether conditional-on-referee recommendation characteristics, such as gender, have any predictive power. Intuitively, if referees impartially evaluate papers according to their quality, author characteristics such as gender should not predict citations, once referee recommendations are controlled for. On the other hand, if female-authored papers that are conditional on referee recommendations accumulate more/fewer cites than all-male papers, then we could argue that female-authored papers face a higher/lower bar in the publishing process.
What do the data tell us? For the sake of brevity, we compare all-female papers with all-male papers but note that the results for mixed-gender papers (especially those whose most senior author is female) tend to lie in between all-female and all-male papers. We find that all-female papers receive about 25% more citations than observably similar all-male papers. This is consistent with female researchers facing a higher bar in the publishing process. Based on a theoretical model, we can translate this effect into R&R probabilities. Our calculations suggest that the R&R rate for all-female papers would rise from 12.3% to 18.6% – a 50% increase – if editors were to maximise citations. We also find that while editors do not maximise citations when it comes to evaluating male and female-authored papers, they largely follow the referees’ recommendations (which underpredict the citation potential of female-authored papers).
One potential caveat could be that cites depend on gender. To explain our results, it needed to be the case that female-authored papers ceteris paribus receive more citations. However, in a survey of economists and editors that we administered prior to receiving the data, the opposite proved true: survey respondents clearly believed that female authors receive fewer citations than male authors for a paper of comparable quality. Hence, our estimate for the higher bar for women turns out to be a lower bound.
So far, we have interpreted the results as being consistent with a higher bar for females. An alternative explanation could be that female-authored papers have characteristics that lead to higher citations but are not as highly rewarded in the review process. As we do not have access to the actual papers in our submissions data (and cannot infer more detailed paper characteristics), we have to rely on another dataset. Though much smaller in sample, we can analyse the set of published papers (that largely correspond to our submitted papers, assuming a two-year delay between submission and publication). Among this set of papers (which is about one tenth of the size of our submissions data), we collect very fine JEL codes for field as well as objective (word counts) and more subjective (RA evaluations) measures of theoretical (as opposed to empirical) paper content. Overall, the results on empirical versus theoretical content provide suggestive evidence that different characteristics of papers play some role in the observed citation premium but do not explain all of the citation premium in our main sample. This conclusion is tentative, given the difficulty of making inferences from published papers alone.
Do editors value female and male referee recommendations equally?
As an important part of the editorial process, we also consider how editors use the information provided by male versus female referees. We first analyse the informativeness of referees – that is, the relationship between referee recommendations and the paper’s future citations, by the referee’s gender. Male and female referees do not seem to differ in their informativeness, and nor do editors place a different value on them. In sum, we conclude that editors are gender neutral in valuing advice from male and female referees.
Conclusions
Are the referees and editors in economics gender neutral? The answer is both “yes” and “no”. If we focus on referee recommendations, we replicate the findings of Abrevaya and Hamermesh (2012), who observed no difference in the way referees of different genders assess papers by female and male authors. We also find no difference in how informative male and female referees are about future citations, and editors rightfully assign them the same weight. However, papers submitted by female authors receive 25% higher citations, conditional on referee recommendations and paper and author characteristics. While female-authored papers differ from male-authored papers in empirical content (which positively affects citations), that difference is unlikely to explain the whole citation gap and suggests that female-authored papers face a higher hurdle in the publishing process.
References
Abrevaya, J and D S Hamermesh (2012), “Charity and favoritism in the field: Are female economists nicer (to each other)?”, Review of Economics and Statistics 94.1: 202-207.
Blank, R M (1991), “The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from the American Economic Review”, American Economic Review 81: 1041–1067.
Boring, A (2017), “Gender biases in student evaluations of teaching”, Journal of Public Economics 145: 27-41.
Card, D, S DellaVigna, P Funk and N Iriberri (forthcoming), “Are referees and editors in economics gender neutral?”, Quarterly Journal of Economics.
Chari, A and P Goldsmith-Pinkham (2019), “Gender Representation in Economics Across Topics and Time: Evidence from the NBER Summer Institute.” NBER Working Paper No. 23953.
Hengel, E (2019), “Publishing while female”, Working Paper.
Hospido, L and C Sanz (2019), “Gender Gaps in the Evaluation of Research: Evidence from Submissions to Economics Conferences”, IZA Discussion Paper No. 12494.
Mengel, F, J Sauermann and U Zölitz (2018), “Gender bias in teaching evaluations”, Journal of the European Economic Association 17.2: 535-566.
Sarsons, H (2017), “Gender differences in recognition for group work”, Working Paper.
Wu, A H (forthcoming), “Gender Bias in Rumors among Professionals: An Identity-based Interpretation”, Review of Economics and Statistics.
