The demise of Doing Business: Goodhart’s Law in action

Thorsten Beck 29 September 2021

a

A

Twenty years ago, researchers trying to study the importance and effectiveness of ‘institutions’ in the economic growth process were mostly limited to survey-based measures from Business Environmental Risk Intelligence (BERI) and International Country Risk Guide (ICRG). These typically captured very broad concepts, such as rule of law or political stability. They could not really serve as a basis for specific policy reforms.

This is where the Doing Business project came in.1 Focusing either on specific institutions (e.g. credit registries) or specific business cases (enforcement of bounced check, registering a property of specific value), the Doing Business data provided a benchmarking tool for cross-country analysis of specific dimensions of the business environment. The database allowed researchers to use consistent indicators of specific institutions across countries to both gauge the determinants of institutional frameworks and their effects.

The original index was built for the World Bank’s 2002 World Development Report, Building Institutions for Markets, and the first stand-alone Doing Business report was published in 2004. Over time, Doing Business has become by far the most successful data-collection exercise and report of the World Bank, but also, as former World Bank chief economist Kaushik Basu (2018) stated, its most contentious publication. Controversies over Doing Business resulted in the resignation of Kaushik’s successor Paul Romer and have resulted in numerous inquiries and reviews, ultimately resulting in the pausing of the report in August 2020 and its permanent suspension in September 2021.

The data

The indicators are based on expert judgement and legal documents (where relevant) and in several cases refer to a hypothetical firm (to allow for cross-country comparability). They refer to de jure regulations and can thus be linked to specific policy actions. Capturing a variety of different dimensions, they also allow researchers to disentangle different dimensions of the institutional framework. While initially only cross-sectional, the updating of the database over time has added another useful dimension that can be used for cross-country panel analysis.

Many of the indicators have been shown to be correlated in the cross-section with certain outcome variables. For example, Djankov et al. (2002) show that countries with heavier regulation of entry have higher corruption and larger unofficial economies, but not better quality of public or private goods. Djankov et al. (2007) show that credit reference bureaus and creditor rights are correlated with financial sector development. Overall, the data-collection exercise was based on theoretical concepts and empirical research (Besley 2015, Djankov 2016). The indicators were in turn used by other researchers in influential papers, such as Klapper et al. (2006) and Nunn (2007).  They have given rise to an expansive literature that has explored the importance of different growth constraints and specific policy reforms across the globe. 

As cross-country indicators such as the Doing Business variables are open to the usual critique against cross-country work, the recent literature has focused more on the assessment of country-specific institutional reforms, exploiting either sub-national variation in the business environment such as labour-market regulation across Indian states (Besley and Burgess 2004) or country-specific policy reforms (e.g. Bruhn 2013 for Mexico, Gine and Love 2010 for Colombia). Many of the country-specific analyses, however, are consistent with the Doing Business philosophy, notwithstanding country-specific analyses of China and India that point to the effective functioning of alternative business structures (Allen et al. 2005, 2006). Overall, the Doing Business project has been part and an important driver of increasing interest in institutions and policies to foster private sector development across the globe.

The ranking

In order to maximise the impact of the data collection, the Doing Business reports included rankings of countries based on Doing Business data. The reports and ranking became part of a broader effort of the World Bank and other donors to assess countries’ investment climate and foster private sector development and thus growth in developing countries.  Countries that moved up the most in the ranking were crowned reformer of the year, which in turn has been used by governments to attract foreign direct investment.   When Narendra Modi was elected Prime Minister of India in 2014, he explicitly targeted achieving 50th place in the ranking as a benchmark for his administration (Besley 2015). What started as a research data-collection exercise not only became an important tool for policy reforms, but also entered the political debate.

Changes in rankings, however, can be misleading. If a country improves its business environment, it will only move up the ranking if it improves more than other countries. Similarly, a country can keep its rank or even improve it only if other countries fall behind in their business environment.  

Importantly, governments that care about these rankings might start gaming the system – rewriting laws with an eye on improvements in the Doing Business ranking rather than focusing on the most important constraints for private sector development. As Basu (2018) points out, focusing on mechanical improvements in Doing Business rankings cannot substitute for a national economic strategy. 

A more fundamental question is whether the institutional dimensions captured by the Doing Business ranking are really the relevant ones for a large share of the enterprises in a country, including the large number of informal firms in many developing economies. Hallward-Driemeier and Pritchett (2015) show that the de facto implementation of rules and policies (as captured by Enterprise Surveys, another data-collection project by the World Bank and other international financial institutions) varies more within countries and across firms than across countries. Constraints as perceived by enterprises often do not correlate with Doing Business scores (Besley 2015) and neither do changes in Doing Business scores and constraints in enterprise surveys (Hallward-Driemeier and Pritchett 2015).

The controversies

The Doing Business reports and ranking attracted increasing criticism over the past decade.  Some observers accused it of a deregulation bias, especially with respect to the labour market and taxation components of the data collection. There have been continuous methodological changes over the years. While improving the usefulness of the indices, these also make comparability over time difficult; in some cases, they might also raise suspicions, as in the case of Chile where methodological changes resulted in a fall in ranking during left-wing governments. There are also data-collection concerns, as the number of sources per country varies and judgement calls have to be made by Doing Business staff in some cases. 

There have been several internal and external evaluations of the Doing Business project (IEG 2008, World Bank, 2013, 2021a). Some changes have been made following these reports, although suggestions to drop the ranking have been rejected. 

Conflicts of interest

Credit rating agencies face a conflict of interest if they rate securities while providing advisory services to the issuing companies. Similarly, the Doing Business team faces a conflict of interest if it collects data used for country ranking while at the same time offering advisory services to governments on how to improve their business environment and thus their Doing Business Ranking.  This conflict of interest is not theoretical, but has been clearly shown in the Wilmer Hale (2021) report.

However, the conflict of interest is even deeper and cannot necessarily be remedied by introducing a firewall between data collection efforts and advisory services. Ultimately, the World Bank is owned by its member countries, who are represented on the Executive Board.  The (ultimately successful) attempt by certain governments to take influence on the ranking (and thus the underlying data) clearly speaks to this conflict of interest. 

Goodhart’s Law

Beyond these conflicts of interest, there is a broader concern of whether the political targeting of the Doing Business ultimately undermines the usefulness of the index.  Charles Goodhart stated in 1975 that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”.  Originally, this regularity (known nowadays as Goodhart’s Law) was used in monetary policy, but it can be applied in any economic policy area. Marilyn Strathern (1997) summaries Goodhart’s Law in a discussion on the UK evaluation framework for universities: “When a measure becomes a target, it ceases to be a good measure”.

The events described in the World Bank report read like Goodhart’s Law in action.  As rankings were published on an annual basis and as improvements in the ranking became political targets, so did attempts at influencing ranking and thus data. The conflict of interests mentioned above facilitated such influence-taking; all that was needed were persons in place that would be open to such influences.  Beyond these conflicts of interests, however, it is hard to see how one can get around this more fundamental problem of declaring an index as policy target, without this index subsequently losing its usefulness.  

Conclusions

Where do we go from here?  The original idea of collecting data on the business environment in which companies across different countries (or even across different regions within countries) operate continues to be good and important – again for research purposes and to inform the policy dialogue (though not to pre-empt policy outcomes). In Beck (2013), I called for moving away from rankings to focus exclusively on data. Researchers do not need rankings, we need data.  Policymakers need data and comparisons, while questionable rankings not only do not help a constructive reform process, but narrow it down to what are not necessarily the best reforms. Finally, Doing Business data should be regarded as one of several gauges of the business environment firms face, forming part of a more broad-based approach that draws on multiple data sources and analyses. Such an approach might not hit the headlines as often, but might be more effective in driving policy reforms.

References

Allen, F, J Qian and M Qian (2005), “Law, Finance and Economic Growth in China”, Journal of Financial Economics 77: 57-116.

Allen, F, R Chakrabarti, S De, J Qian and M Qian (2006), “Financing firms in India”, World Bank Policy Research Working Paper 3975.

Basu, K (2018), “The ease of doing business comes with trade-offs”, Brookings Institution, 12 March

Beck, T (2013), “Doing Business: less icing, more cake”, VoxEU.org, 6 June. 

Besley, T (2015), "Law, Regulation, and the Business Climate: The Nature and Influence of the World Bank Doing Business Project", Journal of Economic Perspectives 29: 99-120.

Besley, T, and R Burgess (2004), “Can Labor Regulation Hinder Economic Performance: Evidence from India”, Quarterly Journal of Economics 119: 91-134.

Bruhn, M (2013), “A Tale of Two Species: Revisiting the Effect of Registration Reform on Informal Business Owners in Mexico”, Journal of Development Economics 103: 275-83.

Djankov, S (2016), "The Doing Business Project: How It Started: Correspondence", Journal of Economic Perspectives 30(1): 247-48.

Djankov, S, R La Porta, F Lopes-de-Silanes, and A Shleifer (2002), “The Regulation of Entry”, Quarterly Journal of Economics 117(1): 1–37

Djankov, S, C McLiesh, and A Shleifer (2007), “Private Credit in 129 Countries”, Journal of Financial Economics 84(2): 299–329.

Gine, X and I Love (2010), “Do Reorganization Costs Matter for Efficiency? Evidence from a Bankruptcy Reform in Colombia”, Journal of Law and Economics 53: 833-64.

Goodhart, C (1975), "Problems of Monetary Management: The U.K. Experience", Papers in Monetary Economics 1, Reserve Bank of Australia.

Hallward-Driemeier, M and L Pritchett (2015), “How Business is Done in the Developing World: Deals versus Rules”, Journal of Economic Perspectives 29: 121-40

IEG – Independent Evaluation Group (2008), Doing Business: An Independent Evaluation, The World Bank.

Klapper, L, L Laeven, and R Rajan (2006), “Entry Regulation as a Barrier to Entrepreneurship”, Journal of Financial Economics 82: 591–629.

Nunn, N (2007), “Relationship-Specificity, Incomplete Contracts and the Pattern of Trade”, Quarterly Journal of Economics 122(2): 569–600.

Strathern, M (1997), "'Improving ratings': audit in the British University system", European Review  5(3): 305–321

WilmerHale (2021), “Investigation of Data Irregularities in Doing Business 2018 and Doing Busdiness 2020”, 15 September. 

World Bank (2013), Independent Panel Review of the Doing Business report, June.

World Bank (2021) Doing Business: External; Panel Review, 1 September.

Endnotes

1 https://www.doingbusiness.org/en/doingbusiness

a

A

Topics:  Development Institutions and economics

Tags:  World Bank, Doing Business, Goodhart's Law, data collection

Professor of Financial Stability and Director, Florence School of Banking and Finance, EUI; CEPR Research Fellow

Events

CEPR Policy Research