Many spheres of economic activity rely on expert ratings to guide choices when the quality of alternatives is otherwise difficult to assess. Examples range from credit rating agencies to college and university rankings. But an obvious concern arises when evaluators have incentives to distort ratings for private gain at the expense of those who rely on them. Such conflicts of interest have been found in recent studies of credit rating agencies prior to the collapse of the market for mortgage-backed securities in 2007 (eg Ashcraft et al 2010, Griffin and Tang 2010 and 2011).
More generally, conflict-of-interest studies are notoriously challenging because data are difficult to collect, incentives are not easy to define, and what constitutes a biased or distorted evaluation is hard to measure. We overcome these challenges in a recent study of how conflicts of interest distort the way coaches rank US college football teams in the USA Today Coaches Poll of the top 25 teams (Kotchen and Potoski 2011). We find robust evidence that reputation benefits and direct financial payoffs bias coaches’ rankings.
The USA Today Coaches Poll is based on the weekly ballots of approximately 60 coaches that are selected each year to submit their votes for the top 25 teams. These rankings are closely followed by millions of football fans, television executives seeking to market and broadcast the games of highly ranked teams, and other observers interested in university reputations. Moreover, the final poll of the regular season has additional importance because it is used to determine the eligibility of teams for the high-profile Bowl Championship Series, including the national championship. Selection into a Bowl game is not only prestigious, it comes with substantial financial rewards for participating teams and conferences, as indicated by the nearly $200 million that was disbursed after the 2010–11 season.
While coaches are assumed to have expert knowledge, and are thus relied upon to provide what is intended to be an unbiased and objective ranking, there are a host of potentially distorting private incentives that create conflicts of interest. These fall into the broad categories of improving the standing of one’s own team and athletic conference and receiving direct financial payouts by influencing which teams are invited to play in Bowl games. While questions about bias have been considered before in the Coaches Poll (Witte and Mirabile 2010, Sanders forthcoming), our study is the first to use individual coach ballots as the unit of analysis.
We use the final regular season ballots of coaches for the 2005–10 seasons. These ballots are all those that are publically available. We find that, on average, coaches rank teams from their own conference nearly a full position more favourably and boost their own team’s ranking more than two full positions. We also find that it does not matter if a coach’s team simply plays a team during the season, but coaches rank teams they defeated more favourably by more than half a position. Coaches thus make their own teams look better by ranking more favourably teams they defeated.
Above and beyond the effect of reputation concerns, coach rankings respond to the structure and amount of direct financial incentives created by invitations to the Bowl games. When a coach’s university receives a greater financial payoff if a particular contending team receives an invitation, coaches rank that team higher, thereby increasing their chance of receiving the payoff. On average, an additional payoff between $3.3 and $5 million buys a more favourable ranking of one position. Moreover, for each increase in a contending team’s payoff equal to 10% of a coach’s football budget, coaches respond with more favourable rankings of half a position. Finally, this effect is strongest, more than twice as large, when coaches rank teams outside the top 10, where consensus about the relative standing of teams is lower.
Beyond providing insight into conflicts of interest, our results have implications for the importance of information disclosure requirements. It was only after a wave of controversy about the integrity of the Coaches Poll that individual ballots for the final regular season poll were made publicly available starting in 2005. Looking at periods before and after public disclosure, we find that making the coaches’ ballots public reduces the difference between the aggregated results of the final Coaches Poll and unbiased computer polls. It appears, therefore, that coaches are aware of their biases because they become attenuated when the ballots are publically available. That coaches’ rankings are noticeably different under public disclosure raises further questions about how much stronger the biases are in settings when there is no disclosure, and underscores the importance of public disclosure to minimise bias.
While we recognise that the rankings of NCAA football teams may not be of immediate concern to most economists and policy analysts, the Coaches Poll provides a unique setting in which many agents are evaluating the same thing, private incentives to distort evaluations are clearly defined and measurable, and there exists an alternative source of computer evaluations that is bias free. The study is thus an example of the new ‘forensic economics’ that takes advantage of data availability in a sports setting to study more general economic phenomena (Zitizewitz forthcoming), in this case the distorting influence of conflicts of interest. Concern and debate about these issues have clear applicability beyond NCAA football, as the task of ranking teams closely mirrors that in more immediately relevant domains where similar conflicts are likely to emerge.
Ashcraft, Adam, Paul Goldsmith-Pinkham, and James Vickery (2010) “MBS Ratings and the Mortgage Credit Boom”. Federal Reserve Bank of New Work Staff Report 449.
Griffin, John M and Dragon Yongjun Tang (2010) “Did Subjectivity Play a Role in CDO Credit Ratings?” University of Texas at Austin, McCombs Research Paper FIN-04-10.
Griffin, John M. and Dragon Yongjun Tang (2011) “Did Credit Rating Agencies Make Unbiased Assumptions on CDOs?” American Economic Review: Papers & Proceedings 101(3):125-130.
Kotchen, M J and M Potoski (2011) “Conflicts of Interest Distort Public Evaluations: Evidence from the Top 25 Ballots of NCAA Football Coaches” NBER Working paper 17628.
Sanders, J (fothcoming) “Raking Patterns in College Football’s BCS Selection System: How Conference Ties, Conference Tiers, and the Designation of BCS Payouts Affect Voter Decisions” Social Networks
Witte, M D and M P Mirabile (2010) “Not So Fast, My Friend: Biases in College Football Polls”. Journal of Sports Economics 11:443-455.
Zitzewitz, E (forthcoming) “Forensic Economics”, Journal of Economic Literature.