Field experiments in economic research

<p><em>Romesh Vaitilingam interviews John List for Vox<br />
<br />
January 2009<br />
<br />
Transcription of an VoxEU audio interview [http://www.voxeu.org/index.php?q=node/3522</em></p>
<p><strong>Romesh Vaitilingam</strong>: Welcome to Vox Talks, a series of audio interviews with leading economists from around the world. My name is Romesh Vaitilingam, and today's interview is with John List, who is a professor of Economics at the University of Chicago and a pioneer in the use of field experiments in economic research.</p>
<p>John and I met at the American Economic Association's Annual Meetings in San Francisco in January 2009. I began by asking him to explain why we might want to use field experiments in research.</p>
<p><strong>John List</strong>: The way I view field experiments is that they're a recent methodological innovation that helps to bridge the gap between laboratory experimentation and empirical methods that use naturally occurring data.</p>
<p>So, in economics, you're exactly right. When you're looking back at the history, most of the insights that we've gained using data would be using empirical techniques like regression analysis, instrumental variable analysis, or the analysis and structural modelling.</p>
<p>And on the other side, more recently, people have engaged in laboratory experimentation. Vernon Smith was one of the pioneers of lab experiments and through the late 40s and 50s he published several fundamental insights associated with using lab methods with students.</p>
<p>One of the major criticisms of that line of work is, well, these are students - they might be unmotivated, and we might not be able to generalize our results outside of the lab to the extra-lab world. In fact, when I was working in the White House, I was trying to push the White House into thinking about using results from the laboratory experimental literature to make some policy insights on benefit-cost analysis.</p>
<p>A major official told me, &quot;Well, while I understand these results, they seem to be similar to scientific numerology because students are not real people.&quot; Of course students are real people. The point is that the representativeness of the population might be important. So, a first step that we should think about in moving from the lab is &lsquo;let's go out and gets a non standard subject pool&rsquo;. What I mean by that is maybe a collection of experts, CEOs, traders at the Chicago Board of Trade, maybe farmers, whichever group we're interested in. Bring those folks into the lab and test the economic theory with these folks. That's what I've called an &lsquo;artifactual&rsquo; field experiment.</p>
<p>So, it's a normal lab experiment, but it's using a non-standard subject pool. Now, the properties of that situation may still be foreign to the participants because coming into a lab setting, I think by and large, is peculiar for people. So that might change their behaviour.</p>
<p>Now, we need to think about slowly changing the properties of the situation to move toward the natural environment. So the next step after an artifactual field experiment does just that. That's what we call a framed field experiment. The framed field experiment adds naturalness in the task or in the commodity that they're purchasing, or in the stakes or in the information.</p>
<p>But importantly, people still know they're taking part in an experiment, in a framed field experiment. So then, that begs a question. If you go to the next level and experiment with people without them knowing about it, can you have your cake and eat it too?</p>
<p>What I mean by that is, can you have randomization as your instrument - which is the key behind the experimental method - and can you have realism, the realism that naturally occurring data actually have. And yes we can, we call that a natural field experiment.</p>
<p>That's going to a setting, going to a market or any other type of individual setting, and perturbing the situation and exploring how people change their behaviours. So, we have randomization and we have realism. So now we've effectively bridged the lab and naturally occurring data.</p>
<p>Now, you might want to say, can you give me some examples of a natural field experiment?</p>
<p><strong>Romesh</strong>: I was just about to say that.</p>
<p><strong>John</strong>: And that's a good question. One example is with the recent work that I've done on the economics of charity. So, right now, economists know very little about the source of why people give to a specific charity.</p>
<p>Some might argue that it's altruism. Some might argue that it's warm glow; that people get utility out of just giving money. They don't really care what the charity does with that money, but they just get utility out of giving money. Some folks say then that you might want to think about modelling these charitable gifts just like you're purchasing a private good.</p>
<p>We know very little about this. This is a prime way to think about testing that theory by teaming with charities and setting up an experiment whereby you randomize households into different treatment cells. So, you team with a charity to send letters to households. The households are used to getting these letters. But the households get, of course, a different type of letter. And then you can parse the alternative hypotheses about why people are giving.</p>
<p>Now, this is valuable to a charity because they need to know, of course, why do people give and how can we induce people to give more? So, one example of a result that we've gotten just recently is that when you change the price of giving&hellip;the law of demand over private goods says if you raise the price, people will purchase fewer of the goods.</p>
<p>So the intuition is that if you lower the price for giving to a charity, people might give more. That's actually not what we found. What we found is that over pretty liberal price changes, people give about the same. So that suggests that they're giving out of purely a warm glow component, that they just get utility out of the dollar gift, the $50 that they send in to the Sierra Club rather than &lsquo;how far will my $50 go?&rsquo;</p>
<p>Of course there are some givers in that world, but by and large the data suggests that warm glow is a very important component of giving. Now, you can say OK that's an interesting example of a natural field experiment. Now we can see how we can use the internet, or use smaller markets to think about other types of questions. And, you can back up and say, &lsquo;How can you use artifactual field experiments or frame field experiments to answer questions?&rsquo; One example of an artifactual experiment that we've been working on recently is we've gone to various cultures to look at the differences between men and women, in terms of their competitive inclinations.</p>
<p>So we went to a matrilineal society in India called the Khasi society. This particular society is unique in the sense that women have a lot more power within the household, and within government than women traditionally have in patriarchal societies or societies in the Western world.</p>
<p>As an example, in a Khasi family all of the wealth runs through the youngest female. If you have two kids, a boy and a girl, and you can only send one to college, you're going to send the girl to college. So the idea is we have a result in the literature that says, &lsquo;Men are more competitively inclined than women.&rsquo; Some people have argued that's one of the reasons why men earn more money than women in the labour market.</p>
<p>The famous statistic is that women earn 78 cents for every dollar that men earn in the labour market. Some people argue, well, it could be discrimination. It could be risk. It could be women have time outside of the labour force, and employees recognize that. It could be that the two genders have different inclinations for competitiveness and they don't fight as hard to get up the run.</p>
<p>So, we run similar experiments in the Western world and in the matrilineal society. What we find is a direct reversal of the results in the matrilineal society. Instead of men being more competitively inclined than women, women are actually more competitively inclined than men.</p>
<p>Now this doesn't answer the key question about nature vs. nurture, but I think it does move us in that direction to think about the power of socialization, and how important it is in peoples&rsquo; behaviours markets.</p>
<p>Now, what about a frame field experiment? We mentioned discrimination, and I think one of the important open questions in economics is &quot;What is the source of discrimination in product markets or labour markets?&quot; There are two very compelling theories about why certain races are treated differently, why women are treated differently than men or why the elderly are treated differently from middle-aged men. One theory is Becker's theory that people just have a taste for discrimination. So entrepreneurs will forgo profits to hurt a particular class of person.</p>
<p>Now, the other theory of course is called &lsquo;third-degree price discrimination&rsquo; and Pigou talked about this years ago. The literature also calls it &lsquo;statistical discrimination.&rsquo; So under this particular theory, the entrepreneur actually uses the observable signals from the labourer or the consumer, and that helps determine the price that the entrepreneur quotes.</p>
<p>The entrepreneur is not discriminating because they have a taste for discrimination, as in Becker's model, but they're discriminating because they're trying to maximize their profits. So if you know someone will pay more for a car, you tend to start them off at a higher price. That's third-degree price discrimination.</p>
<p>Now, when you see naturally occurring data it's very, very difficult to parse. Why does one class of person receive a higher price than the other? It's just about impossible. Using a series of field experiments you can do that. In my 2004 QJE paper we set up a series of field experiments, and you can parse why one class of person or people is treated differently than another.</p>
<p>I think the power of field experimentation is not only that it adds more insights from other types of data domains, but it also helps you parse the different theories that might be explaining behaviour that we observed, and our data patterns that we observed in the real world.</p>
<p><strong>Romesh</strong>: You mentioned the three different types of field experiments that you use. Is there a preferred one that you would use or does it depend on the situation? Would you ideally for example like to use the natural field experiment?</p>
<p><strong>John</strong>: Exactly, exactly. I think a lot of it does depend on the situation, but I think the most appealing to me is a natural field experiment, because it does have randomization but it does have realism. I can understand that there are points where you do want to run an artifactual field experiment, because in some sense you have more control over exactly what is happening in that environment. You can, for example, induce values. And when people are overbidding in auctions in a natural field experiment, if you couple that with an artifactual field experiment, you can actually find out why indeed are they overbidding in those auctions.</p>
<p>So I do have a slight preference for natural field experiments, but I also understand that all three of the field experimental types are useful. And I think the most power can be gotten by combining insights gained from all three of these, and not only insights from the three types of field experiments, but also a laboratory experimentation, and insights that we gain from naturally occurring data.</p>
<p>Now, it's interesting that we spent so much time using naturally occurring data and so much time now in the lab in the last 40 years, but very little time in the middle. Of course, in the last decade or so now, there is a wave happening, and I'm quite happy to be part of that wave.</p>
<p>I think as economists recognize that the everyday phenomena that they approach and they think about it as, &lsquo;What economic question can I ask in this particular market? How can I perturb it to learn something fundamental about economics, or make a fundamental economic insight, or test an economic theory?&rsquo;</p>
<p>I think the growth will continue in field experimentation, for all three types of field experimentation, (a) because it has been under-utilized, and not even used as you mentioned at the beginning of the interview, but also because there are a lot of low apples still to pick by using field experimentation.</p>
<p><strong>Romesh</strong>: Going back to the natural field example that you mentioned: the charitable giving. It seemed the finding was basically the 'warm glow' was the outcome. So presumably the charities are learning from this research, and using in the way they go about trying to raise funds.</p>
<p><strong>John</strong>: Yeah, so far so good. So, we're working now roughly 20 or 30 charities. They've become more open minded now to testing their conventional wisdoms. Because there is a lot of conventional wisdom in that area, and when you ask people, &lsquo;where did this conventional wisdom come from?, none of them really know and it's very hard for them to pinpoint any solid empirical evidence for where this conventional wisdom has come from.</p>
<p>In that line, the charities that we've been working with have pretty much greeted us with open arms because they really want to know what works. Is our conventional wisdom correct, and if not, how do we amend the situation to (a) raise more money and (b) provide more of the public good?</p>
<p>So, I think, have they put this in place? Certainly they have. They've used some of these insights and they're changing their ways. But there still is a lot to learn, because, when you look at the charitable sector, we're talking about a sector that is roughly 2-3% of GDP. This is a sector that economists haven't focused a lot of their attention on over the years.</p>
<p>It's really interesting when you think about the economics of charity, on what I call the demand side, we really haven't explored it except for, say, the last decade. Now on the supply side, you had Marty Feldstein and Charles Clodfelter doing seminal work on how changes in tax codes influence how much people give to specific charities.</p>
<p>What I'm talking about is, what are the optimal actions of a charity to induce people to give, and can we think about models that predict why people are giving? What are the underlying sources for why people are giving?</p>
<p>So it's a compliment to these older supply side studies, which were excellent. But the demand side has not really been looked at much. It's sort of puzzling, but I'm happy about it, of course, because I can fill that area as well.</p>
<p>And field experiments compliment that of course, because, you could learn something using naturally occurring data but there isn't a lot of naturally occurring data out there to look at. You can use some laboratory experimentation as well, but there's really no need to because you have a lot of - Sierra Club, NAACP - a lot of non profits that are open to experimentation. So now, of course, people who give to these organizations are really my experimental subjects.</p>
<p><strong>Romesh</strong>: So, John, it sounds like you have had great success working with the charities on this and some very positive news to report. Are there other possible applications for this kind of natural field experiment that you're using?</p>
<p><strong>John</strong>: Certainly, and that's a good question to ask because we have now partnered with multiple for-profit companies such as Chrysler, United Airlines, Ambassadors Group. And these are groups that are interested in maximizing feedback.</p>
<p>It's interesting because, when I was saying before that charitable foundations and various charities, they haven't been very good at maximizing feedback in the past. For-profit companies have some of the same problems.</p>
<p>So, Chrysler, for example, we're doing an experiment in their Wellness program. We're exploring what are some financial and non-financial factors that Chrysler can use to induce their employees to lose weight. And, we've been quite successful. Two or three of our interventions have worked very well, and then we can look back after the employee loses weight, are they more productive on the line?</p>
<p>Of course, Chrysler is very interested in this. We have some preliminary results that suggest that we're finding something interesting for Chrysler. United, of course, is also interested in things like their frequent flyer program and how do they induce people to sign up for the frequent flyer program?</p>
<p>Pricing is important. What are the optimal prices that they should use? And, what should be their response to their competitors' price changes? And really, it's limitless the number of examples that you can think of in the business world where you can use field experimentation, learn something fundamental about economic theory.</p>
<p>Now I'm not talking about a consulting project here. Everything I'm doing with the companies, the only payment I get is the data, and I can publish the data. I'm talking about using experiments to test theory, and also to inform the companies about what are some optimal strategies that they should be pursuing to maybe increase the health of their workers, or increase the bottom line.</p>
<p>I think there will be several more opportunities in the future. Questions like, does advertising work? We're working with a few companies trying to explore the ROI of ads. It's a very difficult question, but with randomization and proper experimentation, you can answer these types of questions. So, I think the number of questions you can answer is limitless, and the number of partners, I think they are more or less lining up right now.</p>
<p>I think as we move on and show serious results, and show how the results can help the bottom line, of course, there will be more and more field experimentation. That's one way to get businesses to think about field experimentation. I'm also starting to teach a class, actually on Monday, to MBAs on how to use experiments in firms.</p>
<p>Steve Levitt and I are co-teaching this class. We figure, we're going directly to the firms to run field experiments, but if we can also get 75 or 100 people every year coming out of the University of Chicago with an understanding about how experimentation can be used to maximize feedback, then we have people at the entry level as well.</p>
<p>Now, we have the CEOs interested, we have someone at the entry level who can actually carry it out. So, this is new for me to go teach MBAs, because I'm in the Economics department at the University of Chicago. But, I think the opportunities of teaching the best young minds going into the business world in the next several years is a very important way to push field experimentation into the for-profit world, and actually some of these kids will go to non-profits as well.</p>
<p>In the end, I see this idea about maximizing feedback and field experimentation will be a very important component of that.</p>
<p><strong>Romesh</strong>: John List, thank you very much.</p>
<p><strong>John</strong>: Thank you.</p>