Learning to value value-added: Information and preferences in school choice

Robert Ainsworth, Rajeev Dehejia, Cristian Pop-Eleches , Miguel Urquiola 07 February 2021

a

A

Friedman (1955) famously argued that giving households freedom to choose schools would improve outcomes, including educational performance. At first pass, this is a straightforward claim, since it extends standard results from markets for consumer goods to education. Yet rigorous empirical work has produced mixed results on Friedman’s prediction. For example, voucher experiments suggest that choice can impact students’ measured skills in ways that are highly positive (Eyles and Machin 2015, Bettinger et al. 2017), modest (Muralidharan and Sundararaman 2015, Cohodes et al. 2019), or highly negative (Abdulkadiroglu et al. 2018).

Considering the mixed evidence on the effects of access to selective schools, Beuermann and Jackson (2020) observe that “the lack of robust achievement effects of attending schools that parents prefer is something of a puzzle”. The same puzzle emerges in other realms, including choosing residential neighbourhoods that improve children’s prospects later in life (Chetty et al. 2018) and choosing effective healthcare providers (Chandra et al. 2016).

Research has found that, in making choices, households often leave alternatives on the table that are better (as deemed by researchers), available, and (where relevant) cheaper 

For instance, while better neighbourhoods often have higher housing prices, there are opportunity bargains that produce good outcomes for children and are relatively affordable (Chetty and Hendren 2018). Similarly, while more-selective colleges tend to be better at increasing earnings, some less-selective colleges are also effective (Chetty et al. 2020). 

Finally, in the domain that we study, parents do not systematically select the schools in their choice set that improve student outcomes most over and above the baseline knowledge that students brought with them. Or to put it differently, while the most popular schools often attract the best students, they are not necessarily the schools that improve students’ outcomes to the greatest extent beyond their starting points – a concept known in the education literature as value-added (Bruhn et al. 2020).

These findings raise the question of why households do not always favour their highest value-added options. What constraints or factors lead them to other options? We explore these questions in the context of school choice in Romania. We consider two reasons why households may not choose the schools that researchers deem most productive. 

  • First, households may simply lack information. Value-added is considerably more difficult to observe than other school attributes, such as the quality of the school’s facilities or the achievement level of its students. Thus, it is possible that households do wish to attend high value-added schools, but do not know which those are. 
  • On the other hand, it may be that households’ preferences lead them to prioritise other school traits. For example, a given school might not provide the largest gains in skill, but it may provide a safe environment or a short commute. In this case, households may willingly give up value-added in exchange for alternative dimensions of quality (MacLeod and Urquiola 2019, Beuermann et al. 2019).

Distinguishing between preferences and information has important policy implications. If information is the obstacle to good choices, then making it available would improve households’ choices and spur providers to compete on value-added. By contrast, if preferences are the constraint, then policy options to boost value-added may be more limited. For instance, school choice may cause schools to invest in other, possibly less desirable, quality dimensions.

To investigate further, in Ainsorth et al. (2020), we ran an informational experiment in Romania in which we randomly provided a subset of middle-school families with information on the value-added of their high-school options. While there is a longstanding tradition of running informational experiments in economics (Deserranno 2015), ours is among the first to randomly intervene with value-added information in school choice, and it is informed by the goal of distinguishing information and preferences.

Before discussing the results of our experiment, it is useful to sketch two key features of the Romanian school-choice setting. High school is bookended by high-stakes standardised exams. Before entering high school, students take a national admissions test, known as the transition exam. The transition score is used to determine high-school assignments. Students rank their preferred choices and are assigned based on a serial dictatorship algorithm, which considers applicants one at a time from highest to lowest transition score, assigning each to his/her most-preferred school that has not yet reached capacity. What is relevant for us is that households have an incentive to reveal their true ranking of schools, since the algorithm will assign them to their highest-ranked school that is still available.

Before graduating high school, students take a national exit test, known as the baccalaureate exam. Performing well in the baccalaureate exam is crucial for moving on to higher education. Passing is required for admission to any university, and a high score helps win merit scholarships and entry to selective programmes. We use the two exams to calculate school value-added with respect to the probability of passing the baccalaureate exam; i.e. we look at the extent to which a high school increases the probability of a student passing the baccalaureate exam among students with similar entering transition scores.

In addition to using administrative data on the universe of high-school admissions, we implemented a detailed survey in our study sample. The two together offer a nuanced view of school choice. The administrative data reveal that, on average, households choose schools at the 68th percentile of value-added among their feasible options. This means that they could choose schools with about one standard deviation’s worth of additional value-added, i.e. there were choices available that would have increased their probability of passing the baccalaureate exam by 13 percentage points relative to a mean pass rate of around 60%. 

To understand why households leave so much value-added unexploited, we assessed households’ knowledge of this school trait. In our survey, we asked households to rate schools by value-added and find that these rankings explain less than a fifth of the actual cross-school variation. In other words, household behaviour is consistent with both caring partially about valued-added and with being imperfectly informed about value-added.

To break this logjam, we conducted an experiment in which we randomly informed selected households about the value-added of the schools in their towns. Our data show that providing information on value-added improves the accuracy of households’ beliefs and leads them to assign higher preference ranks to high value-added schools. In other words, the experiment achieved its proximate goal. Effects are larger for households with low-achieving children and for options that were initially less preferred. Specifically, our experimental treatment had no effect on beliefs or preference ranks for the two options that households ranked highest in the baseline survey.

As a result, the experiment also succeeded in its ultimate goal of shifting households’ high-school assignments toward higher value-added schools, albeit along the lines suggested by the uneven impact of the experiment on beliefs regarding value-added. For low-achieving students who were rejected by their two top choices, the experiment caused students to attend schools with 0.2 standard deviations’ worth of additional value-added. This translates to attending schools that increase the probability of passing the baccalaureate exam by 2.4 percentage points (9.6%). For all other students, the treatment had no effect.

The experiment is a powerful tool to demonstrate that providing information does lead to improved choices. However, we cannot rule out that our information intervention unintentionally changed household preferences. This is a limitation of most information interventions in the social sciences – when a group of researchers from foreign universities descend on your village or town and start talking about value-added, it is difficult to disentangle two effects: learning about value-added and learning to value value-added (i.e. for any level of information, simply placing more importance on this attribute of schools).

To tackle this issue, we turn again to our survey. Both before and after the experiment, we asked households to rank schools by value-added and by other attributes such as proximity, choice of subjects, and presence of friends or siblings. We first estimate preferences (using a discrete choice model). Then we compare predicted school assignments using estimated preferences combined in one case with households’ actual (imperfect) beliefs on value-added and in the other with our values estimated from the administrative data. 

We predict that fully correcting households’ beliefs would spur low-achieving (high-achieving) students to attend schools with 0.13 (0.11) standard deviations’ worth of additional value-added, representing only a modest fraction (18% for low-achieving students, 11% for high-achieving students) of the value-added that these households would otherwise leave on the table. 

Since our experiment had, in fact, a larger impact on low-achieving students than would have been predicted under the benchmark case of perfect information, it suggests that our experiment affected both the information and preferences of these students.

For any experiment or empirical finding, it is important to assess its external validity (Bryan 2019). In this regard, our conclusions are consistent with those that emerge in other contexts. For instance, Abdulkadiroglu et al. (2014) and Abdulkadiroglu et al. (2020) find that there is an overall correlation between value-added and selectivity in New York City, although not necessarily among elite schools. Beuermann and Jackson (2020) find that the most popular schools in Barbados are not those that improve test scores to the greatest extent. Our key results are also consistent with work finding that information on value-added may affect school markets less than data on absolute achievement (Mizala and Urquiola 2013, Imberman and Lovenheim 2016).

In conclusion, both information limitations and preferences seem to matter in school choice. Improving the choices that households make and creating an environment in which schools compete for productive dimensions of quality is likely to require deep interventions.

References

Abdulkadiroglu, A, J Angrist and P Pathak (2014), “The elite illusion: Achievement effects at Boston and New York exam schools”, Econometrica 82(1): 137–96.

Abdulkadiroglu, A, P A Pathak, Jonathan Schellenberg and Christopher R Walters (2020), “Do parents value school effectiveness?”, technical report.

Abdulkadiroglu, Attila, P Pathak and C R Walters (2018), “Free to choose: Can school choice reduce student achievement?”, American Economic Journal: Applied Economics 10(1): 175–206.

Ainsworth, R, R Dehejia, C Pop-Eleches and M Urqiola (2020), “Information, preferences, and household demand for school value added”, NBER Working Paper 28267. 

Bettinger, E, M Kremer, M Kugler, C Medina, C Posso and J E Saavedra (2017), “Can educational voucher programs pay for themselves?”, mimeo, Harvard University.

Beuermann, D W, and C K Jackson (2020), “Do parents know best? The short and long-run effects of attending the schools that parents prefer”, Journal of Human Resources, no. 2.

Beuermann, D, C K Jackson, L Navarro-Sola and F Pardo (2019), “What is a good school, and can parents tell? Evidence on the multidimensionality of school output”, technical report, Northwestern University.

Bruhn, J, C Imberman and M Winters (2020), “Regulatory arbitrage in teacher hiring and retention: Evidence from Massachusetts charter schools”, VoxEU.org, 1 November.

Bryan, K (2019), “What randomisation can and cannot do: The 2019 Nobel Prize”, VoxEU.org, 29 October.

Chandra, A, A Finkelstein, A Sacarny and C Syverson (2016), “Health care exceptionalism? Performance and allocation in the US health care sector”, The American Economic Review 106(8): 2110–44.

Chetty, R, J N Friedman, N Hendren, M R Jones and S R Porter (2018), “The opportunity atlas: Mapping the childhood roots of social mobility”, NBER Working Paper 25147.

Chetty, R, J N Friedman, E Saez, N Turner, and D Yagan (2020), “In- come Segregation and Intergenerational Mobility Across Colleges in the US,” The Quarterly Journal of Economics 135(3): 1567–1633

Chetty, R, and N Hendren (2018), “The impacts of neighborhoods on intergenerational mobility II: County-level estimates”, The Quarterly Journal of Economics 133(3): 1163–228. 

Cohodes, S, E Setren and C Walters (2019), “Replicating successful charter schools in Boston”, VoxEU.org, 22 August.

Eyles, A, and S Machin (2015), “Academy schools and pupil performance”, VoxEU.org, 22 September.

Friedman, M (1955), “The role of government in education”, in Robert Solow (ed.), Economics and the Public Interest, NJ: Trustees of Rutgers College.

Imberman, S, and M Lovenheim (2016), “Does the market value value-added? Evidence from housing prices after public release of school and teacher value added”, Journal of Urban Economics 91: 104–21.

MacLeod, W B, and M Urquiola (2019), “Is education consumption or investment? Implications for the effects of school competition”, Annual Review of Economics 11: 563–89.

Mizala, A, and M Urquiola (2013), “Parental choice and school markets: The impact of information on school effectiveness”, Journal of Development Economics 103: 313–35.

Muralidharan, K, and V Sundararaman (2015), “The aggregate effect of school choice: Evidence from a two-stage experiment in India”, The Quarterly Journal of Economics 130(3): 1011–66.

a

A

Topics:  Education

Tags:  education, educational outcomes, Romania, schools, secondary school, student performance

Assistant Professor of Economics, University of Florida

Professor of Public Policy and Economics, New York University

Associate Professor at SIPA, Columbia University

Professor and Chair, Department of Economics, Columbia University

Events

CEPR Policy Research