The public debate is dominated by controversial and polemic issues in politics, science, and health. With growing evidence that Americans hold strongly polarised views about policies (Nyhan 2020, Alesina et al. 2020), it is becoming of critical importance for researchers and policymakers to develop new and rigorous methods to listen to citizens of different backgrounds and to better understand their views.
Surveys: A unique research tool
To this end, surveys are a unique tool to understand what is happening in society and in people's minds – what they think, how they reason, and what their concerns are. Few other research tools allow researchers to take such a snapshot of people's minds. When designed well, calibrated properly, and targeted to relevant samples of interest, surveys enable researchers to conduct online, large-scale investigations in real-time. They are intuitive, clear, and adaptive to participants.
Surveys have so far helped shed light on many relevant policy topics. One particularly fruitful line of research has leveraged survey data to explore people's perceptions and preferences about tax policy and redistribution (Cruces et al. 2013, Karadja et al. 2017, Alesina et al. 2018, Roth and Wohlfart 2018, Fisman and Kuziemko 2020, Hvidberg et al. 2021).
The value of open-ended survey questions
The backbone of surveys often consists of closed-ended questions that provide a fixed set of answer options. The advantages of these questions are that the answer options are standardised and streamlined across respondents and they easily lend themselves to quantitative analysis. However, in some settings, we may unintentionally prime respondents to think about answer options that they would otherwise not have thought about. Conversely, we may omit relevant options that we do not know about. In open-ended questions, respondents are not offered answer options, but rather an empty text entry field in which they can write freely. Open-ended survey questions can therefore circumvent some of the above-mentioned issues. By being less guided, they may teach us things that we may otherwise have missed and that we may not be used to thinking about. In this sense, answers to open-ended questions may help to shed light on the first-order considerations that come to people's minds without constraining them to think about a limited set of answer options.
What are some best practices when designing open-ended questions? Open-ended questions can range from very broad to narrow and specific. Broader open-ended questions are useful to elicit first-order, intrinsic concerns that people have before they are prompted to think of a particular policy aspect with more directed questions. Thus, it makes sense to start by asking people big picture questions such as the ‘main considerations’ that come to their minds when they think about an issue (e.g. income or estate taxes). It is then useful to narrow down the focus by asking people what they think about specific aspects of a policy, its advantages and disadvantages, or even more targeted questions about its consequences. Ideally, open-ended questions should be complemented with closed-ended questions for cross-validation.
Answers to open-ended questions may capture two different things. First, the answers of respondents who have not previously thought about the topic may be ‘gut reactions’. These reactions are informative, as they reflect what a respondent thinks and will keep thinking, absent more learning or focused reflection. Second, the answers of respondents who have already thought about the topic or who take time to think about it during the survey before answering may reflect more profound views. Either way, answers to open-ended questions capture the first-order considerations that matter to people and the aspects of an issue that are at the top of their minds.
Methods of analysis
In a recent project (Ferrario and Stantcheva 2022), we apply these open-ended question methods to data from two surveys on income and estate taxes conducted in 2019 on large and representative samples of US residents aged 18 to 70.
We focus on answers to the broad question, “What are your main considerations?”, when thinking about income or estate taxes, respectively, and we propose three different methods of analysis.
Word clouds are best used as a first step in visualising the data and for scanning answers quickly. Figures 1 and 2 show the word clouds derived from the responses. For income taxes, respondents express disagreement with the current levels of taxes, views on the direction in which to change them (“lower tax” or “higher tax,” depending on the groups they refer to), and concern about the impacts on the “middle class.” For estate taxes, respondents' first-order reaction centres around the issue of “double taxation” and the fairness of estate taxes, followed closely by concerns about the “middle class” and “family.”
Figure 1 Main considerations about income and estate taxes
a) Income tax
b) Estate tax
Notes: Word clouds based on answers to open-ended questions about respondents' main considerations on income and estate taxes.
Figure 2 Main topics respondents think about regarding the income and estate taxes
a) Income tax
b) Estate tax
Notes: The figure shows the distribution of topics mentioned in the answers about the main considerations on income and estate taxes. The bars represent the number of times a topic was mentioned out of the total mentions of any topic by political group.
It is then particularly interesting to study whether people belonging to different groups (e.g. of different political affiliations, ages, and incomes) use a systematically different lexicon. We focus our attention on the heterogeneity by political affiliation and cut our sample using a combination of the presidential candidate supported in 2016 (Clinton versus Trump) and of the self-reported degree of conservatism versus liberalism.
We consider two ways of investigating heterogeneity: keyness analysis and topic analysis. Keyness analysis is based on a relative frequency analysis that compares the use of words between two groups of people. This method attaches a score to each term, which measures how characteristic the term is of one group relative to the other. Topic analysis is based on a keywords-count model, where topics are defined by sets of keywords.
First-order concerns about income and estate taxes
For income tax, we identify eight distinct topics: Distribution, Fairness, Government Spending, Social Safety, Efficiency, Loopholes, Flat Tax, Public Goods, and Don't Know. The Distribution topic, for example, contains keywords such as “middle class”, “low income,” and “millionaire.” The Fairness issue contains the words “fair” and “unfair.” Public Goods captures “infrastructure”, “education”, and “health care.” Efficiency is represented by words such as “hurt economy”, “work less”, “competition”, and “spend less”, among others. The final category is for respondents who express that they do not know enough about the policy to give a meaningful answer.
The topics of Distribution and Fairness, as well as Government Spending and Loopholes, are at the top of many people’s minds. The Efficiency topic does not appear to be as salient. These results echo Stantcheva (2021), who finds that distributional and fairness considerations dominate efficiency concerns in shaping people's tax policy views.
For estate tax in Panel b, the topics are similar to those from the income tax survey. Specific to estate tax is the Double Tax topic, captured by keywords such as “already taxed”, and the Grieve topic, which is captured by terms such as “grieve”, “bury”, and “funeral”. The distribution of topics is again heavily centred around issues of Distribution and Fairness, but Double Tax is also important. Much rarer are mentions of Government Spending, Efficiency, Loopholes, Grieve, and Public Goods. Furthermore, more people express a lack of knowledge about estate tax than income tax.
Partisan differences in first-order concerns
There are clear political differences in the topics mentioned. Issues related to Distribution are much more prevalent on the left side of the political spectrum than on the right, while Government Spending is a more pressing concern for respondents on the right. Efficiency is not frequently mentioned by any political group. Fairness mentions are quite evenly distributed across the political spectrum. However, even though everyone cares about fairness, the meaning of this concept greatly differs across respondents; fairness is in the eye of the beholder.
Figure 3 shows the keywords that are most specific to Democrats and Republicans. Consistent with the topic distribution, the keywords on the Democrat side centre around issues of Distribution (“poor rich,” “lower class,” “middle class,” and “tax wealthy”). Republicans tend to emphasise Government Spending, Government Waste, and the Economic Costs (“people work”, “economy spend”). For estate tax, there are also clear differences in the topics mentioned by political leanings. Distribution issues are most prevalent among Clinton liberals and diminish rapidly and monotonically towards Trump conservatives. Conversely, Double Taxation concerns are prevalent among Trump conservatives, but quite rare among Clinton liberals. The mentions of Fairness are again evenly distributed across the political spectrum. These patterns are confirmed by the keyness analysis in Panel b of Figure 3.
Figure 3 Keywords mentioned by democrats and republicans regarding income and estate taxes
a) Estate tax
b) Income tax
Notes: The figure shows keywords among Democrats and Republicans in answers to the question about respondents' main considerations on the income and estate taxes. The score reported for a set of two words is the χ^2-test statistic, testing the null hypothesis that the occurrence of the given keywords is the same among Democrats and Republicans. * p<0.1, ** p<0.05, *** p<0.01.
Open-ended questions as a guide to in-depth research
The answers to open-ended questions can guide subsequent research. In the case of income and estate taxes, they reveal that people care about the distributional impacts of taxes and fairness. This prompts us to dig deeper into these issues using detailed (closed-ended) survey questions and experiments to better understand these concerns (Stantcheva 2021). Many research areas in economics could benefit from the use of open-ended survey questions to elicit people’s first-order concerns and guide more in-depth research.
References
Alesina, A, A Miano, and S Stantcheva (2020), "The Polarization of Reality", AEA Papers and Proceedings 110: 324-28.
Alesina, A, S Stantcheva, and E Teso (2018), “Intergenerational mobility and preferences for redistribution", American Economic Review 108(2): 521-554.
Cruces, G, R Perez-Truglia, and M Tetaz (2013), “Biased Perceptions of Income Distribution and Preferences for Redistribution: Evidence from a Survey Experiment", Journal of Public Economics 98(C): 100-112.
Ferrario, B and S Stantcheva (2022), “Eliciting People's First-Order Concerns: Text Analysis of Open-Ended Survey Questions”, AEA Papers and Proceedings 112: 1-8.
Fisman, R, K Gladstone, I Kuziemko, and S Naidu (2020), “Do Americans Want to Tax Wealth? Evidence from Online Surveys", Journal of Public Economics 188: 104207.
Hvidberg, K, C Kreiner, and S Stantcheva (2020), “Social Position and Fairness Views", NBER Working Paper 28099.
Karadja, M, J Mollerstrom, and D Seim (2017), “Richer (and Holier) Than Thou? The Effect of Relative Income Improvements on Demand for Redistribution", Review of Economics and Statistics 99(2): 201-212.
Nyhan, B (2020), "Facts and Myths about Misperceptions", Journal of Economic Perspectives 34(3): 220-36.
Roth, C and J Wohlfart (2018), “Experienced Inequality and Preferences for Redistribution", Journal of Public Economics 167: 251-262.
Stantcheva, S (2021), “Understanding Tax Policy: How Do People Reason?", The Quarterly Journal of Economics 136(4): 2309-2369.
