Download data and study materials from OSF
Sample size: 916
Field period: 10/13/2005-2/1/2006
These survey experiments probe the application of anchoring vignettes as a technique to reduce differences in how respondents understand survey questions. By asking individuals to assess hypothetical individuals on the same response scale as they assess themselves, anchoring vignettes can quantify differences in how survey respondents interpret the same question. Typically, however, individuals have been asked to assess themselves before assessing the hypothetical vignettes. Our experimental module employed vignettes placed before or after a self-assessment question to determine whether answering vignettes first improved respondents’ capacities to provide meaningful responses. We were especially interested in responses about abstract, multidimensional entities such as health, political efficacy, and social class. We also experimented with direct comparison questions, where respondents were asked to compare themselves directly to a hypothetical individual after hearing about that individual. We find that: 1) asking individuals to assess themselves immediately after hearing the vignettes produced responses that were more closely related to key covariates; and 2) inconsistent responses were more likely when individuals were asked to compare themselves directly to hypothetical individuals.
One central hypothesis is that the placement of the self-assessment question (e.g. “how much say do you have in getting your local government to consider issues important to you?”) immediately after vignettes about hypothetical individuals will produce responses that are more closely correlated with theoretically relevant variables, such as homeownership, income, and education. This is because vignettes prime individuals to think about the underlying dimension in the way that the researcher intended, and also because the vignettes familiarize the respondent with the response scale. We further hypothesized that one could eliminate the need to ask all of the vignettes and a separate self-assessment by instead asking a subset of “direct comparison questions,” in which respondents placed themselves with respect to a hypothetical individual.
For three different types of dependent variables of interest to different fields, we experimentally varied whether the respondent first was asked the self-assessment question or the vignettes assessing hypothetical individuals.
The three dependent variables are: 1) how much external political efficacy the respondent feels with respect to his or her local government; 2) how rested/energetic the respondent is; and 3) what economic class the respondent categorizes himself as.
One key finding is that for some concepts (such as political efficacy), hearing a set of vignettes immediately prior to answering the self-report makes it more likely that a respondent will understand the concept in the same way as the researcher. We demonstrate this effect by showing that standard covariates (such as education, income, and homeownership) are more predictive of the responses of those who reported their own efficacy after hearing the vignettes. We do not find the same improvement for individuals' self-reported class, suggesting that this technique works best on concepts that are relatively unfamiliar to respondents.
In analyzing the format of the self-assessment question, our findings show that there is more information contained in the responses to vignettes where respondents assess themselves separately. This is because responses to questions that make
direct comparisons are more likely to be inconsistent. That is, it appears better for researchers to ask people to rate hypothetical individuals separately and then to rate themselves, as opposed to making a direct comparison between themselves and
the hypothetical individuals. The apparent reason is that when making direct comparisons, respondents are likely to rate themselves as “equal” to multiple hypothetical individuals, even when doing so is inconsistent with previous responses.
When conducting analyses with vignettes, self-assessment questions should be placed immediately after the vignettes, so that the vignettes can prime the respondent to think about the question in the same way as intended by the researcher. Although more research into “direct comparison” questions is required, they present researchers with a trade-off. In theory, one can assess respondents with fewer questions, since no separate self-assessment is needed. In practice, however, biases toward acquiescence appear to lead responses to such questions to be less informative than standard vignettes.
Anchoring vignettes require post-processing to be appropriately interpreted. Specifically, each individual’s self-assessment must be re-scaled based on his or her responses to the vignettes. This creates a new dependent variable with 2n+1 categories, where n is the number of categories in the original question. Instead of knowing that a respondent placed himself as “very energetic,” we know that a respondent placed himself as more energetic than vignette three but less energetic than vignette one, for example. We then used a multiple-response ordered probit to estimate the predictive power that various covariates had on the rescaled dependent variable for the treated and control units. This study is ongoing, as we have obtained experimental data on the same questions from an internet panel.
Hopkins, Daniel, and Gary King. 2008. "Testing for Hierarchical Structure Among Individual Value Choices." Paper presented at the Annual Meetings of the American Political Science Association.
Hopkins, Daniel, and Gary King. 2010. Improving anchoring vignettes: Designing surveys to correct interpersonal incomparability. Public Opinion Quarterly. 74(2): 201-222.