Gender Identity and Diversity
Download data and study materials from OSF
Grand Valley State University
Sample size: 2144
Field period: 11/24/2014-12/30/2014
This study tested the feasibility of asking alternative measures of sex and gender in nationally representative surveys. Conventionally, self-completion surveys include a single item that asks whether respondents are male or female. This standard measure conflates the distinct concepts of sex and gender and also masks diversity not captured by a male/female dichotomy. Our experimental module asked respondents to report their sex at birth and current gender separately, a two-step approach advocated by experts from UCLA's Williams Institute. Sex response options in our module included female, male, and intersex, while gender response options included woman, man, transgender, and a gender not listed here (please specify). The module also included a third question intended to confirm whether or not respondents could be considered transgender – whether or not they identify explicitly as such. This item included a definition of transgender similar to that fielded previously by the Behavioral Risk Factor Surveillance System in Massachusetts. Question order and placement of our module within the questionnaire were randomized to assess their effect, if any, on responses. Overall, the study was intended to add to the growing literature on best practices for measuring sex and gender in surveys.
The key question motivating this study was whether a two-step approach to measuring sex and gender could feasibly be incorporated in self-completion surveys administered to nationally representative samples. We hypothesized, based on previous pilots with national samples, that item non-response would not compromise data collection.
We also aimed to assess the overall level of response in categories other than male/man and female/woman, and the consistency of responses across multiple items, including those previously recorded in the respondents' demographic profiles.
Finally, we were interested in whether reporting consistency or non-conventional responses were more frequent when the sex at birth question preceded the question on current gender identity, or vice versa.
Question order and placement within the questionnaire were randomized. Half of respondents were asked first to report their sex at birth, then their current gender, and finally to confirm whether or not they could be considered transgender according to a standard definition. The remaining half of the sample was asked to report current gender first, followed by sex at birth, and the confirmation question. In terms of overall placement in the survey, half of respondents answered our questions first, before completing two other studies, and half of respondents answered our questions after they completed two other studies.
Sex at birth, current gender identity, and transgender status were the key outcomes measured. Item non-response was also assessed.
Summary of Results
Non-response was negligible, as expected. Seven cases were missing on sex at birth and current gender, with six respondents out of 2,144 (0.3%) refusing to answer either question. Notably, five of those six cases occurred when our module came at the end of the survey. A similar pattern of non-response was observed for the transgender confirmation question.
Based on the sex at birth and current gender questions, 18 people (0.8% of the sample) reported in ways that would not be captured by conventional measures. These included eight people who reported being male at birth who did not currently identify as men, seven people who reported being female at birth who did not currently identify as women, and three people who reported being intersex at birth. Of these, three people identified explicitly as transgender and two selected "a gender not listed here." The frequency of gender variant respondents did not differ by question order (i.e., sex at birth asked first or second), or by when the module appeared during the survey. Indirect transgender responses – answering female and man, or male and woman – also were equally common across conditions; although female/man responses were more common when our module appeared first in the survey (5 of 7), while male/woman responses were more common when the module appeared last (4 of 5).
The sex at birth and current gender identity questions, as posed here, were piloted on two samples of U.S. adult workers from Amazon Mechanical Turk. In general, the frequency of gender variant responses based on these items was similar between the Mturk and TESS samples, and in line previous research. However, the frequency of affirmative responses to our transgender confirmation question was unexpected and potentially troubling from a methodological perspective. Nearly 4 percent of respondents said "yes" they could be considered transgender, after being provided with a definition of the term. This estimate is larger, by an order of magnitude, than previous research would have predicted, and it varied significantly by whether or not our module appeared first or last in the survey (2.2% and 5.4%, respectively).
Additional research on best practices for measuring sex and gender in surveys is needed. There are, as yet, no national benchmarks for the frequency of gender variant responses to alternative measures, and the expected rareness of experiences that do not fall into conventional dichotomous response categories (male/man and female/women) further complicates assessments of reliability and validity.
Nevertheless, the level of inconsistency across the various items measuring sex and gender was notable. Some inconsistency between old and new measurement methods is to be expected: given that the demographic profile data on respondents conflates sex and gender, older responses to a single question asking "Are you male or female?" might not match up exactly with separate responses for sex at birth and current gender identity. A response of "male" on the profile item could be referencing sex at birth, current gender, or both. That said, we did not expect that people who are classified as cisgender on our items (i.e., female and woman or male and man) would have reported using the other category in their demographic profile. Yet 23 people who previously identified as male (2 percent of previous males) are recorded as cisgender women in our data, while 14 people who previously identified as female (1 percent of previous females) are recorded as cisgender men. This type of inconsistent response between old and new measures appears to be mitigated, but not entirely eliminated, when our module was presented first in the survey.
The relatively high rate of transgender reporting on our follow-up item that included a definition of transgender was even more problematic in terms of its inconsistency. Given the lack of concordance between it and all of the other available measures of sex and gender, as well as its clear sensitivity to module placement in the questionnaire, we cannot recommend the transgender confirmation item for use in self-completion surveys. We are currently testing modifications to this question to rule out concerns about survey fatigue, response comprehension, or a combination of the two that could have influenced our results.