Sample size: 2030
Field period: 11/19/2015-02/10/2016
This module was a follow-up study testing the feasibility of asking alternative measures of sex and gender in nationally representative surveys (see also “Gender Identity and Diversity,” Saperstein and Westbrook, 2014). Our previous study asked respondents about their sex assigned at birth and their current gender, and then asked a direct question about their transgender status after providing a definition of the term. The proportion of respondents who provided an affirmative response to the transgender question was unexpectedly large in our first study (3.8%), and was not consistent with responses to the other two items. The present study sought to test whether changing the answer options for the transgender question might yield results that were more consistent across measures and more in line with estimated prevalence from previous research using a similar question in telephone surveys. Participants in our follow-up study were randomly assigned to see the transgender question as it was originally structured or a version of the transgender question with the order of the answers reversed. Overall, as with our previous study, we aimed to assess the performance of separate sex and gender items on a nationally representative sample, and to determine whether a direct question about transgender status would be a useful supplement to help identify possible false positives (or false negatives) from the two-step sex at birth and current gender approach.
The present study sought to test whether changing the order and wording of answers would decrease the number of participants who said “yes” they could be considered transgender after reading a definition of the term. We hypothesized that potential false positives would be reduced if “no” preceded “yes” among the two available answers and if the answers were written out as complete sentences to make it clearer what respondents were selecting “yes” or “no” to mean.
In our previous study, we also found that affirmative responses to the transgender definition question were significantly higher when our module came toward the end of the survey compared to when it came at the beginning (2.2% vs. 5.4%, module order was randomized). So, we sought to assess whether this result was related to the specific modules that preceded ours in the previous study, or might be more generally related to “survey fatigue,” which would it inadvisable to ask a transgender question that includes a two-sentence definition near the end of an online survey of the general population.
The text of the transgender definition question was the same in both conditions and the same as in our previous study. However, in this study, half of respondents were randomly assigned to see the answer options as “Yes” and “No”, while the other half were given the answers “NO, I would not be considered transgender.” and “YES, I would be considered transgender.”
Sex at birth, current gender identity, and transgender status were the key outcomes measured. Item non-response was also assessed.
Non-response was relatively low, as before. Thirteen respondents (0.6%) did not provide their sex at birth, ten (0.5%) did not provide their current gender, and twelve respondents (0.6%) did not respond to the transgender definition question.
Based on combined information from the sex at birth and current gender questions, 20 respondents (1%) identified in ways that would not be captured by a standard single-item sex/gender measure. These included 11 people who reported being male at birth who did not currently identify as men, seven people who reported being female at birth who did not currently identify as women, and two people who reported being intersex at birth. The proportion of respondents who identified as transgender indirectly (answering female and man, or male and woman) versus directly (identifying as transgender on the gender question) did not vary significantly by survey condition; thus unbalanced samples do not explain the differences below.
Results from the experimental manipulation supported our hypothesis in the sense that about 40% fewer respondents reported that they were transgender when they saw the full-sentence “no first” answer options compared to the one-word “yes first” answer options (3.6% compared to 6.1%). However, the 3.6% estimate is still much higher than other estimates of the transgender population from state-specific telephone surveys, and higher than the transgender population as estimated by comparing responses to the sex at birth and current gender questions in our own study. This persisting high rate of affirmative responses could be explained by our module always appearing toward the end of the survey in this study. It is therefore possible that survey fatigue lead to erroneous answers. Had our module appeared first in this study, and with a similar 40% reduction in affirmative responses using the adjusted answer options, we might have seen something closer to 1.3% identifying as transgender in response to the definition question.
There is a need for research on best practices for measuring sex and gender in surveys of the general population. However, because experiences that do not fall into conventional dichotomous response categories (male/man and female/women) are relatively rare, assessing the reliability and validity of such measures has proven complicated.
As we found in our 2014 TESS survey, there was inconsistency in how participants identified their gender over time. Twenty-one participants reported their sex as male in demographic profile data but identified as cisgender women in this survey, while 6 participants who had previously identified as female identified here as cisgender men. However, these cases are significantly more likely to occur in multi-adult households (26 cases) than single-adult households (1 case). It is possible, then, that these inconsistencies arose because a household member other than the account-holder completed this survey.
We were also puzzled by the high rate at which participants identified as transgender on the transgender definition question, especially because these results barely overlapped with those who were transgender according to the first two gender measures of our survey. Of the 98 people who said that they were transgender on the yes/no definition question, just 4 identified as non-cisgender according to the two-step sex at birth and current gender questions. Responding affirmatively to the transgender definition question occurred about twice as often in the yes-first answer option condition, suggesting that our manipulation did lower false positives, but even in the no-first condition 35 participants identified as transgender on the yes/no question but as cis-gender according to the sex and gender questions. Given its unexpectedly high prevalence estimate, and this lack of agreement between the questions, we cannot recommend the transgender definition item for use in online, self-completion surveys.