Index of TESS FAQ items

Key Linked Documents

  1. Size limits for TESS studies
  2. Variables in TESS Standard Data Delivery
  3. Variables in AmeriSpeak's Public Affairs Profile
  4. Variables in AmeriSpeak's Health Profile
  5. Variables in AmeriSpeak's Financial Profile
  6. Variables in AmeriSpeak's Core Adult Profile
  7. List of TESS Associate PIs
  8. IRB Support Information from NORC
  9. Template of project specification for authors of successful proposals to NORC

Introduction

  1. What are the overall goals of TESS?
  2. What sampling methodology is used for TESS surveys?
  3. What kinds of proposals are appropriate?
  4. Who pays for TESS?
  5. Who runs TESS?
  6. What is a general population experiment?
  7. What are advantages of general population experiments?
  8. Why is time-sharing beneficial?

How to Submit

  1. Who is eligible to submit a proposal?
  2. When is the deadline for submission of a proposal?
  3. How do I submit a proposal?
  4. How long can my proposal be?
  5. What information must my proposal include?
  6. Can I submit more than one proposal at a time?
  7. What kinds of proposals are most likely to be successful?
  8. Are examples of successful proposals available?
  9. My proposal is neither experimental nor quasi-experimental. Will TESS accept it?
  10. How many items can I include? How many subjects can I have?
  11. Can I do a study of members of [specific subgroup]?
  12. Why are there limits on what I can ask for?
  13. What other items are included in every TESS data delivery?
  14. What additional items can be added to the TESS data delivery for my study?
  15. What if I need more respondent-questions than described above?
  16. What if my survey employs open-ended questions?
  17. What if I want to sample a specific subpopulation?
  18. What if I want to conduct an endowment experiment?
  19. What if I want to conduct an experiment that involves deception?
  20. What if I want to do an experiment with an open-ended prompt?
  21. Can TESS provide information on response times?
  22. Can TESS experiments include tasks implemented using specialized software?
  23. Can I tailor the information a participant receives based on where they are located?
  24. Can I do randomized block designs using TESS?
  25. Are there limits on the number of experiments that I can run on TESS?
  26. How do I submit a proposal to TESS's Short Studies Program (SSP)?
  27. Can I conduct my study by web and phone?
  28. Is it possible to conduct a second wave follow-up with respondents?

Short Studies Program

  1. How do I submit to the Short Studies Program?
  2. What are formatting guidelines for Short Studies Program proposals?
  3. Do projects have to be experiments?
  4. Do I have to get IRB approval for my short study?
  5. How will projects be reviewed?
  6. How will my study be fielded?
  7. How are special stimuli (video clips, news articles) counted as items?
  8. What if I need more than 3 items or 2000 respondents?
  9. What if my experiment requires a subsample rather than a general population sample?

Our Review Process

  1. What happens after I submit a proposal?
  2. How long will I have to wait to hear whether or not TESS accepts my proposal?
  3. How are resubmissions of previous proposals treated?
  4. Can I resubmit my rejected proposal to TESS?
  5. How many proposals will TESS accept?

After Acceptance

  1. What is provisional acceptance?
  2. What is the requirement for institutional review board (IRB) documentation?
  3. What happens after I get human subjects approval?
  4. Can I control the timing in which an experiment is fielded?
  5. Can I exclude smartphones from answering the survey?
  6. When will I receive the data from my experiment?
  7. For how long will I have exclusive access to the data from my proposal?
  8. What are my obligations to TESS after I receive my data?
  9. Will my proposal also be made available online?

Miscellany

  1. What's the story of your logo?

Introduction

What are the overall goals of TESS?

1. to provide social scientists with a new opportunity for original data collection and discovery
2. to promote innovative experimentation in social science
3. to increase the precision with which fundamental social, political and economic dynamics are measured and understood
4. to increase the speed and efficiency with which advances in social scientific theory and analyses can be applied to critical social problems
5. to maximize financial efficiency by combining otherwise separate studies, thereby radically reducing the average cost per study
6. to provide an online collection of exemplary experiments for teachers and students who want to learn more about social science experimentation

What sampling methodology is used for TESS surveys?

To achieve a representative sample, we contract with NORC, which conducts surveys using its AmeriSpeak Panel. AmeriSpeak is a nationally representative, probability-based panel based on NORC’s National Sample Frame, an area probability sample funded and managed by NORC and used for several NORC studies. These other studies include the General Social Survey funded by the National Science Foundation and the Survey of Consumer Finances sponsored by the U.S. Federal Reserve Board in cooperation with the U.S. Treasury Department. For the recruitment methodology, A summary of the AmeriSpeak Panel sample design used for the TESS projects can be accessed here.

While the AmeriSpeak Panel does include online and telephone interviewing, for all TESS surveys, data will be collected using online interviewing only. NORC will select the most representative sample available from panel members who have elected to participate in online surveys. The final data will be weighted to the appropriate population benchmarks for the target sample of interest in order to provide each investigator a representative sample.
What kinds of proposals are appropriate?

Proposals may come from any substantive area within any discipline in the social sciences so long as they utilize experimental designs and seek to make a valuable contribution to knowledge. 

In the most common experimental design in social science, different subjects are randomly assigned to groups that receive some different stimulus, and then differences between groups in some outcome are assessed.  Designs that involve within-subject experimental manipulations are also acceptable. 

TESS does not consider proposals that involve presenting the same survey questions to all respondents and comparing non-assigned subgroups (for example, men vs. women, or lower vs. higher educated respondents).

Experimental research often involves testing specific hypotheses, but this is not required of TESS proposals. 

Who pays for TESS?
TESS is funded by the Social, Behavioral, and Economic Sciences Directorate of the National Science Foundation.
Who runs TESS?

Jeremy Freese of Stanford University and James Druckman of Northwestern University are the current Principal Investigators.  A multidisciplinary team of Associate PI's assists Freese and Druckman in managing TESS. The team includes an accomplished and diverse roster of over 65 Associate PIs from across the social sciences. Team members span several generations and multiple disciplinary boundaries, and each member has established a reputation in his or her respective field. Most importantly, they share our enthusiasm for this project.

What is a general population experiment?
General population experiments allow investigators to assign representative subject populations to experimental conditions of their choosing.  TESS uses Internet technology to present randomly selected respondents with experimental stimuli of some kind. These stimuli can take any form; thus far, general population experiments have used systematic variation in the information that is given to respondents, alterations in question wording, pictorial stimuli, and differing incentives and instructions, just to name a few possibilities.
What are advantages of general population experiments?

General population experiments are intended to combine the strengths of experimental and survey designs in supporting causal inferences in the social sciences.  Surveys provide an efficient and well-studied means of gathering descriptive information about populations.  There are, however, often substantial obstacles to drawing strong causal inferences from conventional survey data. Over the years, many have hoped that advances in statistical methods would help scholars use such survey data to "partial out" or control for all plausible rival interpretations of a potentially causal relationship. Despite massive advances in statistical methods over the years, few people are as optimistic today that statistics can solve all such causal inference problems. Instead, throughout the natural and social sciences employ experimental designs in order to combat the challenges of causal inference, experiments are widely regarded as the most decisive design for adjudicating competing hypotheses about what causes what.

Social science has conducted a broad variety of different kinds of laboratory experiments.  While these laboratory experiments provide strong tests of causal propositions, scientific audiences, policy makers, and the public sometimes request more than a causal demonstration.  In many cases, science and society benefit from knowing that our laboratory observations survive exposure to myriad conditions outside of the lab. Moreover, some critics have questioned the extent to which the usual subjects in social science experiments resemble broader, more diverse populations ).

General population experiments offer a powerful means for researchers to respond to such critiques.  They allow researchers to assign large subject populations to experimental conditions of their choosing.  TESS uses Internet surveys to engage randomly selected respondents with randomly selected stimuli. By moving the possibilities for experimentation outside of the laboratory in this way, we can strengthen the internal validity of social science research and interest a much broader group of social scientists in the possibilities of experimentation.

As such, general population experiments offer many advantages to social scientists. Laboratory experimenters, for example, can use general population experiments to show that observations generated in a laboratory can be replicated in very different conditions. They can also test new hypotheses that emerge from their work with smaller groups of subjects. Alternatively, investigators who use the internet to run experiments on "knowledge networks" (e.g., markets, which aggregate diffuse individual behavior into prices and social systems that aggregate many individual characteristics into social hierarchies) can reinforce their research agendas by using general population experiments to evaluate individual-level hypotheses that inevitably follow from network-level observations. Likewise, scholars can use general population experiments to clarify the causal implications of findings from conventional surveys. Additionally, for a wide range of scholars, general population experiments offer new opportunities to innovate. A special advantage of general population experiments is the broad and diverse subject pools that they allow researchers to contact.

Such experiments are particularly effective at documenting differences in the status of causal hypotheses between the type of people who are usually selected for laboratory experiments and those who are not. Though not all social scientists require large and diverse subject populations to accomplish their research goals, many do. Finally, TESS also provides opportunities to strengthen and improve a wide range of measurement issues. For example, those interested in how to assess race and ethnicity in an increasingly diverse society can use experimental methods to understand how the method of data collection affects response attributes. Moreover, although the philosophy of early survey research was to attempt to create a social vacuum in which people could express their "true" beliefs and opinions, the more recent acknowledgment of attitudes, beliefs and preferences as a function of both the person and situation has led to an interest in the systematic study of how context alters the opinions and preferences that are expressed.

Why is time-sharing beneficial?

A critical element of the TESS research strategy is multiple studies from different disciplines sharing common observational platforms, all exploiting the inferential power and measurement efficiencies of experimental designs. This time-sharing on data collection platforms is the key to economic efficiencies of TESS. By distributing the costs of sampling and fielding over a large number of studies, the marginal cost of each study can be greatly reduced.   The start-up costs for launching any kind of large data collection effort are substantial. Time-sharing allows social science to experience the considerable economies of scale that come from paying such start up costs only once.

In this respect, time-sharing follows an established scientific tradition. The natural sciences, for example, promote progress in many areas by instituting time-sharing on expensive instruments (e.g., particle accelerators and telescopes). Time-sharing of university computer resources allowed many researchers to experience the benefits of computation before PC's became feasible and cost effective, and it is still used for many especially computing-intensive scientific applications.

Our time-sharing strategy also promotes efficiency in data collection by collecting demographic information that all investigators can share. If the experiments were conducted independently, each investigator would use time to collect such data on their own - increasing redundancy and reducing overall efficiency.

TESS provides other economic efficiencies by reducing waste. Most national surveys collect responses from some relatively large, round number of respondents. Because most surveys are not designed to test a small number of specific hypotheses, there end up being more cases than was actually needed for some purposes, and perhaps too few for others. The flexibility of our instruments allow people to roll on for as many or few cases as they need, and they must justify the sample size requested as part of the proposal. Flexibility in the instruments will allow more investigators to be served more efficiently.

How to Submit

Who is eligible to submit a proposal?
Any faculty member, postdoctoral fellow, or graduate student of any social science or social science-related department anywhere in the world. We regret that we cannot provide opportunities to any other individuals and groups.
When is the deadline for submission of a proposal?
Proposals are being accepted now and on a continuous basis. Excepting special competitions, TESS has no submission deadlines.
How do I submit a proposal?

All proposals must be submitted through our proposal handling system.

Each proposal must also designate a contact author in the submission system. The contact author must be listed first on all proposal documents and is the person to whom all official TESS corresponds will be held.

How long can my proposal be?

Proposals are limited to five pages of text (including footnotes/endnotes), plus references, up to two pages of tables, and the actual survey items to be included.

Power analyses, which are encouraged, may also be referenced in the main text and placed in a short appendix that does not count against the five page limit. When revisions are invited to proposals, these may be accompanied by a memo that details changes; concision here is encouraged. The entire proposal with all appendices or supplements of any sort may not exceed twenty pages under any circumstances.

Proposals must be double-spaced and 12 point font. A smaller font is allowed in footnotes and endnotes, and there are not specific formatting requirements for tables.

Proposals that exceed these limits will be returned by TESS staff.

What information must my proposal include?
To be successful, a proposal must include: A title, provided at the top of the first page of the proposal. A thorough description of the study design. An explanation of how the study will make a valuable contribution to science and society. An explanation of how people in other scientific disciplines will benefit from this study. A request for a particular number of respondent-items with justification. An appendix with actual questions and description of stimuli. In sum, the proposed experiments must evaluate important and clearly-stated hypotheses and be likely to generate new and broadly-applicable knowledge. To preserve the anonymity of the review process, we ask that the main text of proposals be stripped of content that identifies the proposer. Since proposals are linked to their authors by their user profiles, there is no need to include any of this information in the proposal itself.  Proposer names should not be listed on the front page or any page of the proposal, although references to previous research that are in stated in the third person are acceptable. If a proposal includes self-identifying content, it will be returned to the contact author along with a request that it be resubmitted without this information.
Can I submit more than one proposal at a time?
Yes.
What kinds of proposals are most likely to be successful?

We seek proposals that break new ground in the hypotheses they investigate, the procedures they employ, or both.

The key to TESS success is to win over reviewers in your chosen field. Ideally, your proposal should offer the potential for a clear scientific advance whose relevance expands beyond any one discipline.

Proposals that report trial runs of novel and focal ideas will be viewed as more credible.

While not required, it is desirable if the proposal is conducted in coordination with non-TESS data collection endeavors, such as traditional laboratory experiments or field work.

Are examples of successful proposals available?
Starting with proposals accepted in mid-2015, TESS proposals are made available a year after the data are delivered. They should be included along with study materials in the OSF pages for each project. (If you notice one is missing, please let us know.)
My proposal is neither experimental nor quasi-experimental. Will TESS accept it?

No. See the introduction for an elaboration of what we mean by an experimental design. 

How many items can I include? How many subjects can I have?

The "size" of a TESS experiment is a function of both the length of the experiment and the number of respondents (N). The shorter the experiment, the more respondents on which it can be conducted. This page provides the maximum N for studies of different length, as well as guidelines for how study length is calculated. Note that experiments that involve subsampling will involve some % reduction in the maximum N.

This allotment does not include the demographic and socioeconomic data that TESS provides for all studies. The standard delivery includes:

  • Gender
  • Race/ethnicity
  • Age
  • Education
  • Household income
  • Employment status
  • Marital status
  • # of members in household
  • Housing type
  • Home ownership
  • Home internet access
  • State of residence
  • Census region
  • Metropolitan Statistical Area (MSA) status
  • Telephone service
  • Device type (used to take survey)
  • Party affiliation
  • Political ideology
  • Religious affiliation
  • Religious service attendance
  • More information on the precise measurements provided for family income is available here.

    Can I do a study of members of [specific subgroup]?
    If the population subgroup can be identified by data collected by NORC, it might serve as the basis for an experiment. One issue is whether there are enough members of the subgroup in the AmeriSpeak Panel, after taking usual patterns in fielding and recruitment into account. For an experiment of 500 individuals, our experience suggests that being able to field an experiment on a subgroup representing less than 10% of the general US adult population is very unlikely. For experiments that need 1000 respondents to be adequately powered, these figures should be doubled.
    Why are there limits on what I can ask for?

    TESS provides a free service to investigators whose proposals are endorsed by the external reviewers, relevant Associate PI's and can meet standard human subjects requirements without placing an extraordinary burden on TESS resources. That TESS is a collective endeavor implies that there are strict limits on what services we can provide to any one investigator.

    The resource limits stated herein are rTESS provides a free service to investigators whose proposals are endorsed by the external reviewers, relevant Associate PI's and can meet standard human subjects requirements without placing an extraordinary burden on TESS resources. That TESS is a collective endeavor implies that there are strict limits on what services we can provide to any one investigator. The resource limits stated herein are real. Proposals are more likely to succeed, both in the review process and then once out in the field, if these limits are strictly interpreted.eal. Proposals are more likely to succeed, both in the review process and then once out in the field, if these limits are strictly interpreted.

    What other items are included in every TESS data delivery?

    A list of the items included for free in the TESS data delivery from NORC is provided here.

    What additional items can be added to the TESS data delivery for my study?

    TESS is conducted in the course of NORC AmeriSpeak Panel surveys, which also include sets of "profile" variables. Investigators in TESS studies may add profile variables to their studies, which are counted as just a fraction of an original survey item. The profile variables expected to be of most interest to TESS investigators are those on the Public Affairs Profile, the Health Profile, and the Financial Profile.

    What if I need more respondent-questions than described above?
    In a limited number of cases, TESS can provide additional respondent-questions. Such requests, however, are required to pass higher review standards than regular proposals. If the request entails substantial additional costs on TESS, we will have to reject the proposal or ask the proposer to pay the additional cost. More systematically, opportunities for larger studies have been provided through TESS special competitions.
    What if my survey employs open-ended questions?
    Researchers who intend to employ open-ended questions in their surveys should be aware that the maximum character limit for responses to such questions is 4000 characters. Researchers who anticipate needing over 1000 characters for these questions, or who will use prompts longer than a few sentences, should specify this in their proposal. Such questions may count as multiple units.
    What if I want to sample a specific subpopulation?
    TESS can provide samples of subpopulations, depending upon the type of subpopulation sought and the expense involved in reaching an adequate number of people within such a group. A few things of note on subpopulations: First, not all subpopulations can be reached with an adequate sample size, given the extant panel. Second, if the request is feasible but entails substantial additional costs for TESS, we will have to reject the proposal or ask the proposer to pay the additional costs. Finally, if a subsample is taken, we cannot guarantee it will be perfectly representative of that subpopulation since the probability sample is based on US population, in full, and not particular subsamples.
    What if I want to conduct an endowment experiment?
    An endowment experiment is an experiment in which a real-stakes reward is offered to participants. For example, participants may be offered a choice between some payoff for sure and a larger payoff that is subject to a gamble, where a payment in real money is made to the participant in accordance with their choice. TESS can be used to perform such experiments, but the investigator will have to provide funds (1) to cover the actual payments to respondents and (2) a 15% surcharge to NORC for handling the distribution of payments to respondents. In other words, if an endowment effect distributes $5 on average to 1000 respondents, the investigator will need to provide $5750 ($5000 for the payments and $750 for the surcharge).
    What if I want to conduct an experiment that involves deception?
    All TESS studies require IRB approval from the investigator's home institution and so would any deception would likewise need to be approved. In addition, any deception would need to be approved for fielding by NORC, which has conducted studies that provide a short 1-2 paragraph textual debriefing afterward.
    What if I want to do an experiment with an open-ended prompt?
    A common experimental technique is to have subjects write for a few minutes in response to a prompt, for example in order to induce a particular emotional state. We think such manipulations are very valuable when they work, but have become worried that they are not a good fit for TESS. Many respondents on web panel surveys often provide extremely cursory responses, and stipulations of our survey vendor do not allow investigators to press them to spend a particular amount of time on the item or write responses of a particular length. Of course, if the writing task is not taken seriously, there is no chance for the experiment to work, even if the hypothesis is correct. While we do not prohibit proposals for such experiments, we have become wary of fielding them without reason to believe they will surmount the problem of engaging participants sufficiently for the manipulation to work.
    Can TESS provide information on response times?
    Yes. For an extra charge, TESS can provide information on response times in milliseconds and can also present stimuli to experiments for a length of time specified in milliseconds.
    Can TESS experiments include tasks implemented using specialized software?
    TESS has run experiments that include a component that links to software implemented outside the AmeriSpeak platform.  An example would be a survey that includes a component linking to an Implicit Attitudes Task.  For purposes of calculating length with respect to TESS's size limits, each minute of task time is counted as 4 units.
    Can I tailor the information a participant receives based on where they are located?

    Maybe! The platform can be used to do this for information at various geographic levels, down even to the census tract.  However, this must be done in a way that the information can still be included as data that TESS can release publicly following the embargo period.  For any geographic information below the state level, the information given to a respondent cannot be so specific that it might identify their area uniquely.  For example, if an experimental stimulus was to include information about the racial composition of a census tract, the information that the tract was "56.45% White" might identify it uniquely, as opposed to "55-60% White."

    Can I do randomized block designs using TESS?
    Yes. In a randomized block design, randomization is done within categories of a variable and can improve the efficiency (power) of a study. Blocking can be done on profile variables that have been collected by NORC prior to the administration of your study.
    Are there limits on the number of experiments that I can run on TESS?
    There are no limits on the number of times investigators may use TESS. In fact, we encourage investigators to build on their previous TESS findings for subsequent proposals.
    How do I submit a proposal to TESS's Short Studies Program (SSP)?

    To submit a proposal to TESS's Short Studies Program please follow the instructions provided here.

    Can I conduct my study by web and phone?
    In some cases, yes. NORC’s AmeriSpeak Panel is a multi-mode panel, in which respondents may take surveys by web or phone. Multi-mode surveys can be advantageous in reaching a representative sample of U.S. adults, including non-internet households. However, not all projects are suitable for phone mode, such as experiments that use visual stimuli and TESS Short Studies, which are often bundled with other short studies. Projects will incur a 20% surcharge for including phone mode. Those additional costs may require you to reduce the sample size or provide supplemental funding to cover the difference, if the overall project reaches our limits. When you submit your project description, be sure to include that you would like your study conducted by web and phone. Otherwise, we will assume applicants prefer web mode only.
    Is it possible to conduct a second wave follow-up with respondents?
    Follow-up waves with TESS are possible in theory. In practice, these additions tend to be expensive and researchers can expect imperfect retention rates to decrease the overall sample size.

    Short Studies Program

    How do I submit to the Short Studies Program?

    Proposals should be e-mailed to tess.experiments@gmail.com, with the subject line “Short Study Proposal”. Proposals for this Program should not be submitted through the TESS Manuscript Central site, and any proposals submitted through the Manuscript Central site will be treated as regular TESS proposals, not SSP proposals.

    What are formatting guidelines for Short Studies Program proposals?

    Short study proposals must be 1-2 pages in length (single-spaced), plus the proposed items and any references included at the end as a separate page.  The experimental design of the study must be clear.  All proposals should include a title for the project on the first page, but no identifying information of the author.  Other information permitted as appendices to regular TESS proposals may also be appended to a Short Study proposal.  Proposals should be in PDF format.

    Do projects have to be experiments?

    Yes. See here for an elaboration of what we mean by an experimental design.

    Do I have to get IRB approval for my short study?

    Investigators for all TESS studies must receive appropriate approval from their institutional IRBs before we field their study.  Typically, investigators wait until after their study has been approved for fielding before pursuing IRB approval.

    IRB support information from NORC

    How will projects be reviewed?

    While regular TESS proposals are externally reviewed, for purposes of expediency, we will typically review these projects internally -- that is, by the Principal Investigators and/or Associate Principal Investigators. Feedback on proposals apart from the decision likewise should be expected to be brief and quick.

    How will my study be fielded?
    Short studies may be fielded in the same instrument as other short studies, or may be appended to the end of regular TESS survey experiments. Randomization to conditions will be independent across all short studies.
    How are special stimuli (video clips, news articles) counted as items?

    For regular TESS proposals, the ultimate arbiter of length for any TESS study are length determinations made by our survey vendor (NORC) for pricing studies. For purposes here, a stimulus will be considered one item unless its estimated time of administration is longer than 45 seconds, in which case it will be counted as more than one item. A stimulus that requires more than 90 seconds to administer needs to be submitted using the regular TESS mechanism rather than the Short Study Program.

    What if I need more than 3 items or 2000 respondents?
    SSP is a special mechanism of the larger TESS project, which allows proposals of larger projects (although still subject to rigorous size limits).
    What if my experiment requires a subsample rather than a general population sample?

    For ease of fielding, the Short Studies Program is for projects that involve no special subsampling.  If you have a short study that requires subsampling, you should submit a regular proposal using our regular mechanism.

    Our Review Process

    What happens after I submit a proposal?

    Upon receipt, we ask an appropriate Associate PI to provide us with the e-mail addresses of two prospective reviewers. The Associate PI is chosen from the proposer's home discipline. For co-authored proposals, we will choose an Associate PI from the home discipline of the first named author. We contact reviewers electronically, attaching preformatted request letters outlining the criteria for review. Completed reviews are received electronically by the co-PIs, and shared electronically with the appropriate Associate PI to obtain his or her advice about whether the project is both worthy and ready to earn a place in the study.

    Our review process is double-blind. The two referees will review proposals without knowledge of the identity of potential user, and the reviews will be returned to the potential users stripped of their identities as well. The Associate PIs and co-PIs will know the identities of potential users so that they can use that information to make sure all proposals receive fair consideration. The PIs will take care of conveying the good or bad news back to the potential users.

    In some circumstances, we also may ask for additional pre-testing of proposed projects. This includes sometimes offering to work with proposers to pre-test measures using Amazon Mechanical Turk in cases where the PIs deem it desirable to evaluate or strengthen a project.

    In sum, our organizational goal is to minimize both paperwork and the amount of time between when a good research design reaches us and when the experimental data are in investigators' hands.

    How long will I have to wait to hear whether or not TESS accepts my proposal?
    Our goal is to have an answer in investigators' hands approximately a month after receiving an application. At that point, TESS will state that the proposal is rejected, invite the proposer to send a revised version of the proposal for further review (i.e., revise and resubmit), or provisionally accept the proposal. Investigators who make submissions during or just before holiday periods and the summer months will likely experience longer waits.
    How are resubmissions of previous proposals treated?
    If you receive a "revise and resubmit" decision on your initial proposal, submit the revised version in the regular manner, through this web site. During the submission process, you will have an opportunity to check a box denoting the proposal as a resubmission. Do this. In most cases, the "R&R" letter from TESS will also instruct you to address reviewer concerns on an additional page. Please submit your response letter and your revised proposal letter as a single document. Such letters also specify how we will process the revised submission (e.g., whether we will send it back to one or both of the initial reviewers).
    Can I resubmit my rejected proposal to TESS?
    TESS's policy is that all decisions are final and not open to appeal.  Also, we do not accept re-submissions of declined proposals unless the new proposal differs fundamentally in the research question asked and/or the study's design
    How many proposals will TESS accept?

    For researchers, TESS is free. As a result, the number of requests for our services are typically far greater than the number of opportunities we can supply. Among the consequences of this:

    1. Our ability to provide opportunities for so many people depends on the internal economies of scale generated by having researchers share data collection instruments. We are open to a wide range of innovative ideas, but are constrained financially in what we can do regarding any single proposal. For proposals that go beyond our basic offering we require a higher review standard.  In addition, for options that impose unusual costs (such as streaming video presentations), we may make acceptance contingent on the proposer agreeing to pay the additional costs.

    2. In the event that the number of proposals received overwhelms the administrative capacity that our funding allows, we may find it necessary to revise the requirements for proposal submission. So please be sure to check this page for updates before submitting future proposals.

    After Acceptance

    What is provisional acceptance?

    When the review process results in a positive judgment for a proposal, we say that it is provisionally accepted. After TESS makes such a judgment, we notify the contact author with a letter of congratulations.

    Provisional acceptance is necessary but not sufficient for data collection to begin. Two additional steps must be taken: the proposal must obtain human subjects approval and it must have a successful trial run.

    What is the requirement for institutional review board (IRB) documentation?

    As with any empirical research done by faculty or graduate students, TESS cannot place an experiment on a data collection instrument, until it has in hand a document indicating that the project has received human subjects approval from a university human subjects committee or internal review board (henceforth, we use the term IRB to refer to such administrative bodies.)

    This documentation must be sent to TESS electronically to us at tess@ tessexperiments.org.

    There are no exceptions to this rule. There are also no circumstances in which TESS or its data collecting subcontractors can provide IRB reviews. IRB approval is the sole responsibility of the investigators.

    We encourage investigators to obtain IRB approval as soon as possible. The longer a researcher waits to get such approval, the longer will be the delay in collecting their data.

    NORC provides extensive information for use when applying for IRB approval of studies conducted using its panel (including TESS experiments).

    If your project is deemed non-exempt from review by your institution’s IRB, a representative of your institution will need to sign an Inter-Institutional Agreement (IIA) with NORC that establishes your institution as the reviewer of record. NORC will provide a template agreement that you are welcome to use.

    To repeat, if a proposal is accepted for placement on a TESS data collection instrument and if the investigators cannot obtain IRB approval, then it is ineligible for inclusion on a TESS data collection instrument.

    What happens after I get human subjects approval?

    You will be asked to submit a project description that can be used to obtain pricing information from NORC and that can be used to program the instrument. It is very important that this project description provide not only the questions but a clear description of the randomizations used and which groups of respondents receive which questions. The basic template for the needed project specification is here. The project specification should be provided in Microsoft Word or other standard word processor format (not as .pdf, as this is harder for programmers to work with).

    With the order of final acceptance as their guide, the co-PIs will exercise discretion in combining individual investigators' modules into a given version of the Internet data collection efforts. The TESS PI's and data collection subcontractors will work closely with individual scholars to make sure their modules are appropriately situated within larger instruments.

    For programming an instrument, multimedia materials will need to be provided to NORC in accordance with their specifications. Please provide a file with the best image quality and least compression possible. Preferred file types are as follow:

    Video: .mp4 or .ogg

    Audio: .wav or .mp3

    Image: .jpg, .gif , tif, .pct, .psd, .tga, .eps, or .ai

    Can I control the timing in which an experiment is fielded?
    (An example of this would be wanting an experiment to field after a specific Supreme Court decision is handed down.) Yes, it is possible to program and test ahead of time and then wait for an event to launch. The extra project management time that this involves may end up increasing the cost of the project to TESS, which might in turn necessitate a modest reduction in sample size or number of items to fit within our budget for a given project.
    Can I exclude smartphones from answering the survey?
    Yes. As of early 2018, about half of surveys are done by smartphones when respondents are given the choice (which is our default). Excluding smartphones will come with an increased cost to TESS that will likely result our needing to reduce the sample for the project by 10-20%. Excluding smartphones also is not feasible for certain subpopulation samples.
    When will I receive the data from my experiment?
    Our goal for investigators who have their proposals accepted is for them to enter the field and have the data back in their hands as soon as possible. We anticipate projects will take about six weeks from the time the final description of the approved project is received from investigators and when the data are delivered. As soon as your experiment is conducted, we will send the data to investigators along with subjects' demographic information and an electronic codebook.
    For how long will I have exclusive access to the data from my proposal?

    You will have exclusive access to the data for one year after the data are delivered. After that time, the data will become available to others via TESS's partnership with the Open Science Foundation. Our goal is to encourage you, the original researchers, to analyze, present, and publish your results as quickly as possible while making innovative data available to the larger scientific community.

    Researchers sometimes augment TESS data collection by providing supplemental funds from other sources. TESS policy is that all data from studies for which it provides any funding are posted, regardless of the fraction covered by TESS and the fraction covered by other sources.

    What are my obligations to TESS after I receive my data?

    All publications using TESS should include the following citation: "Data collected by Time-sharing Experiments for the Social Sciences, NSF Grant 0818839, Jeremy Freese and James Druckman, Principal Investigators."

    Investigators must also agree to prepare a summary description of their study and results. This summary must include a description of the treatment effects. Investigators will first be asked to provide a summary at the end of the one-year period in which the data are embargoed from other users. We recognize that investigators may not always be ready to provide a summary at that point; we do, however, require that all investigators provide the summary within three years after their project was accepted. The summary information is then posted on our website.

    To increase the speed at which new knowledge is made broadly available, we also require the investigators to notify us of any working papers or publications based on the experiments.

    Will my proposal also be made available online?
    Starting with proposals submitted/resubmitted after July 15, 2014, TESS posts accepted proposals for fielded projects online at the same time it posts the raw data. That is, one year after the data are delivered to investigators. We do this because of the growing interest in pre-registration of studies. While TESS studies do not contain all the elements of pre-registration, our investigators have from the beginning of TESS been articulating their hypotheses before fielding their study, and the policy change credits their doing so.

    Miscellany

    What's the story of your logo?
    TESS has had several logos over its history and they have all involved a tesseract, a hypercube-shape that is prominently featured in Madeleine L'Engle's book A Wrinkle In Time. The current logo was designed by Sasha Johfre.