Consent to Participate in Research
This study supports the Intelligence Advanced Research Project Activity’s (IARPA’s) Hybrid Forecasting Competition (HFC) program by evaluating the accuracy of systems that combine human judgments with automated tools and data to generate high-accuracy forecasts about the likelihood of real-world events. With your help, this research may allow for more accurate predictions concerning future events in global security-relevant domains. This effort is conducted by the HFC Evaluation Team, led by the MITRE Corporation in collaboration with Good Judgment Inc. and Cultivate Labs.
During this study, you will be asked to engage in web-based activities that support the generation of forecasts about the likelihood of various real-world events occurring. Over the course of an approximately seven-month (28-week) forecasting tournament that is expected to start in February 2018, you will have the opportunity to work on numerous real-world forecasting questions, released at various times throughout the tournament. After this tournament ends, we anticipate more forecasting research opportunities (additional tournament seasons) for interested participants.
Intake Data Collection and Preparatory Activities
Prior to the start of the formal forecasting tournament, you will be asked to (a) complete a battery of intake questionnaires measuring personality, thinking style, and reasoning skills; and (b) complete a 45-60-minute forecasting skills training session—both via the web. The intake battery will be presented immediately after this informed consent, and the training will be presented approximately 1-3 weeks before the start of the tournament. In addition, if you enroll in the study by December 2017, you will have the opportunity to begin honing your forecasting and analysis skills by participating in a “pre-season” Hybrid Forecasting Challenge that will be active from August through December 2017, prior to the start of the tournament.
Forecasting Tournament and Third-Party Systems
At the start of the tournament, you will be randomly assigned to participate on a specific hybrid forecast system. This random assignment process, which is vital to the integrity of the research design, means that you will not be able to select the system to which you are assigned. As the tournament start date approaches, we will notify you of your hybrid system assignment and show you how to navigate to that system website. This is the main event for which we are recruiting you.
Please note that most participants will be randomly assigned to participate on a hybrid system that is designed, hosted, and managed by an independent research organization not directly affiliated with the HFC Evaluation Team. This is the goal of the HFC Program: for a set of independent research organizations to each propose and test its own hybrid system in competition with the others. Although these independent research organizations are also funded by IARPA, the HFC Evaluation Team plays no direct role in the design or oversight of their systems. Rather, the Evaluation Team’s role is to evaluate those systems, and your participation is vital to that evaluation. For this reason, when you are assigned to an independent research organization’s hybrid system (an interactive website), you will be asked to complete an additional informed consent form covering that independent system. Once assigned to an independent research organization’s system, any questions or issues that you encounter on that website during the tournament should be directed to that organization.
All HFC forecasting systems will be designed to require about two (2) hours of participation per week. Participation in this study is completely voluntary, and individuals may engage in forecasting activities for as much time as they like.
HFC participants like you may benefit by acquiring knowledge and reasoning skills through completion of forecasting training and by learning from your forecast experiences. In addition, it is hoped that the results of this study will provide valuable insight into reasoning, judgment, and decision making processes that will improve planning and anticipatory intelligence in all sectors of society.
There are minimal foreseeable risks to participating in this study. Although many forecasting questions will include descriptions of real-world global security issues (e.g., warfare, disease spread), these descriptions should be no more disturbing than similar descriptions appearing in the news. In addition, there is always some risk that the confidentiality of your data could be compromised (see next section). If for any reason you are uncomfortable or wish to withdraw from the study, you may stop participating at any time.
Confidentiality and Data Sharing
As part of this registration process:
- The HFC Evaluation Team will collect your full name and an email address.
- We will not share your full name with anyone outside the Evaluation Team.
- We will share the email address and other data you provide with the independent, third-party research organization that manages the forecasting system to which you are assigned.
- Additionally, after removing any information that directly identifies you, such as your name or email address, we may publicly post or share the data you produce. More details are as follows:
Your forecast data – Any forecast data you generate on an HFC system may be shared with the independent (third-party) research organization to which you are assigned or with the general public. Only the specific third-party organization to whom you are assigned will be able to link this forecast data to your email address. The third-party research organization to which you are assigned is responsible for securing such data once we transmit it to them.
Your email address – Your email address will be shared only with the third-party research team to which you are assigned, so that they can set up your account on their hybrid system website and provide “alerts” about the status of forecast questions or your participation activities. If you are concerned about this independent research organization having your email address, you may choose to use an address that does not reveal your name, and you may additionally set that account to automatically forward emails to your primary email address (simple instructions for doing so can be found on the web).
Your intake questionnaire data – Intake questionnaire data includes your responses to questions about demographics (age, gender, etc.), personality, attitudes, or reasoning problems. This data will be shared with the independent research organization separately from your email address and may be used to see how your intake data correlates with your forecasting behavior. Note that although your email address will not be directly attached to this intake questionnaire data when we share it, it will be possible for the independent research organization to whom you are assigned to link your email address to these responses.
Data generated on third-party systems - In addition to sharing information with the third-party organization to which you are assigned, we will request that these organizations share data with the HFC Evaluation Team so that we can maintain a complete data archive across the HFC effort. We will ask your assigned independent research organization to securely send us the data you generate there. Further, we will instruct your assigned independent research organization to not include your email address on any data they submit to us: Instead, we will ask that data be linked to an HFC Evaluation Team-generated participant id number that allows us to match your records but does not directly identify you. In addition, all independent research organizations will be instructed to protect your email address, limit the number of team members with access to it, and not disclose it outside of their research teams.
Sharing Non-Identifying Data
As noted above, we may publicly share or post the data you produce. Public sharing of data will not include any information that directly reveals your identity, such as your name or email address.
As with all research, there is a chance that confidentiality could be compromised. The HFC Evaluation Team is taking precautions to protect the data we collect or transmit, such as storing and transmitting information with encryption; limiting the number of people with access to your email address; and not directly linking email addresses to files containing other data you submit. Your confidentiality can be even further secured if you elect to use an email address that does not directly identify you by name, as described above.
Participation in research is completely voluntary. You have the right to decline to participate or to withdraw at any point in this study.
If you have any questions about participation in this study, you may contact the HFC Evaluation Team at [email protected]. If you have any questions or concerns about your rights and treatment as a research participant, you may contact [email protected].