Consent to Participate in Research

Purpose
This study supports the Intelligence Advanced Research Project Activity’s (IARPA’s) Hybrid Forecasting Competition (HFC) program by evaluating the accuracy of systems that combine human judgments with automated tools and data to generate high-accuracy forecasts about the likelihood of real-world events. This effort is conducted by the HFC Evaluation Team, led by the MITRE Corporation in collaboration with subcontractors Cultivate Labs, TurkPrime, and Good Judgment, Inc.

Study Overview
Over the course of an approximately 9-month forecasting tournament that is expected to start in late November / early December 2018, invited participants will have the opportunity to work on numerous real-world forecasting questions, released at various times throughout the tournament. After this tournament season ends, we anticipate more forecasting research opportunities (additional seasons) for those interested. Prior to the start of the formal tournament, interested participants will complete an intake questionnaire and participate in a limited forecasting “pre-season” that is expected to begin sometime in October 2018.

Eligibility
Please note that accepting the terms listed on this screen (and completing the intake questionnaire presented afterwards) does not automatically qualify you to participate in the 9-month forecasting tournament season. Here are further details:

  • If you are signing up between August and October 2018, you will have access to sample geopolitical forecasting questions, where you can practice making predictions by expressing the probability of an event occurring.
  • In October 2018, we will open a “pre-season” forecasting experience to select participants for the formal tournament season. Qualifying participants will be selected based on pre-season engagement levels, including number of forecast questions answered and frequency of forecast updates. The final selection formula is to be determined and will be announced at the start of the pre-season in October 2018.
  • All participants who participate in the pre-season will be notified of their selection status before the formal tournament begins in late November / early December 2018.
  • Note: A critical mass of active pre-season participants will be necessary to move forward with the formal HFC forecasting tournament. If pre-season engagement is insufficient to meet our overall project needs, the tournament will be cancelled.
  • HFC forecasting season participants are not eligible to participate in the IARPA Global Forecasting Challenge (GFC; https://www.iarpa.gov/challenges/gfchallenge.html). See below for further details.

Independent Research Organization (IRO) Involvement
At the start of the tournament (late November / early December 2018), invited participants will be randomly assigned to participate on a specific hybrid forecast system. Although one of the systems is directly managed by the HFC Evaluation Team, most participants will be randomly assigned to participate on a system that is hosted and managed by an independent, third-party research organization not directly affiliated with or under contract to the HFC Evaluation Team. Although these IROs also are funded by IARPA, the HFC Evaluation Team plays no direct role in the design or oversight of their systems and is not responsible for overseeing participant experiences on third-party IRO websites. For this reason, participants assigned to an IRO will be asked to complete an additional informed consent / terms of participation form created by the IRO to address your experiences and expectations for that independent system. Once assigned to an IRO’s system, any questions or issues that you encounter on that website during the tournament should be directed to that organization.

Rights
Participation in research is completely voluntary. You have the right to decline to participate or to withdraw at any point in this study without penalty, though this may impact your eligibility for gift card awards (see below). Please see below for information on eligibility, risks, confidentiality, and benefits of participation.

Benefits
HFC participants may benefit from their participation by acquiring knowledge and reasoning skills from their forecast experiences. In addition, once the regular forecasting tournament season starts in late November / early December, up to 3,000 participants will be eligible for an end-of-season Amazon Gift Card. Award amounts will be based on activity level, according to the following tiers:

  • $300 for high-frequency engagers (Tier 1): Make a first-time forecast in at least 10 questions every calendar month of the season. The most active 1,000 participants who meet this requirement will receive this gift card.
  • $50 for regular engagers (Tier 2): Forecast in at least 25 unique questions during the season AND in at least 1 new question every calendar month of the season. The most active 1,000 participants who meet this requirement and do not receive the $300 gift card will receive this one.
  • $15 for light engagers (Tier 3): Forecast in 10 or more unique questions during the season. The most active 1,000 participants who meet this requirement and do not receive one of the larger gift cards will receive this one.
Important notes and limitations concerning gift cards:
  • The awards are not additive: Each participant's award will be based on the highest activity tier achieved. All gift card awards are expected to be disbursed within 8 weeks of the season's conclusion.
  • We are not able to offer gift cards redeemable on non-U.S. websites (e.g., Amazon.uk).
  • If the tournament is cancelled prior to its commencement, no gift cards will be awarded.
  • If the tournament begins but is terminated early, the 3,000 overall most active forecasters (as of termination date) will be eligible for pro-rated gift card awards, as follows: Tier 3 gift card requires minimum 10 questions forecasted; Tier 2 and Tier 1 gift cards require minimum 25 questions forecasted; Tier 1 gift card value will be $30 multiplied by the number of forecast season calendar months completed. Only the 3,000 overall most active forecasters will qualify and only if they meet the above requirements.
  • Although our current plan is to award gift cards via Amazon.com, we reserve the right to use an alternative gift card format/platform in lieu of Amazon.com at our sole discretion.

Risks
Although many forecasting questions will include descriptions of real-world global security issues (e.g., warfare, disease spread), these descriptions should be no more disturbing than similar descriptions appearing in the news. In addition, there is always some risk that the confidentiality of your data could be compromised (see below for details). Again, if for any reason you are uncomfortable or wish to withdraw from the study, you may stop participating at any time. If you withdraw from the study and have met all the requirements to qualify for any of the gift card awards, you will receive that award (see above for details).

Confidentiality and Data Sharing
As part of this registration process, the HFC Evaluation Team will collect your email address.

  • We will share your email address with other parties under the specific circumstance that you have cross-registered with both HFC (this registration process) AND the IARPA Global Forecasting Challenge (see below for details).
  • Although we are not directly overseeing or responsible for the third-party IRO to which you are assigned, that IRO will share with us the data you produce on their website, including but not limited to forecasts and comments. We will instruct the IRO not to link this data to personally identifying information (e.g., email address) in transit to us, but we will be able to link the data to your email address by a common participant identification number.
  • After removing any information that directly identifies you, such as your email address, we will publicly post or share the data you produce for research purposes. This data may include but is not limited to: your intake questionnaire responses, forecast probabilities, and unstructured text comments, including data transmitted to us from the third-party IRO to which you are assigned. Important note: If you choose to identify yourself by name on your assigned platform—such as by choosing a personally identifying username or by referring to yourself in comments—you should assume that this data will become public.
  • Confidentiality risks – In addition to the data sharing practices noted elsewhere, there is always a chance that confidentiality could be compromised in other ways (e.g., hacking, data leaks). The HFC Evaluation Team is taking precautions to protect the data we collect or transmit, such as storing and transmitting information with encryption; limiting the number of people with access to your email address; and not directly linking email addresses to files containing other data you submit.

About the IARPA Global Forecasting Challenge
As noted above, HFC participants are not eligible to participate in the IARPA Global Forecasting Challenge (GFC; https://www.iarpa.gov/challenges/gfchallenge.html), which is managed by a third-party. The organizations administering the GFC may share GFC participant email addresses with the HFC Evaluation Team, and the HFC Evaluation team may notify the GFC Team of any HFC participant whose email address is cross-registered to both HFC and GFC. If this applies to you, your email address will be identified in any such communications with the GFC Team. In addition, for cross-registered individuals, the HFC Team may share aggregate participation data about that person to characterize the duration, timeframe, and level of that participant’s HFC activity. If you do not accept this practice or if you wish to participate in GFC instead of HFC, you should not accept these HFC terms of participation. If you do not cross-register on both sites, the HFC Evaluation Team will not share your email address or any other identifying information with the GFC Team.

Questions
If you have any questions about participation in this study, you may contact the HFC Evaluation Team at [email protected]. If you have any questions or concerns about your rights and treatment as a research participant, you may contact [email protected].