Study Overview
- Title:
- An experiment to promote the uptake of 3ie’s TREE review framework
- Study is 3ie funded:
- Yes
- Study ID:
- RIDIE-STUDY-ID-662bea2d35a67
- Initial Registration Date:
- 04/26/2024
- Last Update Date:
- 11/27/2023
- Study Status:
- In Development
- Location(s):
- India
United States
- Abstract:
Globally, there is now greater emphasis on integrating evidence into policy making. Government policies and programs affect society at large, research generating evidence therefore needs to be credible and implemented in a transparent and ethical manner. Concerns, however, exist around the credibility, rigor, and reproducibility of existing evidence. Experts have pointed out various issues, such as selective reporting, p-hacking, lack of reproducibility, and ethical concerns plaguing social sciences research. Also, awareness and use of best practices in evidence generation remains low. Promoting adoption and integration of best practices (such as, pre-analysis plan, computational reproducibility) into research workflow becomes important to produce credible and ethical evidence. 3ie’s initiative on Transparency, Reproducibility, and Ethical Evidence (TREE) aims to facilitate the adoption of TREE best practices by building capacity and generating resources such as the TREE review framework. Piloted in 2022, the TREE review framework defines a set of questions that evaluators can use to analyze their compliance with standard ethical requirements and TREE best practices. 3ie is actively seeking funding to refine, test and scale the TREE review framework. We are doing this study is an experiment to test the effectiveness of an information campaign on the uptake of the TREE review framework amongst researchers. Findings from this experiment will inform 3ie’s approach to scaling the initiative and support fundraising activities.
- Registration Citation:
- Categories:
- Other
- Additional Keywords:
- Secondary ID Number(s):
Principal Investigator(s)
- Name of First PI:
- Chandan Jain
- Affiliation:
- 3ie
- Name of Second PI:
- Sebastian Martinez
- Affiliation:
- 3ie
Study Sponsor
- Name:
- 3ie
- Study Sponsor Location:
- United States
Research Partner
- Name of Partner Institution:
- 3ie
- Location:
- United States
Intervention Overview
- Intervention:
The intervention is designed as an information campaign aimed at raising awareness about TREE best practices and the TREE review framework. The information campaign will also provide the treatment group with a link to the TREE review framework, which they can use to register on the framework’s portal and conduct a TREE review for their on-going studies. The information campaign will be delivered to the treatment group via an email.
- Theory of Change:
The information campaign aimed at raising awareness about the TREE review framework is expected to increase the integration of these practices into the research workflow, and thereby improve the process of evidence generation in the long run.
- Multiple Treatment Arms Evaluated?
- No
Implementing Agency
- Name of Organization:
- 3ie
- Type of Organization:
- NGO (International)
Program Funder
- Name of Organization:
- 3ie
- Type of Organization:
- NGO (International)
Intervention Timing
- Intervention or Program Started at time of Registration?
- No
- Start Date:
- 12/10/2023
- End Date:
- 12/09/2024
Evaluation Method Overview
- Primary (or First) Evaluation Method:
- Randomized control trial
- Other (not Listed) Method:
- Additional Evaluation Method (If Any):
- Other (not Listed) Method:
Method Details
- Details of Evaluation Approach:
The study is based on a cluster-randomized control trial, wherein we will randomly allocate on-going research studies to either the treatment group or control group. The researchers on these studies will comprise the participants in this experiment. If a study is assigned to the treatment group, then all associated researchers on that study will be assigned to the treatment group as a cluster. The treatment group consisting of researchers on studies assigned to the treatment will be exposed to an information campaign
- Outcomes (Endpoints):
Outcomes of interest
1. Awareness about the TREE Review Framework,
2. Use of the TREE Review Framework in upcoming or ongoing impact evaluations,
3. Use of TREE best practices in upcoming or ongoing impact evaluations.
- Unit of Analysis:
- Researchers
- Hypotheses:
The following are the research hypotheses of interest in the study -
1. What is the impact of exposure to the information campaign on the researcher's awareness of the TREE review framework?
2. What is the impact of exposure to the information campaign on the researcher's interest in using the TREE review framework?
3. What is the impact of exposure to the information campaign on the researcher's interest in incorporating TREE best practices in their future or ongoing research?
- Unit of Intervention or Assignment:
- Study level
- Number of Clusters in Sample:
- 453 studies
- Number of Individuals in Sample:
- 1244 researchers
- Size of Treatment, Control, or Comparison Subsamples:
- Equal split of the sample into treatment and control group
Supplementary Files
- Analysis Plan:
- Other Documents:
Outcomes Data
- Description:
- A baseline and an endline survey, conducted using online survey forms, will be used to measure outcomes.
- Data Already Collected?
- No
- Data Previously Used?
- Data Access:
- Data Obtained by the Study Researchers?
- Data Approval Process:
- Approval Status:
Treatment Assignment Data
- Participation or Assignment Information:
- No
- Description:
- We use the AEA study registry data to identify researchers for program participation.
- Data Obtained by the Study Researchers?
- Yes
- Data Previously Used?
- No
- Data Access:
- Not restricted - access with no requirements or minimal requirements (e.g. web registration)
- Data Obtained by the Study Researchers?
- Yes
- Data Approval Process:
- Approval Status:
Data Analysis
- Data Analysis Status:
Study Materials
- Upload Study Materials:
Registration Category
- Registration Category:
- Prospective, Category 1: Data for measuring impacts have not been collected
Completion Overview
- Intervention Completion Date:
- Data Collection Completion Date:
- Unit of Analysis:
- Clusters in Final Sample:
- Total Observations in Final Sample:
- Size of Treatment, Control, or Comparison Subsamples:
Findings
- Preliminary Report:
- Preliminary Report URL:
- Summary of Findings:
- Paper:
- Paper Summary:
- Paper Citation:
Data Availability
- Data Availability (Primary Data):
- Date of Data Availability:
- Data URL or Contact:
- Access procedure:
Other Materials
- Survey:
- Survey Instrument Links or Contact:
- Program Files:
- Program Files Links or Contact:
- External Link:
- External Link Description:
- Description of Changes:
Study Stopped
- Date:
- Reason: