Download Study

General

Study Overview

Title:

Can a socioemotional skills based curriculum boost the academic performance and aspirations of Indonesian adolescents?

Study is 3ie funded:

No

Study ID:

RIDIE-STUDY-ID-5ce4f41bdd8f1

Initial Registration Date:

05/22/2019

Last Update Date:

05/29/2020

Study Status:

Ongoing

Location(s):

Indonesia

Abstract:

This study is a large-scale randomized controlled trial that aims to improve the academic achievement and aspirations of 9th grade students in Indonesia. Indonesian students perform worse on international achievement tests than their counterparts in most other countries (PISA, 2015). Moreover, the Indonesian labor market is characterized by low female labor force participation and occupational sorting along gender lines. This gender-focused impact evaluation tests whether two socioemotional skills based interventions delivered under the Semua Bisa Sukses program (SBS, or “All Can Succeed”) can improve school outcomes and raise students' aspirations for their future education and career. The first intervention consists of a 6-week socioemotional skills curriculum based in the concept of growth mindset. Provided in combination with the student curriculum, the second intervention includes tools and activities that teachers can integrate in their daily interactions with students. The goal of these tools and activities is to shift the classroom environment to be conducive to adopting a growth mindset and to reinforce the principles of the training daily.

Change History for Abstract

Changed On

Previous Value

05/29/2020

This study is a large-scale randomized controlled trial that aims to improve the academic achievement and aspirations of 9th grade students in Indonesia. Indonesian students perform worse on international achievement tests than their counterparts in most other countries (PISA, 2015). Moreover, the Indonesian labor market is characterized by low female labor force participation and occupational sorting along gender lines. This gender-focused impact evaluation tests whether two socioemotional skills based interventions delivered under the Semua Bisa Sukses program (SBS, or “All Can Succeed”) can improve school outcomes and raise students' aspirations for their future education and career. The first intervention consists of a 6-week socioemotional skills curriculum based in the concept of growth mindset. Provided in combination with the student curriculum, the second intervention includes tools and activities that teachers can integrate in their daily interactions with students. The goal of these tools and activities is to shift the classroom environment to be conducive to adopting a growth mindset and to reinforce the principles of the training daily.

Registration Citation:

Johnson, H.C., Pervora, E., Vakis, R., De Martino, S., Trzesniewsk, K. and Molina, C., 2019. Can a socioemotional skills based curriculum boost the academic performance and aspirations of Indonesian adolescents?. Registry for International Development for Impact Evaluations (RIDIE). Available at: 10.23846/ridie175

Categories:

Education

Additional Keywords:

Gender, socioemotional skills, growth mindset, middle school, adolescent, Indonesia

Secondary ID Number(s):

Principal Investigator(s)

Name of First PI:

Hillary C. Johnson; Elizaveta Perova; Renos Vakis; Samantha De Martino

Affiliation:

World Bank

Name of Second PI:

Kali Trzesniewski; Diego Catalan Molina

Affiliation:

University of California, Davis

Study Sponsor

Name:

World Bank

Study Sponsor Location:

United States

Research Partner

Name of Partner Institution:

Indonesia Ministry of Education and Culture

Type of Organization:

Government agency (eg., statistics office, Ministry of Health)

Location:

Indonesia

Intervention

Intervention Overview

Intervention:

The first intervention consists of a 6-week socioemotional skills curriculum based in the concept of growth mindset and delivered to 9th grade students. Growth mindset refers to the perception that intelligence is malleable and that it is possible to improve one’s ability through practice and effort. The curriculum is rolled out in one 45-minute session per week for six weeks. Provided in combination with the student curriculum, the second intervention includes tools and activities that teachers can integrate in their daily interactions with students. The goal of these tools and activities is to shift the classroom environment to be conducive to adopting a growth mindset and to reinforce the principles of the training daily. The teachers receive tip sheets with ways to make their feedback to students promote growth mindset and materials for two activities: “I see SBS behavior” and “Sharing SBS experiences.” The schools also receive instructions for creating a Tim Perubahan, or change team, tasked with organizing a training workshop for teachers, supporting teachers in implementing the program, and organizing school-wide celebrations of classrooms that are particularly active.

Change History for Intervention

Changed On

Previous Value

05/29/2020

The first intervention consists of a 6-week socioemotional skills curriculum based in the concept of growth mindset and delivered to 9th grade students. Growth mindset refers to the perception that intelligence is malleable and that it is possible to improve one’s ability through practice and effort. The curriculum is rolled out in one 45-minute session per week for six weeks. Provided in combination with the student curriculum, the second intervention includes tools and activities that teachers can integrate in their daily interactions with students. The goal of these tools and activities is to shift the classroom environment to be conducive to adopting a growth mindset and to reinforce the principles of the training daily. The teachers receive tip sheets with ways to make their feedback to students promote growth mindset and materials for two activities: “I see SBS behavior” and “Sharing SBS experiences.” The schools also receive instructions for creating a Tim Perubahan, or change team, tasked with organizing a training workshop for teachers, supporting teachers in implementing the program, and organizing school-wide celebrations of classrooms that are particularly active.

Theory of Change:

The team expects the program to influence students’ educational outcomes, such as their performance on test scores. The team expects the program to bring about this change in the following way. The program will teach students that their skill level is malleable and that they can improve their abilities through effort and persistence. The students will also learn how to set goals, deal with distractions, and overcome setbacks. These skills can motivate students who are struggling in certain subjects in school to put in more effort and work to improve their knowledge and performance in these subjects. Rather than thinking there is no point in studying because they are either “good” or “bad” at a subject, students will be encouraged to adopt a growth mindset, understanding that they can improve their abilities. We hope that this change will translate into increased effort in school and diligence in studying. Improved effort and study habits will hopefully be reflected in the form of higher grades and better national test scores.

Change History for Theory of Change

Changed On

Previous Value

05/29/2020

The team expects the program to influence students’ educational outcomes, such as their performance on test scores. The team expects the program to bring about this change in the following way. The program will teach students that their skill level is malleable and that they can improve their abilities through effort and persistence. The students will also learn how to set goals, deal with distractions, and overcome setbacks. These skills can motivate students who are struggling in certain subjects in school to put in more effort and work to improve their knowledge and performance in these subjects. Rather than thinking there is no point in studying because they are either “good” or “bad” at a subject, students will be encouraged to adopt a growth mindset, understanding that they can improve their abilities. We hope that this change will translate into increased effort in school and diligence in studying. Improved effort and study habits will hopefully be reflected in the form of higher grades and better national test scores.

Multiple Treatment Arms Evaluated?

Yes

Implementing Agency

Name of Organization:

Indonesia Ministry of Education and Culture, with support from the World Bank

Type of Organization:

Public Sector, e.g. Government Agency or Ministry

Program Funder

Name of Organization:

World Bank

Type of Organization:

Foreign or Multilateral Aid Agency

Intervention Timing

Intervention or Program Started at time of Registration?

Yes

Start Date:

10/01/2018

End Date:

02/28/2019

Evaluation Method

Evaluation Method Overview

Primary (or First) Evaluation Method:

Randomized control trial

Other (not Listed) Method:

Additional Evaluation Method (If Any):

Other (not Listed) Method:

Method Details

Details of Evaluation Approach:

To evaluate the impacts of the program and compare the two packages of interventions, the team uses RCT methodology. Due to both concerns over spillovers and implementation constraints, the randomization is carried out at the school level. The impact evaluation takes place in 2,097 schools in 60 districts in Java and Sumatera: 699 schools are randomly selected to receive a 6-week program teaching growth-mindset and related concepts (treatment arm 1); 699 schools are randomly selected to receive additional teaching tools and activities for classroom teachers in addition to the 6-week program offered in the first treatment arm (treatment arm 2); 699 schools are randomly selected to serve as a control group for the impact evaluation and will not receive the interventions. Only public (Negeri) middle schools (Sekolah Menengah Pertama – SMP) on the islands of Java and Sumatera are targeted, due to logistics and cost issues. Sample schools are selected using probability proportional to size sampling, and random assignment is done stratifying by district.

Change History for Details of Evaluation Approach

Changed On

Previous Value

05/29/2020

To evaluate the impacts of the program and compare the two packages of interventions, the team uses RCT methodology. Due to both concerns over spillovers and implementation constraints, the randomization is carried out at the school level. The impact evaluation takes place in 2,097 schools in 60 districts in Java and Sumatera: 699 schools are randomly selected to receive a 6-week program teaching growth-mindset and related concepts (treatment arm 1); 699 schools are randomly selected to receive additional teaching tools and activities for classroom teachers in addition to the 6-week program offered in the first treatment arm (treatment arm 2); 699 schools are randomly selected to serve as a control group for the impact evaluation and will not receive the interventions. Only public (Negeri) middle schools (Sekolah Menengah Pertama – SMP) on the islands of Java and Sumatera are targeted, due to logistics and cost issues. Sample schools are selected using probability proportional to size sampling, and random assignment is done stratifying by district.

Outcomes (Endpoints):

Primary outcomes include: 1) academic performance 2) students' mindsets, perceptions, and socioemotional skills 3) school drop-out Secondary outcomes include: 1) Students' learning and studying behaviors 2) Teachers' mindsets, teaching practices, and perceptions of students 3) Students' aspirations and expectations for educational attainment and future occupation 4) Change in students' career choices

Change History for Outcomes (Endpoints)

Changed On

Previous Value

05/29/2020

Primary outcomes include: 1) academic performance 2) students' mindsets, perceptions, and socioemotional skills 3) school drop-out Secondary outcomes include: 1) Students' learning and studying behaviors 2) Teachers' mindsets, teaching practices, and perceptions of students 3) Students' aspirations and expectations for educational attainment and future occupation 4) Change in students' career choices

Unit of Analysis:

Individual level, with standard errors clustered at the school level

Hypotheses:

Primary hypotheses: 1) The interventions improve students' academic performance 2) the interventions improve students' mindsets, perceptions and socioemotional skills 3) The interventions decrease drop-out rates Secondary hypotheses: 1) The interventions change students' learning and studying behaviors 2) The interventions change teachers' mindsets, teaching practices and perceptions of students 3) The interventions increase students' aspirations and expectations for educational attainment and future occupation 4) The interventions change students' career choices

Change History for Hypotheses

Changed On

Previous Value

05/29/2020

Primary hypotheses: 1) The interventions improve students' academic performance 2) the interventions improve students' mindsets, perceptions and socioemotional skills 3) The interventions decrease drop-out rates Secondary hypotheses: 1) The interventions change students' learning and studying behaviors 2) The interventions change teachers' mindsets, teaching practices and perceptions of students 3) The interventions increase students' aspirations and expectations for educational attainment and future occupation 4) The interventions change students' career choices

Unit of Intervention or Assignment:

Random assignment was conducted at the school level, and the intervention is delivered at the classroom level

Number of Clusters in Sample:

2097 clusters (schools)

Number of Individuals in Sample:

56,000

Size of Treatment, Control, or Comparison Subsamples:

699 schools in each intervention arm

Supplementary Files

Analysis Plan:

Indonesia Aspirations Preanalysis Plan v5 clean.docx

Data

Outcomes Data

Description:

The impact evaluation uses a combination of survey and administrative data. Unique survey data was collected at baseline and endline and focused on measuring student aspirations and socio-emotional skills using both self-reported and behavioral measures. The baseline survey was collected in February and March 2018, when the students were in 8th grade. The endline survey was collected in February and March 2019, when the students are nearing the end of their 9th grade year.

Data Already Collected?

Yes

Data Previously Used?

No

Data Access:

Not restricted - access with no requirements or minimal requirements (e.g. web registration)

Data Obtained by the Study Researchers?

No

Data Approval Process:

Approval Status:

Treatment Assignment Data

Participation or Assignment Information:

Yes

Description:

Data Obtained by the Study Researchers?

Data Previously Used?

Data Access:

Data Obtained by the Study Researchers?

Data Approval Process:

Approval Status:

Data Analysis

Data Analysis Status:

Study Materials

Upload Study Materials:

Registration Category

Registration Category:

Prospective, Category 2: Data for measuring impacts have been collected by others but not obtained or analyzed by the research team

Completion

Completion Overview

Intervention Completion Date:

Data Collection Completion Date:

Unit of Analysis:

Clusters in Final Sample:

Total Observations in Final Sample:

Size of Treatment, Control, or Comparison Subsamples:

Findings

Preliminary Report:

Preliminary Report URL:

Summary of Findings:

Paper:

Paper Summary:

Paper Citation:

Data Availability

Data Availability (Primary Data):

Date of Data Availability:

Data URL or Contact:

Access procedure:

Other Materials

Survey:

Survey Instrument Links or Contact:

Program Files:

Program Files Links or Contact:

External Link:

External Link Description:

Description of Changes:

Study Stopped

Date:

Reason: