Download Study

General

Study Overview

Title:

Evaluating the Design and Impact of the Secondary School Teacher Training Initiative under the Government of Nepal’s School Sector Development Program

Study is 3ie funded:

Yes

Study ID:

RIDIE-STUDY-ID-5d2ff6ec3a751

Initial Registration Date:

07/17/2019

Last Update Date:

07/21/2020

Study Status:

Ongoing

Location(s):

Nepal

Abstract:

This RCT-based study is designed to evaluate the impacts on the subject knowledge and pedagogical practices of teachers of 9th and 10th grade math and science, and on student learning, of Nepal's SSDP teacher training (TT) program.  The main teacher training (TT) intervention requires all 9th and 10th grade teachers in government schools to attend government-run in-service teacher training modules that are intended to raise their subject knowledge and to motivate and equip them to use practical, demonstration-based teaching methods rather than more traditional teaching methods. The training includes a 10-day session at an Education Training Center (ETC), and then completion by participating teachers of the equivalent of five days of “selfstudy project work,” which includes independent lesson plan development and other classroom research and practice activities, on which they must submit a report within 45 days of completing their training at the ETC.  The supplementary Video Assignment (VA) treatment requires each trained teacher to submit (before he or she can receive full credit for the training) a video of himself or herself (during an entire class session) implementing one of the new lesson plans that he or she is expected to create as part of the self-study project. The aim of adding the VA is to increase teachers’ motivation for investing serious effort in the self-study project activities, which may be important for translating what teachers learn at the ETC into new and improved classroom practices.  The evaluation is a randomized controlled trial, using 204 schools in 16 different districts in Nepal.

Registration Citation:

Glewwe, P. and Schaffner, J., 2019. Evaluating the Design and Impact of the Secondary School Teacher Training Initiative under the Government of Nepal’s School Sector Development Program. Registry for International Development for Impact Evaluations (RIDIE). Available at: https://www.doi.org/10.23846/ridie180

Categories:

Education

Additional Keywords:

Secondary ID Number(s):

Principal Investigator(s)

Name of First PI:

Paul Glewwe

Affiliation:

University of Minnesota

Name of Second PI:

Julie Schaffner

Affiliation:

Tufts University

Study Sponsor

Name:

3ie

Study Sponsor Location:

India

Research Partner

Name of Partner Institution:

Center for Policy Research and Consultancy (CPReC)

Type of Organization:

NGO (local) or other civil society organization

Location:

Nepal

Intervention

Intervention Overview

Intervention:

The study examines two interventions. The Teacher Training (TT) treatment provides new training for teachers of grade 9 and 10 math, science, English and Nepali. This study focuses on math and science. All teachers of these classes were required to take a course on teaching their main subject at the secondary level. This course, which includes subject content, more interactive teaching methods, and assessment methods, was developed by the National Center for Educational Development in collaboration with Education Training Centers (ETCs). The training provides teachers 10 days of classes at an ETC, who then do 5 additional days of practical application in their own classrooms, and submit a report on their experiences within 45 days. Before returning to their schools, teachers must create specific improvement plans based on what they learned. Within 30 days of returning to their schools, they must create 15 lesson plans (including objectives, activities and process for student evaluation); employ the plans in their classrooms; have a peer or head teacher, or resource person observe and give them feedback; conduct action research on some problem; and write a report on these experiences (which must be verified by the head teacher). Teachers are graded on their training: up to 50 points. The Video Assessment (VA) treatment is an addition to the TT treatment. It is intended to raise trained teachers’ accountability for practicing in their schools what they learned in training.  It specifies that teachers receive credit for their classroom practice and associated report only if they submit a video of themselves employing some part of their improvement plan in their classrooms. Each ETC assigns an employee to arrange with teachers to record the required videos. 

Theory of Change:

The theory of change for the TT treatment is the following. The intervention will improve student learning only if the following assumptions hold. Math and science secondary school teachers currently lack knowledge of subject content, interactive pedagogical practices, and/or assessment methods, or motivation to experiment with new practices in their classrooms.  The training curricula must be of high quality and address these needs, and must be implemented by high quality trainers. Teachers must participate in the trainings. Teachers must complete the practical application part of the trainings. After starting to use the new methods in their classrooms, they must continue to teach more effectively, and must have adequate materials, and must not run into opposition from parents or others. Students must respond to these new teaching practices by learning more from a given quantity of their time in learning activities, or by increasing the time spent on those activities. Students must be neither so poorly nourished nor so tired (from long work hours outside of school) that they cannot learn, and must expect that studying will improve their lives. For the VA treatment to strengthen the impact of the TT treatment, the following assumptions must hold. The schools must obtain video cameras to use from the ETCs or other sources. Teachers must care enough about obtaining credit for training courses that they arrange for a video to be taken of their use of the new teahcing practices. They must understand what they must show in the videos to receive credit, and must be able to deliver the videos to the ETCs and believe that their use of new teaching methods in the videos will be assessed accurately by the ETCs. 

Multiple Treatment Arms Evaluated?

Yes

Implementing Agency

Name of Organization:

Ministry of Education, Nepal

Type of Organization:

Public Sector, e.g. Government Agency or Ministry

Program Funder

Name of Organization:

Ministry of Education, Nepal

Type of Organization:

Public Sector, e.g. Government Agency or Ministry

Intervention Timing

Intervention or Program Started at time of Registration?

Yes

Start Date:

08/17/2017

End Date:

07/16/2019

Evaluation Method

Evaluation Method Overview

Primary (or First) Evaluation Method:

Randomized control trial

Other (not Listed) Method:

Additional Evaluation Method (If Any):

Other (not Listed) Method:

Method Details

Details of Evaluation Approach:

The evaluation will be based on a randomized controlled trial (RCT) with two main study arms: Phase 1 Schools, which received the TT treatment shortly after baseline data collection; and Phase 2 Schools, which during the two years of the study will receive no new teacher training at the secondary level and serve as the control group for the RCT.  The treatment will be subdivided randomly into two arms, one receiving only the basic TT treatment and the other also receiving the VA supplement to the TT treatment. We are gathering data on school management practices, teacher subject knowledge, teacher attendance and time on task, teaching practices and student learning outcomes from all study schools. Our primary focus is to estimate the impact after two years, focusing on students entering grades 8 and 9 at the beginning of the study and who complete grades 9 and 10 at the end of the second year.  Students completing grade 9 at endline will have been treated by trained teachers for one year; students completing grade 10 at endline will have been treated for two years. We collected baseline data before allocating study schools to treatment arms. This ensures that survey organization procedures are identical across treatment arms at baseline. We hope to complement our RCT-based impact estimates with difference-in-differences estimates of impact on grade 10 student scores on the School Leaving Certificate (SLC) exams, which are mandatory for students in Grade 10. Similar difference-in-differences estimation may also be possible using other end-of-year test scores for grade 9 and 10 students routinely carried out by Nepalese schools.

Outcomes (Endpoints):

Our primary outcome variables are student scores on academic achievement tests, which are designed to measure performance at five cognitive levels, from knowledge gained through memorization up through critical thinking skills and application of ideas to new contexts, and which recognize that many students may be functioning below grade level.  To study the many channels through which the treatments might impact the outcomes, we will also study impacts on teacher characteristics, teacher attendance and time on task, teaching practices, student attendance, and student dropout rates. 

Unit of Analysis:

School

Hypotheses:

The main evaluation questions are:

• What are the impacts on teacher knowledge, teacher attendance, teaching practices, and student learning outcomes of the TT treatment?

• To what extent do TT treatment impacts increase or decrease when the VR treatment is added?

•  How do TT and VR treatment impacts differ across schools with stronger and weaker initial school management?

• What are the strengths and weaknesses of the TT and VR treatment implementation processes?

Unit of Intervention or Assignment:

Schools

Number of Clusters in Sample:

204

Number of Individuals in Sample:

About 8,000

Size of Treatment, Control, or Comparison Subsamples:

102 control schools, 51 TT schools, 51 VA schools

Supplementary Files

Analysis Plan:

PAP_April30_2019.pdf

Other Documents:

Data

Outcomes Data

Description:

We have a sample of 204 schools distributed across 16 of Nepal’s 75 districts. We are administering tests in math and science in grades 8, 9 and 10, as well as administering a short socio-economic questionnaire to students, at both baseline (April-May 2017) and endline (February 2019). Tests will be administered to students who are entering grades 8 and 9 at endline. We are attempting to collect data on school- and government-sponsored end-of-year exams for students finishing grade 10.

Data Already Collected?

Yes

Data Previously Used?

No

Data Access:

Not restricted - access with no requirements or minimal requirements (e.g. web registration)

Data Obtained by the Study Researchers?

Yes

Data Approval Process:

Approval Status:

Treatment Assignment Data

Participation or Assignment Information:

Yes

Description:

Data Obtained by the Study Researchers?

Data Previously Used?

Data Access:

Data Obtained by the Study Researchers?

Data Approval Process:

Approval Status:

Data Analysis

Data Analysis Status:

No

Study Materials

Upload Study Materials:

Registration Category

Registration Category:

Prospective, Category 3: Data for measuring impacts have been obtained/collected by the research team but analysis for this evaluation has not started

Completion

Completion Overview

Intervention Completion Date:

Data Collection Completion Date:

Unit of Analysis:

Clusters in Final Sample:

Total Observations in Final Sample:

Size of Treatment, Control, or Comparison Subsamples:

Findings

Preliminary Report:

Preliminary Report URL:

Summary of Findings:

Paper:

Paper Summary:

Paper Citation:

Data Availability

Data Availability (Primary Data):

Date of Data Availability:

Data URL or Contact:

Access procedure:

Other Materials

Survey:

Survey Instrument Links or Contact:

Program Files:

Program Files Links or Contact:

External Link:

External Link Description:

Description of Changes:

Study Stopped

Date:

Reason: