Download Study

General

Study Overview

Title:

Cluster randomized controlled trial of the Data Must Speak community friendly school profiles program in Zambia

Study is 3ie funded:

No

Study ID:

RIDIE-STUDY-ID-5b70f662f3bd0

Initial Registration Date:

08/12/2018

Last Update Date:

08/08/2018

Study Status:

In Development

Location(s):

Zambia

Abstract:

The Zambian Ministry of General Education (MoGE), with technical support from UNICEF’s Data Must Speak (DMS) initiative, aims to improve education quality through increased accountability by providing school-level profiles to all primary schools in Zambia. To make the data accessible for communities, the MoGE and DMS developed simplified community friendly school profiles that use icons (pictures) rather than text and graphs. We will implement a mixed-methods evaluation that combines experimental and qualitative approaches to rigorously examine the impacts of the interactive DMS community training program. The experimental approach comprises a cluster randomized controlled trial (RCT) to measure the impact of the DMS training program on parental/caregiver profile awareness and understanding as well as the impacts on engagement with student schooling decisions and school management. Interviews and focus group discussions will inform the design of the community training prior to the start of the evaluation, and allow us to explore individual and community experiences with school profiles and the causal mechanisms through which changes occur as a result of the program.

Registration Citation:

Brudevold-Newman, A. and Spier, E., 2018. Cluster randomized controlled trial of the Data Must Speak community friendly school profiles program in Zambia. Registry for International Development for Impact Evaluations (RIDIE). Available at: 10.23846/ridie152

Categories:

Education
Public Sector Management

Additional Keywords:

Secondary ID Number(s):

Principal Investigator(s)

Name of First PI:

Andrew Brudevold-Newman

Affiliation:

American Institutes for Research

Name of Second PI:

Elizabeth Spier

Affiliation:

American Institutes for Research

Study Sponsor

Name:

United Nations Children's Fund

Study Sponsor Location:

Zambia

Research Partner

Name of Partner Institution:

Type of Organization:

Location:

Intervention

Intervention Overview

Intervention:

The intervention being evaluated is a community-level training on the use of Community Friendly School Profiles. The Ministry of General Education (MoGE) in Zambia, with technical support from UNICEF’s Data Must Speak (DMS) initiative, aims to improve education quality through increased accountability by providing school-level profiles to all primary schools in Zambia. The profiles are based on education management information system (EMIS) data collected by the MoGE via the annual school census. The profiles also incorporate examination and learning outcome measures provided by the Examinations Council of Zambia. Presently, it is unclear whether communities are aware of the profiles or how to use the information they contain. The intervention comprises an experiential-based training session to teach community members about the profiles and how to use the information they contain. The training will start by presenting a generic profile and then ask individuals to fill in their perception of the profile for their local school and will finish by presenting the information for the local school and discuss how the community can use that information.

Theory of Change:

The DMS community training programme aims to address the fact that providing community profiles to schools does not guarantee that the information they contain will be disseminated, understood, and used by communities. The programme aims to support the integration of community-friendly profiles into school management and community dialogues using a short community-level training. The community trainings target PTAs, school staff, and the community at large, and encourage participants to use the data and make evidence-based decisions to create more student-friendly learning environments. The Ministry of General Education expects the provision of profiles and training to support communities to improve the school environment, making it more conducive to learning through the rectification of missing school inputs and facilities. The profiles are also expected to improve the advocacy abilities of communities by helping them better understand how their school’s performance and resources compare to others in their district. The final stage of UNICEF’s theory of change involves multiple intermediate and final outcomes combining to yield improved learning outcomes and healthier behaviours. We note that improving academic outcomes and inducing behaviour change are typically long-term processes and are unlikely to manifest within the timeframe of the project, even if they intervention is highly successful. It is assumed in the model that if parents/caregivers are given information about their school’s performance, and if they believe that they have the power to act on the information, then they actually do have the power to influence what happens at their school. Further, it is assumed that the parents/caregivers and the school then know how to make effective improvements to address issues, and schools have the power and resources to implement the change.

Multiple Treatment Arms Evaluated?

No

Implementing Agency

Name of Organization:

American Institutes for Research

Type of Organization:

Research Institution/University

Program Funder

Name of Organization:

United Nations Children's Fund

Type of Organization:

Foreign or Multilateral Aid Agency

Intervention Timing

Intervention or Program Started at time of Registration?

No

Start Date:

10/01/2018

End Date:

11/30/2018

Evaluation Method

Evaluation Method Overview

Primary (or First) Evaluation Method:

Randomized control trial

Other (not Listed) Method:

Additional Evaluation Method (If Any):

Other (not Listed) Method:

Method Details

Details of Evaluation Approach:

We will implement a mixed-methods approach that combines experimental and qualitative approaches to rigorously examine the impacts of the interactive DMS community training programme. The experimental approach comprises a cluster randomised controlled trial (RCT) where we will randomly assign of 42 of 84 evaluation communities to receive the community training and then enroll a random sample of households with primary-school-aged children into the study. We will measure the impact of the DMS training programme on parental/caregiver profile awareness and understanding as well as engagement with student schooling decisions and school management. We will also leverage the random assignment of schools to the DMS training to test for impacts of the programme on academic achievement using the publicly posted examination results at government schools. Interviews and focus group discussions will inform the design of the community training prior to the start of the evaluation and allow us to explore individual and community experiences with school profiles and the causal mechanisms of changes which occur as a result of the programme.

Outcomes (Endpoints):

1. Primary Outcomes: a. Community member and school administrator awareness and understanding of the school profiles (do community members know that the profiles are a potential source of information, how useful do they consider the information to be) b. Parent/caregiver and broader community engagement in school education decisions (such as parent/caregiver involvement in PTA or other school management committees) 2. Intermediate Outcome: a. School management and change i.e. parent/caregiver perceptions of school decision making processes b. School management and change i.e. school administrators’ perceptions of power to affect change, school decision making practices 3. Final Outcome: a. Changes in student/teacher attendance, repetition rates, dropout rates, student academic achievement

Unit of Analysis:

Household

Hypotheses:

1. Training on CFSP will improve profile awareness and understanding 2. Training on CFSP will have a positive effect on community engagement with school administrators and decisions 3. Training on CFSP will have a positive effect on the school level decision-making process 4. Training on CFSP will have a positive impact on school-level indicators associated with learning

Unit of Intervention or Assignment:

School catchment area

Number of Clusters in Sample:

84

Number of Individuals in Sample:

1680

Size of Treatment, Control, or Comparison Subsamples:

42 schools in control group and 42 schools in treatment group

Supplementary Files

Analysis Plan:

Other Documents:

Data

Outcomes Data

Description:

The main data set will comprise two rounds of a household survey that will be administered to 1680 households in the catchment areas around 84 schools.

Data Already Collected?

No

Data Previously Used?

Data Access:

Data Obtained by the Study Researchers?

Data Approval Process:

Approval Status:

Treatment Assignment Data

Participation or Assignment Information:

Yes

Description:

Data Obtained by the Study Researchers?

Data Previously Used?

Data Access:

Data Obtained by the Study Researchers?

Data Approval Process:

Approval Status:

Data Analysis

Data Analysis Status:

Study Materials

Upload Study Materials:

Registration Category

Registration Category:

Prospective, Category 1: Data for measuring impacts have not been collected

Completion

Completion Overview

Intervention Completion Date:

Data Collection Completion Date:

Unit of Analysis:

Clusters in Final Sample:

Total Observations in Final Sample:

Size of Treatment, Control, or Comparison Subsamples:

Findings

Preliminary Report:

Preliminary Report URL:

Summary of Findings:

Paper:

Paper Summary:

Paper Citation:

Data Availability

Data Availability (Primary Data):

Date of Data Availability:

Data URL or Contact:

Access procedure:

Other Materials

Survey:

Survey Instrument Links or Contact:

Program Files:

Program Files Links or Contact:

External Link:

External Link Description:

Description of Changes:

Study Stopped

Date:

Reason: