Download Study

General

Study Overview

Title:

Impact Evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development (RAPID) Activity

Study is 3ie funded:

No

Study ID:

RIDIE-STUDY-ID-5fad890349eb6

Initial Registration Date:

11/12/2020

Last Update Date:

10/23/2020

Study Status:

Ongoing

Location(s):

Kenya

Abstract:

Reliable and sustained water service delivery in rural Kenya areas is an on-going a challenge, particularly in drought-prone areas. To address this issue, the United States Agency for International Development (USAID) commissioned the Kenya Resilient Arid Lands Partnership for Integrated Development (Kenya RAPID) Activity, which aims to contribute to sustainable and resilient livelihoods for communities in five of Kenya’s arid and semi-arid lands (ASALs) counties by improving water availability and water service delivery to people and livestock and by improving rangelands in those regions. One component of Kenya RAPID uses sensors on water borehole pumps to notify sub-county and county staff about pump stoppages. In this study, we implement a quasi-experimental propsensity score matching design to rigorously test the effects of this ICT intervention on borehole pump on-time and management decisions during the drought season. The evaluation uses data on borehole characteristics to match Kenya RAPID boreholes with one or more comparison boreholes that are as similar as possible, and then estimates differences in borehole on-time across Kenya RAPID boreholes and matched controls. Data from comparison boreholes in eight non-Kenya RAPID counties are collected through in-situ sensors that measure the same outcome--borehole pump on-time--as Kenya RAPID's sensors. Qualitative focus groups and key informant interviews also provide insight into user perceptions and borehole management in both treatment and comparison counties 

Registration Citation:

Categories:

Water and Sanitation

Additional Keywords:

sensors, water management, Kenya, USAID

Secondary ID Number(s):

Principal Investigator(s)

Name of First PI:

Katherine Dickinson

Affiliation:

University of Colorado

Name of Second PI:

Jacob Patterson-Stein

Affiliation:

Management Systems International

Study Sponsor

Name:

USAID

Research Partner

Name of Partner Institution:

Ipsos Kenya

Type of Organization:

Private firm

Location:

Kenya

Intervention

Intervention Overview

Intervention:

Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s arid and semi-arid lands (ASALs) by improving water availability and water service delivery to people and livestock and by improving rangelands in those regions. As part of its approach, Kenya RAPID is committed to making data and information and communication technology (ICT) tools available and accessible to improve decision-making for better water service delivery. The activity installed approximately 400 sensors to measure flow rates and water extraction rates and detect system failures on water boreholes. Of these 400 sensors, 69 are in areas identified as strategic by local authorities due to the risk of drought in those areas and the subsequent importance of the water boreholes. The activity developed customized data dashboards for each county to display water borehole status in near-real time. Kenya RAPID makes these data accessible to relevant authorities such as county governments and the appropriate service providers. County and sub-county officials will, in theory, be able to use the sensor data to improve their management and deployment of staff and resources—areas that are receiving support through other Kenya RAPID interventions. This improved water management will eventually help mitigate drought risk for thousands of Kenyans who live in arid and drought prone areas by improving access to reliable water. 

Theory of Change:

The theory of change envisions that if the Kenya RAPID activity, 1) installs sensors on strategic boreholes, shares the data through mobile applications and online dashboards, and provides training on sensor data use; 2) supports the development of county operations and maintenance teams; and 3) facilitates a dedicated budget for strategic borehole repairs, this will lead to increased strategic borehole functionality, including more borehole pump on-time during critical drought periods and reduced drought impacts on ASAL communities. The impact evaluation focuses on the first and second parts of this theory of change by comparing remotely sensed borehole data across intervention and a matched set of comparison boreholes to understand whether stoppages in pumping change as a result of improved access to performance data among county operations and maintenance teams. 

Multiple Treatment Arms Evaluated?

No

Implementing Agency

Name of Organization:

MWA

Type of Organization:

NGO (International)

Program Funder

Name of Organization:

USAID

Type of Organization:

Foreign or Multilateral Aid Agency

Intervention Timing

Intervention or Program Started at time of Registration?

Yes

Start Date:

10/01/2015

End Date:

10/01/2020

Evaluation Method

Evaluation Method Overview

Primary (or First) Evaluation Method:

Matching

Other (not Listed) Method:

Additional Evaluation Method (If Any):

Other (not Listed) Method:

Method Details

Details of Evaluation Approach:

The evaluation’s quasi-experimental propensity score matching design attempts to identify a set of boreholes that are as similar as possible to the treatment group such that we would expect outcomes (in this case, borehole functionality) to be similar in the absence of treatment. If this is achieved, the comparison group’s outcomes can serve as a proxy for the treatment group’s counterfactual, allowing the team to estimate the specific effect of the ICT intervention. To assess similarity, the evaluation team collected data on the following observable borehole characteristics from 132 boreholes in eight non-intervention counties that are expected to affect functionality during droughts:

• Type of pump
• Populations served (human and livestock)
• History of breakages/repair times
• Tariff scheme

Using the borehole characteristics data that the evaluation collected, as well as remotely sensed rainfall data, and data on travel to the county seat (where borehole repair decisions are made), the evaluation aims to select a comparable set of boreholes with similar observable characteristics and controlled for these characteristics through a matching algorithm. The identifying assumption for this design is that, conditional on these observable characteristics, we would expect similar functionality outcomes across boreholes in Kenya RAPID and non-RAPID counties in the absence of the intervention.

Outcomes (Endpoints):

The percentage of the day that a borehole pump is turned on, a measure we call “percent on,” is the main outcome variable for this evaluation. This provides the percentage of on-time, defined as time within a 24-hour period that the meters recorded the borehole pump as running; for example, a value of 50 percent would indicate the borehole pump ran for 12 hours (i.e. 0.5 x 24 hours).

Unit of Analysis:

water boreholes

Hypotheses:

If water borehole pump information is made accessible to operators and managers, then borehole pump on-time will increase compared to a set of boreholes where there is no information sharing. 

Unit of Intervention or Assignment:

county

Number of Clusters in Sample:

NA

Number of Individuals in Sample:

201 boreholes, with daily pump data from each

Size of Treatment, Control, or Comparison Subsamples:

68411 comparison observations, 47068 treatment observations

Supplementary Files

Analysis Plan:

Other Documents:

Data

Outcomes Data

Description:

Remotely sensed borehole pump data and borehole characteristics survey data.

Data Already Collected?

Yes

Data Previously Used?

No

Data Access:

Not restricted - access with no requirements or minimal requirements (e.g. web registration)

Data Obtained by the Study Researchers?

Yes

Data Approval Process:

Approval Status:

Treatment Assignment Data

Participation or Assignment Information:

Yes

Description:

Data Obtained by the Study Researchers?

Data Previously Used?

Data Access:

Data Obtained by the Study Researchers?

Data Approval Process:

Approval Status:

Data Analysis

Data Analysis Status:

Yes

Study Materials

Upload Study Materials:

Registration Category

Registration Category:

Non-Prospective, Category 4: Data for measuring impacts have been obtained/collected by the research team and analysis for this evaluation has started

Completion

Completion Overview

Intervention Completion Date:

Data Collection Completion Date:

Unit of Analysis:

Clusters in Final Sample:

Total Observations in Final Sample:

Size of Treatment, Control, or Comparison Subsamples:

Findings

Preliminary Report:

Preliminary Report URL:

Summary of Findings:

Paper:

Paper Summary:

Paper Citation:

Data Availability

Data Availability (Primary Data):

Date of Data Availability:

Data URL or Contact:

Access procedure:

Other Materials

Survey:

Survey Instrument Links or Contact:

Program Files:

Program Files Links or Contact:

External Link:

External Link Description:

Description of Changes:

Study Stopped

Date:

Reason: