Download Study

General

Study Overview

Title:

Measuring the Impact of a Health Delivery Intervention in the Ahafo, Ashanti, Bono East, Central Eastern, Northern, North East, Upper East, and Volta Regions of Ghana

Study is 3ie funded:

No

Study ID:

RIDIE-STUDY-ID-5f84429e27881

Initial Registration Date:

10/12/2020

Last Update Date:

06/11/2021

Study Status:

Ongoing

Location(s):

Ghana

Abstract:

In 2019, Zipline, a medical product drone delivery service, expanded its operations from Rwanda into Ghana. It has since established four distribution centres to ultimately serve approximately 2,000 health facilities across the country, with the goal of improving access to medical products such as blood, antivenom, vaccines, and essential medicines. Since March 2020, Zipline has also been distributing Covid-19 related products, including PPE, test samples, and supportive treatments, and it is considering the potential for distribution of Covid-19 vaccines. Despite the growing popularity of the use of drone technology to deliver medical products, little evidence exists that quantifies its impact on medical stocks and quality of care. Zipline, LLamasoft, and IDinsight are partnering to study the impact of Zipline’s operations on health facilities.

The study will leverage Zipline’s non-universal roll-out in Ghana to estimate impacts using difference-in-differences with matching strategy. This involves sampling health facilities with certain observable characteristics within Zipline service zones and matching these facilities with similar health facilities outside the service zones on baseline characteristics and pre-intervention trends. To identify the impact of Zipline, we will compare the differences in pre-post outcomes between the two types of facilities using survey and administrative data. A discrete event simulation model of the Ghana public health supply chain will be used to estimate Zipline’s effect on national supply chain costs. 

Results from the study will contribute to a nascent but growing body of evidence in global public health on the effectiveness of on-demand product fulfillment, and further, specifically those involving drone delivery.

Registration Citation:

Categories:

Health, Nutrition, and Population

Additional Keywords:

Secondary ID Number(s):

Principal Investigator(s)

Name of First PI:

Dr Valentina Brailovskaya

Affiliation:

IDinsight

Name of Second PI:

Dr Alison Connor

Affiliation:

IDinsight

Study Sponsor

Name:

Bill & Melinda Gates Foundation

Study Sponsor Location:

United States

Research Partner

Name of Partner Institution:

LLamasoft

Type of Organization:

Private firm

Location:

United States

Intervention

Intervention Overview

Intervention:

Zipline is a drone delivery service that provides third-party logistics (3PL) services, holding customers’ products in warehouses to minimize the lead time between order and delivery for end recipients. In Ghana, Zipline has partnered with Ghana Health Service (GHS) to provide last mile logistics services and improve access to crucial health products in health facilities. Zipline is establishing four distribution centres (Omenako, Mampong, Walewale, and Sefwi Wiawso) to serve approximately 2,000 health facilities across Ghana.

Zipline provides emergency and non-emergency on-demand medical products (such as blood, emergency and routine vaccines, and emergency and essential medicines). This delivery system is intended to improve timely access to medical products, reduce instances and duration of stockouts in health facilities, and ultimately improve health outcomes. The process is initiated when a healthcare worker submits an order for a particular product via SMS or a phone call. The order is recorded at the closest distribution center, and the required medical products are packed into a drone. The automatically operated drone drops off the medical products in an allocated location nearby the facility. The medical products are picked up by the facility staff within about 30 minutes.

Theory of Change:

The overall theory of change operates as following:

Health workers and health facilities use Zipline to order health care products, which are supplied by the  Ghana Health Service. Zipline receives the order, then processes and confirms it. The package is then launched and dropped at the health facility. This results in a faster and more convenient delivery system for health facilities. The main outcome is the reduction of stockout instances and duration, and improved product availability and a more equitable medical supply system. This can lead to fewer patient referrals and less waiting time before treatment. Ultimately this feeds into the a final impact of a) better quality treatment and improved health outcomes, and b) less wastage of medical products and lower supply chain costs, leading to cost savings for the Ghanaian public health system. 

Multiple Treatment Arms Evaluated?

No

Implementing Agency

Name of Organization:

Zipline

Type of Organization:

Private for profit organization

Program Funder

Name of Organization:

Bill and Melinda Gates Foundation

Type of Organization:

Other

Intervention Timing

Intervention or Program Started at time of Registration?

Yes

Start Date:

04/01/2019

End Date:

Evaluation Method

Evaluation Method Overview

Primary (or First) Evaluation Method:

Other (specify)

Other (not Listed) Method:

Difference in differences on a matched sample of facilities

Additional Evaluation Method (If Any):

Other (not Listed) Method:

Method Details

Details of Evaluation Approach:

The research team conducted a two stage matching process.

The first stage of matching was completed before the baseline data collection using  District Health Information Management System (DHIMs). The dataset contained basic information about facilities and health records for selected outcomes. Researchers used 1:1 coarsen matching  to create a comparable treatment and control sample based on observable characteristics. The process created a sample of 452 facilities across the two groups, interviewed during baseline visits. The baseline sample consisted of district hospitals, healthcare centers, and Health Service Community-Based Health Planning and Services (CHPS). During the data collection, the following staff members were interviewed to document outcomes of interest: heads of facilities, staff members in charge of all dispensaries, health workers and patients. Patients were also interviewed. The final sample of baseline facilities consisted of  422 health facilities ( 238 and 184 within and outside of Zipline zones).

The second 1:1 coarsen matching is completed using information collected during the baseline and is used to further narrow down to the sample of the most comparable facilities. 

The impacts on outcomes of interest will be measured using data from  high frequency phone surveys and an endline survey. To quantify impacts of Zipline surveys we will either use difference-in-differences and compare trends in outcomes of interest between treatment and control groups or ANCOVA for quantifying the impacts on variables which were not collected during the baseline pretreatment period. We plan to use Conley estimation of standard errors to adjust for spatial correlation of the error term. 

Outcomes (Endpoints):

Zipline’s near-term effects on health facilities, by exploring differences between facilities served by Zipline and control facilities across outcome groups:

Primary outcomes:

1. Product stocking - whether or not mandated medical products are stocked at the facility

2. Product stockouts - instances and durations of stockout of mandated products

3. Stocking behavior - the actions taken by heath facility workers to resolve stockouts

4. Treatment behavior - the actions taken by health workers to treat patients for certain medical conditions

Secondary outcomes:

1. Attitudes and perceptions - the views of health workers and local community members towards Zipline

2. What effect do Zipline’s operations have on supply chain costs? Cost components to be considered include: transport costs; storage and physical infrastructure costs; expiry, waste, and loss of potency; inventory holding cost; and management, overhead, and labour.

3. How is the supply chain and facility-level patient demand for services impacted by the COVID-19 pandemic? Are drone deliveries an impactful intervention during a public health emergency and after? 

Exploratory outcomes:

1. Treatment seeking behaviour - patient decisions to visit particular health facilities to seek care, and the costs associated with these decisions

Unit of Analysis:

Health facility

Hypotheses:

Primary Research Questions:

1. Product Stocking: Did Zipline improve equity of healthcare delivery through improving stocking rates of essential products/vaccines?
2. Did Zipline improve the supply chain? Specifically,
   1. Product Stockouts: are there reduced instances and duration of stockouts for relevant products?
3. Treatment Behaviour: Did Zipline improve health worker’s ability to deliver better care (patients experiencing less referrals due to lack of medical products)?

 

Secondary Research Questions:

1. Stocking Behaviour: Are healthcare workers more likely to take action to resolve stockouts? Are stockout resolutions more timely?
2. Attitudes and perceptions: What are the views of Zipline by health workers
3. Costs: What effect do Zipline’s operations have on supply chain costs? Cost components to be considered include: transport costs; storage and physical infrastructure costs; expiry, waste, and loss of potency; inventory holding cost; and management, overhead, and labour. This piece will be completed by Llamasoft and is outside of the scope for IDInsight
4. Healthcare worker satisfaction: Are healthcare workers more satisfied with their jobs?

 

Change History for Hypotheses

Changed On

Previous Value

06/11/2021

H1: Does Zipline improve healthcare’s capacity to stock all mandated products? H2: Does Zipline reduce instances and duration of medical product stockouts? H3: Does Zipline improve healthcare worker’s ability to treat patients? H4: Is there an indication that changes in healthcare systems lead to reduced supply chain costs? H5: Is there an indication that changes in healthcare systems lead to improved patient outcomes? H6: Is drone-delivery service impactful during emergencies (such as COVID19 pandemic?)

Unit of Intervention or Assignment:

Health facilities

Number of Clusters in Sample:

Zipline currently has 4 distribution centers (only 3 are in the study sample) and all healthcare facilities located within the flying can be served by Zipline.

Number of Individuals in Sample:

422 facilities for baseline. Final sample between 200 - 300 facilities.

Size of Treatment, Control, or Comparison Subsamples:

200 and 300 facilities in total

Supplementary Files

Analysis Plan:

Zipline_PAP_submitted.pdf

Other Documents:

Data

Outcomes Data

Description:

Primary data will be collected on Health worker stocking, treatment behaviour, attitudes and perceptions of healthcare workers and patients through interviews with the heads of health facilities, staff members in charge of dispensaries, health workers and patients. Data collection activities include the following: In-person baseline survey conducted at health facilities: At selected treatment facilities At control facilities that share similar characteristics to tmt

Data Already Collected?

No

Data Previously Used?

Data Access:

Data Obtained by the Study Researchers?

Data Approval Process:

Approval Status:

Treatment Assignment Data

Participation or Assignment Information:

No

Description:

List of facilities obtained from Zipline.

Data Obtained by the Study Researchers?

Yes

Data Previously Used?

Not sure

Data Access:

Restricted -- Access requires a formal approval process

Data Obtained by the Study Researchers?

Data Approval Process:

Contact Zipline and IDinsight team.

Approval Status:

Yes-obtained approval and have received the data

Data Analysis

Data Analysis Status:

Study Materials

Upload Study Materials:

Registration Category

Registration Category:

Prospective, Category 1: Data for measuring impacts have not been collected

Completion

Completion Overview

Intervention Completion Date:

Data Collection Completion Date:

Unit of Analysis:

Clusters in Final Sample:

Total Observations in Final Sample:

Size of Treatment, Control, or Comparison Subsamples:

Findings

Preliminary Report:

Preliminary Report URL:

Summary of Findings:

Paper:

Paper Summary:

Paper Citation:

Data Availability

Data Availability (Primary Data):

Date of Data Availability:

Data URL or Contact:

Access procedure:

Other Materials

Survey:

Survey Instrument Links or Contact:

Program Files:

Program Files Links or Contact:

External Link:

External Link Description:

Description of Changes:

Study Stopped

Date:

Reason: