Community health workers form the backbone of healthcare systems globally. The rapid expansion of mobile communications systems represents an opportunity to improve the productivity of community health workers in rural areas. Here, we describe a programme in rural Nepal that aimed to implement a mobile phone system for collecting health surveillance data, yet did not reach its fullest potential due to several programme management challenges during the implementation of the surveillance programme. Despite early successes with the mobile phone system itself, the programme ultimately failed due to leadership transitions, poor process design and a lack of consistent vision of how to operationalise the data. This field report provides important insights into the design, maintenance and pitfalls of similar community-based mobile health initiatives and technology innovation projects in general.
Statistics from Altmetric.com
There is substantial interest in mobile health (mHealth) applications for improving healthcare systems in settings of poverty. The field of mHealth consists of interventions that apply cellular phones and other mobile devices for healthcare purposes such as data collection,1–3 clinical decision support,4–6 self-care7 ,8 and community health worker (CHW) management.9
There are several challenges that have been identified in the literature towards the success of mHealth intervention. These include implementation, acceptability and adoption by providers and patients10 and scalability.11 Recent systematic reviews of the mHealth literature found that while in some categories mHealth interventions might have been successful, there appears to be a dearth of high-quality research into mHealth interventions globally to assess what true impacts might exist.12 ,13
Failed programmes typically go unreported in the literature. Few mHealth studies are registered online beforehand, and the results of some interventions might not be reported on.13 While there have been some initiatives to address this,14 it remains unclear how many unsuccessful mHealth interventions are actually reported. This publication bias may lead researchers, policymakers and healthcare organisations to understate the challenges, under plan or under budget for mHealth interventions and misunderstand why programmes fail and how to help them succeed. In order to avoid this bias and to add to the body of literature on mHealth interventions, we describe here a failed programme in rural Nepal that highlights some of the challenges of implementing interventions in global health settings.
The programme was implemented in rural Achham, Nepal, by Possible, a non-governmental organisation that manages public sector healthcare services in partnership with Nepal's Ministry of Health. Achham is a district of 270 000 people located in the Far-Western Development Region.15 The district struggles with some of the lowest development indices and literacy rates, and the highest malnutrition rates and mortality rates in the country.16
The Ministry of Health of Nepal manages a national network of ∼50 000 community health workers (CHWs, who are known locally in Nepal as Female Community Health Volunteers) who perform largely preventative health activities at the community and household level. Possible has expanded on this model in 14 village clusters (known locally in Nepal as Village Development Committees) through offering additional training to the CHWs, a stipend for their work and the introduction of a community health worker leader (CHWL) in each village cluster. More detail on Possible's training of these CHWLs can be found in Schwarz et al.17 The CHWLs supervise the CHWs and provide them with additional training with the support of Possible and its staff. During the time of this mHealth programme, there were 9 CHWLs and 81 CHWs across nine village clusters serving 25 000 people.
Possible's Community Health Programme faced a lag time between when data were collected by the CHWs and when the data were received at the district level hospital called Bayalpata Hospital. The CHWLs reported patient encounters weekly on paper forms that were error prone and often misplaced, which would eventually need to be entered by hand into a computer database. Owing to the highly isolated region where the CHWs were based, many of these reports failed to make it back to the hospital. The delays and errors in this process reduced the usefulness of the data collected in impacting hospital actions and patient outcomes. This workflow is illustrated in figure 1A.
To address these issues, Possible designed a mobile phone-based programme for CHW data collection and analysis as shown in figure 1B. They hoped that this would be useful to Bayalpata Hospital in improving community-based healthcare across this resource-limited region.
Possible partnered with the non-profit technology company ‘Medic Mobile’ to create a personalised mobile phone-based application for Possible's CHWLs. CHWLs were given mobile phones that could collect data which could be sent remotely to be aggregated and visualised on a server at the hospital, as shown in figure 2. CHWLs were to use these phones to collect the data in a timelier manner, so that the results could be used to help make population-level health decisions at the hospital and to inform design of community-level programmes. It was originally hypothesised that the intervention would lead to a reduction in CHWL data reporting times and allow for the data to be put into action by the hospital. This would lead to quicker treatment times and a reduction in overall mortality among community members within the hospital's catchment area population. More details on the intervention can be found in online supplementary appendix A. The Nepal Health Research Council (number 79/2012) and the Brigham and Women's Hospital IRB (2013P000709) approved the study of the intended intervention.
An initial pilot programme was launched in June 2012 and the CHWLs were trained over a 2-week period. The training included detailed instruction and demonstrations in Nepali in use of the technology provided by ‘Medic Mobile’ staff. The training was followed by a period of practice meant to last for 3 months under close supervision, during which data would be entered into the phones at the Bayalpata Hospital.
Throughout the process, ‘Medic Mobile’ served as a close partner in the design and implementation of the intervention. Possible's community health staff members met with ‘Medic Mobile’ representatives to present local site-specific and context-specific challenges in order to design a system that would work in Achham.
Despite some initial hesitation, the CHWLs responded positively to the programme. CHWLs generally reported enthusiasm about the potential for this programme. They did express concern that it took a considerable amount of time to enter the records into the phone during the first few months due to unfamiliarity with the use of mobile phones. After several months of the programme, they reported they could enter the data at a comparable pace to the paper forms.
Programme supervision disconnect
Soon after the programme began, Possible's Community Health Director's contract expired and there was a delay in replacement of this post. Additionally, Possible's administrative and clinical leadership experienced a significant turnover. This instability in programme leadership led to poor communication regarding the intent of the programme as well as significant difficulty in establishing regular processes for data analysis and use.
While the technical components, described in further detail in online supplementary appendix B, of the intervention worked well in the context, several glitches did occur early in the pilot phase. A technical fault was found in the phone application that led to the duplication of up to 50% of reports submitted by the CHWs. Owing to a modem issue, no data were collected in November of 2012. It was also discovered during the initial training that the two major Nepali telecom providers vary in coverage of each of the village clusters and thus some CHWLs required different SIM cards than those used during the trainings.
The first two problems were quickly remedied and solved due to immediate and direct communication with the technical team at ‘Medic Mobile’. The third issue was not identified quickly; however, owing to lapses in communication, it was not resolved for several months. In this instance, the managers of the intervention did not have awareness of the full technological situation before proceeding.
Results and aftermath
Figure 3 shows the number of reports submitted by the CHWLs over the span of the full programme period. At the start of the study, there was an accurate encounter rate of 62% and by the end it had increased to an average accuracy rate of 92%. The start-up cost was $8884 for the development of the tool, the purchase of the equipment and all training. The average annual cost of running this programme after launch was $18 148. This amounts to $2017 per year per village cluster, $224 per CHW annually or $0.82 per submitted encounter.
After operating from June 2012 to February 2014, Possible ended the programme and discontinued the use of the mobile phones. While data were successfully collected, Possible's leadership no longer felt that the programme was worth the financial and human resources costs that the programme incurred as the data had not actually been used towards the stated goal of improving population health.
Data usage was hindered by several factors. Up until this point, the community health department did not have experience in analysing and acting on data at this scale. Compounded by the fact that population level data would have required a longer time frame than the project's duration for real impact to be reached, it took too long to visualise the data and put them back into the hands of the CHWs. The hospital's data team was originally supposed to have a role in the analysis of collected data; however, these responsibilities were never communicated to them.
After 2 years of little tangible progress, ‘Possible’ discontinued the programme in favour of a modified version, designed instead to specifically monitor patient follow-up rather than population health. From this experience, and with the belief that mHealth interventions do offer a potential for great patient benefit, we present the following insights that we hope will be informative for future mHealth interventions in similar contexts. While this evaluation does not provide evidence that these lessons if mitigated would lead to a successful intervention, addressing them would help mitigate the challenges that led to the failure of this intervention.
The first lesson: with adequate support, community health workers can become adept at using mobile phone-based data collection tools. After the first 3 months of the programme, the CHWLs reported interest and success in using the data collection tool with a low error rate. If the training programme is well implemented, then we are confident that CHWs in similar contexts can achieve similar successful results.
The second lesson: mHealth interventions cannot succeed without adequate organisational process development. Organisations must plan the processes by which data will be collected as well as the workflow and structures to ensure how such data will be used. While this programme had success with the actual data collection, the effectiveness of the programme was ultimately limited as comprehensive plans were not put into place to analyse, report and act on those data. Mobile data collection may ultimately be more efficient, but it is unwise to try to build out such a tool without routine data collection, analysis and distribution processes already in place.
The third lesson: the goals of any intervention must be unified across the entire organisation. The low degree of data usage may have been related to different goals between Possible's Community Health and Research Programmes. Possible's Research Programme was heavily involved in the design of the mobile data collection tool, emphasising population health surveillance metrics over individual patient data. In recent discussions, however, Possible's Community Health Programme team felt they would be better served by having patient management and follow-up data. Possible should have held more conversations prior to programme implementation to reach common ground on programme goals among all team members.
The fourth lesson: close engagement of technological partners is vital to designing a tool useful in local contexts. From the early stages of planning the programme, ‘Medic Mobile’ was actively involved in the process. This allowed for ‘Medic Mobile’ to fully customise their software and hardware for use by Possible's Community Health Programme, in a way that would be useful for CHWs. An important piece of this relationship was that the ‘Medic Mobile’ staff actually spent time in Nepal to train the CHWs, address technical issues and to listen to feedback. It was because of this relationship and level of interaction that technical issues and application design did not serve as the ultimate bottlenecks to the programme.
The final lesson: leadership turnover must be anticipated, and new leadership must be adequately informed of all programme details. Possible's Community Health Programme leadership transitioned several times during the programme, presenting significant instability for the programme. Ultimately, this led to the programme never transitioning out of the pilot phase and left much of the data unused. Leadership transitions are a natural stage of any organisational life cycle that must be anticipated. When Possible's leadership shifted, the new leaders needed to have been more adequately engaged so they fully understood the scope and the goals of the programme.
We present this field report as an example of an mHealth programme that had potential to impact the efficiency of a community health programme in rural Nepal, but did not reach fruition as a result of poor process planning and management. This case study is limited by the lack of a formal, prospective evaluation. More research is also needed to demonstrate what other factors do lead to the successful management of mHealth interventions.
In this intervention, the use of mHealth technology itself appeared to be feasible, but in the absence of adequate management, the programme did not meet its intended goals. While this is not a feature of mHealth-based interventions exclusively, technology without adequate process is a means to an end, not the end itself. For mHealth, or any intervention in low-resourced global settings to be successful, appropriate and actionable data collection, organisational buy-in and effective process management are necessary.
The authors wish to express our appreciation to the Nepal Ministry of Health for their continued efforts to improve the public sector healthcare system in rural Nepal; to the Nepal Health Research Council and the Brigham and Women's Hospital Institutional Review Board for granting ethical approval; and to Dr Mark Woodin for crucial advice and assistance during the design of the research study. The authors also wish to give our deepest thanks to the staff of Bayalpata Hospital, notably the Community Health Programme including the community health workers, whose tireless efforts and unwavering commitment to the patients of Achham continue to inspire us.
Contributors DJM, AH, RS, AB, DS, RKS and DSRM conceived and designed the programme. DJM, AH and and RS oversaw the initial implementation of the programme and led the training. AB and SP managed the day-to-day operations of the programme. DJM and MF evaluated the programme. DJM, MF and SH drafted the manuscript. All the authors edited and revised the manuscript. All the authors approved the final draft and agree with the conclusions.
Funding This work was supported by a ‘Stars in Global Health’ grant from Grand Challenges Canada (0257-01). Grand Challenges Canada is funded by the Government of Canada and is dedicated to supporting Bold Ideas with Big Impact in global health. This work was also partially supported by a Thrasher Early Career Award provided by the Thrasher Research Fund.
Competing interests DJM is currently a doctoral student at a private university (Brown University School of Public Health). MF is a currently medical student at a public university (University of Massachusetts Medical School). AHB is a resident at a public hospital (Contra Costa Medical Center). SH is employed by and DS, RKS and DSRM work in partnership with a non-profit healthcare company (Possible) that delivers free healthcare in rural Nepal using funds from the Government of Nepal and other public, philanthropic and private foundation sources. At the time of programme implementation, RS, AB and SP were employed by Possible. RS is employed by a non-profit technology company (Medic Mobile) that receives philanthropic funding. SP is employed by a multilateral aid agency (United Nations Population Fund) that receives multilateral funding from numerous countries. SH is also employed part-time at a public university (University of Washington). DS, RKS and DSRM are employed at an academic medical centre (Brigham and Women's Hospital) that receives public sector research funding, as well as revenue through private sector fee-for-service medical transactions and private foundation grants. DS and DSRM are also employed at a separate academic medical centre (Boston Children's Hospital) that receives public sector research funding, as well as revenue through private sector fee-for-service medical transactions and private foundation grants. RKS is also employed at an academic medical center (Massachusetts General Hospital) that receives public sector research funding, as well as revenue through private sector fee-for-service medical transactions and private foundation grants. RKS and DSRM are also faculty members at a private university (Harvard Medical School). DSRM is a non-voting member on Possible's board of directors, but receives no compensation. All authors have read and understood BMJ Innovation's policy on declaration of interests, and declare that we have no competing financial interests. The authors do,however, believe strongly that healthcare is a public good, not a private commodity.
Ethics approval Nepal Health Research Council (number 79/2012) and the Brigham and Women's Hospital IRB (2013P000709).
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.