For a few months in 2016 community health workers working with children in northern Malawi had a helping hand.

A smartphone app led them through a standard protocol of examinations helping to boost both the accuracy of diagnoses, as well as their confidence. They told researchers they liked the app because it reassured them that their decisions were solid.

But, after eight months, the project was over and the app removed for fear that — with the funding gone — it could soon become out of date.

This is hardly an isolated incident. mHealth makes a tantalising offer: to plug gaps in health provision in impoverished countries using cheap, widely available tools. But across Sub-Saharan Afrika, Asia and South America hundreds — perhaps thousands — of mHealth experiments done in the hope of improving healthcare disappear after a year or two.

Methodological woes

A 2016 review cast a harsh light on mHealth [2]. Led by Alain Labrique, a public health professor who heads the Global mHealth Initiative at Johns Hopkins University in the United States, it concluded that — with some exceptions — “the strength of the findings to support the use of mobile interventions over alternate strategies is still limited”.

The review found that many interventions go undocumented; many are designed without academic rigour, making it hard to gauge their success; and often the chosen outcome measures mean they cannot be compared with other studies.

Caroline Perrin, a PhD student in telemedicine at Geneva University in Switzerland, says that instead of measuring outcomes, studies tend to measure how much health professionals like their new apps, and are often too small to have the statistical sensitivity to detect a fall in mortality, for example.

And this, she says, adds to the difficulty of demonstrating that a single digital element woven into a complex health system actually makes a difference.

Perrin’s professor, Antoine Geissbühler, adds that rigorous evidence is important not just to prove effectiveness, but to avoid harm. Even at pilot stage, he says, training sessions that focus on one health activity can eat up time and distort priorities. And governments need to make hard decisions between deploying eHealth and other activities, such as running vaccination campaigns. “Some countries cannot do [it] all.”

Few studies ever get scaled-up because they cannot prove to donors that they are effective, says Labrique’s review. And many projects are simply unscalable due to poor choices during the design phase.

Choice of software, for example, is crucial, says Garrett Mehl, a specialist in digital health at the World Health Organisation (WHO). A bespoke bit of standalone software might be fine for a few villages. But could it withstand the demands of 100,000 health workers — the help desks, the training and re-training, the ‘community of knowledge’ that’s in the background, keeping it ticking over?

“The reality is that unless they are using software that’s a ‘global good’ it’s unlikely to be able to scale,” he says. But even then, other factors may come into play.

Take the app that proved useful in diagnosing children in northern Malawi — the Supporting LIFE app (Low-Cost Intervention for Disease Control). “Even though the software was open source, the ministry of health just doesn't have any internal capacity to roll out the software or even maintain it,” says project’s coordinator John O’Donoghue, now an eHealth specialist at University College Cork, in Ireland.

“Realistically if the project was to be modified and maintained and implemented we're probably talking about a multimillion Euro effort per year.”

Chasing success

Digital and health communities are trying to find better ways to capitalise on mHealth’s promise.

For example, Perrin is helping researchers choose useful outcome measures when they set up pilots. Although a study may be too small to spot a drop in maternal or neonatal mortality, it might be able to detect an improvement in another parameter shown to decrease mortality.

She has identified 70 such variables, all of which can be influenced by digital interventions — having four check-ups during a pregnancy, for example, or identifying danger signs that predict neonatal illness. These are set to be adopted by WHO as part of upcoming guidance from its Department of Reproductive Health and Research on digital maternal health interventions.

O’Donoghue says such proxies were essential for Supporting LIFE. “Thankfully,” he says, “no child passed away during our study”, so they measured hospital referral rates for evidence that the digital system improved the accuracy of health workers’ referrals.

Adding to efforts to impart rigour to mHealth pilots, a few years ago the WHO, with the International Telecommunications Union, published advice on how to scale mHealth non-Communicable Disease projects. The agency has also developed a lexicon for both health and digital specialists, whose first version was released earlier this year.

Meanwhile, Principles for Digital Development written by a group of donors and implementers were endorsed by many international organisations in 2015, and a supporting forum launched last year.

And the WHO produced the mHealth Assessment and Planning for Scale (MAPS) toolkit in 2015, which has been adopted by the new Malawi eHealth Research Centre at Mzuzu University, which O’Donoghue helped found.

This means that anyone proposing an mHealth intervention must go to the Centre and spend several days assessing it against the MAPS toolkit. If it fails to demonstrate it can meet the demands in areas such as human resources, finance and technical capacity, and no solution can be found, the proposal is abandoned.

Toolkits and better study design may improve pilots and perhaps allow some scale-up. But if mHealth is to make a real difference in the developing world it requires scale-up to national level.

Many people fail to grasp how formidable that challenge is, says Mehl. “It requires the enabling environment of policies, the infrastructure, the tools — and knowing which work and which don’t — and the training.”

As a rare example of success, he highlights the Indian government’s messaging service Kilkari, designed by BBC Media Action, which delivers free audio messages about pregnancy, childbirth and childcare to women from the second trimester of pregnancy and continues until the child is one year old. Developed and tested in the state of Bihar in 2013, it went on to reach two million subscribers within 12 months of its launch. The government went on to adopt it in 2016, footing the bill for call costs and rolling it out in 13 states.

Sarah Chamberlain, director of digital for BBC Media Action India, highlights three factors in its success. First, the government had already invested in registering pregnancies and births, and it saw Kilkari as a way of incentivising women to supply this data; second, long-term and “patient” investment by donors; and third, spending years cultivating a relationship with the Ministry of Health. “You cannot imagine how much energy it takes to nurture a relationship with a national government,” she says. “You can't just fly in and fly out.”