Article Text

Original article
Conducting a fully mobile and randomised clinical trial for depression: access, engagement and expense
1. Joaquin A Anguera1,
2. Joshua T Jordan1,
3. Diego Castaneda1,
5. Patricia A Areán2
1. 1Departments of Neurology and Psychiatry, University of California, San Francisco, California, USA
2. 2Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, Washington, USA
1. Correspondence to Dr Patricia A Areán, Department of Psychiatry and Behavioral Sciences, University of Washington, 1959 Northeast Pacific Street, Seattle, WA 98195, USA; parean{at}uw.edu

## Abstract

Importance Advances in mobile technology have resulted in federal and industry-level initiatives to facilitate large-scale clinical research using smart devices. Although the benefits of technology to expand data collection are obvious, assumptions about the reach of mobile research methods (access), participant willingness to engage in mobile research protocols (engagement), and the cost of this research (cost) remain untested.

Objective To assess the feasibility of a fully mobile randomised controlled trial using assessments and treatments delivered entirely through mobile devices to depressed individuals.

Design Using a web-based research portal, adult participants with depression who also owned a smart device were screened, consented and randomised to 1 of 3 mental health apps for treatment. Assessments of self-reported mood and cognitive function were conducted at baseline, 4, 8 and 12 weeks. Physical and social activity was monitored daily using passively collected phone use data. All treatment and assessment tools were housed on each participant's smart phone or tablet.

Interventions A cognitive training application, an application based on problem-solving therapy, and a mobile-sensing application promoting daily activities.

### Procedures to reduce gaming

‘Gaming’ is a situation where a user fraudulently enrols in a study solely to acquire research payment. We utilised the following safeguards to prevent this: (1) locking the eligibility survey if a participant tried to change a submitted answer, (2) using study links that are valid for one user/device, and (3) tracking IP addresses to minimise duplicate enrolment.

### Statistical analyses

To assess participant access, we describe the sample demographics, clinical characteristics and sample comorbidities using the appropriate descriptive statistics. To assess participant engagement, we examined the proportion of study drop-outs and the proportion of enrolled individuals who responded to the primary mood outcome measures at each time point using a mixed-model analysis of variance (with Greenhouse-Geisser corrections when needed). To calculate time to drop-out, we tested a survival analysis model with the distribution of the ‘survival’ times for those assigned each app estimated and non-parametric estimates of the survivor function computed by the Kaplan-Meier method, with curves tested using the log-rank test using Stata V.14.0. We also examined whether there was a significant difference in drop-out rates among the three interventions using Pearson's χ2 test. Pairwise log-rank tests were conducted to determine where there were significant differences between the distributions, and a Bonferroni correction was set at p<0.017 to correct for multiple comparisons. We also compared these outcomes for the entire sample and by sample type (randomised and non-randomised). To assess issues surrounding cost, we describe a total study cost approach factoring in β testing, staff time and efforts beyond those payments made for recruitment and participant remuneration.

## Results

### Access

#### Recruitment rate

National recruitment began in August of 2014, and was conducted in five, 2-week advertising waves (total of 5 months of recruitment). We recruited a total of 2923 participants. Of these recruited individuals, 1098 were enrolled to the randomised (N=626) and non-randomised (N=472) arms of the study (see figure 1). Eighty-nine per cent of the sample came from traditional recruitment approaches, <1% came from social networking, <1% came from search engine-based methods, and 10.3% came from unanticipated means (own search, referrals). We were able to successfully recruit individuals from 8 of the 15 most rural states in the USA29 without any targeted recruitment efforts (see figure 2A).

Figure 1

CONSORT diagram (HT, health tips; PHQ, Patient Health Questionnaire).

Figure 2

Demographic characteristics. (A) Percentage of recruited participants across the USA. (B) Percentage of participants within different age ranges from the recruited sample. (C) Ethnic composition of the recruited individuals, and its comparison to the observed ethnic composition reported in the 2013 US Census.

#### Sample demographics

Participants were primarily young adults (see figure 2B), although age ranged from 18 to 76, with 79% identifying as female. Fifty-eight per cent of our sample was non-Hispanic white, and an ethnicity distribution comparable to the 2013 US Census (see figure 2C). Fifty-seven per cent of our participants obtained a 4-year college degree or higher, with a mean annual income of $30–$35 000 (see table 1). Sixty-seven per cent of our participants were employed at the time of enrolment. There was a difference in age between randomised and phone-only assigned participants, with randomised participants slightly older than phone-only participants, t(954.38)=−3.22, p=0.001. However, there was no difference in gender between these groups (=0.08, p=0.77). Enrolled individuals who were single/never married reported greater symptoms of depression (t[528.17]=2.96, p=0.003).

Table 1

Baseline demographics

#### Clinical characteristics

The sample was moderately depressed at baseline, with a PHQ-9 mean score of 13.9 (SD=5.1). There was a significant association between age and depression severity, Spearman's r=−0.11, p<0.001. There was no significant difference in depression severity among gender (t(365.85)=0.63, p=0.53) or ethnic groups, (F(6, 1091)=1.37, p=0.22). Fifty-one per cent reported comorbid anxiety, 53% reported comorbid alcohol misuse, 16% reported a history of psychosis or mania. In total, 54.5% of our sample was receiving mental health treatment for their depression. This sample mirrored the ethnic disparities in mental health service use found in the general population, with 63% of non-Hispanic white participants in treatment, and only 42% of ethnic minorities were in treatment (=28.6, p<0.001, OR=2.29). There were no significant differences in depression severity among individuals randomised to the three primary arms (F[2, 623]=0.14, p=0.87; see table 1).

### Engagement

Sixty-six per cent of the sample completed the 4-week assessment, 50% completed the 8-week assessment and 41% completed the 12-week assessment (see figure 3A). There was no adherence difference by group (F(2, 241)=2.50, p=0.08) and no time by group interaction (F(3.55, 428.14)=1.93, p=0.11). We found similar adherence to the cognitive assessment tool, with neither a group (F=0.46, p=0.63) nor interaction effect present (F=0.91, p=0.42). Although lower assessment adherence was observed in the more depressed participants, younger participants, and participants with lower education, the effects sizes were small (see table 2).

Table 2

Baseline demographics*

Figure 3

Intervention and assessment adherence. (A) Percentage of individuals who responded to their mood assessment during the treatment phase (first 4 weeks) and follow-up periods (weeks 8 and 12). (B) Kaplan-Meier survival estimates per study arm illustrating survival distributions of time to drop-out (last day of recorded activity) over the course of the study (84 days).

Kaplan-Meier survival analysis was conducted to determine whether intervention assignment or any baseline demographic variables predicted drop-out status. The log-rank test revealed a significant difference between the survival distributions between groups (=19.27, p<0.001), with the EVO arm having significantly earlier time to drop-out than the PST arm (=7.45, p=0.01) or HT (=17.51, p<0.001) arms (see figure 3B). We did not find a significant difference in survival distributions for those with high versus low PHQ-9 scores (using a PHQ-9 score of 10 as a cut-point, =2.29, p=0.13). There was no significant difference in survival distributions between non-Hispanic whites and ethnic minorities (=2.13, p=0.14). Participants who received bonus pay remained in the study longer than those who did not receive a bonus (=11.82, p<0.001). Bonus pay was for assessment completion, not intervention app use.

### Cost

Total study costs included participant payments ($23 320), website/enrolment portal/database development ($46 507), and salaried staff time (3; 2 student volunteers also assisted) over the 9 months the study was active ($58 917), summing to a total of$128 444. The total amount spent on for this project, including staff costs, development and β testing of the UCSF developed apps (ACE and iPST), and licensing fees for the use of the other apps (EVO and Ginger.io) was \$314 264 over 2 years.

## Discussion

The results from this study have a number of important implications for the future of RCTs in mental health. First, we recruited a large sample of depressed participants in a short period of time and with minimal cost and effort. Currently, the typical RCT takes 4–5 years to complete, and another 1–2 years before the outcomes from these trials are reported publically. Rapid recruitment has the potential for quickly testing intervention efficacy and effectiveness, and ultimately moving effective treatments into practice while identifying and preventing the proliferation of ineffective, even unsafe, treatments. Second, we were able to recruit a highly representative sample of the US population, without any specific cultural adaptations or targeted advertising. Remote research methods could address decades-long concerns about the generalisability of clinical findings to minority samples not typically represented in clinical research. Finally, the cost of a fully remote RCT could allow for greater distribution of dwindling clinical research and development funds from federal, foundation and industry sponsors. Investment in large-scale clinical trials is a costly endeavour, resulting in the need to focus funds on only a few research areas. Although not all mental health RCTs should be fully remote, particularly those that test hypotheses about biological or neurological processes that can only be measured with immobile devices, the methods presented here, such as automated data collection of neuropsychological processes could result in substantial savings, which in turn could be invested in a diverse research portfolio.

This research method is not without its limitations. Primary among the challenges of fully remote research is the ability to keep participants engaged in the study protocol over time. Although there is an appeal to quickly recruiting and retaining large numbers of participants in an RCT, researchers and developers need to be cautious when interpreting outcomes from samples with a drop-out rate greater than 70%.30–33 However, it is important to point out here that the project was completely automated with very little contact between the participant and research team, and our retention rates were higher than in the typical internet-based RCT.34 Internet-based studies have shown that when there is more direct contact between the research team and participant through technologies such as video-over-internet protocols, retention rates are greater and less subject to bias,35 suggesting a hybrid approach may provide an optimal response.

Although we experimented with two incentive models to encourage retention, we determined that participant payment was not enough to keep engagement from waning across the course of the study, as bonus pay only encouraged participants to complete their assessments, and did not engender any additional motivation to utilise the training apps. Previous work has demonstrated that externalised benefits in the form of compensation can dull motivation,36 ,37 indicating that the creation of internalised reward structure to enhance motivation (eg, individualised presentation of study progress, personalised encouragements) is critical for improved adherence.

## Conclusions

Mobile technology has an important role in broadening the reach and representativeness of RCTs, while substantially reducing the time to determine intervention effectiveness and reducing study costs. Although study retention remains challenging for technology-based research, innovative methods to increase motivation and study engagement could easily address this important limitation.

## Acknowledgments

The authors thank A Brandes-Aitken (UCSF), M Brander (UCSF) and A Bodepudi (UCSF) for assistance in data monitoring, M Gross (UCSF) and J Camire (Ekho.me) for their help in participant recruitment, J Steinmetz (Ekho.me) for database architecture, D Ziegler (UCSF) for helping with app deployment, D Albert (UCSF) for assistance in web design, L Slavik (UCSF) for accounting assistance, C Catledge (UCSF) for administrative oversight, A Piper, E Martucci, S Kellogg, J Bower, M Omerick and the entire Akili Interactive team as well as I Elson, L Kaye, S Goobich and the rest of the Ginger.io team for helping with data collection and partnering with us on this project. They also thank all the participants whose time and efforts made this work possible. Support for this research was provided by the National Institute of Mental Health (P.A.A.; R34-MH100466, T32 MH0182607, K24 MH074717). JAA and JTJ had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

View Abstract
• ## Supplementary Data

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

## Footnotes

• Competing interests AG is co-founder, chief science advisor and shareholder of Akili Interactive Labs, a company that develops cognitive training software. AG has a patent pending for a game-based cognitive training intervention, ‘Enhancing cognition in the presence of distraction and/or interruption’, on which the cognitive training application (PROJECT: EVO) that was used in this study was based.

• Ethics approval UCSF Committee for Human Research.

• Provenance and peer review Not commissioned; externally peer reviewed.

## Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.