Article Text
PDF
Abstract
We undertook a systematic review evaluating the effectiveness and cost of health-focused innovation design contests. We followed Cochrane guidance and systematically searched eight databases. Articles were included if they reported an open contest focused on improving health, required participants submit finished design solutions and contained a prize structure. We searched 3579 citations, identifying 29 health-focused innovation design contests which collectively received 15494 contest submissions. Contests solicited submissions worldwide (7) and exclusively from North America (13), Asia (4), Africa (2), Australia (2) and Europe (1). Submissions per contest ranged from 3 to 11354 (median of 73). Contest entrants were tasked with helping develop health promotion messages (HPM) (25) and improve predictive clinical models, protocols and/or algorithms (4). Two types of contests emerged—those focused on high-quality, innovative outcomes and those focused on the process of mass community engagement. All outcome-oriented contests had innovation design contest solutions equivalent or superior to the comparator (100%; 7/7). Two of two studies found that innovation design contests saved money (100%; 2/2). Five of seven process-oriented contests concluded the contest improved at least one health indicator (71%; 5/7). Innovation design contests are an effective way to solicit innovative solutions to health problems and spur mass community engagement.
- crowdsourcing
- prize
- challenge
Statistics from Altmetric.com
Introduction
In 2011, an 11-year-old boy in the USA was living with centronuclear myopathy, a rare genetic muscle disorder, which made it impossible for him to walk. Clinicians struggled for over a decade to identify the precise genes responsible for his condition, but without success. Then, in 2012, researchers at Boston Children’s Hospital organised an open contest for researchers worldwide to identify the genes causing the young patient’s physical ailment. The contest solicited genetic mapping solutions from 23 teams across the world, 8 of which successfully helped pinpoint the exact genes causing the muscle disorder in less than 9 months.1 The contest helped inform future research on the genetic muscle disorder2 and contributed to greater awareness about the disease.
This contest suggests that innovation design contests might be useful for improving health. We define innovation design contests as open contests that solicit finished solutions to design problems and then award prizes to finalists based on predetermined criteria.3 This prize structure and requirement to submit a creative solution is largely what distinguishes innovation contests from crowdsourcing models in general.4 Innovation design contests have been used by public, non-profit and private organisations to shape health campaigns targeting issues ranging from cardiac arrest to cancer.5 6 Innovation design contests are also being used with greater frequency.7 Between 2010 and 2013, the number of innovation design contests organised by the USA government increased by over 250%.3 Such rapid growth has largely been driven by advances in information technology that now enable organisations to more widely promote contests and interact with potential contest entrants worldwide.8
Despite increased interest, recent queries of the Cochrane Database of Reviews (http://www.cochranelibrary.com/) and the Centre for Reviews and Dissemination (https://www.york.ac.uk/crd/) indicate that there have been few systematic reviews formally evaluating the process and outcomes of health-focused innovation design contests. Such reviews are necessary for understanding when and how to implement innovation design contests for health. We undertook a systematic review of studies evaluating the effectiveness and cost of health-focused innovation design contests.
Methods
Search strategy and selection criteria
On the basis of the PRISMA checklist9 and Cochrane guidance,10 a search algorithm was used in eight electronic databases: Pubmed, CINAHL, Embase, Web of Science, PsycINFO, the Cochrane Library, Conference Papers Index and Dissertations & Theses Global (ProQuest). The search algorithm included the following categories: (1) health, (2) contest and (3) design. The full list of search terms is presented as supplemental data (online supplementary file 1). Databases were electronically searched 11 March 2016 without date restrictions. After articles outside of the study scope had been excluded, bibliographies of relevant articles identified from the electronic search were manually reviewed by an individual in order to identify additional pertinent studies.
Supplementary file 1
Inclusion criteria were defined prior to the study in a protocol registered on PROSPERO, an international prospective register of systematic reviews (42016038107). Articles were included if they reported on a contest that focused on improving health, required participants to submit a finished design solution, contained a prize structure and had sufficiently detailed methods. After duplicate articles were removed, screening proceeded in two stages (figure 1). First, one individual reviewed the abstract and title of each article according to the criteria mentioned above. A full-text review was then conducted with two individuals independently evaluating each article. Disagreements on whether to include a full-text article were resolved by a third individual. Non-English articles were excluded during the full-text screening phase.
Flow chart of research study selection.
Data analysis
The following data fields were independently double extracted by two individuals: citation information, study setting, target population, task assigned to contest entrants, health condition or behaviour of interest, type of contest submission, type of contest prizes, number of submissions, funding source, judging process, contest engagement metrics, evaluation outcomes of contest submission finalists and evaluation outcomes of innovation design contests among contest observers and entrants. We also extracted data on whether contest organisers were focused on the process of community engagement or the contest finalist’s efficacy. Decisions regarding classification of contests as process or outcome oriented were guided by Goldhammer’s report on design contests in the public sector.3
Study quality assessment
Study quality was independently assessed by two individuals for all articles that reported health-related outcome evaluations of (1) contest submission finalists or (2) the innovation design contest itself as a tool to increase engagement.
Qualitative studies were scored using the following criteria from the Critical Appraisal Skills Programme (CASP) qualitative appraisal tool11: (1) defined aims of research, (2) qualitative research appropriate, (3) appropriate research design, (4) recruitment strategy appropriate, (5) data collection is described and appropriate, (6) relationship between researcher and participants considered, (7) ethical issues considered, (8) data analysis is described and appropriate, (9) findings described and (10) value in research. Studies could score 0 or 1 for each criterion for a maximum study quality assessment score of 10. Final scores were averages of the two individuals’ independent scores.
Quantitative studies were scored using the following criteria developed from a validated quality assessment tool12: (1) defined target population, (2) probability sampling, (3) sample characteristics match the target population, (4) sufficient response rate, (5) standardised data collection, (6) reliability of data collection instrument, (7) validity of data collection instrument and (8) appropriate statistical methods. Studies could score 0 or 1 for each question for a maximum study quality assessment score of 8. Final scores are averages of the two individuals’ independent scores and normalised to a scale of 0–10. Results for specific scoring criteria are presented in online supplementary file 2.
Supplementary file 2
Funnel plots were not used to assess reporting bias because data were too heterogeneous for formal meta-analysis.
Role of the funding source
The funder of the study had no role in the study design, data collection, data analysis, data interpretation or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Results
Process of study selection
The database search and selection of articles from references of selected studies yielded 3579 citations (figure 1). After screening abstracts, the full texts of 43 articles were reviewed by two researchers (SWP and GS). Sixteen articles were excluded during full-text screening: three lacked sufficient information about the contest,13–15 one did not use primary data,16 four were not innovation design contests,17–20 one lacked a prize structure,21 two lacked a design component,21 22 one was in a language other than English23 and four were about previously included contests.24–27 Ultimately, 27 citations describing 29 contests were included.
Innovation design contest characteristics
Contests were conducted in five high-income countries,28–44 six middle-income countries43–47 and three low-income level countries.46 48 Seven contests allowed worldwide submissions.5 6 49–52 In sum, 15 494 contest submissions were reported. The number of submissions ranged from 3 to 11 354, with a median of 73. Contests were intended to promote health-messaging (25 contests),5 28–49 51 to improve predictive clinical models (three contests)6 50 and to improve models of computational data analysis (one contest).52 Health messaging contests focused on sexual health (10 contests),28 30 31 42–44 46–48 cancer (3 contests),35 36 49 smoking (3 contests)32–34 and other topics (8 contests).5 29 37–41 45 51 Fourteen studies were evaluated for study quality. One study was low quality (quality score of <8.0),48 five were moderate quality (quality score between 8.0 and 9.0)5 6 33 35 41 and seven were high quality (quality score of >9.0)32 43–45 47 50 52
The use of innovation design contests for health appears to be increasing. Since 2000, every 5-year period showed an increase in published contests (figure 2). Thirteen contests targeted individuals between 10 and 24 years of age.31–35 39–41 46 48 49 51 Contest entries included videos (11 contests),28 31 33 35 37 41–45 51 posters (4 contests),5 33 34 39 stories (4 contests),30 31 40 46 clinical algorithms (4 contests),6 50 52 photos (3 contests)36 38 47 and other media (4 contests).29 33 38 49 One contest allowed for multiple types (video, photo, text, so on) of submissions. Posters were defined as static visual graphics of any dimension and format. All contests provided a prize structure.5 6 28–52 Out of the contests that listed prize structures, only four31 32 48 had a single prize. Eighteen contests offered monetary prizes,5 6 28–35 39–42 45 50 52 13 contests5 6 28 29 31 33 39–42 46 50 reported sponsorship or support from a commercial entity and 11 contests27–29 33 36 42 44 46 50 53 reported receiving ethical approval. Contest finalists were selected by judging panels (20 contests),6 29 31–35 39–41 43–46 48–50 52 crowd votes (one contest)37 or a combination of a judging panel and crowd votes (eight contests).21 28 30 36 42 47 Additional contest characteristics are shown in table 1.
Number of innovation design contests over time (n=26).
Contest characteristics
Process-oriented and outcome-oriented innovation design contests
Two distinct types of contests were identified1: outcome-oriented contests organised primarily to elicit a small number of highly innovative contest finalists and2 process-oriented contests organised primarily to enhance community health knowledge and behaviours through mass engagement with the contest itself. Outcome-oriented contests focused on soliciting high-quality entries that could be directly incorporated into programmes, whereas process-oriented contests focused on attracting attention through marketing and a high volume of entries that encouraged communities to better understand the health issue of interest and spur discussion. A comparison of process-oriented and outcome-oriented innovation design contests is presented in table 2.
Comparison of process-oriented and outcome-oriented innovation design contests for health
Evaluation of outcome-oriented innovation design contests
Seven outcome-oriented innovation design contest studies evaluated finalist contest entries.6 43–45 50 52 Six were evaluated with quantitative data,6 43 44 50 52 and one was evaluated with qualitative data.45 Quantitative evaluations included randomised control trials (two studies),43 44 observational cohort studies (two studies)6 50 and observational cross-sectional studies (two studies).6 52 Among the finalist contest entries of quantitative contests with control groups, two were superior to a control,6 44 one demonstrated non-inferiority44 and two had comparable results.43 50 Qualitative evaluation data were based on focus group transcripts, the results of which indicated that crowdsourced contest submissions had strong potential to improve health knowledge, attitudes and behaviours.45 Additional findings from the evaluation of outcome-oriented innovation design contests are presented in table 3.
Outcome evaluations of contest finalists
Evaluation of process-oriented innovation design contests
Seven process-oriented innovation design contest studies evaluated the effects of contests for at least one health indicator.5 32 33 35 41 47 48 Measures of contest engagement varied. Three contests did not specify the type of contest engagement,33 35 48 two contests measured contest engagement as observing the contest or visiting the contest website5 32 and two contests measured contest engagement as submitting an entry.41 47 Outcome measures included both health-related behaviours32 33 48 and attitudes.5 35 41 47 All six quantitative contest evaluations were based on observational data, four of which used a pre–postevaluation32 33 35 48 and two of which used a post-only evaluation.5 41 Of the three contest evaluations which had a comparator group,32 33 48 two indicated that the innovation design contest had improved some health-related indicator33 48; results from the third evaluation failed to demonstrate that innovation design contests statistically significantly improved health-related outcomes.32 Results from the one qualitative study showed that the contest had helped increase knowledge and healthy attitudes about sexual health.47 Additional findings from evaluation of process-oriented innovation design contests are presented in table 4.
Health-related effects of innovation design contests
Discussion
The use of innovation design contests for health has grown rapidly in recent years, but the pertinent academic literature remains relatively underdeveloped and lacking in standards of evaluation and reporting. This appears especially true for process-oriented innovation design contests for health. Past reviews and conceptual papers have examined how entries submitted in innovation design contests can be used to identify novel solutions to public health problems (outcome-oriented contests),4 54 55 but none discussed how implementation of an innovation contest itself is able to directly impact health knowledge and attitudes through contest engagement (process-oriented contests). In contrast, business and non-profit organisations have explicitly recognised innovation design contests as an effective means of marketing and engaging with the community.3
In response to the relatively nascent literature on innovation design contests for health, we conducted a systematic review that identified a wide range of health-oriented innovation design contests, including those focused on innovative outcomes (outcome-oriented contests) and those focused on the process of mass contest engagement (process-oriented contests). Our review expands on the limited crowdsourcing literature3 7 54–56 by focusing specifically on innovation design contests for health, examining study quality and using a systematic review. Overall, contest evaluations indicated that innovation design contests for health were effective and low cost, but many lacked outcome or process evaluation.
Evidence from seven outcome-oriented contest studies suggests that across disparate health domains, innovation design contests for health produce novel and effective solutions. This is consistent with innovation design contests from other fields.7 8 In principle, a larger and more diverse pool of problem-solvers increases the likelihood of identifying an innovative solution, often in a limited time.8 55 In 5 months, one contest initiated by the National Cancer Institute received 52 submissions from 127 participating scientists worldwide and subsequently identified a novel predictive algorithm for cancer drug sensitivity that was superior to previously established models.57 Findings also suggest that innovation design contests may also save more money compared with more conventional approaches.43 50
Results of five process-oriented contests indicate that innovation design contests are effective at directly enhancing health knowledge and/or behaviours, especially among young people. In one contest designed to promote sexual health in the community, condom use among young people increased from 46% to 69% before and after the contest, respectively.48 These findings are consistent with data from innovation design contests in other fields.58 Compared with more conventional marketing campaigns, the participatory nature of innovation design contests facilitates campaigns that may be more authentic and locally relevant.58 Given that many health behaviours are often initiated during school age,59 school-based innovation design contests could engage youth and create more effective health campaigns.
Few health-focused innovation design contests have been rigorously evaluated. Among the 25 health promotion messaging contests, only 3 reported an outcome evaluation of the top-ranked submissions27 43–45 47 and 7 evaluated health-related impacts of the innovation design contest.5 32 33 35 41 47 48 Furthermore, three quantitative evaluations of innovation design contests lacked a comparison group.5 33 41 Many health promotion messaging contests reported metrics important for assessing the reach of the contest (eg, number of contest submissions, website hits and video views), but these process-oriented indicators alone were unable to assess how contests may have impacted the health indicators of individuals who interacted with the contest. Future studies of health promotion messaging contests should conceptualise the contest implementation itself as a type of intervention and at minimum use comparison groups or pre–postassessments to evaluate how the contest itself may have affected health indicators of contest observers and active contest participants.
Our review findings have implications for research and practice. Results demonstrated that design contests can be an effective strategy for eliciting novel and effective solutions to health problems and directly promoting health within specific communities. However, success of a design contest will depend on key decisions concerning contest design and implementation. Generally, contests should be customised according to organiser’s strategic goals.3 For example, if organisers were chiefly concerned with maximising awareness among a large population (ie, a process-oriented contest), then it may be beneficial to use public voting rather than a judging panel and to have a greater number of prizes rather than a single prize.3 However, a single prize of high-value may be more useful for identifying technical solutions to a specific design problem (ie, outcome-oriented contest). In any case, contest organisers must ensure that the complexity of the design task is commensurate with participant motivations, incentives and resources.3 Given the private sector’s experience with design contests, greater interdisciplinary collaboration between business and public health sectors may improve success of design contests for health. Finally, as innovation design contests for health are applied with increasing frequency in the future, it is important that norms for reporting about contest details be established. Table 5 enumerates a list of potential data types for future studies to report.
Potential data to report for health innovation design contest research studies
Our review has several limitations. First, publication bias may have led to under-representation of innovation design contests with poor effectiveness. Nonetheless, we were still able to identify studies in which the contest finalist failed to demonstrate significant improvement50 and in which the contest failed to provide evidence of improving a health-related outcome.32 35 Second, innovation design contests remain a relatively new research field in the health sciences, and a lack of established terminology may have led to inadvertent exclusion of innovation design contest studies that used descriptors not included in our search. However, by reviewing 3579 citations extracted from multiple databases in a wide search, we are confident that this study is among the most comprehensive reviews of published health innovation design contests to date. Third, high heterogeneity in outcomes between studies precluded us from conducting more precise quantitative analyses, such as a meta-analysis. Nonetheless, evidence from 12 of 14 evaluation studies strongly suggests that health innovation design contests are overall an effective strategy for developing novel health solutions and promoting healthy behaviours.
In conclusion, health-focused innovation design contests may help to create effective, low-cost health campaigns. Several US government leaders have realised this opportunity, introducing legislation to expand health-focused innovation design contests. For example, the recently funded ‘21st Century Cures Act’ mandates health-focused innovation design contests to be organised by the US National Institutes of Health.60 61 Similar government initiatives to support contests in the UK,62 Canada63 and Taiwan64 suggest that innovation design contests for health have traction. Health-focused innovation design contests may help organisations to design more people-centred public health programmes.
Acknowledgments
We would like to thank Jennifer Walker for her assistance with conducting the literature search. We would also like to thank the two anonymous reviewers for their feedback.
References
Footnotes
SWP and GS contributed equally.
Handling editor Balram Bhargava
Contributors JDT conceptualised the study. SWP, GS and JDT drafted the manuscript and provided substantial contribution to data acquisition and data interpretation. BB, WT, AM, C Wang and C Wei provided substantial contribution to the interpretation of data and revising the letter for important intellectual content. All authors approved the final version of the manuscript to be published and accept accountability for all aspects of the letter.
Funding This study was supported by the National Institutes of Health Fogarty International Center (R25TW009340).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.