Co-creation of patient engagement quality guidance for medicines development: an international multistakeholder initiative
==========================================================================================================================

* Katherine Deane
* Laure Delbecque
* Oleksandr Gorbenko
* Anne Marie Hamoir
* Anton Hoos
* Begonya Nafria
* Chi Pakarinen
* Ify Sargeant
* Dawn P Richards
* Soren Eik Skovlund
* Nicholas Brooke

## Abstract

**Introduction** Meaningful patient engagement (PE) can enhance medicines’ development. However, the current PE landscape is fragmentary and lacking comprehensive guidance.

**Methods** We systematically searched for PE initiatives (SYNaPsE database/publications). Multistakeholder groups integrated these with their own PE expertise to co-create draft PE Quality Guidance which was evaluated by public consultation. Projects exemplifying good PE practice were identified and assessed against PE Quality Criteria to create a Book of Good Practices (BOGP).

**Results** Seventy-six participants from 51 organisations participated in nine multistakeholder meetings (2016–2018). A shortlist of 20relevant PE initiatives (from 170 screened) were identified. The co-created INVOLVE guidelines provided the main framework for PE Quality Guidance and was enriched with the analysis of the PE initiatives and the PE expertise of stakeholders. Seven key PE Quality Criteria were identified. Public consultation yielded 67 responses from diverse backgrounds. The PE Quality Guidance was agreed to be useful for achieving quality PE in practice, understandable, easy to use, and comprehensive. Overall, eight initiatives from the shortlist and from meeting participants were selected for inclusion in the BOGP based on demonstration of PE Quality Criteria and willingness of initiative owners to collaborate.

**Discussion** The PE Quality Guidance and BOGP are practical resources which will be continually updated in response to user feedback. They are not prescriptive, but rather based on core principles, which can be applied according to the unique needs of each interaction and initiative. Implementation of the guidance will facilitate improved and systematic PE across the medicines’ development lifecycle.

*   patient engagement
*   quality guidance
*   multi-stakeholder initiative
*   patient engagement medicines

## Introduction

The value and necessity of engaging patients during the development and life cycle of medicines is increasingly recognised1–3; many discrete programmes have been initiated,4–8 which are often specific to a particular phase of drug development (eg, clinical trials, regulatory or health technology assessment) and to a region.9–13 For example, in the USA, the Food and Drug Administration has conducted patient-focussed drug development (PFDD) meetings to gather patient perspectives on specific conditions and their treatment more systematically.14 In Europe, the Innovative Medicines Initiative has launched PARADIGM (Patients Active in Research and Dialogues for an Improved Generation of Medicines) to develop a framework for patient input in early dialogue, research priority setting and clinical trials design.15 Many more initiatives exist and though complementary, these separate activities have resulted in a fragmented PE landscape, with a lack of continuity, efficiency and coordination. Investment in a holistic, integrated and sustainable initiative is required to cover the entire medicines lifecycle and to connect all stakeholders across geographies. Though there is no widely accepted definition of patients’ engagement in the medicines’ development continuum, in this paper, the term ‘PE’ refers to the active and meaningful involvement of patients and carers as active participants and collaborators in developing medicines. Patient Focused Medicines Development (PFMD;  [www.pfmd.org](http://www.pfmd.org) ) is a global multistakeholder collaboration of health stakeholders that aims to synergise PE efforts by working with stakeholders to co-create a meta-framework for PE that spans the entire medicines lifecycle. PFMD is taking a disciplined four-step approach to this co-creation, in order to build on existing work and prevent duplication. The steps are: (1) mapping and connecting the PE landscape to learn from existing efforts and identify needs or gaps; (2) convening multistakeholder workshops to co-create tools that meet identified needs; (3) reiterative refinement of tools to develop and pilot a draft meta-framework for PE and (4) creation of a practical PE Actionable Framework and implementation tools. Here we describe the first two steps of the approach culminating in the development of the PE Quality Guidance which is a core element of the meta-framework. The PE Quality Guidance was co-created as a practical tool to facilitate and improve the quality of PE activities across the medicines’ lifecycle. The principles outlined within the PE Quality Guidance are intended to be applicable for use by any stakeholder involved in the design, execution or evaluation of PE activities (regardless of role, position, experience or geographical location) and relevant for all interactions of stakeholders with patients.

## Methods

### Landscape review and identification of priority needs in PE

Multistakeholder Working Groups were established through invitation via PFMD and PFMD member networks, existing contributors to meta-framework co-creation activities and owners of PE initiatives. Invitees were required to have some PE experience and expertise in at least one of the medicine development phases. A literature search was conducted to provide Working Group participants with an overview of the PE landscape. Published literature and existing frameworks were identified using keyword searches on PubMed and *British Medical Journal* (BMJ) *Open* (timeframe January 2011–December 2016). Keywords were PE, patient and public involvement, clinical trials, patients benefit, PE benefit for pharmaceutical industry, PE benefit for patients, PE benefit for research. Articles relevant to PE in the context of medicines lifecycle that provided the most detail and granularity and were most informative for Working Group interrogation were identified.

SYNaPsE (SYNergising Patient Engagement) was used to augment published information available. SYNaPsE was designed as a platform for capturing PE practices and is a dynamic user-populated repository of objective, well-documented multistakeholder PE initiatives. It categorises and ‘maps’ PE initiatives and frameworks, organisations active in PE, experts and resources. Initiatives in SYNaPsE, follow a standard template for documentation of methods, challenges and outcomes, which gives a structure for understanding each initiative. Owners of relevant initiatives in SYNaPsE were contacted to provide more information on their initiative(s) to capture a deeper level of insight and understanding that then informed the co-creation process. Each Working Group included participants with expertise in the relevant phase(s) of medicines development (figure 1): there was no overlap of individuals in the different Working Groups. Working Groups were tasked with using their PE experience and review of the PE landscape to identify and prioritise PE needs. Co-creation of practical PE Quality Guidance to help stakeholders improve the quality of their PE activities and provision of illustrative examples of good practice were identified as priorities independently across Working Groups.

![Figure 1](http://innovations.bmj.com/https://innovations.bmj.com/content/bmjinnov/5/1/43/F1.medium.gif)

[Figure 1](http://innovations.bmj.com/content/5/1/43/F1)

Figure 1 
Process for development of Patient Engagement Quality Guidance (PEQG) tool.

### PE Quality Guidance co-creation

Task Forces comprised Working Group participants (or their nominees) who volunteered to contribute to the iterative elaboration of the PE Quality Guidance, sharing their example PE initiatives and testing the PE Quality Guidance in development with these examples. Task Force nominees were invited based on their having the relevant expertise to complete prioritised actions and willingness and capacity to contribute. A Core Team comprising participants of Working Groups or Task Forces with extensive experience in PE was also established to facilitate finalisation of tools. There was intentional overlap with Working Group, Task Force and Core Team members to ensure continuity and incorporate learnings from each stage. A preliminary PE Quality Guidance tool was developed by Task Forces in an iterative process which included validation, feedback and refinement at each step, with input from the Working Groups (figure 1). The guidance introduces PE Quality Criteria to assess PE practices consolidated from published literature and co-developed further by Task Forces. The Quality Criteria were validated through retrospective application to real PE activity with feedback and refinement in an iterative process. Further feedback and validation were secured through public consultation.

### Public consultation on draft PE Quality Guidance

A public online, survey-based consultation on the draft PE Quality Guidance was undertaken (from 20 November 2017 to 1 January 2018) to gather wider input from PE stakeholders. This consultation was open to all regardless of experience in PE. The survey had two sections: section one asked general questions about the PE Quality Guidance and its usability; section two asked detailed questions about the PE Quality Criteria specifically (see online supplementary file 1 for Survey Questions). Consultation was invited via the PFMD website and across stakeholder groups via Working Group, Task Force and PFMD networks; and through advertising (21 589 reach), media (40 259 reach) and a social media campaign. In addition, a separate focus group (organised and funded by Parkinson’s UK) with representation from patient organisations, a government health research agency, academic research, healthcare professionals and a regulatory agency was held in October 2017. Feedback was submitted for independent review by an external expert from the Working Group and a patient representative/academic from the Core Team to finalise the PE Quality Guidance.

### Supplementary data

[[bmjinnov-2018-000317supp001.htm]](pending:yes) 

### Development of the Book of Good Practices

Examples of good practice were collected through screening of initiatives in SYNaPsE (entered between SYNaPsE launch in January 2017 and workshop commencement in September 2017) and via Task Force and PFMD networks. All initiatives independently submitted by PE initiative owners were eligible for inclusion in the Book of Good Practices. Preliminary selection criteria were pragmatic and included initiatives: being specific to PE in medicines’ development; involving more than one stakeholder group and having comprehensive description of methodology for PE practice (and outcomes for completed initiatives where available). Final selection for inclusion in the first iteration of the Book of Good Practices was undertaken by the Core Team who reviewed each initiative against the following key criteria: being extensively described by the owner using the PE Quality Guidance; demonstrating PE Quality Criteria as assessed by majority consensus (>half) of the multistakeholder Core Team; focusing on involving patients in decision processes around medicines development or lifecycle management; involving multiple stakeholder groups; owners/organisations agreeing to participate in the Book of Good Practices and being willing to provide additional information to further enrich the examples; being approved for public dissemination. The evaluation process was anonymised to avoid bias.

## Results

### Summary of PE Quality Guidance co-creation workshops

Nine Working Group, Task Force or Core Team meetings were held (November 2016–June 2018) involving 76 unique participants, representing 51 organisations (including patient charities, academic researchers, funders, pharmaceutical companies). Table 1 summarises objectives and outputs from these meetings.

View this table:
[Table 1](http://innovations.bmj.com/content/5/1/43/T1)

Table 1 
Summary of Working Group/Task Force workshops and key outputs

#### Step 1: mapping and connecting the PE landscape to learn from existing efforts

Literature search yielded 239 results: 134 were available through PubMed, of which 12 were relevant or somewhat relevant to the subject and 105 through *BMJ Open*, of which 13 were relevant or somewhat relevant to the subject. Eight additional resources were identified and assessed to understand the progress of earlier developed approaches and concepts beyond the 2016 cut-off of the search period.1 6 7 16–20 The 33 relevant resources yielded seven conceptual frameworks/models (summarised in table 2). Although the importance of unifying PE approaches was highlighted within several models and frameworks, methodology gaps and variations were identified, for example, within objectives, outputs and applicability to different stages of medicines lifecycle. Working Groups agreed that taken together, the seven frameworks represented comprehensive tools that reflect the complexity of medicines research and development, and capture all stages of medicines lifecycle. They jointly demonstrated the following PE characteristics: diversity and representativeness, joint working and partnership, reciprocity, capabilities development and learning, continuity and sustainability, transparency, meaningfulness, metrics for impact assessment, standardisation and communication plans.

View this table:
[Table 2](http://innovations.bmj.com/content/5/1/43/T2)

Table 2 
Overview of frameworks and resources assessed for PE quality guidance co-creation

#### Step 2: Multistakeholder co-creation of PE guidance and good practice identification

Task Forces adapted and augmented the characteristics and related indicators from the frameworks into the PE Quality Guidance. The first section of the guidance tool explains how to use it: for planning a new PE project; for gap analysis to compare projects within an organisation and to assess the impact of ongoing or completed PE projects. The INVOLVE recommendations21–23 were agreed by Working Groups to be the most advanced, providing well documented and sufficiently detailed information (particularly on the practical ‘how to’ of PE) and were selected as the basis for development of the first iteration of PE Quality Criteria. The INVOLVE recommendations were developed for use in the UK and for advancing public involvement specifically in research and were therefore adapted and enriched with the analysis of PE initiatives collected in SYNaPsE and from Working Groups and Task Forces to reflect the needs of the broader PE community and stakeholders and to be relevant across all phases of medicines development. Seven PE Quality Criteria were agreed as a manageable number, comprehensive (without being overwhelming), relevant for the majority of PE stakeholders and accessible to all regardless of experience in PE. The PE Quality Criteria describe the core elements that should be integrated for individual PE initiatives or across several (related) initiatives that involve the same partners. They form the backbone of PE methodology and provide a set of basic principles to help standardise the levels of PE, assess the quality of PE in existing projects, and document and share outcomes in a standardised way. Descriptions for each of the seven PE Quality Criteria are summarised in table 3. Tools for assessment of initiatives (either planned, in preparation, ongoing or completed) against the PE Quality Criteria were also developed. Tools contain a definition, a rationale and questions for consideration by the initiative owners used for planning and/or evaluation purposes for each criterion.

View this table:
[Table 3](http://innovations.bmj.com/content/5/1/43/T3)

Table 3 
Patient engagement (PE) quality criteria summary and description

### Examples of good practice

Of 170 available initiatives screened, 40 were selected based on preliminary criteria then shortlisted to 20 using key criteria and anonymised for Core Team review. Eight initiatives were selected for inclusion in the first iteration of the Book of Good Practices. Table 4 provides a summary of the PE examples selected for the Book of Good Practices and results of their assessment using the PE Quality Criteria.

View this table:
[Table 4](http://innovations.bmj.com/content/5/1/43/T4)

Table 4 
Summary of the patient engagement (PE) examples selected for the Book of Good Practices and results of their assessment using the PE quality criteria

### PE quality guidance public consultation

The PE Quality Guidance public consultation resulted in 851 website visits and 67 responses from patients, patient advocates/organisations (n=25); pharmaceutical and biotech industry (n=19); research/academia (n=12) and other (n=11). The majority (69%) of responders were from Europe and the remainder from Canada, the USA, Algeria and Australia. Over 80% of respondents indicated that they were ‘advanced’ in terms of their PE experience (‘actively part of PE projects’), while 16% were a ‘beginner’ (‘currently I have no previous experience, but I am planning to do PE projects within the next year’) and, 3% ‘none’ (‘I have no experience in working with patients but would like to do more’).

Of 51 respondents, the vast majority (range 85%–96%) agreed or strongly agreed that: the PE Quality Criteria are useful for achieving quality PE practice; the descriptive language used is comprehensive and easy to understand, and the format is clear. Overall, 56% of respondents felt they did not need help to use the PE Quality Guidance. Specific feedback included needing practical examples and specific tips based on how others have used it. Most respondents (range 65%–84%) agreed or strongly agreed that the PE Quality Guidance could be used to: improve the quality and consistency of PE activities; better plan and develop PE activities; better assess the quality and impact of PE initiatives; identify gaps and opportunities for PE activities; capture and share learnings beyond the project or project team and structure work with partnering organisations (figure 2).

![Figure 2](http://innovations.bmj.com/https://innovations.bmj.com/content/bmjinnov/5/1/43/F2.medium.gif)

[Figure 2](http://innovations.bmj.com/content/5/1/43/F2)

Figure 2 
Public consultation feedback on patient engagement (PE) quality guidance.

Of 51 respondents, 86% agreed or strongly agreed that having practical PE guidance was important in their work and 90% indicated they would use such guidance. Almost half (49%) of respondents already use some type of PE guidance in their work. The vast majority (92%) of respondents believed that the PE Quality Guidance should be used by diverse health stakeholders (all those involved in medicines development, patients/patient organisations, pharmaceutical industry, regulators, payers, research and academia). Specific feedback noted that US and EU standards for engaging patients and patient organisations differ and should be recognised; the tool is a ‘shared guidance, understood and applied by all stakeholders’ involved in PE engagement and the tool may be especially useful for less experienced patient representatives. Overall, 21 respondents gave detailed feedback on the PE Quality Criteria including 85 comments on specific criteria and four general comments. Responses were generally evenly distributed across all seven PE Quality Criteria ranging from 67% (shared purpose) to 48% (respect and accessibility, representativeness of stakeholders). Comments ranged from simple agreement (eg, ‘This is crucial’) to detailed suggestions on each criterion and were overall positive. The feedback from all partners, representing varying levels of PE experience, has informed the final versions of the PE Quality Guidance. We have created the Book of Good Practices to provide examples of good quality practical applications of the guidance.

## Discussion

The development of the PE Quality Guidance adhered to the core principle of respect for all stakeholders’ knowledge. It also ensured a diversity of viewpoints from: a systematic search of the literature; a wide range of stakeholders co-creating the guidance (table 1) and public consultation, thereby minimising potential bias. This practical guide aims to improve PE quality during planning and development of new medicine development projects or assess the quality and impact of ongoing/completed projects. Examples of good practice of the PE guidance were rated against the quality criteria to provide high-quality exemplars of PE in a range of contexts. This is not intended to be an exhaustive set of examples of how to apply the guidance, rather to provide real-world case studies showing the diversity of approaches and techniques that can be used to generate meaningful PE. Finally, the PE Quality Guidance was assessed for utility and practicality by a wide range of stakeholders. They determined it would be of use, particularly to those new to PE and those wishing to document, and quality appraise their PE processes. In response to feedback, the PE Quality Guidance incorporates good practice examples and tips to aid practical implementation of PE across diverse activities and groups. The PE Quality Guidance is a dynamic tool and is currently being piloted and implemented across several PE initiatives. Learnings and feedback will be used to further refine the tool and we will report on the outcomes of these pilots.

PE is not ‘one size fits all’ and consequently, the PE Quality Guidance is not prescriptive, rather it is based on core principles that should be adapted and applied according to the unique needs of each interaction and project. These principles are relevant and applicable beyond medicines development specifically, for example, also in clinical research settings. The Guidance can be tailored by adding specific modules to deliver consistently high-quality PE activities. This modular approach facilitates a long-term strategy for more systematic and meaningful PE. We hope the detailed specificity of the PE Quality Guidance and supporting Book of Good Practices provides not only sufficient structure to allow genuine engagement but also sufficient flexibility to be adapted to many types of medicines development, cultural considerations, national legislation and relevant specific circumstances. We also believe—and our survey confirmed—that the guidance principles have sufficient generalisability to also be applied to the development of other clinical interventions, such as educational interventions and medical devices.

We acknowledge potential limitations of this work: literature screening may not have identified all relevant existing efforts, and differences in Working Group individuals’ experience, confidence and specific challenges (such as language barriers and health conditions) may influence participation during workshops. However, different processes and checkpoints were established to ensure that all participants were able to contribute fully (eg, anonymously if desired). Though there were limited contributions from participants in developing countries, the distribution of contributors reflects current activity patterns across medicines development. Furthermore, our extensive search of literature for PE guidance effectively expanded the pool of contributors to the PE Quality Guidance tool. As such, we believe our guidance is as comprehensive and diverse as is possible. The resulting first iteration of the PE Quality Guidance intentionally focuses on PE characteristics that are relevant across stakeholder groups to be broadly relevant. The next phase of development involves customisation of the Guidance in identified priority activities. Examples for the Book of Good Practices were primarily identified through SYNaPsE: some initiatives (though worthy) may not have included sufficient information to meet initial inclusion criteria. As the PE Quality Guidance is used by research projects worldwide, it will also serve to collect good practices which will be published on the PFMD website to inspire new initiatives.

The need for PE that informs medicine development at all stages of the development process has been widely recognised. It is known that meaningful PE can improve the relevance of medicines in addressing symptoms of importance to patients and improve health outcomes.24–26 It could help address the observation that current medical research does not optimally address the needs and concerns of patients and the clinicians that treat them.27 28 However, it is important that this PE is genuine and not tokenistic in order to realise the potential benefits.29 30 The field of PE is growing and maturing and use of evidence-based methodologies and documentation is becoming increasingly important.31 32 The PE Quality Guidance provides a much-needed global tool for documenting and improving the quality of the process of PE across the medicines’ lifecycle and subsequently, the quality of desired outputs. Future uses should seek to apply scientific and robust methodologies to further refine the guidance and build the evidence base for the value of systematic PE.

We described the methodological four-step approach towards co-creation of a pilot meta-framework for PE: this requires multistakeholder, long-term investment in a global and collaborative venture. Investment in a foundation of mutual respect and trust through transparent governance, in building and synergising the PE ecosystem, and in driving implementation of co-created solutions to deliver better PE. The co-creation of this meta-framework and implementation toolkit (Steps 3 and 4) is underway and will build on outcomes from piloting the PE Quality Guidance and ongoing feedback to provide a comprehensive and actionable resource for more consistent and meaningful PE.

## Acknowledgments

The authors gratefully acknowledge the contribution of all members of the Patient Focused Medicines Development (PFMD) Patient Engagement Meta-framework Co-creation Team including: Karlin Schroeder (Community Engagement Director, Parkinson's Foundation, New York, USA); Carol Meyer Sokolowski (R&D Patient Engagement Office, Takeda Pharmaceuticals, Illinois, USA) and Jean-Christophe Reglier (Amgen S.A.S, Boulogne Billancourt, Paris, France). They also thank Ify Sargeant (ISMEDICA Ltd, UK) for writing and editorial support funded by PFMD.

## Footnotes

*   Contributors AMH, CP and NB identified participants, planned and organised multistakeholder working group workshops to co-create the PE Quality Guidance. KHOD, LD, OG, AMH, BNE, CP, DP, SES and NB participated in PE Quality Guidance co-creation workshops, tested and reviewed iterative drafts of the PE Quality Guidance; KHOD, SES, OG DR, LD, BNE, AMH, CP were part of the Core Team that reviewed and evaluated the example cases/projects for the Book of Good Practices; KHOD, LD, OG, AMH, BNE, CP, IS, DP, SES, NB jointly (and as part of a larger contributor group) reviewed the final versions of the PE Quality Guidance, scenarios and Book of Good Practices before release; KHOD, LD, OG, AMH, AH, BNE, CP, IS, DP, SES, NB jointly planned the manuscript and agreed the outline (with each author having responsibility for writing a specific section of the paper); IS coordinated writing efforts and developed each draft by collating and editing sections written by authors; circulated drafts for author review; collated and implemented author comments at each draft stage. KHOD, LD, OG, AMH, AH, BNE, CP, IS, DP, SES, NB reviewed and approved the final manuscript for submission.

*   Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

*   Competing interests DR: personal fees from various pharmaceutical companies, outside the submitted work; NB: grants from various pharmaceutical companies and personal fees from Innovative Medicine Initiative outside the submitted work.

*   Patient consent for publication Not required.

*   Provenance and peer review Not commissioned; externally peer reviewed.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  1. Boutin M , Dewulf L , Hoos A , et al . Culture and process change as a priority for patient engagement in medicines development. Ther Innov Regul Sci 2017;51:29–38.[doi:10.1177/2168479016659104](http://dx.doi.org/10.1177/2168479016659104) 
    
    

2.  2. Perfetto EM , Burke L , Oehrlein EM , et al . Patient-focused drug development: a new direction for collaboration. Med Care 2015;53:9–17.[doi:10.1097/MLR.0000000000000273](http://dx.doi.org/10.1097/MLR.0000000000000273) 
    
    [CrossRef](http://innovations.bmj.com/lookup/external-ref?access_num=10.1097/MLR.0000000000000273&link_type=DOI) 
    
    [PubMed](http://innovations.bmj.com/lookup/external-ref?access_num=25494232&link_type=MED&atom=%2Fbmjinnov%2F5%2F1%2F43.atom) 

3.  3. Levitan B , Getz K , Eisenstein EL , et al . Assessing the financial value of patient engagement: a quantitative approach from CTTI's patient groups and clinical Trials Project. Ther Innov Regul Sci 2018;52:220–9.[doi:10.1177/2168479017716715](http://dx.doi.org/10.1177/2168479017716715) 
    
    [CrossRef](http://innovations.bmj.com/lookup/external-ref?access_num=10.1177/2168479017716715&link_type=DOI) 
    
    [PubMed](http://innovations.bmj.com/lookup/external-ref?access_num=29714515&link_type=MED&atom=%2Fbmjinnov%2F5%2F1%2F43.atom) 

4.  4.National Health Council (NHC). The patient voice in value: the NHC patient-centered value model rubric, 2016. Available: [http://www.nationalhealthcouncil.org/sites/default/files/Value-Rubric.pdf](http://www.nationalhealthcouncil.org/sites/default/files/Value-Rubric.pdf) [Accessed 4 Jun 2018].
    
    

5.  5. Geissler J , Ryll B , di Priolo SL , et al . Improving patient involvement in medicines research and development:: a practical roadmap. Ther Innov Regul Sci 2017;51:612–9.[doi:10.1177/2168479017706405](http://dx.doi.org/10.1177/2168479017706405) 
    
    

6.  6. Sheridan S , Schrandt S , Forsythe L , et al . The PCORI engagement rubric: promising practices for partnering in research. Ann Fam Med 2017;15:165–70.[doi:10.1370/afm.2042](http://dx.doi.org/10.1370/afm.2042) 
    
    [Abstract/FREE Full Text](http://innovations.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTUvMi8xNjUiO3M6NDoiYXRvbSI7czoyMToiL2Jtamlubm92LzUvMS80My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

7.  7. Kirwan JR , de Wit M , Frank L , et al . Emerging guidelines for patient engagement in research. Value Health 2017;20:481–6.[doi:10.1016/j.jval.2016.10.003](http://dx.doi.org/10.1016/j.jval.2016.10.003) 
    
    

8.  8. Staniszewska S , Brett J , Simera I , et al . GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ 2017;358.[doi:10.1136/bmj.j3453](http://dx.doi.org/10.1136/bmj.j3453) 
    
    

9.  9. Ho M , Saha A , McCleary KK , et al . A framework for incorporating patient preferences regarding benefits and risks into regulatory assessment of medical technologies. Value Health 2016;19:746–50.[doi:10.1016/j.jval.2016.02.019](http://dx.doi.org/10.1016/j.jval.2016.02.019) 
    
    

10. 10. Domecq JP , Prutsky G , Elraiyah T , et al . Patient engagement in research: a systematic review. BMC Health Serv Res 2014;14.[doi:10.1186/1472-6963-14-89](http://dx.doi.org/10.1186/1472-6963-14-89) 
    
    

11. 11. Fergusson D , Monfaredi Z , Pussegoda K , et al . The prevalence of patient engagement in published trials: a systematic review. Res Involv Engagem 2018;4.[doi:10.1186/s40900-018-0099-x](http://dx.doi.org/10.1186/s40900-018-0099-x) 
    
    

12. 12. Bloom D , Beetsch J , Harker M , et al . The rules of engagement: CTTI recommendations for successful collaborations between sponsors and patient groups around clinical trials. Ther Innov Regul Sci 2018;52:206–13.[doi:10.1177/2168479017720247](http://dx.doi.org/10.1177/2168479017720247) 
    
    

13. 13. Huang GD , Bull J , Johnston McKee K , et al . Clinical trials recruitment planning: a proposed framework from the clinical trials transformation initiative. Contemp Clin Trials 2018;66:74–9.[doi:10.1016/j.cct.2018.01.003](http://dx.doi.org/10.1016/j.cct.2018.01.003) 
    
    

14. 14.Patient-focused drug development: disease area meetings held in fiscal years 2013-2017. Available: [https://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm347317.htm](https://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm347317.htm) [Accessed 22 Jan 2019].
    
    

15. 15.PARADIGM. Patients active in research and Dialogues for an improved generation of medicines. Available: [https://imi-paradigm.eu/](https://imi-paradigm.eu/) [Accessed 21 Jun 2018].
    
    

16. 16. Bagley HJ , Short H , Harman NL , et al . A patient and public involvement (PPi) toolkit for meaningful and flexible involvement in clinical trials – a work in progress. Res Involv Engagem 2016;2.[doi:10.1186/s40900-016-0029-8](http://dx.doi.org/10.1186/s40900-016-0029-8) 
    
    

17. 17. South A , Hanley B , Gafos M , et al . Models and impact of patient and public involvement in studies carried out by the medical Research Council clinical trials unit at University College London: findings from ten case studies. Trials 2016;17.
    
    

18. 18. Gorbenko OV . The role of patients and community representatives in value-based assessment of health technologies and respective decision-making processes within healthcare. In: Economy and legislation of health care. 2, 2016: 10–23.
    
    

19. 19. Perfetto EM , Oehrlein EM , Boutin M , et al . Value to whom? The patient voice in the value discussion. Value Health 2017;20:286–91.[doi:10.1016/j.jval.2016.11.014](http://dx.doi.org/10.1016/j.jval.2016.11.014) 
    
    

20. 20.Patient-Perspective value framework (PPVF). Available: [http://www.fastercures.org/assets/Uploads/PPVF-Version-1.0-Methodology-Report-Final.pdf](http://www.fastercures.org/assets/Uploads/PPVF-Version-1.0-Methodology-Report-Final.pdf) [Accessed 1 Aug 2018].
    
    

21. 21.INVOLVE 2018a. National standards for public involvement. Available: [http://www.invo.org.uk/wp-content/uploads/2018/06/Public\_Involvement\_Standards\_v1.pdf](http://www.invo.org.uk/wp-content/uploads/2018/06/Public_Involvement_Standards_v1.pdf) [Accessed 21 June 2018].
    
    

22. 22.INVOLVE 2018b. Guidance on co-producing a research project (involve). Available: [http://www.invo.org.uk/wp-content/uploads/2018/03/Copro\_Guidance_Mar18.pdf](http://www.invo.org.uk/wp-content/uploads/2018/03/Copro_Guidance_Mar18.pdf) [Accessed 21 Jun 2018].
    
    

23. 23.INVOLVE. Public involvement in research: values and principles framework (October 2015), 2015. Available: [http://www.invo.org.uk/wp-content/uploads/2017/08/Values-Principles-framework-Jan2016.pdf](http://www.invo.org.uk/wp-content/uploads/2017/08/Values-Principles-framework-Jan2016.pdf) [Accessed 2 Jul 2018].
    
    

24. 24.INVOLVE, 2013. Exploring the impact of public involvement on the quality of research: examples, Eastleigh. Available: [http://www.invo.org.uk/wp-content/uploads/2014/11/INVOLVEExploringImpactQualityResearchExamples\_updated-July-2014.pdf](http://www.invo.org.uk/wp-content/uploads/2014/11/INVOLVEExploringImpactQualityResearchExamples_updated-July-2014.pdf) [Accessed 29 Jun 2018].
    
    

25. 25. Pushparajah DS . Making patient engagement a reality. Patient 2018;11:1–8.[doi:10.1007/s40271-017-0264-6](http://dx.doi.org/10.1007/s40271-017-0264-6) 
    
    

26. 26. Greene J , Hibbard JH , Sacks R , et al . When patient activation levels change, health outcomes and costs change, too. Health Aff 2015;34:431–7.[doi:10.1377/hlthaff.2014.0452](http://dx.doi.org/10.1377/hlthaff.2014.0452) 
    
    [Abstract/FREE Full Text](http://innovations.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjg6IjM0LzMvNDMxIjtzOjQ6ImF0b20iO3M6MjE6Ii9ibWppbm5vdi81LzEvNDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

27. 27. Tallon D , Chard J , Dieppe P . Relation between agendas of the research community and the research consumer. Lancet 2000;355:2037–40.[doi:10.1016/S0140-6736(00)02351-5](http://dx.doi.org/10.1016/S0140-6736(00)02351-5) 
    
    [CrossRef](http://innovations.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(00)02351-5&link_type=DOI) 
    
    [PubMed](http://innovations.bmj.com/lookup/external-ref?access_num=10885355&link_type=MED&atom=%2Fbmjinnov%2F5%2F1%2F43.atom) 
    
    [Web of Science](http://innovations.bmj.com/lookup/external-ref?access_num=000087519400012&link_type=ISI) 

28. 28. Crowe S , Fenton M , Hall M , et al . Patients’, clinicians’ and the research communities’ priorities for treatment research: there is an important mismatch. Res Involv Engagem 2015;25:1–2.
    
    

29. 29. Ocloo J , Matthews R . From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ Qual Saf 2016;25:626–32.[doi:10.1136/bmjqs-2015-004839](http://dx.doi.org/10.1136/bmjqs-2015-004839) 
    
    [Abstract/FREE Full Text](http://innovations.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI1LzgvNjI2IjtzOjQ6ImF0b20iO3M6MjE6Ii9ibWppbm5vdi81LzEvNDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

30. 30. Supple D , Roberts A , Hudson V , et al . From tokenism to meaningful engagement: best practices in patient involvement in an EU project. Res Involv Engagem 2015;25:1–5.
    
    

31. 31.FDA, 2017. Enhancing FDA’s approach to patient engagement current state analysis and recommendations. Available: [https://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/UCM589533.pdf](https://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/UCM589533.pdf) [Accessed 8 Jul 2018].
    
    

32. 32.EMA, 2017. Guidance for parallel consultation. Available: [http://www.ema.europa.eu/docs/en\_GB/document\_library/Regulatory\_and\_procedural\_guideline/2017/07/WC500230375.pdf](http://www.ema.europa.eu/docs/en\_GB/document\_library/Regulatory_and_procedural_guideline/2017/07/WC500230375.pdf) [Accessed 8 Jul 2018].
    
    

33. 33.CTTI (Clinical Trials Transformation Initiative), 2015. CTTI recommendations: effective engagement with patient groups around clinical trials. Available: [https://www.ctti-clinicaltrials.org/files/pgctrecs.pdf](https://www.ctti-clinicaltrials.org/files/pgctrecs.pdf) [Accessed 21 Jun 2018].
    
    

34. 34.M-CERSI, 2015. Assessing meaningful patient engagement in drug development: a definition, framework, and rubric. Available: [http://www.pharmacy.umaryland.edu/media/SOP/wwwpharmacyumarylandedu/centers/cersievents/pfdd/mcersi-pfdd-framework-rubric.pdf](http://www.pharmacy.umaryland.edu/media/SOP/wwwpharmacyumarylandedu/centers/cersievents/pfdd/mcersi-pfdd-framework-rubric.pdf) [Accessed June 21, 2018].
    
    

35. 35.NHC/Genetic Alliance, 2015. Dialogue / advancing meaningful patient engagement in research, development, and review of drugs. Available: [https://www.nationalhealthcouncil.org/sites/default/files/PatientEngagement-WhitePaper.pdf](https://www.nationalhealthcouncil.org/sites/default/files/PatientEngagement-WhitePaper.pdf) [Accessed 21 Jun 2018].
    
    

36. 36.PCORI 2014a. PCORI engagement rubric for applicants (updated June 6, 2016). Available: [https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf](https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf) [Accessed 21 Jun 2018].
    
    

37. 37.PCORI 2014b. PCORI funded projects: sample engagement plans from methods portfolio. Available: [https://www.pcori.org/sites/default/files/PCORI-Sample-Methods-Engagement-Plans.pdf](https://www.pcori.org/sites/default/files/PCORI-Sample-Methods-Engagement-Plans.pdf) [Accessed 21 Jun 2018].
    
    

38. 38. Dubois RW , Lauer M , Perfetto E . When is evidence sufficient for decision-making? a framework for understanding the pace of evidence adoption. J Comp Eff Res 2013;2:383–91.[doi:10.2217/cer.13.39](http://dx.doi.org/10.2217/cer.13.39) 
    
    [PubMed](http://innovations.bmj.com/lookup/external-ref?access_num=24236680&link_type=MED&atom=%2Fbmjinnov%2F5%2F1%2F43.atom) 

39. 39.FasterCures, 2016. Integrating the patient perspective into the development of value frameworks. Available: [http://www.fastercures.org/reports/view/56](http://www.fastercures.org/reports/view/56) [Accessed 21 Jun 2018].
    
    

40. 32. Gregory J . Conceptualising consumer engagement: a review of the literature. Melbourne Australia: Australian Institute of Health Policy Studies, 2007.
    
    

41. 41. Greenhalgh T , Snow R , Ryan S , et al . Six ‘biases’ against patients and carers in evidence-based medicine. BMC Med 2015;13.[doi:10.1186/s12916-015-0437-x](http://dx.doi.org/10.1186/s12916-015-0437-x) 
    
    

42. 42.Deane KHO for the Access All Areas Team. Making meetings accessible. Norwich: University of East Anglia, 2018.
    
    

43. 43. Burns E . Making sure your research priorities are built on diversity. James Lind Alliance, 2017.
    
    

44. 44.All Trials, 2013. All trials registered. All results reported. Available: [http://www.alltrials.net//wp-content/uploads/2013/09/What-does-all-trials-registered-and-reported-mean.pdf](http://www.alltrials.net//wp-content/uploads/2013/09/What-does-all-trials-registered-and-reported-mean.pdf) [Accessed 7 Jun 2018].
    
    

45. 45.World Medical Association. Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 2013;310:2191–4.[doi:10.1001/jama.2013.281053](http://dx.doi.org/10.1001/jama.2013.281053) 
    
    [CrossRef](http://innovations.bmj.com/lookup/external-ref?access_num=10.1001/jama.2013.281053&link_type=DOI) 
    
    [PubMed](http://innovations.bmj.com/lookup/external-ref?access_num=24141714&link_type=MED&atom=%2Fbmjinnov%2F5%2F1%2F43.atom) 
    
    [Web of Science](http://innovations.bmj.com/lookup/external-ref?access_num=000327404400028&link_type=ISI) 

46. 46. Oliver S , Clarke-Jones L , Rees R , et al . Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach. Health Technol Assess 2004;8:1–148.[doi:10.3310/hta8150](http://dx.doi.org/10.3310/hta8150) 
    
    [PubMed](http://innovations.bmj.com/lookup/external-ref?access_num=15080866&link_type=MED&atom=%2Fbmjinnov%2F5%2F1%2F43.atom) 
    
    [Web of Science](http://innovations.bmj.com/lookup/external-ref?access_num=000224870900001&link_type=ISI)