Article Text
PDF
Abstract
Background Feedback drives learning in medical education. More regular multisource feedback would improve appraisal for doctors-in-training in the UK. Trainers currently receive little feedback on supervision they provide to doctors-in-training. Barriers against providing feedback include shortage of time and lack of clear mechanisms to do so.
Methods A literature review was undertaken to investigate whether a Smartphone App to provide feedback on training was a new concept. Different training environments were identified and feedback proformas for each of these areas developed for trainers and trainees. A pre-existing logbook App identified during literature searching was modified with permission and pilot phases completed. Professional software engineering was subsequently undertaken.
Results The App developed is Healthcare Supervision Logbook. Trainers can use it for providing feedback about a trainee's performance in a clinical session and trainees can use it for providing feedback about their perceived value of the educational content of a clinical session, including the educational and/or clinical supervision they have received. Healthcare Supervision Logbook also contains a comprehensive logbook of practical procedures for all doctors using the App. It can also be used to collect patient and colleague feedback.
Conclusions Healthcare Supervision Logbook is a new concept. It allows trainers to provide daily feedback about a trainee's performance which can be used to facilitate appraisal and highlight areas for development. Feedback provided by the trainees can be used similarly to identify patterns and improve the quality of training provided. The feedback and logbook content is useful for purposes of appraisal and revalidation.
- Medical Apps
- mHealth
- Accessible
Statistics from Altmetric.com
Introduction
It is clear that receiving high-quality feedback is important in driving learning and development for doctors-in-training.1 Likewise, those providing educational and clinical supervision require feedback on their performance as trainers in order to develop into highly effective teachers and leaders. Feedback in an education context means getting advice about a performance that then allows the recipient to reflect, improve and develop.2 This is relevant for both doctors-in-training and their trainers.
Feedback to doctors-in-training is provided in a number of ways, including face-to-face, written feedback and work-based assessment (WBA) tools3 ,4 and 360° multisource feedback. These methods all require a reasonable outlay of time and the WBAs require the completion of either a written or online form. It is clear that a shortage of time contributes to the lack of feedback that doctors-in-training often receive about their performance;5 this has been exacerbated by the European Work Time Directive's impact on junior doctors’ working hours in the UK.6
Doctors-in-training in the UK complete an annual feedback form about their training circulated by the doctors’ regulatory body The General Medical Council,7 and from 2014 trainers will also complete a formal annual feedback survey.8 Health Education England (HEE) regional authorities are responsible for the quality assurance and delivery of educational and clinical supervision for doctors-in-training. Local feedback from doctors-in-training and their trainers is also collected on an annual basis. In addition, Royal Colleges for different specialties co-ordinate the structure of training for doctors and are responsible for delivering training for doctors providing educational and clinical supervision; often this has been disorganised and the quality of educational supervision has been inconsistent.9 A clear need is present for a tool which enables trainees to provide more frequent feedback assessing the educational content and level of supervision given for their training. A similar tool is also required for trainers, who currently receive very little feedback on the quality of the educational and clinical supervision they provide.
A Smartphone is a mobile phone with enhanced computer processing capability and connectivity. Most devices have a touch screen technology, Internet access and can run custom-built software applications or ‘Apps’. The UK Department of Health has stated that “…Smartphones can enhance patient care by providing opportunities for students, trainees and staff to accumulate essential knowledge, skills, values and behaviours.”10
The aim of this research project was to develop a novel Smartphone App which could be used by doctors-in-training to provide feedback about their clinical training and supervision, and for trainers to provide feedback about the trainees’ performance. This App would be simple to use and very quick to access, meaning that synergistic feedback between doctors-in-training and their trainers could easily be collected on a day-to-day basis after every clinical session. This would overcome the problems of placement feedback being collected infrequently, often at the end of the placement. This feedback could then be used to identify patterns of concern or good practice and areas for development, on a much more regular basis, improving on the current annual review process and providing a much more accurate assessment of training. Ultimately, the aim of this project is to improve patient care and promote excellent healthcare for all11 through better medical training.
Methods
A systematic literature review was first conducted. MEDLINE, PubMed and Web of Science databases were searched for English language, peer-reviewed publications from August 1983 to August 2013. Prior to writing this article, this search was updated to include new publications until August 2014.
The search terms “Smartphone”, “mobile phone”, “cell phone”, “cellular phone”, “hand-held computer”, “PDA” and “Personal Digital Assistant” were used and in addition were each individually combined using AND with “education”, “medical education”, “feedback”, “training” and “applications”. All titles and abstracts generated were read to identify those of relevance to the project.
Online App stores: The Apple App Store,12 Google Play Store,13 Windows Store,14 Blackberry World15 and SamsungApps16 were all searched for Apps for healthcare professionals. The results of the literature search were used to confirm that the product we were planning to develop would indeed be novel.
The initial plan was to software engineer the App ourselves using free software and then ensure its functionality before making the outlay of professional software engineering. A medical logbook App (used to record practical procedures performed in different medical specialties), identified during literature review, had a function for changing the fields on the logbook's input form. This function could be used to adapt this App to work as a feedback tool for doctors-in-training and their trainers. The owner of the logbook App agreed that we could use this function to develop a prototype feedback App.
Different training environments were identified and feedback proformas for each of these training areas were developed for both trainers and trainees. The proformas were then software-engineered into the existing logbook App. Three pilot phases enabled the feedback App to be refined with regard to content and layout. Feedback on the trainee's perceptions of the App was also evaluated.
Sheffield Teaching Hospitals NHS Foundation Trust then funded the formal professional software engineering of the App. The App, its supporting desktop program and cloud system are now operational, and it is shortly to be rolled out, initially within the Obstetrics and Gynaecology Department of the Sheffield Teaching Hospitals NHS Foundation Trust.
Results
Literature review
The literature review results showed that a Smartphone App to provide feedback to doctors-in-training and their trainers is an entirely novel entity. The literature search also identified several well-designed systematic reviews of current medical Apps available,17–19 however, the use of a mobile technology to provide feedback on training was a new concept.
A small study evaluating the use of a PDA-based WBA3 tool for medical students was identified,20 but this was not the focus of the feedback App, as WBAs can already be completed on Smartphones.
Reassuringly, the literature search confirmed that usage levels of Smartphones among doctors are high18 ,21 ,22 and that acceptability of the use of Smartphone Apps in the workplace is also high.21
Pilot
The evaluation forms completed by the trainees provided positive feedback about the use of a Smartphone App to provide day-to-day feedback on perceptions of educational and clinical supervision (table 1). The trainees questioned the value of the App for reporting undermining, but felt confident that it would be a useful tool for recording evidence of training activities.
Twelve trainees using the prototype version of Healthcare Supervision Logbook completed a feedback form
System and security
The App developed is called Healthcare Supervision Logbook (figure 1).
A screenshot showing the home screen for Healthcare Supervision Logbook Smartphone App.
Healthcare Supervision Logbook works on the Apple iOS and Google Android operating systems with data automatically uploaded to a secure cloud-based system via the Internet. ‘Safe’ data can be accessed from the user's Smartphone or the desktop version, which has an advanced content management system so users can be added and managed by Trust educational leads supported by the IT department. Varying degrees of access to different components and data can be controlled via this system. The App is secure and users require a username and Personal Identification Number (PIN) to log on (figure 1).
Menus
Healthcare Supervision Logbook exists in two different versions—one for trainers and one for trainees. Trainers have access to five menu items: Logbook, Sessions, Training, Patient and Archive (figure 2). Trainees have access to four menu items: Logbook, Sessions, Patient and Archive.
A screenshot showing the trainer's main menu on Healthcare Supervision Logbook.
Logbook
The Logbook section of the App has been designed to be specialty specific and modifiable. The system is currently mapped for obstetrics and gynaecology (figure 3). Details recorded in the Logbook section include the time and date of the procedure selected, the level of supervision, the details of complications and an optional free-text comment.
A screenshot showing an example of the logbook function of Healthcare Supervision Logbook.
Sessions
The Sessions section of the menu can be completed by both trainers and trainees. There are six session options: ‘clinic session’, ‘on-call session’, ‘theatre session’, ‘teaching session-L’ (as a learner), ‘teaching session-T’ (as a teacher) and ‘incident’. Each of these menu items has a tailored feedback form and comprises the feedback component of Healthcare Supervision Logbook.
For each individual session, the level of supervision provided in each session (eg, consultant, specialty trainee grade, etc) is recorded. All ‘Session’ feedback elements of the App include whether or not the session started and ended on time, a global rating for the session on a 5-point scale and an optional free-text comment.
The ‘Clinic session’ feedback captures items such as the number of new and follow-up patients seen, whether there was direct observation of their practice in the clinic and whether they received any feedback or were offered WBAs.
The ‘Theatre session’ feedback includes the number of minor and major cases, the level of supervision received, whether the training session was compromised (eg, due to time pressures or complications), whether there was an opportunity to review the patients post-op and whether WBAs were offered or asked for.
The ‘Teaching session’ includes the topic, teacher (or learners) and type of teaching (eg, small group, lecture, bedside, etc) in the context of either being the teacher or the learner.
The ‘On-call session’ includes feedback about satisfaction with the clinical supervision provided during the session, details of complications occurring during the on-call session and whether WBAs were completed or requested.
For each session, trainees are asked to record whether they had experienced any undermining during the session and, if so, whether it has been reported. If undermining is recorded, but not logged as reported, the trainee is automatically emailed a copy of the local undermining policy highlighting mechanisms for formal reporting.
The ‘incident’ form in the ‘sessions’ section is used to record involvement in clinical incidents. This includes free-text details of the incident, whether it was reported and whether reflective practice has been completed.
Each feedback log completed is uploaded into the personal ‘archive’ section and onto the content management system where review can be undertaken by the doctors co-ordinating the training programme.
Training (Sessional educational feedback)
This component of Healthcare Supervision Logbook is only available on the trainers’ version of the App and enables the trainer to provide feedback on the trainee's performance in a clinical session. The trainer selects the relevant trainee from the menu, and feedback is then provided for the appropriate clinical area (‘clinic’ session, ‘theatre’ session or ‘on-call’ session). The trainer's perspective of the trainee's performance is collected on items including empathy/respect, team working, verbal communication skills and clinical judgement (figure 4). The data are automatically uploaded onto the cloud system, but are not automatically visible to the trainee. This component of Healthcare Supervision Logbook effectively collects daily multisource (360°) feedback about each trainee. This produces a large amount of data, which can be analysed at a later date to provide a summary to facilitate formal educational supervision meetings and inform the annual review of training.
An extended screenshot from Healthcare Supervision Logbook showing an example of the sessional feedback form completed by a trainer regarding a trainee's performance in a theatre session.
Patient
The patient component of Healthcare Supervision Logbook is available for both trainers and trainees. An evaluation questionnaire to obtain patient feedback on your performance as a doctor can be delivered electronically to a patient (with their permission). Once the feedback form has been completed, it is uploaded to the archive section of Healthcare Supervision Logbook.
Archive
The archive section of Healthcare Supervision Logbook has three sections—log, sessions and patient. Within these sections are all of the entries made; they can be ordered by date or title and can be searched and scrolled back through.
Discussion
It is clear that using a Smartphone App to provide feedback on training, from both the perspective of the doctor-in-training and their trainer, is a novel entity.
Apple iOS and Google Android are by far the two most popular and best-established operating systems for Smartphones; therefore, Healthcare Supervision Logbook has been engineered for these platforms only. Trainees and trainers will use the App on their own personal device, making it possible to integrate the App into frequent everyday use.
It is important that any computer system being used in the workplace has been through an appropriate IT governance process, particularly with regard to the safe storage of any confidential data. Before Healthcare Supervision Logbook is rolled out across the Trust, this process will be completed. There is currently no clear governance process for the use of Smartphone Apps in the workplace. A governance review process with “internal review at institutional level, with subsequent external review by experts in the field” has been suggested by the literature.23
Healthcare Supervision Logbook is able to act as a comprehensive record of training experiences and can be used to identify trends and patterns of behaviour of doctors-in-training and their trainers on a day-to-day, session-by-session basis. This provides a vastly more accurate picture of perceived training quality than current processes allow.
The App is quick to use and, as it is accessed on the trainee's or trainer's personal device, it will be possible to complete feedback immediately after a session. This will allow for more accurate and timely reflection than current processes.
It is essential that users of Healthcare Supervision Logbook feel free to honestly assess the clinical session without fear of consequences. This applies to both trainees and trainers. Trainees are used to being continuously reviewed as part of their annual specialty training requirements and, although it is envisaged that this form of assessment will occur much more frequently with Healthcare Supervision Logbook, the principle is the same. This concept, however, will be new to most trainers who may feel uncomfortable at having the quality of the educational and clinical supervision they provide assessed by a trainee in such a regular manner. In order to promote the initial use of Healthcare Supervision Logbook, the decision has been made for the trainee feedback of the supervision provided to be anonymised at present. This will not detract from its value as departments will still be able to provide reports on perceived levels of educational and clinical supervision provided to trainees by different levels of seniority across different clinical sessions (clinic, theatre, on-call). Once integrated into day-to-day training, it is anticipated that the trainer would be named on this sessional quality assessment process completed by the trainee.
The way that Healthcare Supervision Logbook will be used in practice is as follows. After each session (clinic, theatre, on-call, etc), both the trainer and the trainee will separately complete a feedback form. The completed feedback form will act as a record of the trainee's clinical activities and provide a record of the trainer's/trainee's perceived educational value of that particular clinical session.
The doctors managing the training programme will have access to the content management system, which would allow triangulation of the data for both doctors-in-training and their trainers. For instance, all of the different trainees’ feedback forms with regard to a specific session can be reviewed. This will provide an accurate long-term assessment of the training that is occurring. For example, with regard to a theatre training session, if there is a persistent problem with the session being compromised due to time pressures, this can be clearly evidenced from the trainee feedback and then addressed if it is shown to be a problem. Similarly, all trainers’ logs for a particular trainee can be reviewed to identify trends regarding items such as team working and verbal communication skills with much greater frequency and accuracy than the current annual multisource feedback. This allows for patterns in training and in clinical supervision to be easily identified and problems addressed. Trainees in difficulty can be quickly highlighted and provided with help and support.
Healthcare Supervision Logbook also highlights good practice for both trainers and trainees, providing robust evidence that trainers are providing good-quality clinical supervision and that the trainee's work is of high quality. These data will prove very useful for purposes of revalidation and appraisal.
The logbook function of Healthcare Supervision Logbook is useful for all doctors using the App. Both trainers and trainees perform practical or specialty-specific procedures regularly. The App provides a simple way of keeping a very accurate daily record of the practical procedures undertaken.
Undermining or bullying can be a problem for trainees: in the 2013 GMC National Training Survey, 26.5% of doctors-in-training reported experiencing undermining from a senior colleague.24 Healthcare Supervision Logbook provides a novel, simple, non-confrontational way for undermining to be logged. Any trainee session feedback which records undermining which has not been reported will automatically trigger an email to be sent to the trainee who is reporting the behaviour. This will include a copy of the Trust's undermining policy. It is hoped that the App will allow for persistent patterns of behaviour to be demonstrated over time, allowing issues around undermining to be addressed quickly, for example, a trainee who is persistently denied the opportunity to operate in a particular theatre session.
Another novel function of Healthcare Supervision Logbook is the ability to collect feedback from patients. The NHS Patient Feedback Challenge25 has had much publicity and feedback from patients is becoming an important part of revalidation and appraisal. Healthcare Supervision Logbook can be used to send a questionnaire survey on the doctor's performance to a patient's email address (figure 5). When it is completed it is uploaded into the desktop content management system and the personal archive on the App. It can be viewed by the doctor who has sent it and used for revalidation and appraisal. This function can also be used to collect feedback from colleagues to provide evidence of the 360° colleague feedback for purposes of appraisal and revalidation.
A screenshot of the patient feedback system in Healthcare Supervision Logbook.
Healthcare Supervision Logbook can be adapted to work in other disciplines such as nursing and midwifery. The principles of the trainer and the trainee providing synergistic feedback on a day-to-day basis are applicable to any scenario where training is being given. It would also be highly feasible to use the App in low-resource settings, as Smartphone ownership and internet access is widespread across both developed and developing countries.26 Healthcare Supervision Logbook would provide a model of documented clinical supervision in settings where medical training processes are less rigorous and well defined.
Healthcare Supervision Logbook collects feedback on training activities from both trainers’ and trainees' perspectives; it also acts as a comprehensive logbook of practical procedures and as a tool for collecting patient and colleague feedback. The data that it generates can be used to identify trends in training on a session-by-session basis; these data can be used to optimise training experiences for doctors-in-training. New technology is often a source of anxiety and this App needs to be introduced with good support and sensitivity. If used as intended, it will promote excellent clinical practice through improved training and clinical supervision. It will also be valuable for providing evidence at appraisal and revalidation.
Healthcare Supervision Logbook does not replace high-quality face-to-face feedback, which is of huge importance in training, but it does promote a culture where day-to-day feedback on training is part of the everyday routine. Ultimately, this will be reflected in improved patient care as a result of improved training.
Acknowledgments
The authors would like to acknowledge Dr Raj Burgul for permitting us to use the Universal Logbook App as the initial software platform for Healthcare Supervision Logbook during the pilot phase of the project.
Footnotes
Contributors TGG and TF designed the concept and content of Healthcare Supervision Logbook and organised software engineering and piloting. GH provided educational leadership advice on the content. All authors have reviewed a final copy of this manuscript.
Funding The professional software engineering of Healthcare Supervision Logbook was funded by Sheffield Teaching Hospitals NHS Foundation Trust.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.