Statistics from Altmetric.com
The role of feedback in medical education is well established. Good quality feedback drives learning for doctors-in-training and helps those supervising their training to develop as teachers and leaders.1
Previously, feedback on medical training in the UK has been collected annually by the General Medical Council (GMC)2 and Health Education England (HEE). The changes implemented as a result of the feedback are usually made after the cohort of doctors providing it have rotated to an alternative post and therefore they often do not benefit from these changes.
Individual trainers providing supervision to doctors-in-training do not currently receive specific feedback via this process. Negative feedback from doctors-in-training usually triggers visits to the relevant hospital/departments from the GMC or the Local Education and Training Boards (LETB).
Increasingly, both the GMC and LETBs are requesting evidence from hospital trusts on the quality of the educational and clinical supervision provided to trainees in different departments.3 Likewise, doctors providing educational supervision to doctors-in-training are increasingly required to provide evidence of their involvement and development in medical education.3 This can be difficult to achieve without significant effort.4 These issues led to the development of Healthcare Supervision Logbook Smartphone App at Sheffield Teaching Hospitals, UK.
Healthcare Supervision Logbook
Healthcare Supervision Logbook (HSL) is a Smartphone App and desktop content management system (CMS) developed at Sheffield Teaching Hospitals NHS Foundation Trust, UK with support from Health Education Yorkshire and Humber, UK.5 There is a version for trainees and a version for supervisors (figure 1).
HSL was originally developed in obstetrics and gynaecology, but the content of the App is fully modifiable and can be adapted for any specialty/discipline.
HSL allows doctors-in-training to provide feedback on their perceived value of the educational content, including the educational or clinical supervision provided by a supervisor, after each session they attend5 (figure 2). This means that at the end of each clinical session (theatres, clinic, on-call, etc) doctors-in-training use HSL App on their smartphone/tablet to complete a short form about the educational value of the session. This will include details about the activities completed during the session, questions about whether undermining has occurred in the session and ratings on a 10-point scale for their perceived educational value of the session and perceived quality of supervision for the session. This data is currently collected anonymously (ie, the trainee does not name the supervisor when recording feedback) and individual supervisors cannot review the data provided by the trainee. Therefore, there are no concerns with trainees being adversely affected as a result of providing negative feedback.
The data generated from trainees feedback can only be reviewed by senior doctors organising the training programme via the content management system. This data can be used to identify training issues in a continuous manner, as doctors-in-training provide feedback daily, on a session-by-session basis.
HSL also allows supervisors to provide feedback on the performance of doctor-in-training on a session-by-session basis.5 At the end of each session, the supervisor uses HSL to complete a short form about the trainee's performance during the session. This data is collated on the CMS and a report can be created regarding the performance of an individual trainee. This works by combining the reports from each supervisor to create an enhanced 360° multisource assessment of the trainee. These reports can then be used to inform educational supervision meetings and appraisal for the doctor-in-training.
Supervisors are also able to obtain a record of the feedback they have recorded about trainees, which helps them to provide evidence of their involvement and development in medical education.
Integration of HSL into clinical practice
Following three successful pilot phases, HSL was rolled out into clinical practice within the obstetrics and gynaecology directorate at Sheffield Teaching Hospitals from January 2015. This project was registered as a service evaluation with the clinical effectiveness unit at Sheffield Teaching Hospitals and ethical approval was not required. Immediately prior to the rollout of HSL, a survey to assess current perceptions regarding feedback on training was undertaken of all doctors in specialty-training posts in the South and West regions of Health Education Yorkshire and Humber, UK (n=3026). The response rate was 16.5% (n=499). When asked how frequently they were currently able to provide formal feedback about the quality of the educational and clinical supervision they received as part of their training, 65.6% said annually and 21% said monthly. Asked if they felt that they had enough opportunities to provide feedback on the educational and clinical supervision received as part of their training, 42.2% disagreed or strongly disagreed compared with 34.6% who agreed or strongly agreed with this.6
A launch event was held for HSL in December 2014, which all specialty trainees in obstetrics and gynaecology and their supervisors at Sheffield Teaching Hospitals were invited to attend. HSL was demonstrated and details given of how to download and use the App. Each trainee and supervisor then received an email with this information. Additionally, instructional videos demonstrating how to use the App were produced and made available to all users. A clinical leadership fellow working on the HSL project met with each user individually to ensure that they had been able to download HSL and were able to use it. An email address to contact for technical support was provided and the clinical leadership fellow answered any queries and provided support.
Overcoming barriers to using HSL
As with any new tool, there were inevitably some issues with integrating HSL into clinical practice. The main issue was concerns from users about the time needed to use HSL, in addition to their clinical duties and other clerical tasks. Using HSL to provide feedback after attending a session, whether to provide feedback on the session as a doctor-in-training or feedback on a trainee as a supervisor, takes approximately 45–55 s—including entering a PIN number to access HSL. Once this was demonstrated to users, particularly via the video tutorials and through face-to-face support, concerns about extra time needed was no longer a concern for the majority of users.
Other concerns included the need for Internet access to use HSL and the data requirements that the App would have. As Internet access was only required for the initial log on and thereafter HSL could be used offline, this did not turn out to be an issue as users could continue to record feedback in areas of the hospital where phone signal and internet access were unavailable. The App only uses a few KBs of data to run, so did not have a significant impact on users data supply on their personal Smartphones.
The most difficult part of the roll out into clinical practice was prompting users to use HSL in a timely manner to provide feedback after each session they attended. Eventually this was overcome by emailing regular reminders to users and through face-to-face support from the clinical leadership fellow.
At 6 months after introducing HSL all the core specialty trainees (n=22) in obstetrics and gynaecology were using the App to provide feedback on their training.
During the first 6 months of the roll out, feedback was recorded by trainees for 136 clinic sessions (41% of all clinics held with a supervised trainee present), 88 theatre sessions (61% of all theatre sessions attended by a trainee) and 127 on-call sessions (76% of supervised daytime on-call sessions). This feedback helped to highlight issues with training in the obstetrics and gynaecology department; specifically infrequent completion of work-based assessments during on-call sessions, lack of opportunities to operate under supervision and provision of face-to-face feedback to trainees after theatre sessions, particular clinic sessions which frequently finished late and an overall lack of direct observation in clinic sessions. Equally, the evidence collected by HSL has helped to support training too; demonstrating areas where this is occurring effectively as well as highlighting these areas of deficiency. A number of these areas have subsequently been addressed; for example, trainees were allocated to theatre sessions more likely to provide them with opportunities to operate and scrub nurse practitioners were allocated to theatre lists where training was not provided.
Feedback was provided by supervisors using HSL about the performance 21 of the 22 specialty trainees. There are future plans to use this data alongside the current workplace assessments to enhance the annual review of trainees, but this is an area which requires further evaluation.
It would be simple to present the data recorded by trainees and supervisors using HSL in the event of a GMC/LETB visit, which would be able to demonstrate regular feedback about the quality of the training occurring in the department.
Evaluation of HSL
At 6 months, all of the trainees using the App (n=22) completed an evaluation form with identical questions to the survey completed prior to the roll out of HSL. This was registered as a service evaluation and ethical approval was not required.
When asked how frequently they were currently able to provide formal feedback about the quality of the educational and clinical supervision they received as part of their training, 82% said daily, 14% said weekly and 6% monthly. Asked if they felt that they had enough opportunities to provide feedback on the educational and clinical supervision received as part of their training, 95% agreed or strongly agreed with this. The results showed a significant improvement (see table 1), with trainees able to provide feedback more frequently and feeling that they had adequate opportunities to do this compared with before. Eighty-two per cent of trainees agreed or strongly agreed that they found HSL's functions easy to use and 91% agreed or strongly agreed that they would be willing to continue using HSL in the future.
The future for HSL
HSL will continue to be used in the obstetrics and gynaecology directorate at Sheffield Teaching Hospitals (STH). From August 2015 it has been rolled out within obstetrics and gynaecology to the four other hospitals in the South locality of Health Education Yorkshire and Humber with all obstetrics and gynaecology specialty trainees (n=51) using HSL.
HSL has so far been adapted for use in anaesthetics, midwifery, neurosurgery and Foundation Training. From August 2015 the anaesthetics department at Sheffield Teaching Hospitals NHS Foundation Trust has been using HSL for all of their specialty trainees (n=53) following a successful pilot phase in June and July 2015. Adaption of HSL for use in other directorates in Sheffield Teaching Hospitals is continuing with further departmental rollouts planned for late 2015.
The authors would like to acknowledge Pipe and Piper—mobile and cloud software specialists of Sheffield, UK for their role in software engineering Healthcare Supervision Logbook.
Twitter Follow Thomas Gray at @ThomasGray007
Contributors Healthcare Supervision Logbook was designed and developed by TF and TG with support from GH. The App was developed at Sheffield Teaching Hospitals NHS Foundation Trust, UK, with support from Health Education Yorkshire and Humber, UK. Software engineering was undertaken by Pipe and Piper, Sheffield, UK. All authors reviewed a final copy of this manuscript.
Funding Sheffield Teaching Hospitals NHS Foundation Trust.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.