Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
What are the new findings?
The literature suggests that augmented reality (AR) is a feasible and engaging platform for foundational learning events, particularly where the visualisation and recognition of devices, medications and tools are the objectives.
This study found that mobile-based AR training can provide familiarity with low-frequency and high-risk equipment and its location, to optimise speed and efficiency.
How might it impact on healthcare in the future?
Educators may wish to leverage mobility-based AR for foundational learning which may ‘layer’ experience and create efficiencies for learners.
Use of ‘bring your own device’ asynchronous training has the advantages of time, place and learner readiness often preferred by adult learners.
Paediatric resuscitation is recognised as a low-volume, high-stress environment for even the most experienced practitioners. Stress can be amplified if rescue equipment is not readily available. Time is critical and delays in the delivery of medications have been shown to decrease the odds of survival in children and adults.1 While most providers working in healthcare areas are certified to provide life support, knowledge of the resuscitation cart contents and timely access to the right equipment are challenging. These factors may lead to delays in providing optimal patient care and compromise patient safety.2
Simulation is one way to identify factors that potentially affect patient safety,3 and our experience with simulation at our institution has demonstrated that many providers do not feel comfortable accessing the code cart and locating items within it. The Joint Commission has emphasised the development of standardised staff education of cart contents and drawer organisation for patient safety.4 There is evidence that weight-based dosing carts are superior in paediatrics.5 6 The use of weight-based dosing carts is endorsed by the American Heart Association.7 8 However, there is neither a standard for training on cart contents nor is there standardisation on how the paediatric code cart is stocked, with variation between and within hospital systems. Current constraints include the lack of stocked carts available for exploration, time for educators to review carts repetitively for diverse learner groups, irregularity of real resuscitation efforts and expense and safety concerns prohibiting use of a functional cart for training.
A paediatric code cart augmented reality (AR) application was developed to overcome these constraints. AR is a technology that has been increasingly used in medical education.9 10 The premise of AR is that it overlays interactive digital elements onto real-world environments. The AR code cart application aimed to familiarise practitioners with the contents of the weight-based code cart and its use. Our hospital system recently standardised the paediatric code cart contents across all facilities, making such a similarly standardised approach to education highly desirable. The objective of the application was twofold: First, to provide access to the contents of a weight-based paediatric code cart in an asynchronous, self-directed manner. Second, to create a standard educational prototype of a weight-based code cart using an innovative technology to prevent costly restocking and recommissioning of code carts.
We opted to use this technology for a number of reasons. AR leverages the nearly ubiquitous personal smartphone carried by adult professional learners and permits learner assessments through gamified educational experiences. This avoids the constraints of fixed-time classes and fixed resources such as training equipment and educators. Additionally, it is available entirely at the learner’s discretion and therefore may be viewed as less of a burden than other, more traditional training methods.
To determine the effectiveness of this approach, participants from multiple cohorts within our institution were asked to identify and retrieve items from a real code cart both before and after exposure to the AR application. Speed and accuracy were assessed and compared between the preassessment and postassessment. We hypothesised that accessibility and repetitive exposure to the paediatric code cart using AR would improve accuracy and time to retrieve the items in the actual code cart.
Application description and development
In the application, the code cart and curricula appear within the smart device screen as a three-dimensional form. The learner identifies a location on a nearby surface on which the code cart appears and can then open and close drawers, walk entirely around the instantiated object and manipulate objects within by tapping or swiping on their device. The application design included three modes (see figure 1 for application screenshots):
Free explore: This mode allows the learner to explore the code cart at their own pace. They can reveal tool tips or other cues as drawers are opened or objects are manipulated by tapping on the mobile device screen.
Timed search: This mode leverages a gaming mechanic whereby learners are asked to retrieve specified items from the cart by tapping on them. Their list retrieval time and accuracy is recorded and can be compared with other learners.
Scenarios: This mode includes a storyline revolving around the care of a particular simulated paediatric patient. For a patient of a certain age and size, the learner demonstrates proficiency by retrieving objects needed in a sequence as if a resuscitation is underway.
Subject matter and technical experts collaborated to develop the application. The initial design was tested by the study team and colleagues and an early version piloted with medical students as a separate study. Further information on the testing and development of the application will be the focus of future publications and related human factors work has been described, including qualitative participant experience.11–13
To study the effectiveness of this application as an adult learning tool, we recruited paediatric and emergency medicine residents, attendings, nurses and nurse educators. Participation was voluntary and there was no incentive. Participants were recruited through email and advertisements to the target groups. Our institution is a children’s hospital within a hospital, and subjects were recruited from all paediatric units including general care, intermediate care, intensive care, haematology/oncology and the pediatric emergency department. We asked participants demographic information including their role and prior experience with life support training and actual paediatric resuscitations. Each participant was asked to find an ordered list of items from an actual paediatric weight-based cart (table 1). The time it took to find each item and whether it was the correct item was recorded. The time was defined as that from being asked to locate the item to placing on top of the cart; an incorrect item found was considered a failed attempt and the participant was not corrected until after completing all items. After this exercise, the participant was given access to the AR application and was instructed to use it as often as desired, no formal training in the application or defined expectations for use was provided. In-application data were tracked for each participant, though these data were not linked to the individual user. Several weeks later, the participant performed the same timed search task with the actual code cart; the exact time of follow-up was dependent on participant availability. We then compared preapplication and postapplication performance.
The metrics used to test application effectiveness were accuracy (per cent of participants correctly locating items) and speed (average time per correctly located items). Univariate analyses of preapplication and postapplication performance differences were conducted using Wilcoxon signed rank tests for overall speed and accuracy (the average of each metric, for all items), as well as for speed per individual item. Proportion tests (χ2 and Fisher’s exact) were also conducted to analyse the univariate effects for accuracy on each individual item. Multivariate regression was performed to determine if any of the demographic variables were associated with performance.
Patient and public involvement
There was no patient or public involvement in this study.
Fifty-six participants were included in the study: 8 attending physicians, 13 nurses, 9 nurse educators and 26 resident physicians. Overall, 64% of participants had prior paediatric advanced life support (PALS) training. Overall, 46% of participants reported prior attendance at an actual paediatric resuscitation.
The number of times each user launched the application was recorded. Overall, 91% (51/56) of users accessed the tutorial feature and 86% (48/56) accessed the free explore feature, which was the most common application mode used.
The average time between preapplication and postapplication assessments was 33.1 days. Table 1 shows the accuracy and speed with which participants located and identified items in the code cart both before and after getting access to the application. Overall, preapplication accuracy was 81.8%. Certain items were more challenging than others both in terms of accuracy and speed, for example, the needle decompression kit was the most difficult item in both metrics. When we compared the preassessment and postassessment, the overall accuracy significantly increased to 93.2%. Overall speed also significantly improved, decreasing from an average of 26.8 s per item to 14.3 s per item. As outlined in the table 1, accuracy and speed significantly improved for specific items as well.
We performed a multivariate analysis to determine if any of the demographic factors significantly affected either preapplication or postapplication performance. For preapplication multivariate analysis, with regard to accuracy, the only significant finding was that having previously taken PALS was correlated with better than average accuracy (p<0.001). With regard to speed, nurses were significantly faster (p=0.0065) and residents were significantly slower (p=0.039) than average. Attendings were marginally significant slower than average as well (p=0.087). Having attended more paediatric resuscitations in the prior year was also associated with better than average speed (p<0.001).
For postapplication multivariate analysis, preapplication performance was included as a covariate. With regard to accuracy, nurses were more accurate than average (p=0.053). Several factors also evidenced marginally significant effects on speed: previous PALS experience (p=0.098), previous paediatric code blue experience (p=0.065) and being a nurse educator (p=0.092) were all associated with faster than average times.
Though it was limited, the application usage data were also included in the postapplication multivariate analysis. The only significant finding with regard to application usage was that more plays on timed search were associated with faster than average times in the postapplication follow-up assessment (p=0.013).
Our results demonstrated improved speed and accuracy to retrieve the code cart contents across various groups of healthcare professionals after one attempt at the activity and exposure to the application. Somewhat surprisingly, the pretest accuracy was high despite participants having varying levels of clinical experience. This supports previous reports that weight-based carts were easier to use even for less experienced providers.5
The years of clinical experience and the role of the healthcare professional were the strongest predictors of performance in the preassessment. Among the four cohorts, the nurses, who are generally more familiar with the contents and medications in the cart, performed the best with the highest accuracy in both assessments. Moreover, participants who had more PALS and resuscitation experience performed better than the rest of the group. This is consistent with other work showing that more recent PALS experience led to improved performance in simulated resuscitation.14 On multivariate analysis looking at application usage, we found that more plays on timed search were associated with increased overall speed in the postapplication follow-up assessment, demonstrating at least partial benefit of using the application for asynchronous, self-directed learning.
We were unable to fully assess application usage and its impact on performance due to the nature of the application itself and our study design. While participants improved from the preassessment to postassessment, this may have simply been due to exposure to the item list and activity in the preassessment, rather than application usage itself. However, this further confirms that activities like this one can improve knowledge of the cart contents and speed to retrieval of critical items, whether using a real cart or a simulated one. While we did not demonstrate that our application itself is a superior method for training compared with other methods, real code carts are not plentiful and costly to restock. Therefore, applications such as the one described here can be an efficient and cost-conscious way to provide this kind of training.
There are limitations in our study that merit attention. The group of participants was small with varying levels of experience. We used a convenience sample for this preliminary study to maximise enrolment; future work could power the study to identify more significant results. The study was conducted at a single institution where weight-based paediatric cart usage is the existing norm and the cart is already standardised. The usage of the application was left to the participants’ discretion; results may have been more significant if increased application usage was required. Neither the time between the preassessment and postassessment was set nor there was a control group that did not have access to the application. Finally, our pretests and post-tests were performed in a controlled environment; therefore, more studies will be required to study the outcomes in real situations.
The development of this application leverages the immersive technology of AR to create a cost-conscious, immediately accessible learner platform to disseminate education surrounding paediatric resuscitation. We found that a training activity and exposure to the application improved participant speed and accuracy in locating items within the paediatric weight-based cart. Future work could focus on determining whether this educational platform impacts learners’ skills, rather than just knowledge, in the real-world clinical environment and whether these gains are retained over the long term.
Data availability statement
Detailed data from this study is available upon reasonable request to the corresponding author.
Patient consent for publication
Contributors TC, KH, NS, HK, MM, TR, SMK, EV-M and GB served as clinical content experts for application development, designed the study and ran the study activity. AW served as human factors expert and participated in study design. RE-A contributed to the design and execution of the study. KF and ZA participated in application design and technical development. DW participated in data analysis. All authors contributed to and reviewed the final manuscript.
Funding Funding for this project was obtained from local institutional grants: a Dean's Award for Innovative Medical Education (University of Illinois College of Medicine at Peoria) and a Jump Applied Research through Community Health through Engineering and Simulation grant. There are no applicable grant award numbers.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.