Article Text

other Versions

PDF

Original article
Wearable technology in an international telementoring setting during surgery: a feasibility study
  1. Henriëtte A W Meijer1,
  2. Juan A Sánchez Margallo2,
  3. Francisco M Sánchez Margallo2,
  4. J Carel Goslings1,
  5. Marlies P Schijven1
  1. 1Department of Surgery, Academic Medical Center, Amsterdam, The Netherlands
  2. 2Jesús Uson Minimally Invasive Surgery Centre, Cáceres, Spain
  1. Correspondence to Professor Marlies P Schijven, Department of Surgery , Academic Medical Center, Amsterdam, 1100DD, The Netherlands; m.p.schijven{at}amc.uva.nl

Abstract

Background Telemedicine holds promise for improving access to care. Telementoring—defined as mentoring by means of telecommunication and computer networks—can be used for remote education of healthcare professionals. Furthermore, it is rapidly establishing itself as a valuable asset in medicine and education. This paper aims to establish a financially and practically feasible, stable telementoring network using wearables for sterile and hands-free remote control, to be used during surgical procedures.

Methods Two stand-alone computer systems, located at an academic hospital in the Netherlands and at a surgical research facility in Spain, were connected using TeamViewer software allowing for remote, hands-free controlling of radiological images using Myo gesture control armband. The operating surgeon consulted the remote surgeon through an audio, video and desktop sharing system during a live surgical procedure on a single porcine model. The system was analysed for feasibility and connection quality.

Results The sensors used were commercially available and relatively cheap, with the integrating computer system being responsible for the majority of costs. A successful connection was established without any downtime and with only a minor time lag, not interfering with the telementoring procedures. The operating surgeon effectively consulted with and was mentored by the remote surgeon, through video, audio and the desktop sharing system, using the wearable sensors.

Conclusions This proof of principle shows the feasibility of using an internet-based remote desktop sharing system in combination with wearable sensors and TedCube technology for telementoring purposes during surgical procedures.

  • telemedicine
  • remote consultation
  • videoconferencing
  • user-computer interface
  • computer-assisted surgery
View Full Text

Statistics from Altmetric.com

Introduction

Telementoring in surgery

Telementoring is defined as the supervision of an inexperienced surgeon or physician by an expert surgeon or physician at a remote location via electronic communications.1–4 Systems used in telementoring need to include options for data collection, processing and display, as well as possibilities for responding to the acquired information. Telementoring enables an expert physician: the mentor, to directly provide feedback and share expertise from a distance with a less experienced professional: the mentee.1 5 Telementoring has been applied successfully numerous times in the past,5 and with the rapid progress in communication technology in recent years, this form of distant mentoring is becoming more readily available, leading to numerous new possibilities.6 7 Telesurgery is defined as local control of procedures happening at a remote operating theatre.1 8 It has enabled surgeons to perform (robot-assisted) laparoscopic procedures on patients over thousands of miles away.9 10 The same information technologies can be used for teaching of new surgical techniques at locations where local expertise is lacking.1 8 11 12

Wearable technology

As the digital era is expanding, wearable sensors have gained much interest in both the patient and professional healthcare communities as well.13–15 These sensors have been used for the purpose of mentoring trainees during surgical procedures7 11 16 and also in the monitoring of outpatients with acute on chronic illness.17 18

In the operating theatre, wearable sensors such as the Myo gesture control armband (Thalmic Labs; Thalmic Labs, Kitchener, Canada) allow for direct, sterile and hands-free interaction with computers housing the electronic patient files, including radiological imaging.19–21 The use of wearable technology for interaction with the hospital computer system is of interest, as the operating surgeon is able to retrieve information and interact with imaging immediately and intuitively, without having to remove the sterile gloves.7 19–21 This may significantly facilitate surgical workflow but does not offer opportunities for a ‘second opinion’ on the spot when the surgeon is in direct need of a colleague’s expertise. A system allowing the local surgeon to interact in a sterile manner with a local computer, that can be controlled reliably by a consulted surgeon at a remote location, is needed.

Objectives

The aim of this study was to test the feasibility of combining wearable technology and telementoring for long-distance remote consultation, between the Netherlands and Spain, during a surgical procedure. Furthermore, the quality and speed of the established connection as well as the costs of the set-up were assessed to determine overall feasibility.

Materials and methods

Set-up

Sensors

Using the combination of Myo gesture control armband, a wireless headset (Plantronics Voyager Legend; Plantronics, Santa Cruz, California, USA) for voice control and TedCube integrating technology (TedCas Medical Systems S.L., Noáin, Spain), a stand-alone, commercially available laptop computer located at the Academic Medical Center (AMC) in Amsterdam was connected to a remote computer using TeamViewer software (TeamViewer GmbH, Göppingen, Germany), which is compliant with the Health Insurance Portability and Accountability Act.22 The remote computer was located in the Jesús Uson Minimally Invasive Surgery Centre (JUMISC) operating suite in Cáceres, Spain (figure 1). At the AMC site, a second laptop computer was running Skype (Microsoft, Palo Alto, California, USA) and screen capture software. This laptop was connected to a separate camera at JUMISC to view and record the laparoscopic video images from the operating suite during the procedure. The set-up allowed both surgeons to see and discuss both the laparoscopic procedure and the radiological images simultaneously (figure 1).

Figure 1

Schematic overview of the set-up at AMC in Amsterdam, the Netherlands (consulted surgeon) and JUMISC in Cáceres, Spain (surgical procedure). AMC, Academic Medical Center; JUMISC, Jesús Uson Minimally Invasive Surgery Centre; PC, personal computer; PC1, used for direct audio/video communication; PC2, used for hands-free interaction with patient-files; SW, software.

Integrating computer

The TedCube is an integrating plug-and-play device that is connected to a computer using a Universal Serial Bus 2.0 connection. The TedCube combines and translates the commands from sensors to evoke actions on the computer it is controlling. In this case, the Myo gesture control armband was used for manipulation of images and for demonstration of specific details using the ‘pointer’ function. The Plantronics Voyager wireless headset was used to activate and deactivate the system, as well as selecting the type of manipulation tool used, such as zoom, rotation or scroll. The TedCube allows for hands-free sterile interaction using voice commands, hand gestures and motion sensors.

Preoperative imaging and surgical procedure

This study was reviewed and approved by the local Animal Welfare and Ethics Committee. Accommodation of the animal and its handling was done in accordance with the European directive (2010/63/EU) regarding the use of animals for scientific purposes, Spanish laws (RD 53/2013) for animal use and care, Animal Research: Reporting of In Vivo Experiments guidelines and according to the Guide for Care and Use of Laboratory Animals.

Both two-dimensional and three-dimensional reconstructions were generated from radiological images of an artificial tumour in the kidney of a porcine model. The artificial tumour was created during a previous surgery in the experimental porcine model (male, 35 kg). The tumour was created with a mixture of saline, alginate and CT contrast medium. With the support of a laparoscopic grasper, a percutaneous 14G needle (51 mm in length) was used to inject the mixture in the superior renal capsule of the left kidney (duration of the procedure: 4.37 min). After the development of the artificial tumour, identification of its exact location, size and other characteristics, as well as identification of renal vascular anatomy, was carried out by means of CT imaging (Brilliance CT; Philips, the Netherlands). The tumour and vascular structures of the left kidney were manually segmented, and 2D and 3D images of these structures were made. All the preoperative images and three-dimensional models were stored in a picture archiving and communication system on a portable workstation (MacBook Pro; Apple, USA; 2 GHz Intel Core i7, 16 GB random access memory).

After radiological imaging had been performed and the 2D and 3D reconstructions were made, the operating surgeon (FMSM) performed a laparoscopic partial nephrectomy in the live experimental porcine model. The procedure was successfully completed in 69.88 min, and no complications occurred. Preoperative radiological imaging was readily available on the computer, allowing for interaction by both the local surgeon (FMSM, mentee) and the consulted surgeon (MPS, mentor) through voice commands and hand gestures. The laparoscopic images were also immediately available to the consulted surgeon.

Training period

Both surgeons were very experienced in laparoscopic surgery and had previously used the gesture control system. The operating surgeon (mentee) had used the gesture control system during multiple different laparoscopic training tasks on a simulator (box trainer) and during three live laparoscopic procedures, including a partial nephrectomy and a partial hepatectomy. The remote surgeon (mentor) had previously received a single training session from the developers of the TedCube and used the gesture control system during multiple surgical procedures before this study took place.

Connection quality parameters

The quality of the established connection, any possible downtime or connection problems were evaluated by both the consulted surgeon at AMC in Amsterdam and the surgical team carrying out the procedure at JUMISC. The consulted surgeon (MPS) also qualitatively evaluated the remote controlling of the radiological imaging, using the wearable sensors and the TedCube system. Parameters used for this were the speed of execution of commands by the remote computer, as well as the definition of the images as they were seen in Amsterdam, defined as good, average or unclear. This distinction was based on whether or not quality of the images was good enough to give live comments and feedback on the procedure. The outcomes were determined by the remote surgeon (MPS) and a second researcher (HAWM). The same criteria were also used for the qualitative evaluation of the audio and video connection. When the speed and quality of images was good enough to give meaningful feedback and comments, the procedure would be regarded a success.

Financial evaluation

The costs of the set-up were determined by contacting the manufacturers of the TedCube. Next to this, the prices of the individual sensors and software programmes used in the experiment were looked up on the different companies’ websites.

Results

Set-up of sensors and remote connection

A successful connection without any downtime was established between AMC in Amsterdam, the Netherlands and JUMISC in Cáceres, Spain (figures 2-4, online supplementary video). The TedCube, in combination with the voice and gesture control, allowed for hands-free remote controlling of the computer located at AMC, which hereby also allowed for hands-free remote controlling of the computer located at JUMISC. The connections were established over the local AMC Wi-Fi network, which has an average of 12 Mbps download speed and 5 Mbps upload speed. Both the Myo gesture control armband and the Plantronics voice control interacted smoothly through the TedCube with the local and the remote computer.

Supplementary file 1

Figure 2

Set-up of the computers for telementoring in the operating room during the partial nephrectomy in an experimental porcine model. The surgeons performing the laparoscopy are seen on the left, with the set-up of the laptop and screen connected to the remote consulted surgeon on the right.

Figure 3

Hands-free interaction by the consulted surgeon at the remote location. The laptop running Skype is seen on the left-hand side of the picture, with the screen on the right showing the radiological imaging (3D reconstruction of renal calix) being controlled by the remote consulted surgeon (also seen on the laptop screen on the left).

Figure 4

Interaction by the local surgeon, seen wearing the Myo gesture control armband for demonstration, with the preoperative images through wearable sensors during the laparoscopic surgery procedure.

Quality of connection

During the telementoring session, there was a minor lag time between the commands given at AMC and the execution of those actions by the JUMISC computer, which did not interfere with the surgical workflow. This delay slightly increased when two-dimensional and three-dimensional CT images were controlled, most likely due to the large size of these files. Both videoconference images and audio were regarded to be of good quality by both researchers (HAWM and JASM) as well as the operating surgeon (FMSM) and the remote surgeon (MPS). No delay in the videoconferencing connection was noted. This made it possible for the consulted surgeon to comment on the procedure while it was being executed and use the radiological images as supporting material.

Costs

The price of the separate sensors is relatively cheap with the Myo gesture control armband available around €200 and the Plantronics Voyager Legend headset at €70. They were included in research combination package with the TedCube, of which the price range including all licenses is dependent on the version of the system, varying around €10 000. In combination with mostly free and readily available software including Skype and TeamViewer, the main costs for this procedure are those of the TedCube.

Discussion

In this proof of principle study, the feasibility of a combination between wearable technology for hands-free controlling and a remote connection between computers located in two different countries was tested. The set-up was found to be feasible for the purpose of telementoring using a live audio and video connection and remote controlling of radiological imaging located on a distant computer. The local surgeon (mentee), while performing the procedure, and the distant consulted surgeon (mentor) could discuss the presented case without significant delay, supported by the video connection and laparoscopic video images, and interact with the available radiologic imaging.

By implementing wearable sensors and remote controlling, the experiment executed in this study adds an extra dimension to previously performed studies on telementoring and teleconsultation.9 12 By using wearable sensors, the mentoring surgeon is also enabled to perform a training procedure, supported by hands-free control of supporting imaging, while simultaneously coaching mentees through their own procedures. Marescaux et al10 were one of the pioneers in this field when they performed a long-distance remote controlled robotic surgery. More recent advances include the use of wearable sensors and head-mounted cameras such as the Google Glass, as well as the use of commercially available communication services like Skype and TeamViewer, as a virtual private network (VPN).23 24

Surgically treatable conditions form 11% of the global burden of disease, with a significant majority of these conditions occurring in resource-poor countries.25Disability-adjusted life years (DALYs) can be used to quantify the morbidity caused by these diseases. The number of DALYs, which is disproportionally high in low- and lower-middle-income countries, could significantly be decreased by improving access to surgical care, as stated by WHO.2 26 By setting up a telementoring network with local surgeons and trainers, surgical needs can be addressed when needed, supported by high quality teaching or second opinion from distant expert surgeons that are immediately available. This would effectively increase levels of patient safety in these areas without needing much effort or resources.2

Though it may not always be necessary for both the mentor and the mentee to have sterile and hands-free control over the computer, the two-sided hands-free controlling is a valuable asset in teaching situations where both surgeons are performing a procedure at the same time. With the TedCube system, a regular computer mouse and keyboard can always be used in conjunction with the wearable sensors. This will also be useful when only the mentee is performing a procedure, and sterile, hands-free interaction by the mentor is not needed. Educational benefits of the sterile, hands-free system used in this telementoring setting are the ability of both mentor and mentee to perform a procedure simultaneously. Although the mentor in this study was not simultaneously performing an operation, the system enables an ‘example’ mentoring operation to be executed at the same time as the mentee performing a ‘practice’ procedure.

A potential addition to this hands-free, wearable computer control system is the option of telestration. Telestration is the ‘technique of drawing freehand commands over still image or video’.27 Though this feature was not used in the current setting, it is a potential powerful tool to be tested in future studies.4 28

While there was a minor delay in the controlling of the imaging, both audio and video were of good quality for the purpose of telementoring. The lag time in the execution of the remote commands by the imaging suite can be explained by the use of a public Wi-Fi network in combination with the large 2D and 3D CT image files. Limits of bandwidth and latency are relevant in telementoring, especially when used in concomitant telesurgery. Previous research on remote controlling in telesurgery showed that a lag time of 300 ms is the maximum delay allowing a safe surgical performance.10 Though this leads to the recommendation that a wired network or a rapid wireless connection should be used in future telementoring sessions, it was most practical to use readily available technology and networks for telemedicine and communication during this study.

In order to implement this remote telementoring system using wearable gesture controlled sensors in standard clinical practice and surgical training, there are several issues that need to be addressed. The latency is something that needs to decrease to below the previously determined lag time of 300 ms. Second, when using this system in a clinical setting, data needs to be secured, and patient privacy should be taken into account, for example, by at least using VPN tunnels and data encryption.

With latency being the most important limitation in this proof of principle, patient privacy was not of important consideration owing to the use of an experimental animal model. In human patients, however, privacy will be a matter of concern. It therefore needs to be investigated whether commercially available and internet-based software for video and audio communication is secure enough to guarantee patient privacy in future telementoring settings.

Conclusions

This experiment shows the feasibility of using wearable sensors in combination with TedCube technology for hands-free computer interaction during surgery and telementoring. The combination of wearable sensors, an integrating device and internet-based remote desktop sharing software proves a feasible set-up for telementoring in situations when sterility for both the mentor and the mentee is necessary, and distance needs to be overcome. While future studies need to determine the security demands to be set for internet networks and software used to transfer patient data, the use of readily available telecommunication technology is pragmatic and useful in our experimental set-up.

Acknowledgments

The authors would like to thank Jesús Perez and Enrique Munoz of TedCas Medical Systems S.L., Spain for their kind and generous advice and feedback during this study.

References

View Abstract

Footnotes

  • Contributors HAWM contributed to the planning, set-up and execution of the study at the Academic Medical Center in Amsterdam, the Netherlands. She assisted the mentoring surgeon (MPS) during the procedure and contributed to the writing of this manuscript. JASM was involved in the planning and set-up of this study. He assisted the remote surgeon (FMSM) during the procedure, prepared the set-up of the computers and sensors at the remote location and contributed to the writing of this manuscript. FMSM was the operating surgeon at the Jesús Usón Minimally Invasive Surgery Center in Cáceres, Spain. He was involved in the planning, set-up and execution of the study. JCG was involved in the planning of the study, data analysis and contributed to the evaluation and writing of the manuscript. MPS was the mentoring surgeon at the Academic Medical Center in Amsterdam, the Netherlands. She was involved in the planning, set-up and execution of the study, as well as the evaluation and data analysis, and she contributed to the writing of the manuscript.

  • Funding The salary of HAWM is funded by the Growing Games Program (Dutch Game Garden, iMMovator, Dutch Games Association and the Economic Board Utrecht), grant no CB00018 and by a grant from the CZ Fund, The Netherlands. JASM, FMSM, JCG and MPS have no financial ties to disclose.

  • Competing interests None declared.

  • Patient consent Detail has been removed from this case description/these case descriptions to ensure anonymity. The editors and reviewers have seen the detailed information available and are satisfied that the information backs up the case the authors are making.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement This feasibility study has no additional unpublished data available. All data generated by this study have been reported in this manuscript.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.