ANGIO MENTOR CLINICAL VALIDATIONS
Simulator-Based Angiography and Endovascular Neurosurgery Curriculum: A Longitudinal Evaluation of Performance Following Simulator-Based Angiography Training
- 1Department of Neurosurgery, University of California, San Diego.
- 2Department of Internal Medicine, University of California, Los Angeles.
- 3Neuroradiology, University of New Mexico.
Diagnostic coronary angiography: initial results of a simulation program
David B. Casey, David Stewart, Mladen I. Vidovich
Division of Cardiology, University of Illinois at Chicago
Cardiovascular Revascularization Medicine, March 2016
BACKGROUND: The use of simulator-based teaching in cardiology has unfortunately lagged behind other procedural specialties. This study investigates the utility of a simulator-based training program for fellows in cardiovascular disease with no prior experience in diagnostic coronary angiography.
METHODS: First-year cardiology fellows at University of Illinois-Chicago (UIC) using ANGIO Mentor™ simulators completed benchmark cases requiring basic coronary engagement. Subsequently, benchmark cases were completed one day later and at 9 months following 2-3 months of training in the cardiac catheterization lab. In addition, 1st year cardiology fellows were compared to 3rd year fellows. Objective measures assessed from benchmark cases were total procedural time, total contrast used, and total fluoroscopy time.
RESULTS: All 1st year fellows improved their total time to complete the benchmark case from initial to second attempt one day later (14:56 on Day 1, 8:30 on Day 2, P=0.03). Total contrast used (60 mL on Day 1, 39 mL on Day 2, P=0.11) and total fluoroscopic time (6:30 on Day 1 and 4:26 on Day 2, P=0.16) also both decreased. Overall procedure time and contrast use was similar among 1st and 3rd year fellows after simulation training. Decreases in procedure and fluoroscopy time were maintained in 1st year fellows after 2-3 months of training.
CONCLUSIONS: Fellows displayed technical and procedural improvement at diagnostic coronary angiography in a short period of time and in a safe, patient free environment. In this study, a computer-based simulator was successfully incorporated into a first year cardiovascular fellowship curriculum and represents a contemporary means to provide the fellow increased procedural training without added risk to the patient.
The model for Fundamentals of Endovascular Surgery (FEVS) successfully defines the competent endovascular surgeon
Cassidy Duran, MD, Sean Estrada, PhD, Marcia O’Malley, PhD, Malachi G. Sheahan, MD, Murray L. Shames, MD, Jason T. Lee, MD, Jean Bismuth, MD
University of South Florida, Tampa, Fla
Journal of Vascular Surgery
OBJECTIVE: Fundamental skills testing is now required for certification in general surgery. No model for assessing fundamental endovascular skills exists. Our objective was to develop a model that tests the fundamental endovascular skills and differentiates competent from noncompetent performance.
METHODS: The Fundamentals of Endovascular Surgery model was developed in silicon and virtual-reality versions. Twenty individuals (with a range of experience) performed four tasks on each model in three separate sessions. Tasks on the silicon model were performed under fluoroscopic guidance, and electromagnetic tracking captured motion metrics for catheter tip position. Image processing captured tool tip position and motion on the virtual model. Performance was evaluated using a global rating scale, blinded video assessment of error metrics, and catheter tip movement and position. Motion analysis was based on derivations of speed and position that define proficiency of movement (spectral arc length, duration of submovement, and number of submovements).
RESULTS: Performance was significantly different between competent and noncompetent interventionalists for the three performance measures of motion metrics, error metrics, and global rating scale. The mean error metric score was 6.83 for noncompetent individuals and 2.51 for the competent group (P < .0001). Median global rating scores were 2.25 for the noncompetent group and 4.75 for the competent users (P < .0001).
CONCLUSIONS: The Fundamentals of Endovascular Surgery model successfully differentiates competent and noncompetent performance of fundamental endovascular skills based on a series of objective performance measures. This model could serve as a platform for skills testing for all trainees.
Validation of a Novel Virtual Basic Skills Simulation Model
Murray L. Shames, Johnathon V. DallaRosa, Michael T. Brannick
University of South Florida, Tampa, Fla
OBJECTIVES: This study determined the construct validity of a novel basic endovascular skills challenge on a virtual reality simulator.
METHODS: Subjects completed a basic endovascular skills challenge on the Simbionix ANGIO Mentor. The subjects were divided into three groups based on endovascular experience: novice (medical students and postgraduate year 1 surgery residents), intermediate (vascular surgery residents postgraduate year 2-7), and expert (vascular surgery faculty). Each task requires advancing a wire, catheter, and sheath to predetermined positions in a virtual vascular tree. The challenge is divided into five cases. Case 1 contains five tasks. Each subsequent task is more complicated than the previous task. Time to completion and points (generated by the simulator) were compared for case 1 (tasks 1-5). Time and points were compared with each other to determine construct validity.
RESULTS: Twenty-two subjects (11 novice, 6 intermediate, and 5 expert) completed the simulation. One-way analysis of variance was computed for time to case completion with the group as the independent variable. The result was statistically significant (P < .05). The results of the Tukey least significant difference post hoc test showed that the novice group was significantly different from the other two groups, which did not differ from one another. One-way analysis of variance was also computed for points accrued for case completion with the same groups. The result was statistically significant (P < .05). Post hoc tests were also computed for the groups, and the results were similar to those for seconds to completion. The novice group earned fewer points than the other two groups, which did not differ from one another. A scatterplot of seconds by points for the case reveals a strong but nonlinear association that is to be expected from the scoring method that assigns points by time.
CONCLUSIONS: A novel endovascular skills challenge on the Simbionix ANGIO Mentor is able to differentiate novice from intermediate and expert operators.
Development of a PROficiency-Based StePwise Endovascular Curricular Training (PROSPECT) Program
Heidi Maertens, MD1, Rajesh Aggrawal, MD, PhD2, Liesbeth Desender1, MD, Frank Vermassen1, MD, PhD, Isabelle Van Herzeele1, MD, PhD.
1 Department of Thoracic and Vascular Surgery, Ghent University Hospital, Ghent, Belgium
2 Department of Surgery, McGill University, Montreal, Canada
Journal of Surgical Education, Published Online: August 11, 2015
DESIGN: A PROficiency-based StePwise Endovascular Curricular Training (PROSPECT) program was developed, consisting of e-learning and hands-on simulation modules, focusing on iliac and superficial femoral artery atherosclerotic disease. Construct validity was investigated. Performances were assessed using multiple-choice questionnaires, valid simulation parameters, global rating scorings, and examiner checklists. Feasibility was assessed by passage of 2 final-year medical students through this PROSPECT program.
SETTING: Ghent University Hospital, a tertiary clinical care and academic center in Belgium with general surgery residency program.
PARTICIPANTS: Senior-year medical students were recruited at Ghent University Hospital. Vascular surgeons were invited to participate during conferences and meetings if they had performed at least 100 endovascular procedures as the primary operator during the last 2 years.
RESULTS: Overall, 29 medical students and 20 vascular surgeons participated. Vascular surgeons obtained higher multiple-choice questionnaire scores (median: 24.5-22.0 vs. 15.0-12.0; p < 0.001). Students took significantly longer to treat any iliac or femoral artery stenosis (3.3-14.8 vs. 5.8-30.1 min; p = 0.001-0.04), whereas in more complex cases, fluoroscopy time was significantly higher in students (8.3 vs. 21.3 min; p = 0.002; 7.3 vs. 13.1 min; p = 0.03). In all cases, vascular surgeons scored higher on global rating scorings (51.0-42.0 vs. 29.5-18.0; p < 0.001) and examiner checklist (81.5-75.0 vs. 54.5-43.0; p < 0.001). Hence, proficiency levels based on median expert scores could be determined. There were 2 students who completed the program and passed for each step within a 3-month period during their internships.
CONCLUSIONS: A feasible and construct validated surgical program to train cognitive, technical, and nontechnical endovascular skills was developed. A structured, stepwise, proficiency-based valid endovascular program to train cognitive, technical, and human factor skills has been developed and proven to be feasible. A randomized controlled trial has been initiated to investigate its effect on performances in real life, patient outcomes, and cost-effectiveness.
Diagnostic angiography skill acquisition with a secondary curve catheter: phase 2 of a curriculum-based endovascular simulation program
Spiotta AM1, Kellogg RT1, Vargas J1, Chaudry MI2, Turk AS2, Turner RD1.
1Division of Neurosurgery, Department of Neurosciences, Medical University of South Carolina, Charleston, South Carolina, USA.
2Department of Neuroradiology, Medical University of South Carolina, Charleston, South Carolina, USA.
J Neurointerv Surg. 2014 Sep 3.
BACKGROUND: We have previously reported the efficacy of a simulator-based training paradigm for residents in neurosurgery with little or no prior experience in diagnostic cerebral angiography with straightfoward arch anatomy. This study investigates the utility of a simulation-based training curriculum for the acquisition of skills employing a secondary curve catheter to navigate more complex arch anatomy.
METHODS: Residents at the Medical University of South Carolina (MUSC) with moderate exposure to diagnostic angiography enrolled into a standardized Institutional Review Board-approved training protocol using SimSuite Compass and Simbionix simulators. The task involved (in order) forming the Simmons catheter in the left subclavian artery and then selecting the brachiocephalic, left common carotid and left vertebral arteries.
RESULTS: All participants improved their total time to complete the task over the course from the first to last trial. Each milestone within the overall task also demonstrated an improvement across trials for each participant. Following the hands-on experience, participants’ rating of their knowledge of arch anatomy and vessel selection technique improved to that between competence and high competence (values of 3.3±0.49 (p<0.005) and 3.1±0.38 (p<0.01), respectively). Comfort with use of the Simmons catheter improved to a value of 2.9±0.38 (p<0.001), between an experienced learner and competence. Participants rated the usefulness of the training environment as very high (4.1±0.90 out of maximum 5).
CONCLUSIONS: Residents became more proficient at vessel selection in a type II and bovine arch over a relatively compressed time period, with both objective and subjective data demonstrating acquisition of skill sets and increased confidence.
Endovascular Simulation Leads to Efficiency and Competence in TEVAR Procedures
Andre F. Gosling, Anil Nagavalli, Daniel Kendrick, Vikram S. Kashyap, John C. Wang
Vascular Surgery, University Hospitals Case Medical Center, Cleveland, Ohio
Journal of Vascular Surgery Volume 59, Issue 6, Supplement, Page 13S, June 2014
OBJECTIVES: We assessed the effects of thoracic endovascular aortic repair (TEVAR) rehearsal with an endovascular surgical simulator in surgical trainees at different levels in training
METHODS: Twelve trainees in three cohorts (student, postgraduate year [PGY]1-3, and PGY 4-7) were oriented to using the Simbionix AngioMentor simulator over four sessions. Likert scale qualitative analysis evaluated participant proficiency. Analysis of data included one-way analysis of variance and paired t-tests.
RESULTS: All groups had reduction of total procedure time (mean 537 ± 148 seconds vs 269 ± 66 seconds, first session vs fourth; confidence interval [CI], 195-341 seconds; P < .05) and fluoroscopy time (201 ± 74 vs 110 ± 37 seconds; CI, 51-132 seconds; P < .05) with case progression. Procedure time significantly decreased in students (551 ± 84 vs 313 ± 65 seconds; CI 189-287 seconds; P < .05) and PGY 1-3 (591 ± 149 vs 264 ± 29 seconds; CI, 113-541 seconds; P < .02) had a significant decrease in procedure time. Fluoroscopy times were brief overall between individuals and groups and did not change significantly with case progression. Participants acquired proficiency after a few runs in almost every step of the procedure. Endograft sizing appeared to be the most challenging task by qualitative analysis. PGY 4-7 trainees had higher technical scores, but this was not statistically significant.
Fig 1. Comparison among groups in TEVAR total procedure time
Conclusions: Practice on endovascular surgical simulators can reduce overall procedure and fluoroscopy time, independent of trainee skill level or experience. Endovascular simulators can be effective tools in residency training and maintenance of proficiency. Further studies are needed to compare simulator performance with outcomes in real cases.
Realism, criterion validity, and training capability of simulated diagnostic cerebral angiography.Nguyen N1, Eagleson R1, Boulton M2, de Ribaupierre S1.
1Department of Electrical and Computer Engineering, Western University, London, ON, Canada.
2Clinical Neurological Sciences, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada.
Stud Health Technol Inform. 2014;196:297-303.
Computer-based simulation is increasingly used in medical education for training, assessment, credentialing, and practice. Compared to medical specialties such as anesthesiology and general surgery, the adoption of simulation for neurointerventional training has been slow. This may be due to the limited number of neurointerventional simulators available and the lack of research assessing their validity and training capability. The objective of this study was to assess the realism, validity, and training capability of computer-based simulation for diagnostic cerebral angiography using a commercially available simulator called the ANGIO Mentor Express.
The following abstract was presented at the Royal College Simulation Summit , November 8 – 9, 2013, Vancouver, Canada
Criterion and face validity of the ANGIO Mentor for diagnostic cerebral angiography
Ngan Nguyen, PhD1, Roy Eagleson, PhD1, Mel Boulton, PhD, MD2, Sandrine deRibaupierre, MD1,2
1Department of Electrical and Computer Engineering,Western University, London, ON, Canada
2Department of Clinical Neurological Sciences, Western University, London, ON, Canada
The objective of this study was to assess the face and criterion validity of a computer-based simulation for diagnostic cerebral angiography using the ANGIO Mentor Express. Participants were divided into two groups: experts (experienced interventional physicians) and novices (residents and fellows). Face validity was addressed by asking experts to rate, on a 5-point Likert scale, the appropriateness of the simulated content as a teaching and training tool. Criterion validity was established by comparing the simulation performance of experts vs. novices. After completing a step-by-step tutorial to become acquainted with the relevant technical features involved in performing a simulated cerebral angiography, all participants practiced performing an angiography of the left internal carotid artery. Subsequently, they completed a simulated angiography of the right middle cerebral artery. The procedure time, fluoroscopy time, amount of contrast, number of fluoroscopic images, and number of roadmaps utilized when performing the right middle cerebral artery were recorded. These allowed us to compute objective measures of performance. Upon completion of the two simulated cases, experts were asked to rate the appropriateness of the simulated content. Experts outperformed novices in nearly all performance variables, but significant differences were found for fluoroscopy time and amount of contrast utilized, p<0.05. Experts reported that the ANGIO Mentor provided content appropriate to the angiography procedure (mean=4.85) and that it is useful as a teaching and training tool (mean=4.71). Preliminary results revealed that the ANGIO Mentor has appropriate face and criterion validity, providing support for the ANGIO Mentor’s use as a tool for teaching diagnostic cerebral angiography.
Two Centre Evaluation of the ANGIO Mentor Electrophysiology Simulator
- Ullah1, R. Marazzi2, R. Hunter1, M. Dhinoja1, S. Sporton1, R. De Ponti2, R. Schilling1
1Department of Cardiology, St Bartholomew’ s Hospital, Barts Health Trust, London, UK; 2Department of Clinical and Experimental Medicine University of Insubria, Varese, Italy.
Presented as a poster at HRS meeting, 2013
Objective Simulator-Based Evaluation of Carotid Artery Stenting Proficiency (from Assessment of Operator Performance by theCarotid Stenting Simulator Study [ASSESS])
Giora Weisz, MD, Nathaniel R. Smilowitz, MD, Helen Parise, ScD, Jacques Devaud,
Issam Moussa, MD, Stephen Ramee, MD, Mark Reisman, MD, Christopher J. White, MD,
and William A. Gray, MD
American Journal of Cardiology, April 2013
Studies have suggested that operator proficiency has a substantial effect on complication rates and procedural outcomes. Endovascular simulators have been used for training and have been proposed as an alternative to the conventional assessment of skills. The present study sought to validate simulation as an objective method for proficiency evaluation in carotid artery stenting. Interventional cardiologists classified as novice, intermediate, or experienced practitioners performed 3 simulated, interactive carotid stenting cases on an Angio Mentor endovascular simulator. An automated algorithm scored the participants according to the technical performance, medical management, and angiographic results. A total of 33 interventional cardiologists (8 novices, 15 intermediates, and 10 experts) completed 82 simulated procedures. The composite scores for the case simulations varied significantly by operator experience, with better scores for the more experienced groups (p <0.0001). The metrics that discriminated between operator experience groups included fluoroscopy time, crossing the carotid lesion with devices other than a 0.014-in. wire before filter deployment, and incomplete coverage of the lesion by the stent. In conclusion, the results of the present study validate that a simulator with an automated scoring system is able to discriminate between levels of operator proficiency for carotid artery stenting. Simulator based performance assessment could have a role in initial and ongoing proficiency evaluations and credentialing of interventional operators of high-risk endovascular procedures.
Simulated diagnostic cerebral angiography in neurosurgical training: a pilot program
Spiotta AM, Rasmussen PA, Masaryk TJ, Benzel EC, Schlenk R.
Department of Neurological Surgery, Cleveland Clinic, Cleveland, Ohio, USA.
J Neurointerv Surg. 2012, May 10.
INTRODUCTION: Surgical simulation provides a zero-risk setting in which technical skills can be obtained through repetition. The feasibility and utility of simulated diagnostic cerebral angiography among neurosurgical residents and fellows was studied using an endovascular biplane angiography simulator.
METHODS: Ten neurosurgical residents and four endovascular neurosurgery fellows were recruited into a standardized training protocol consisting of a didactic, demonstration and hands-on learning environment using the Simbionix simulator. Participants were instructed to catheterize the right internal carotid artery, left internal carotid artery and left vertebral artery. The task was repeated five times.
RESULTS: All participants demonstrated improvement over the five trials. Residents performed actions that were perceived as potentially dangerous (n=8) while fellows performed the procedure with superior technique. Residents performed the task with an initial total procedure and fluoroscopy time of 6.6±4.3 min and 4.9±3.7 min, respectively, and improved on the fifth trial to 3.4±1.3 min (p=0.03) and 2.3±0.78 min (p=0.004), respectively. Residents approximated the efficiency of fellows for the third and fourth trial.
CONCLUSIONS: Incorporating an endovascular simulator is feasible for training purposes in a neurosurgical residency program. This study provides objective documentation of the facilitation of technical angiography skill acquisition by the use of simulation technology.
The following study by Stanford researchers reveal results from endovascular skills training for surgical residents and was presented at the at the 66th Vascular Annual Meeting presented by the Society for Vascular Surgery, June 7-9, 2012, at Gaylord National Resort & Convention Center, National Harbor, MD.
Results from endovascular skills training for surgical residents
Jason T. Lee
Stanford University’s Goodman Simulation Center in the Department of Surgery in Palo Alto, CA,
Training with simulation versus operative room attendance.
Desender LM, Van Herzeele I, Aggarwal R, Vermassen FE, Cheshire NJ.
Department of Thoracic and Vascular Surgery, University Hospital Ghent, Ghent, Belgium.
J Cardiovasc Surg (Torino). 2011 Feb;52(1):17-37.
Reduced training times, increasing complexity of endovascular and open vascular interventions and concerns for patient’s safety have necessitated a modernization in surgical training. A more strategic approach is required to facilitate the acquisition of surgical skills outside the operating room and to minimize the risks to patients as surgeons develop their technical expertise. Virtual reality simulation has been proposed as a means to train and objectively assess technical endovascular performance without risks to patient safety. This article reviews the evidence and the limitations for this adjunctive tool, the implementation in current training programs and future applications to maintain the highest standards of care for treatment of vascular disease.
Simulation in Neurosurgical Residency Training: A New Paradigm
Alejandro M. Spiotta, MD Richard P. Schlenk, MD
The Cleveland Clinic Foundation, Cleveland, Ohio, USA
The Congress of Neurological Surgeons(CNS) Quarterly 2010 page 18-20
In the era of duty hour restrictions and increasing medico-legal pressures, surgical simulation offers a viable alternative to bridge the gap in experience and knowledge of residents. The value of simulation for the re-creation of invasive procedures has a rich history in cadaveric dissection and animal experimentation. While these exposures provide a valuable experience and do not require sophisticated technological support, cadavers and laboratory animals are a scarce and expensive resource. Three years ago, we established a ‘fundamental skills laboratory’ funded by educational grant support. The course was run by our faculty and senior level residents and provided an educational experience to the junior level residents. The course involved intensive hands-on exercises designed to familiarize the trainees with fundamental bedside procedures and operative skills such as external ventricular drain and intracranial pressure bolt placement, suturing, drilling and turning a craniotomy flap. This course was extremely successful and was the forerunner to the SNS Boot Camp that will be offered for its second consecutive year in July 2011 for all incoming post-graduate graduate year 1 (PGY1) neurosurgery residents.
Virtual Reality Simulation in the Endovascular Field
Aggarwal Rajesh , Herzeele Isabelle Van
European Virtual Reality Endovascular Research Team (EVEREST)
US Cardiology, 2008;5(1):41-5
The last decade has witnessed exponential growth in the field of endovascular interventions, although only in the last few years has there been a widespread interest in the carotid artery stent (CAS). Endovascular physicians with different medical backgrounds such as interventional cardiologists, radiologists, and vascular surgeons all recognize the importance of this changing technology.1This procedure is almost unique, as the risks to the patient (stroke and death) as a result of the physician’s learning curve are unacceptably high. This has been summarized by editorials written after the publication of the Carotid and Vertebral Artery Angioplasty Study (CAVATAS)2–4 and the Endarterectomy versus Angioplasty in Patients with Symptomatic Severe Carotid Stenosis trial (EVA-3S).5–7 Recent publications of the rates of medical errors and adverse events within healthcare8 have drawn the spotlight toward methods of establishing credentials for physicians preparing to perform complex procedures. In order to improve patient safety, operators must have the appropriate cognitive and technical skills and experience of CAS. Furthermore, physicians should have previously achieved a high level of proficiency in other catheter-based interventions and completed dedicated training in CAS.9
The Utility of Endovascular Simulation to Improve Technical Performance and Stimulate Continued Interest of Preclinical Medical Students in Vascular Surgery
Jason T. Lee , Mary Qiu , Mediget Teshome, Shyam S. Raghavan, Maureen M. Tedesco, and Ronald L. Dalman
Division of Vascular Surgery, Stanford University School of Medicine, Stanford, California
Journal of Surgical Education Volume 66, Issue 6, November-December 2009, Pages 367-373 Available online 30 January 2010.
Presented at the Association for Program Directors in Surgery/Association for Program Directors in Vascular Surgery Meeting, April 30, 2009, Salt Lake City, UT.
OBJECTIVES: New training paradigms in vascular surgery allow for early specialization out of medical school. Surgical simulation has emerged as an educational tool for trainees to practice procedures in a controlled environment allowing interested medical students to perform procedures without compromising patient safety. The purpose of this study is to assess the ability of a simulation-based curriculum to improve the technical performance and interest level of medical students in vascular surgery.
DESIGN: Prospective observational cohort study of medical student performance.
SETTING: Academic medical center.
PARTICIPANTS: Forty-one medical students (23 first year, 15 second year, 3 other) enrolled in a vascular surgery elective course. Students completed a survey of their interests and performed a renal stent procedure on an endovascular simulator (pretest). The curriculum consisted of didactic teaching and weekly mentored simulator sessions and concluded with a final renal stent procedure on the simulator (posttest). Objective procedural measures were determined during the pre- and posttest by the simulator, and subjective performance was graded by expert observers utilizing a structured global assessment scale. After the course, the students were surveyed as to their opinions about vascular surgery as a career option. Finally, 1 year after the course, all students were again surveyed to determine continued interest in vascular surgery.
RESULTS: The objective and subjective criteria measured on the simulator and structured global assessment scale significantly improved from pre- to posttest in terms of performer technical skill, patient safety measures, and structured global assessments. Before beginning the course, 8.5% of the students expressed high interest in vascular surgery, and after completing the course 70% were seriously considering vascular surgery as a career option (p = 0.0001). More than 95% of the students responded that endovascular simulation increased their knowledge and interest in vascular surgery. In the 1-year follow-up survey (n = 23 medical students), 35% had already entered their clinical years. Seventy percent of the students were still considering vascular surgery, while several other career options were still popular including the surgical subspecialties (70%), interventional cardiology (57%), and interventional radiology (48%). Most respondents indicated the major reasons for continued interest in vascular surgery were the ability to practice endovascular procedures on the simulator (100%) and mentorship from vascular surgery faculty (78%).
CONCLUSIONS: The use of high fidelity endovascular simulation within an introductory vascular surgery course improves medical student performance with respect to technical skill, patient safety parameters, and global performance assessment. Mentored exposure to endovascular procedures on the simulator positively impacts long term medical student attitudes towards vascular surgery. Simulator-based courses may have the potential to be an important component in the assessment and recruitment of medical students for future surgical training programs.
Experienced Endovascular Interventionalists Objectively Improve their Skills by Attending Carotid Artery Stent Training Courses
Van Herzeele I, Aggarwal R, Neequaye S, Hamady M, Cleveland T, Darzi A, Cheshire N, Gaines P.
Department of Biosurgery and Surgical Technology, Imperial College London, UK; Regional Vascular Unit, St. Mary’s Hospital, London, UK.
Eur J Vasc Endovasc Surg. 2008 May;35(5):541-50. Epub 2008 Feb 8.
OBJECTIVES: Carotid artery stenting (CAS) is an advanced endovascular intervention with a steep learning curve. Virtual reality (VR) simulation has been proposed as a means to train and objectively assess technical performance. AIM: To objectively assess psychomotor skills acquisition of experienced interventionalists attending a two-day CAS course, using a VR simulator.
METHODS: Both cognitive and technical skills of 11 interventionalists were trained in a two-day course using didactic sessions, case reviews, supervised VR simulation and live-cases. Pre- and post-course skills were assessed through performance on the same CAS procedure using metrics derived from the simulator.
RESULTS: Significant differences were noted between pre- and post-course performance for procedure (36 vs. 20min., p=0.005), X-ray (20 vs. 11min., p=0.016) and delivery-retrieval time of the embolic protection device (12 vs. 9min., p=0.007). Advancement of the guiding catheter without a leading wire occurred to a greater extent pre- versus post-course (199 vs. 152mm., p=0.050) as did spasm of the internal carotid artery (4 vs. 2, p=0.049).
CONCLUSIONS: This study has objectively proven a benefit for experienced interventionalists to attend CAS courses for skills acquisition measured by a VR simulator. These data can be used to offer participants an insight into their skills and objectively audit course efficacy.
The following abstract was presented in the prize session of the European Society for Vascular Surgery (ESVS) Annual Meeting September 20 – 23, 2007 in Madrid, Spain
Experienced Endovascular Interventionalists Objectively Improve their Skills by Attending Carotid Artery Stent Training Courses
S. Neequaye, I. Van Herzeele, R. Aggarwal, M. Hamady, A. Darzi, T. Cleveland, P. Gaines, N. Cheshire
Department of Biosurgery and Surgical Technology, Imperial College London, U.K.
OBJECTIVES: Carotid artery stenting is an advanced endovascular intervention with a steep learning curve. Its success is tempered by the need to ensure that all interventionalists possess technical proficiency prior to performing cases on patients. Virtual reality simulation has been proposed as a means to train and objectively assess technical performance without risks to patient safety. The aim of this study was, using a commercially available simulator, to objectively assess psychomotor skills acquisition of experienced interventionalists attending a two-day carotid artery stent course.
METHODS: Eleven experienced endovascular physicians from several medical disciplines with minimal experience in carotid artery stenting participated in this study. During two identical two-day courses, cognitive and technical skills were trained using didactic sessions, case reviews, supervised VR simulation and live-cases. Pre- and post-course skills were assessed through performance on the same carotid artery stent procedure using the quantitative and qualitative assessment parameters derived from the simulator. The Wilcoxon signed ranks test was used to compare performance across the two groups.
RESULTS: Significant differences were noted between pre- and post-course performance for total procedure time (medians 36 vs. 20 minutes, p=0.005), fluoroscopic time (20 vs. 11 minutes, p= 0.016) and delivery-retrieval time of the embolic protection device (12 vs. 9 minutes, p= 0.007). Advancement of the guiding catheter or guiding sheath without a leading wire occurred to a greater extent pre-course versus post-course (199 vs. 152 millimetres, p= 0.050) as did spasm of the internal carotid artery (4 vs. 2, p= 0.049).
CONCLUSIONS: This study has objectively proven a benefit for experienced interventionalists to attend carotid artery stenting courses for skills acquisition measured by a virtual reality simulator. In addition to providing participants with an insight into their skills, these data can be used to objectively audit course efficacy.
Analysis of Simulated Angiographic Procedures. Part 2: Extracting Efficiency Data from Audio and Video Recordings
Duncan JR, Kline B, Glaiberman CB.
Mallinckrodt Institute of Radiology, Washington University School of Medicine, 510 S. Kingshighway Blvd., St. Louis, MO 63110, USA.
J Vasc Interv Radiol. 2007 Apr;18(4):535-44
PURPOSE: To create and test methods of extracting efficiency data from recordings of simulated renal stent procedures.
MATERIALS AND METHODS: Task analysis was performed and used to design a standardized testing protocol. Five experienced angiographers then performed 16 renal stent simulations using the Simbionix AngioMentor angiographic simulator. Audio and video recordings of these simulations were captured from multiple vantage points. The recordings were synchronized and compiled. A series of efficiency metrics (procedure time, contrast volume, and tool use) were then extracted from the recordings. The intraobserver and interobserver variability of these individual metrics was also assessed. The metrics were converted to costs and aggregated to determine the fixed and variable costs of a procedure segment or the entire procedure.
RESULTS: Task analysis and pilot testing led to a standardized testing protocol suitable for performance assessment. Task analysis also identified seven checkpoints that divided the renal stent simulations into six segments. Efficiency metrics for these different segments were extracted from the recordings and showed excellent intra- and interobserver correlations. Analysis of the individual and aggregated efficiency metrics demonstrated large differences between segments as well as between different angiographers. These differences persisted when efficiency was expressed as either total or variable costs.
CONCLUSIONS: Task analysis facilitated both protocol development and data analysis. Efficiency metrics were readily extracted from recordings of simulated procedures. Aggregating the metrics and dividing the procedure into segments revealed potential insights that could be easily overlooked because the simulator currently does not attempt to aggregate the metrics and only provides data derived from the entire procedure. The data indicate that analysis of simulated angiographic procedures will be a powerful method of assessing performance in interventional radiology.
The Use of Interventional Cardiovascular Simulation to Evaluate Operator Performance: The Carotid Assessment of Operator Performance by the Simbionix Carotid StEnting Simulator Study (ASSESS)
Giora Weisz, Jacque Devaud, Stephen Ramee, Mark Reisman, William Gray
Cardiovascular Research Foundation, and Center for Interventional Vascular Simulation, New-York Presbyterian Hospital, Columbia University, New York, NY
Journal of the Society for Simulation in Healthcare 2007, Volume2, Issue 1
BACKGROUND: The percutaneous endovascular carotid stenting procedure has only recently been approved by the FDA, and interventional operators from multiple disciplines (Cardiology, Radiology, Vascular Surgery) are getting training in this field. The professional societies of these disciplines are looking for novel ways to assess the competency of an operator, before they grant authorization to perform this complex procedure.
Simulators have been suggested as tools to evaluate operators in specific procedures and operative techniques. The virtual reality machines can be used to capture numerous factors that translate into the success and safety of the procedure. These include selection of devices, accuracy of manual performance, reaction to interactive problems, use of adjuvant pharmacotherapy, management of hemodynamic changes, and prevention and resolution of complications. All of these can be automatically evaluated and scored by a complex software designed virtual reality simulator. The simulators that are used in Interventional Cardiology and endovascular therapy have not been validated yet as such assessment tool.
OBJECTIVE: The main objective of the ASSESS study is to validate the simulator-based metrics as a tool to discriminate between different levels of operators.
METHODS: We used the Angio-Mentor carotid Stenting simulator system (Simbionix). The simulators had been loaded with predefined cases demonstrating increasing complexity for stenting intervention.
Total of 34 operators (10 experts, 12 intermediate level, and 12 novice; based on prior reported experience) were asked each to do three cases of carotid stenting on the simulator. All the operators had the same cases that were selected to demonstrate increased complexity to judge anatomy, correct performance of the technical steps, use of wires, catheters, embolic protection device, balloons, and stents, adjuvant pharmacotherapy, and management of interactive hemodynamic response. During the procedure, multiple features of all of the above elements were automatically captured by the simulator. Each step and parameter captured had a pre-defined score. Scoring was based on penalty points for mal-performance, as pre-defined by a panel of independent experts in carotid stenting, prior to conducting the study.
RESULTS: The study has been completed, and all the data that were captured is being analyzed. Full results will be available at the time of the presentation
CONCLUSIONS: Being able to demonstrate the ability of a hi-tech, virtual reality simulator to differentiate between different levels of interventional operators, may enlighten the simulator as a potential objective “examination tool”, to evaluate the performance and competence of an interventional operator. These will justify the use of Interventional Cardiovascular Simulation for certification and credentialing.