Endoscopic ultrasound (EUS) is a technically demanding procedure requiring the development of advanced endoscopic and ultrasound interpretation skills usually requiring dedicated fellowships to attain procedural competence. Trainees enter fellowships with varying prior experience presenting a challenge to trainers in determining their skill level as they embark on training on patients. Simulation-based training has the potential to accelerate the learning curve. However, to date there is no published evidence on the predictive ability of virtual reality (VR) simulators in determining real-world skillsets. An objective measure of a trainee’s technical skill at the start of fellowship would benefit trainers and potentially reduce the risk to patients. We report an evaluation of the predictive validity of endoscopist technical skills on the GI Mentor II EUS simulator (Surgical Science, Sweden) by comparing trainee performance on simulated and real patients.

A prospective observational study was conducted at a tertiary referral centre. Trainee endoscopists attending a nationally accredited basic skills in EUS course underwent a standardised induction on the EUS simulator before performing supervised EUS examinations on the simulator and then in two real patients. Performance was assessed using a previously validated direct observation of procedural skills (DOPS) form; the TEESAT. Trainees were rated by two trainers on a 1 – 4 Likert from 1 (“achieved without instruction”) to 4 (“unable to complete”) on 13 anatomical landmarks mandatory for a complete EUS examination. A 1–10 global performance score was also rated. Mean TEESAT scores were compared to evaluate predictive and construct validity and inter-rater reliability.

Seven trainees were enrolled (six male; age 30–50 years), most of whom had performed at least 50 EUS procedures. The mean TEESAT score for the 13 landmarks demonstrated excellent predictive validity, with simulator scores strongly correlating with real-patient performance (ρ = 0.96, p = 0.0003). The simulator demonstrated strong construct validity, with both simulator and patient landmark scores correlating closely with lifetime EUS experience (ρ −0.90 to −0.96). Inter-rater reliability ranged from good to excellent (ICC 0.62 overall; improving to 0.91 after exclusion of two outliers). Objective anatomical landmark components of TEESAT showed outstanding validity and acceptable-to-excellent reliability. The global 1–10 assessment showed no predictive validity for real-patient performance (Spearman ρ = 0.18, p = 0.70).

Trainee performance on the GI Mentor II EUS simulator utilising the TEESAT DOPS demonstrates excellent predictive and construct validity. It reliably discriminates skill levels and we would advocate its role in stratifying trainee competence at the outset of fellowships or as a surrogate marker of progress in early EUS training. The global score lacks utility and should be avoided as an assessment measure in this context.