The automation of procedure analysis for endoscopic interventions is a prerequisite for the advancement towards automatic reporting, quality assessment and advanced operator assistence systems. A critical aspect of procedure characteristics is the choice and utilization of endoscopic instruments. The endoscopic submucosal dissection (ESD) is a complex resection technique with a slow learning curve and relevant operator dependent risks for procedure related complications such as bleeding and perforation. During ESD, specific instruments such as electrosurgical knives of different shapes or hemostatic forceps are used. The analysis of the utilized endoscopic instruments may facilitate a more profound understanding of procedure characteristics. Therefore, we aimed to develop an algorithm for the automatic detection and delineation of endoscopic instruments during ESD. 

65 full length ESD videos (15x rectal, 29x esophageal, 16x gastric, 5x duodenal ESD) were extracted from Augsburg University Hospital database. 8.165 single images from these procedures were used for training. On these images, shaft and tip of endoscopic instruments were delineated by two study investigators using Computer Vision and Annotation Tool. Classes included Hook knife, Dual knife (both Olympus, Tokyo, Japan), Flush knife (Fujifilm, Tokyo, Japan), injection needle, hemostatic forceps, hemostatic clip and transparent distal attachment. Annotated images were used for training of a DeepLabV3+ deep learning algorithm. 5-fold internal cross validation was performed on pixel basis, as well as for instrument detection per image (sensitivity, specificity, F1 score). For the instrument detection, a threshhold of 0.2 for the intersection over union of algorithm prediction and ground truth was used. 

Internal validation results on pixel basis are shown in the table. For the detection task, F1 scores of 1.00, 0.90, 0.88, 0.65, 0.45, 0.68 and 0.99 were measured for Hook knife, Dual knife, Flush knife, needle, hemostatic forceps, hemostatic clip and distal attachment. 

In this exploratory study, endoscopic instruments were detected and delineated with varying performance by a deep learning algorithm. This was in part attributed to class imbalances in the training data. However, the incorporation of temporal information may also improve performance. Further research should focus on evaluating this new technology for automatic reporting of ESD procedures.