For locally advanced rectal cancer (LARC), accurate response evaluation is necessary to select complete responders after neoadjuvant therapy (NAT) for a watch-and-wait (W&W) strategy. Algorithms based on deep learning have shown great value in medical image analyses. Here we used deep learning algorithms of endoscopic images for the assessment of NAT response in LARC.214 LARC patients were retrospectively included in the study. After NAT, these patients underwent total mesorectal excision (TME) surgery. Among them, 51 (23.8%) of the patients achieved a pathological complete response (pCR). 160 patients from Shanghai Changzheng Hospital were regarded as primary dataset, and the other 54 patients from Zhejiang Cancer Hospital were regarded as validation dataset. ResNet-18 and DenseNet-121 were applied to train the models based on endoscopic images after NAT. Deep learning models were valid in the validation dataset and compared to manual method.The performances were comparable in AUC between deep learning models and manual method. For mean metrics, sensitivity (0.750 vs. 0.417) and AUC (0.716 vs. 0.601) in ResNet-18 deep learning model were higher than those in the manual method. The deep learning models were able to identify the endoscopic features associated with NAT response by the heatmaps. A diagnostic flow diagram which integrated the deep learning model to assist the clinicians in making decisions for W&W strategy was constructed.We created deep learning models using endoscopic features for assessment of NAT in LARC. The deep learning models achieved modest accuracies and performed comparably to manual method.Copyright © 2023 Asian Surgical Association and Taiwan Robotic Surgery Association. Published by Elsevier B.V. All rights reserved.