Aim We hypothesize that machine learning of histomorphological features can predict response to neoadjuvant therapy (NAT) in locally advanced rectal cancer (LARC). Method This retrospective study included 146 LARC patients who received NAT followed by surgery. The pathologists scanned the H&E slides of pretreatment tumor biopsy into whole slide images (WSIs). We randomly split patients into the primary and validation sets with a ratio of 80%:20%. We cut the WSIs into smaller parts (sample amount: 200-500) and used a convolutional neural network (CNN) to process these blocks directly. Then, a graph neural network (GNN) was applied to train the model in the primary set. The independent validation set was used to assess the performance of the model. Result Our model could provide indicative information to identify the patients who were most likely to benefit from NAT. When the sample amount reached 500, the tile-level classifier for distinguishing poor response from good response produced an AUC of 0.779 in the primary set and 0.733 in the validation set. Conclusion In this pilot study, we propose a novel predictive model of therapeutic response to NAT in LARC using a routine diagnostic tool employed in daily practice.