Background: Immunotherapy has brought about a landmark change in anti-tumor treatment in the past years. High microsatellite instability (MSI-H) is now the only clinically approved biomarker predicting response to immunotherapy in CRC. Increasing evidence suggests that POLE mutations in the exonuclease domain could drive an ultra-mutational phenotype and improve the treatment outcomes of ICI in solid tumors. In this study, we set out to apply a deep learning model using H&E-stained, formalin-fixed, paraffin-embedded (FFPE) whole slide images (WSIs) of CRC primary tumors. Methods: The deep learning model is developed and validated through five-fold cross-validation using WSI of primary tumors from 506 CRC patients and externally validated using 52 WSIs from a prospective cohort. The microsatellite status, tumor mutation burden (TMB) and POLE genotype were determined by next-generation sequencing (NGS). Patients with MSS status and a low TMB (＜20Mutations/Mb) were admitted to the MSS group, and CRCs with a POLE mutation which was defined as an oncogenic mutation referring to the POLE functional mutation list at OncoKB(POLE (oncokb.org) were admitted to the POLE mutant group. Clustering-constrained-attention multiple-instance learning (CLAM) model is employed as the base model, and we conduct the model ensemble by performing a large-scale hyper-parameter search, selecting five models with the highest value in one of the performance metrics, including the AURoC, accuracy, precision, recall, and f1 score, and finally averaging the predictions of the five models. Results: The internal dataset included 237 MSS, 142 MSI-H, and 127 POLE mutant CRC. The three groups had significant differences in primary location (p ＜ 0.0001), histology (p ＜ 0.0001), tumor differentiation (p = 0.002), tumor stage (p ＜ 0.0001), Crohn's-like reaction (p ＜ 0.0001) and tumor growth pattern (p = 0.001). The cross-validation performance of the ensemble model (M_E) in the internal dataset achieves an AURoC of 0.944 for three-way classification task (POLE vs. MSI-H vs. MSS) and 0.940 for two-way classification task (POLE & MSI-H vs. MSS) which were superior to the performance of each single CLAM model. To demonstrate the generalizability of the deep learning model, a domestic perspective cohort consisting of 20 MSS, 17 MSI-H, and 15 POLE mutant CRC H&E images were used to validate the external performance. And the M_E retained robust performance on the external dataset, with an AURoC of 0.904 for three-way classification task and 0.836 for two-way classification task. Conclusions: A CLAM-based deep learning model could directly predict the MSI-H and POLE mutation from histological images that could be used to stratify CRC patients for immunotherapy with faster turnaround time and lower costs compared with traditional sequencing methods.