Background: Previous studies have shown that the presence or absence of genetic mutations is critical for colorectal cancer prognosis. However, genomic testing can be expensive and difficult to perform on all samples. In contrast, hematoxylin and eosin (H&E) staining is relatively inexpensive and can be performed on all tissue specimens. In this study, we designed a novel prognostic method using spatial image features extracted from H&E–stained whole slide images (WSIs) and genetic mutation prediction neural networks. Methods: We obtained H&E–stained WSIs and data on Microsatellite Instability (MSI), BRAF, TTN and APC gene mutations from a clinical cohort of 548 patients with The Cancer Genome Atlas (TCGA) Colon adenocarcinoma and rectum adenocarcinoma. We divided them into training (n=361), validation (n=90), and test (n=115) groups. Classification models were trained to predict the presence or absence of MSI, BRAF, TTN, and APC mutations. The model input comprised features of the H&E–stained WSIs, as obtained via a deep learning–based feature extractor. All resultant models were incorporated into a prognostic model (overall survival: > 60 months (low risk)/< 60 months (high risk)). Our prognostic model’s performance was evaluated against TCGA colorectal dataset, and a survival analysis was performed on the model using the Kaplan–Meier method. Finally, we compared our model’s performance with the end–to–end prognostic prediction of a convolutional neural network (CNN) that also used H&E–stained WSIs as input and provided prognostic prediction as output. Results: Our deep learning–based prognostic prediction model achieved an AUC score of 0.834 with a 95% confidence interval (CI) of 0.734–1.000 alongside TCGA dataset; the survival analysis compared the survival distributions of low–risk and high–risk groups, as predicted by our model; a p–value < 0.01 was obtained. The model could classify low– and high–risk patients and accurately predict patient status as alive (low risk) or deceased (high risk) at 60 months. In contrast, the CNN–based model achieved an AUC score of only 0.502 (95% CI: 0.315–0.690) on the same TCGA dataset, and the p–value obtained for it under the Kaplan–Meier log–rank test was greater than 0.5. The CNN–based method was unable to distinguish between low– and high–risk patients, confirming that our method using spatial imaging features extracted from WSIs was a more effective approach. Conclusions: We developed a novel prognostic prediction method using spatial image features extracted from WSIs and genetic mutation prediction neural networks. Our results demonstrated the advantage of using image features over gene mutation data for prognostic prediction in colorectal cancer patients.