Background: Although recent evidence suggests skeletal muscle depletion predicts the survival of patients with cancer, the retrieval and manual measurement of the computed tomography (CT) images hinder clinical application in routine clinical practice. The advent of recent deep learning applications enables highly accurate noninvasive longitudinal evaluation of skeletal muscle mass (SMM) changes. Here, we evaluated the prognostic impact of DNN-measured skeletal muscle changes in colorectal cancer (CRC) patients. Methods: A total of 6,196 newly diagnosed CRC patients were analyzed in the Yonsei Cancer Registry Database between Jan 1, 2010, and Sep 30, 2020. SMM is measured by the Skeletal muscle index (SMI). The formula used was: L3 skeletal muscle cross-sectional area (cm²)/height² (m²). Patients’ SMI patterns were grouped by difference ratio of initial and last SMI. Patients were also classified by BMI pattern with the result of K-means clustering. Association of baseline SMI, baseline body mass index (BMI), SMI changes, BMI changes, and demographic factors with overall survival (OS) were evaluated. Univariate and multivariate analyses were conducted. Concordance (c) statistics were used to test the predictive accuracy of survival models. Results: Fully automated UNet architecture-based deep learning algorithms were applied for the third lumbar transverse CT detection, skeletal muscle segmentation, and skeletal muscle area quantification in CRC patients undergoing abdominal CT between at the time of diagnosis and one year after the diagnosis. Baseline BMI distribution was 28% obese, 26% overweight, 42% normal weight, and 4% underweight. Patients in all SMI categories varied widely in BMI. Changes in SMI were categorized into three groups: SMI increase (33%), steady (45%), and decrease (22%) group. Similarly, BMI changes were categorized into three groups: BMI increase (24%), BMI stable (57%), and BMI decrease (19%) group. Low baseline SMI, low baseline BMI, SMI decrease, and BMI decrease were independently prognostic of survival. Intriguingly, BMI and SMI changes had a different prognostic impact in men and women. For women, the SMI increase group (hazard ratio [HR], 0.4; 95% CI, 0.3-0.7; P= 0.001) was associated with longer OS, while the SMI decrease group (HR, 1.2; 95% CI, 0.6-2.2; P= 0.619) was not associated with shorter OS, both compared with SMI steady group. Conclusions: Automated CT-derived SMM depletion had a negative prognostic impact independent of BMI and age in CRC patients. A noninvasive automatic deep learning algorithm provides a unique opportunity to apply to routine clinical practice and understand how and when cachexia impacts cancer prognosis.