Background: Persistent inflammatory diseases such as ulcerative colitis (UC) and Crohn’s disease (CD) contribute to the tumor initiation and progression in colitis-associated cancers (CAC). Intestinal damage caused by persistent inflammation promotes clonal expansion of somatic and epithelial cells and alters the immune microenvironment. A comprehensive investigation in comparing the tumor immune environment in CAC with sporadic colorectal cancers (CRC) will provide more clinical usefulness of immunoprofiling in CRC. Methods: Thirteen patients with CAC (6 with UC and 7 with CD) and 10 patients with sporadic colorectal cancer from our hospital were enrolled in the study. FFPE samples from patients were analyzed for the tumor infiltrating lymphocytes [PD-L1+, CD3+, CD4+, CD8+, CD20+, CD68+, CD163+] by multiplex immunofluorescence technique. The core and peritumor areas annotated by pathologists on images. Then artificial intelligence (AI) algorithms delineated the tumor-enriched segments and stroma, and calculated the density and percentage of each marker in two regions. A nonparametric approach (Kruskal–Wallis) was used to compare the density and percentage of immune cells between CAC and sporadic colorectal cancer groups. Results: In the stroma area, the density (counts per mm²) of PD-L1+CD68+ cells in patients with CAC were significantly higher than those in sporadic cancer patients (p < 0.05). However, the density of PD-L1+CD68+ did not show significant differences related to disease types (CAC or sporadic CRC) in the tumor-enriched segments. Significantly, the density of CD3+CD4+ cells and the percentage of CD68+CD163- cells in patients with CAC showed a trend of higher density than those in sporadic cancer patients in the stroma area (all, 0.05 < p < 0.1). The density and the percentage of CD20+ cells in CAC patients had a higher density tendency than those in sporadic cancer patients both in the tumor-enriched segments and stroma (all, 0.05 < p < 0.1). Other indicators in tumor-enriched segments and stroma, such as CD3+, CD4+, CD8+, CD68+CD163+ and PD-L1+, had no significant difference between CAC and sporadic CRC patients, regardless of the percentage or the density. Conclusions: We identified that the tumor immune environment of CAC was different from that of sporadic CRC. Some tumor infiltrating lymphocytes such as PD-L1+CD68+ cells infiltrated more in CAC than those in sporadic cancer, suggesting that CAC may respond better to immunotherapy than sporadic CRC. However, this hypothesis requires a further validation with a larger sample size.