Background: We examined changes in the tumor immune microenvironment during neoadjuvant chemotherapy by comparing immune gene mRNA expression, tumor infiltrating lymphocyte (TIL) count and PD-L1 protein expression in pre- and post-treatment tissues. Methods: Paired pre- and post- treatment tumor samples from 60 patients were profiled using the Nanostring Immune Oncology 360 platform to measure the expression of 770 immune-related genes that also allowed us to test 14 immune cell type and 27 previously published prognostic and immuno therapy response predictive gene signatures. All samples were also assessed for TIL counts and PD-L1 protein expression by immunohistochemistry. Gene expression levels were compared by paired t-test with Bonferroni correction. TIL count, PD-L1 protein expression and immune gene signatures were compared using Wilcoxon signed-rank test. Baseline immune markers were correlated with pathologic complete response (pCR) using estrogen receptor (ER) and treatment adjusted logistic regression. Results: TIL counts were significantly lower in post- compared to pre-treatment samples (17% vs 7%, p = 0.013) but stromal PD-L1 protein expression was not significantly different. At baseline, higher TIL count and PD-L1 expression were associated with pCR. High expression of a mast cell metagene was associated with residual disease (RD). No individual genes or VEGF gene signature were associated with benefit from bevacizumab. In patients with RD (n = 45), genes involved with tissue repair and inflammation (DUSP1, EGR1, IL6, ATF3, CD36, CXCL2, CD69, NGFR, KLF2, THBD, DAB2) showed significantly higher expression after therapy while most other immune markers decreased. The T effector, T-reg, MHC-I, MHC-II, IFNgamma, STAT-1 and M1 macrophage gene signatures were all significantly lower in post- versus pre-treatment samples and only the IL8/VEGFR and T-helper gene signatures showed higher expression after therapy. Conclusions: High mast cell gene expression is associated with lower pCR rate. TIL counts and most immune parameters decrease after neoadjuvant chemotherapy.