Background: Many breast cancer patients receiving neo-adjuvant chemotherapy (NAC) will ultimately fail to achieve pathological complete response (pCR). A pre-treatment clinical marker of pCR could guide NAC without requiring potentially ineffective initial treatment periods. Advances in medical image analysis, such as deep learning (pattern recognition using neural networks) and radiomics (computer-extracted quantitative image features), demonstrate significant potential for non-invasive assessment of NAC outcome. We present a machine learning (ML) approach for pre-NAC response prediction fusing deep learning, radiomics, and clinical variables. Methods: 166 patients with pre-treatment contrast-enhanced MRI from the ISPY1-TRIAL and surgically-confirmed NAC response outcome (ypT0N0, 49 pCR, 117 non-pCR) following anthracycline-cyclophosphamide chemotherapy with or without taxane were retrospectively analyzed. Patients were divided randomly into training (n = 133) and testing (n = 33) cohorts, with proportionate distribution of pCR between cohorts. Multinomial logistic regression integrated DL, radiomics, and HER2 status was assessed by area under the receiver operating characteristic curve (AUC), sensitivity, and specificity within the testing set. Deep learning: A six-layer convolutional neural network was trained to predict response using 65 pixel square patches centered within a tumor. Radiomics: From a pool of 215 radiomic intra-tumoral heterogeneity features, the 8 best-performing features were identified algorithmically and used to train a linear discriminant analysis classifier within the training set. Results: A ML only approach strongly predicted response (AUC = .84) in the testing set. Integrating clinical HER2 status into the combined imaging model yielded optimal pre-treatment response prediction (AUC = .93), with 75% sensitivity and 92% specificity. Conclusions: Further validation of an approach fusing deep learning, radiomic analysis, and clinical information could potentially provide a means of pre-NAC response prediction from MRI.