Background: Patient eligibility for HER2-targeting treatments is commonly informed by testing tumor HER2 expression using immunohistochemistry. As HER2 expression is visually assessed by pathologists, inter- and intra-rater variability might affect treatment decisions. Here, we report the development of an automated machine learning (ML)-based algorithm to quantify HER2 cell membrane expression across a diversity of breast cancer phenotypes as a clinical tool for monitoring HER2 testing quality. Methods: A total of 689 breast cancer tissue samples were either procured (Avaden Biosciences) or were anonymized samples from the AstraZeneca biobank comprising tissues from primary and metastatic tumors, core needle biopsies and surgical resections, lobular and ductal carcinomas, across tumor grades and HER2 expression levels. Samples were stained for HER2 detection (Ventana HER2 (4B5) Assay) and digitized (Leica Biosystems) across 5 laboratories in the US. Whole-slide images (WSIs) were stratified into training (n = 407), validation (n = 110), and test sets (n = 172). Multiple convolutional neural network based ML models (PathAI, Boston, MA) were trained using 190,000 manual annotations provided by 30 board-certified pathologists to identify artifacts, invasive tumor, identify individual cancer cells and measure tumor cell membrane HER2 expression as partial or complete, and negative, weak-or-moderate, or intense. Cell-level scores were validated against a consensus of manual cell counts from 5 independent pathologists in 320 representative regions of test set WSIs. HER2 scores were generated by automatically applying rules derived from 2018 ASCO/CAP guidelines and then compared in the test set with consensus scores from 3 independent pathologists. Results: Cell counts provided by the ML model were strongly consistent with cell counts obtained by pathologist consensus in all cell-types except for faintly positive HER2 cells where ML-based quantification identified more cells on average. Automatically generated ML-ASCO/CAP HER2 scores using WSI showed substantial consistency across IHC categories with the consensus of pathologists (ICC 0.88, 95%CI 0.82-0.92) in the test set and improved further when ML models were trained to agree with pathologists by adjusting cut offs (ICC 0.91, 95%CI 0.89-0.94). The ML-based model was deployed through the PathAI cloud platform to calculate HER2 testing quality control metrics in real-time in multicentric clinical trials. Conclusions: Automated image analysis of HER2-stained breast cancer tissues using ML-based models is consistent with pathologist consensus across breast cancer tissue types. The results support evidence that ML-based algorithms can help pathologists assess HER2 testing reproducibility in clinical trials.