Background: The immune checkpoint inhibitors (ICIs) confer a risk of unique inflammatory immune-related adverse events (irAEs), which are highly distinct from the adverse events historically observed with cytotoxic therapy. To develop a strategy for easier identification and mitigation of irAEs, we sought to understand the frequency of ED visits and hospitalizations in 90 days following ICI start by implementing a rapid learning system (RLS). Methods: We convened an Immunotherapy Toxicities Management Committee with representatives from the Center for Immuno-Oncology, Quality and Patient Safety, and Informatics to draft a series of recommendations for the development of an irAE rapid learning system. The Committee requested an audit of all irAEs between June 2015 and April 2018 by drug, event type (clinical diagnosis), outcomes (including ED visits, hospitalization with length of stay, and death), and time-frame (days since beginning ICI course). An automated pipeline was created to merge structured data from the Electronic Health Record (Epic) and billing system (EPSi). This data was used to design a tool which consisted of an automated dashboard to monitor patient and enable interventions. Results: Over the course of 3 years, a total of 2,020 unique patients receiving ICIs were seen. 918 were treated with Pembrolizumab (45.4%), 768 with Nivolumab (38.0%), 234 with Atezolizumab (11.6%), 111 with Nivolumab & Ipilmumab (5.6%), 68 with Ipilimumab (3.4%), 9 with Durvalumab (0.4%), and 9 with Avelumab (0.4%). ED visits and hospitalization rates over 90 days were similar among the three most prescribed therapies, ranging from 332 unique patient events in the Pembrolizumab Cohort (36.1%) to 96 unique patient events in the Atezolizumab cohort (41.0%). Conclusions: The dashboard is effective tool to build a RLS for irAEs. The immediate output of this tool is using natural language processing (NLP) to distinguish between irAEs related ED visits and hospitalizations or regular disease progression, and measure the impact of interventions including (a) developing standardized algorithms for monitoring for irAEs, (b) designing an educational program for providers, and (c) developing an inpatient and outpatient immunotherapy toxicity management service.