Background: Use of immunotherapy has increased exponentially with survival benefit in many malignancies. As a result, suspected irAEs are commonly encountered, but lack of a gold standard for diagnosis puts patients at risk. We describe preliminary results of a QI project to improve accuracy of irAE diagnosis. Methods: Specialists at our institution created algorithms to define the likelihood of immune-related hepatitis, colitis, and pneumonitis based on clinical data, diagnostic results, and treatment response. Records of patients admitted from June-November 2018 with possible irAE were retrospectively reviewed by an oncology provider and an immunotoxicity specialist. A web-based tool automated irAE likelihood from clinical data based on the aglorithms. Reviewers could agree or disagree with the result and share their clinical impression. We evaluated concordance between providers and between provider and algorithm. Results: Of 65 patients, most had non-small-cell lung cancer (29%), melanoma (15%), or breast cancer (9%) and were treated with pembrolizumab (48%), nivolumab (23%), or ipilimumab/nivolumab (22%). There were 4 irAE-related deaths and 40 patients (71%) discontinued therapy for presumed toxicity. After 14 cases were excluded due to incomplete information, algorithms were applied to the remaining 51 cases (19 colitis, 8 hepatitis, 24 pneumonitis). The algorithm generated the likelihood of an irAE in 63% of reviews. Overall agreement with the algorithm was 86% (84-100% for individual algorithms), Kappa 0.8 (0.7-1.0). Overall concordance between reviewers was 76% (75-88%), Kappa of 0.7 (0.6- 0.8). Conclusions: While algorithm definitions and provider impressions were fairly consistent, cases lacking a generated likelihood suggest that some clinical scenarios are not addressed. Different interpretation of clinical information or varying experience with irAEs may explain inter-reviewer discordance, which underscores the importance of applying a standardized approach to irAE diagnosis across settings and experience levels. In the next improvement cycle, the algorithm will be refined to broaden algorithm scope, clarify criteria, and improve accuracy.