Background: In recent years, the use of tumour molecular profiling within the clinic has allowed for more accurate cancer diagnostics, as well as the delivery of precision oncology. Rapid advances in digital histopathology have allowed the extraction of clinically relevant information embedded in tumor slides by applying machine learning methods, capitalizing on recent advancements in image analysis via deep learning. However, as in many supervised learning approaches, predicting response to therapy using whole slide images (WSI) of tissue stained with hematoxylin and eosin (H&E) requires large datasets comprising matched imaging and response data, severely restricting the applicability of this approach. Methods: To overcome this critical challenge, we introduce here for the first time a generic methodology for generating WSI-based predictors of patients’ response for a broad range of cancer types and therapies, which does not require matched WSI and response data for training. The approach, termed ENLIGHT-DeepPT, consists of: (1) DeepPT, a new deep-learning framework that predicts exome-wide tumor mRNA expression from slides, and (2) ENLIGHT, a recently published response prediction algorithm, applied here to predict response based on the DeepPT predicted expression values, instead of measured values. Results: First, we study the ability to predict tumor expression, showing that DeepPT successfully predicts transcriptomics in all 16 TCGA cohorts tested and generalizes well to two independent datasets. Second, aligned with our key aim, we analyze six independent clinical trial datasets of patients with different cancer types that were treated with various targeted and immune therapies, on which DeepPT was never trained. We show that ENLIGHT, without any adaptation, can successfully predict the true responders from the expression values imputed by DeepPT, using only H&E images: ENLIGHT-DeepPT successfully predicts true responders in these cohorts with an overall odds ratio of 2.19, increasing the baseline response rate by 38.8% on average among predicted responders. Conclusions: ENLIGHT-DeepPT is the first approach to successfully predict response to multiple targeted and immune cancer therapies directly from H&E slides, without requiring any drug-specific training data. ENLIGHT-DeepPT can provide rapid treatment recommendations without requiring tumor sequencing, bringing precision oncology to low-income countries and other situations where NGS is prohibitive.