Background: Improvements in technology can prompt device changes during clinical studies. ON TRAX [NCT04381494] was a prospective pilot study aimed to evaluate the potential benefit of using a multiparametric remote monitoring system to detect early signs of pneumonitis in patients with Stage III NSCLC receiving standard of care durvalumab post-chemoradiotherapy. Here we report learnings from this pilot based on post-hoc analyses comparing changes in device adherence, study retention and user experience after upgrading the wearable device mid-study. Methods: Patients were monitored remotely while on durvalumab, for up to 1 year, with an FDA-cleared system (Current Health) that recorded vital signs continuously (SpO2, respiratory rate, pulse, temperature and motion). During the study, the wearable was upgraded from Generation One (G1) to Two (G2), a smaller and lighter (30 vs 115 grams) device with same capabilities. We compared daily device adherence (defined as median number of hours with wearable data transmission per 24h expressed as %) and total study participation time (days) for G1 vs G2 users by Wilcoxon Rank-Sum test. At end-of-study visit, some patients rated on a 5-point Likert scale three statements regarding comfort of wearable, intrusiveness, and overall experience. Results: Following IRB approvals, 40 patients from 16 US sites were enrolled, but only 39 patients with minimum of 1h of wearable data transmitted were included in this analysis. Nine patients used only G1, 22 only G2, while 8 used both. Collectively, the 39 patients transmitted 76,957 hours of data. The median (interquartile range) length of participation was 84 (47-208) days. Transitioning from G1 to G2 did not affect daily device adherence (G1: 79 (46-96) vs G2: 75 (58-100) %, p = 0.9), but patients using G2 exclusively participated 34 days longer than those using G1 exclusively (47 (2-57) vs 81 (50-161) days, p = 0.03). Seventeen patients responded to the user experience survey, of which 13 used only one device (G1 = 3, G2 = 10). Ten of 10 (100%) G2 users ‘agreed’/’strongly agreed’ that the wearable was comfortable, while 2 out of 3 of G1 users ‘disagreed’. Both groups ‘agreed’/’strongly agreed’ it did not impede their daily activities. Overall experience was ‘good’/’excellent’ for 9 of 10 (90%) G2 users, but varied in G1 users (1 ‘good’, 1 ‘neutral’, 1 ‘poor’). Conclusions: Following the introduction of a smaller and lighter wearable device, patients remained a median 34 days longer in this digital health pilot study and reported greater comfort and better overall experience, however the pilot was terminated early. In light of technological updates, researchers must balance consistency with data yield and patient experience. An interim review of technology and its implementation into study protocols is recommended. Clinical trial information: NCT04381494.