Background: Clinical trials investigating JAK1/JAK2 inhibitors for myelofibrosis (MF) subjects have measured symptom improvement as a minimum 50% reduction in total symptom score (TSS) at the end of a 24-week treatment period. This landmark analysis is based on post-baseline score changes from Weeks 21 (W21) to 24 and requires vastly different absolute TSS improvements for patients with very high or low baseline (BL) TSS to reach responder status. A phase 3 clinical study, SIMPLIFY-1, randomized 432 intermediate and high risk JAK inhibitor (JAKi) naive MF patients 1:1 to momelotinib (MMB) or ruxolitinib (RUX). Non-inferiority on the MPN-SAF TSS response rate endpoint at W24 was not met (MMB: 28% vs RUX: 42%); however, improvement in each of the 7 TSS items was similar for MMB vs RUX. To understand the discrepancy, we applied item analysis and mixed effect models for repeated measures (MMRM) to SIMPLIFY-1. Methods: Analyses were conducted in the intention-to-treat (ITT) population and in a symptomatic subset (selected as subjects with BL TSS ≥ 10). The distributions of TSS items were examined at BL and shift in scores at W24 (health state shifts) were assessed. GEE models were used to estimate item-level odds ratios using multiple predictive imputations for missing data. MMRM compared mean change in TSS from BL to W24 using data from all visits. The meaningful change threshold (MCT) was determined using Patient Global Impression of Change. Results: BL scores across items were heterogenous in the MMB and RUX groups; the proportion of subjects with no or mild symptoms (0–3 on a 0-10 point scale) ranged from 44% (tiredness) to 81% (itching). Distributions of BL scores were different across arms with 6 out of 7 items in the MMB arm reporting more severe or very severe symptoms (scores of 7-10) at BL. Despite the imbalance in BL scores, item-level health state shifts showed similar improvements for MMB and RUX. Categorical responder analysis showed no significant differences on any items. Odds ratios for each between-group comparison ranged from 0.74 to 1.20. MMRM mean TSS change at W24 was 6.35 (MMB) vs 7.87 (RUX) in the ITT and 8.80 (MMB) vs 10.46 (RUX) in the symptomatic subset. Mean TSS were near the within-subject MCT of 8 points in the ITT and exceeded the MCT in the symptomatic subset. The between-group difference was 1.52 (95% CI: (0.196, 2.847)) in the ITT and 1.67 (95% CI: -0.134, 3.468) in the symptomatic subset. Conclusions: Comparable item health state shifts at W24 and similar improvements in mean TSS as shown by MMRM, with a minimal between-group difference of 1.52 on the 70 point scale in context of an 8-point MCT suggest MMB provides clinically relevant and comparable symptom improvements to RUX; these analyses require further validation in independent data sets. Imbalance in BL symptom scores in MMB subjects may have contributed to the inability to demonstrate non-inferiority in TSS response rate at W24. Clinical trial information: NCT01969838