The Reliability and Validity of Lung Cancer and Melanoma Clinical Quality Survival Measures

Objective

To develop a risk adjustment approach and test reliability and validity for oncology survival measures.

Data Sources and Study Setting

We used the National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER)-Medicare data from 2010 to 2013, with mortality data through 2015.

Study Design

We developed 2-year risk-standardized survival rates (RSSR) for melanoma, non-small cell lung cancer (NSCLC), and small cell lung cancer (SCLC). Patients were attributed to group practices based on the plurality of visits. We identified the risk-adjustment variables via bootstrap and calculated the RSSRs. Reliability was tested via three approaches: (1) signal-to-noise ratio (SNR) reliability, (2) split-half, and (3) test–retest using bootstrap. We tested known group validity by stage at diagnosis using Cohen's d.

Data Collection/Extraction Methods

We selected all patients enrolled in Medicare and linked to SEER during the measurement period with an incident first primary diagnosis of stage I–IV melanoma, NSCLC, or SCLC. We excluded patients with missing data on month and/or stage of diagnosis.

Principal Findings

Results are based on patients with melanoma (n = 4344); NSCLC (n = 16,080); and SCLC (n = 2807) diagnosed between 2012 and 2013. The median (interquartile range) for the RSSRs at the group practice-level were 0.89 (0.83–0.87) for melanoma, 0.37 (0.30–0.43) for NSCLC, and 0.19 (0.11–0.25) for SCLC. C-statistics for the models ranged from 0.725 to 0.825. The reliability varied by approach with median SNR 0.20, 0.25, and 0.13; median test–retest 0.59, 0.57, and 0.56; median split-half reliability 0.21, 0.29, and 0.29 for melanoma, NSCLC, and SCLC, respectively. Cohen's d for stage I-IIIa and IIIb+ was 1.27, 0.86, 0.60 for melanoma, NSCLC, and SCLC, respectively.

Conclusions

Our results suggest that these cancer survival measures demonstrated adequate test–retest reliability and expected findings for the known-group validity analysis. If data limitations and feasibility challenges can be addressed, implementation of these quality measures may provide a survival metric used for oncology quality improvement efforts.