Are Ratings from Tiered Quality Rating and Improvement Systems Valid Measures of Program Quality? A Synthesis of Validation Studies from Race to the Top—Early Learning Challenge States
Evidence | Publications

Are Ratings from Tiered Quality Rating and Improvement Systems Valid Measures of Program Quality? A Synthesis of Validation Studies from Race to the Top—Early Learning Challenge States

Published: Apr 29, 2019
Publisher: Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education
Human Services Education School Reform Early Childhood Child Development Early Childhood Systems Quality Measurement
Download
Full Publication
Associated Project

Evaluating Race to the Top and School Improvement Grants

Time frame: 2010-2018

Prepared for:

U.S. Department of Education, Institute of Education Sciences

Authors

Lindsay Fox

Moira McCullough

Pia Caronongan

Mariesa Herrmann

Key Findings
  • All nine states used external measures of quality to examine the validity of their ratings; eight used an independently collected measure of program quality and eight used at least one measure of children’s outcomes.
  • The ratings distinguished between programs with differing quality; higher-rated programs had higher scores on independent measures of quality. However, the overall level of quality for higher-rated programs could not be described as high based on these independent measures.
  • The ratings were not related to differences in children’s outcomes; children who attended higher-rated programs did not have better developmental outcomes than those attending lower-rated ones.
  • Researchers from all nine states reported that their non-experimental designs limited the interpretation of findings. In addition, most researchers perceived recruiting child care providers for the studies and attaining sufficient representation across the rating levels as the most challenging aspects of the studies.

The usefulness of quality ratings for early learning and development programs depends on how accurately they measure programs’ quality, that is, their validity. To inform states’ continued development of tiered quality rating and improvement systems (TQRIS) and future validation studies, this report synthesizes findings from validation studies conducted by nine states that received Race to the Top—Early Learning Challenge (RTT-ELC) grants. It also describes the challenges that researchers faced when conducting these studies.

How do you apply evidence?

Take our quick four-question survey to help us curate evidence and insights that serve you.

Take our survey