Establishing an Evaluation Technical Assistance Contract to Support Studies in Meeting the US Department of Health and Human Services Evidence Standards (Journal Article)

This special issue highlights the results of the Office of Adolescent Health’s (OAH) substantial investment in rigorous evaluations of teen pregnancy prevention (TPP) programs. Through a two-tiered funding strategy, OAH procured cooperative agreements with 94 grantees to replicate programs deemed evidence-based by the US Department of Health and Human Services’ (HHS) TPP evidence review (tier 1) or to implement promising and innovative TPP programs that did not yet have evidence of effectiveness (tier 2). In addition, the Family and Youth Services Bureau, under the Administration for Children and Families (ACF) at HHS, also funded 13 cooperative agreements to implement promising programs through the Personal Responsibility Education Program Innovative Strategies program. A subset of the cooperative agreements required the grantees to evaluate the effectiveness of their funded programs through random assignment or quasi-experimental impact studies led by independent evaluators. The goal of this investment in evaluation was to infuse the field with dozens of new, internally valid studies whose evidence would meet the rigorous research standards established by the HHS TPP evidence review and would inform the field of public health. These new findings would be used to further understand the effectiveness of evidence-based programs when implemented in different contexts and for different populations, and to potentially identify new, effective programs.

OAH’s investment is part of a larger federal effort to use and create evidence through tiered-evidence grant programs. As the government encourages and incentivizes rigorous evaluations, some large-scale federal grant programs provide evaluation technical assistance (TA) to their grantee-led evaluations, including the Investing in Innovation Grants (I3), administered by the US Department of Education; and the Workforce Innovation Fund Grants, administered by the Employment and Training Administration under the US Department of Labor. To support grantees in producing credible evidence of program effectiveness, OAH (with support from ACF) funded Mathematica Policy Research and its subcontractors to be the evaluation technical assistance contractor for the OAH and ACF grantee-led rigorous evaluations that were not part of federally led evaluations, with oversight by the OAH evaluation specialist.

Most of the federal evaluation TA efforts, OAH’s included, are structured around meeting a very specific goal: that completed studies meet a particular set of evidence standards for their field. For OAH, this is the HHS TPP evidence review standards, described below. These TA efforts have broader goals, as well. According to Gibbs et al., evaluation TA on individual studies can lead more broadly to improved capacity in the field to conduct evaluations. For example, the OAH evaluation TA effort can help the grantee organizations and their partner evaluators to lead and produce future evaluations that will meet rigorous evidence review standards, regardless of whether OAH is supporting them. Increasing the rigor of research in the field should lead to a better understanding of what works in field of public health.