Promoting Readiness of Minors in Supplemental Security Income (PROMISE): California PROMISE Process Analysis Report
Download
Associated Project
Evaluation of the Promoting the Readiness of Minors in Supplemental Security Income PROMISE Grants
Prepared for:
Social Security Administration
Key Findings
- The California Department of Rehabilitation (CDOR) was the lead agency for CaPROMISE and the recipient of the cooperative agreement with ED. Representatives from five other state agencies served on a steering committee that supported and worked collaboratively with the program. CDOR contracted with 18 local sites and the Interwork Institute at San Diego State University to implement CaPROMISE. All but one of the local sites were local education agencies; the remaining site was run by a nonprofit organization for a consortium of three adjacent local education agencies. The local sites recruited youth and their families into the evaluation of CaPROMISE and then provided program services to those assigned to the treatment group. The Interwork Institute subcontracted with and managed 16 family resource centers, which provided additional program services to treatment group parents and guardians, and provided technical assistance and training to all program staff. A few months after service delivery began, CDOR established additional contracts with five state universities to hire students as interns to provide administrative support for program operations and direct services to youth and their families. In the second year of program operations, CDOR contracted with four independent living centers to provide youth with training on independent living skills and hired 10 qualified rehabilitation professionals to provide them with employment services.
- CaPROMISE enrolled 3,373 youth in the evaluation of the program, 1,646 of whom were assigned to the treatment group. Three years into program operations, CaPROMISE had engaged 93 percent of treatment group youth as participants in the program. CaPROMISE delivered intensive case management services to youth and their families, consistent with its program design. In addition to case management, CaPROMISE offered (1) benefits counseling and financial education services; (2) career exploration and work-based learning experiences; (3) parent training and information; (4) education services; and (5) other services, such as training on independent living. The program’s management information system did not distinguish program contacts in which program staff discussed services with youth and families from ones in which they delivered services, which precluded the evaluation from measuring service receipt. That said, the number of program contacts related to services was generally high. As of August 2017, the program was very close to being on track to meet its benchmark of placing all youth in a paid work experience by the end of the program; after three years of program operations, 56 percent of participating youth had a paid work experience. Employers paid the youths’ wages for about one-quarter of these experience and CaPROMISE paid wages for the remainder.
- Even before CaPROMISE began, CDOR had relatively strong connections with the state agencies and local contractors that subsequently became partners in the program. Social network survey results indicate that communication frequency and positive views of working relationships among state agency partners increased during the early implementation of CaPROMISE but then declined to pre-implementation levels during late implementation. In the two local sites where the survey was conducted, communication frequency and positive views of working relationships during early implementation were lower among administrators of local level service partners than among state agency partners. Unlike among state agency partners, however, communication frequency and positive views of working relationships among administrators of local level service partners in these sites increased from early to late implementation. As CaPROMISE matured, frontline staff in these two sites communicated with their counterparts at partner organizations with increasing frequency.
- The process analysis suggests that conditions during the operation of CaPROMISE were favorable for finding positive impacts of the program in the later phases of this evaluation. Evidence in three areas implies marked differences in service experiences between treatment and control group youth. First, a large share (93 percent) of treatment group youth actually participated in the program, and most of them had received program contacts within key service areas as well as at least one work experience three years into program operations. Second, control group youth had only limited access to services similar to the intensive case management and employment services treatment group youth could receive through CaPROMISE. Third, there is little risk that control group youth received CaPROMISE services.
PROMISE—Promoting Readiness of Minors in Supplemental Security Income (SSI)—was a joint initiative of the U.S. Department of Education (ED), the Social Security Administration (SSA), the U.S. Department of Health and Human Services, and the U.S. Department of Labor to fund and evaluate programs to promote positive changes in the lives of youth who were receiving SSI and their families. Under cooperative agreements with ED, six entities across 11 states enrolled SSI youth ages 14 through 16 and implemented demonstration programs intended to (1) provide educational, vocational, and other services to youth and their families and (2) make better use of existing resources by improving service coordination among state and local agencies. Under contract to SSA, Mathematica Policy Research is evaluating how the programs were implemented and operated, their impacts on SSI payments and education and employment outcomes for youth and their families (using an experimental design under which we randomly assigned youth to treatment or control groups), and their cost-effectiveness.
In this report, we present findings from the process analysis of the first three years of the implementation and operation of the California PROMISE (CaPROMISE) program. The findings are based on data collected through August 2017 via site visits to CaPROMISE, telephone interviews with and social network surveys of program administrators and staff, and the management information system that the program’s staff used to record their efforts. The report describes how CaPROMISE engaged with youth, the services the program provided to them and their families, and the collaborations the program fostered to support its efforts. It also highlights information about the experiences of control group youth that could have implications for the evaluation’s impact analysis.
More like this from Mathematica
How do you apply evidence?
Take our quick four-question survey to help us curate evidence and insights that serve you.
Take our survey