Identifying students at risk of not meeting academic goals, such as graduation or being college ready, ranks high on the list of school and district priorities. Many districts use early warning systems to identify students who are struggling, to intervene early and provide support to get them back on track. These systems, described in this infographic, typically use academic measures, such as credits earned, to identify students who need support.[1]
Relying exclusively on academic measures to identify at-risk students may miss other determinants of success, such as social and emotional learning (SEL) competencies and school climate.[2] SEL competencies—such as how well students persevere, manage their thoughts and emotions, and understand what others think and feel—are highly related to student outcomes, including achievement, educational attainment, health, earnings, and employment.[3] Evidence also suggests that fostering a positive school climate—the tangible and intangible attributes of a school that support students’ development—can boost SEL competencies. These attributes include relationships among students and staff, school discipline, student engagement, and safety.[4]
At the same time, recent evidence also suggests that the academic measures used in early warning systems may already capture the outcomes of key SEL competencies and school experiences, so—for the particular purpose of identifying students at risk of poor academic outcomes—adding these additional measures may not be necessary.[5], [6] For example, students with higher levels of perseverance are more likely to attend school. Therefore, the reverse is also true—students with higher annual attendance rates are also more likely to have higher levels of perseverance. In other words, their attendance rate captures perseverance to some degree. If the academic measures that schools use to predict outcomes adequately capture SEL competencies and school experiences, then adding measures of the SEL competencies and school experiences to early warning systems would be unnecessary. Using the academic measures alone would allow districts to (1) have simpler predictive models and (2) rely on academic measures they already collect. Yet, previous research has not explored what happens when SEL competencies and school climate measures are added to early warning systems and whether doing so improves these systems’ ability to identify students who are at risk of poor outcomes.
REL Mid-Atlantic partnered with the District of Columbia Public Schools (DCPS) to explore the benefits of incorporating data on SEL competency and school climate into early warning systems. This study drew on survey data that DCPS collects annually to track students’ SEL competencies and experiences with their schools’ climate.[7], [8] Together we examined whether adding these data to models that use academic measures improved predictions of future outcomes, including whether students were college ready based on their English language arts (ELA) achievement two years later.
As shown in the figure below (and described in the study’s report), the analyses revealed that SEL competencies and school experiences are related to academic outcomes but, when included alongside academic measures, they add little accuracy in classifying whether students are at risk of poor future outcomes. For example, models that included SEL competencies and school experiences alone accurately identified students who were not college ready in ELA 64.9 percent of the time. For models with academic measures alone, the corresponding accuracy was 86.3 percent. Adding SEL competencies and school experiences to the academic measures increased the accuracy by less than 0.5 percent (comparing the second and third bars in the figure). The study suggests that districts may not need to incorporate data on SEL competency and school climate into their early warning systems because academic measures already capture these important dimensions.
Data on student SEL competencies and school experiences are related to whether students are college ready in ELA but do not add much accuracy in classifying college readiness beyond academic variables.
ELA is English language arts. SEL is social and emotional learning.
Note: The figure shows the accuracy of predictive models in classifying whether students were not college ready in ELA two years later based on different groups of predictors. The SEL competencies included measures of perseverance, self-management, self-efficacy, and social awareness. The school experiences included measures of rigorous expectations, sense of belonging, and student satisfaction. The academic measures included math and ELA standardized test scores, attendance rate, and number of suspensions.
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 and 2019/20.
Despite these findings, collecting data on SEL competencies and school climate may help districts support students, as these measures can shed light on the root causes of risk that educators could not identify from academic measures alone. Academic measures are predictive of academic outcomes but do not allow districts to distinguish between students’ different SEL competencies and school experiences, which could be valuable when supporting students. For example, after identifying students at risk of poor academic outcomes, districts can use individual measures on SEL competencies and school experiences to determine on which dimensions to focus. These factors also relate to students’ later academic outcomes, so improving them may help districts make progress toward their strategic goals, like boosting on-time graduation.
Footnotes
[1] Allensworth & Easton, 2005; Bowen et al., 2009; Easton et al., 2017; Geiser & Santelices, 2007.
[2] Borghans et al., 2016.
[3] Heckman & Kautz, 2012; O’Conner et al., 2017.
[4] SRI International, 2018.
[5] Jackson, 2018; Kautz & Zanoni, 2018; West et al., 2016.
[6] When referring to an individual student’s report of school climate, the study team uses the term school experiences.
[7] The survey, developed by Panorama Education, was first fielded in spring 2018, and measures four SEL competencies (perseverance, self-management, self-efficacy, and social awareness) and three measures of a school’s climate that could boost SEL competencies (rigorous expectations, sense of belonging, and student satisfaction).
[8] DCPS highlighted the importance of SEL competencies in its 2017-2022 Strategic Plan, a proposal that DCPS uses to outline key goals and hold itself accountable to the public. The data on SEL competency and school experiences allow the district to track progress toward these goals.
References
Allensworth, E. M., & Easton, J. Q. (2005). The on-track indicator as a predictor of high school graduation. Consortium on Chicago School Research, University of Chicago. https://consortium.uchicago.edu/sites/default/files/2018-10/p78.pdf
Borghans, L., Golsteyn, Bart H. H., Heckman, J. J., & Humphries, J. E. (2016). What grades and achievement tests measure. Proceedings of the National Academy of Sciences, 113(47), 13354–13359.
Bowen, W. G., Chingos, M. M., & McPherson, M. S. (2009). Test scores and high school grades as predictors. In W. G. Bowen, M. M., Chingos, & M. S., McPherson (Eds.), Crossing the finish line: Completing college at America’s public universities (pp. 112–133). Princeton University Press.
Easton, J. Q., Johnson, E., & Sartain, L. (2017). The predictive power of ninth-grade GPA. University of Chicago Consortium on School Research.
Geiser, S., & Santelices, M. V. (2007). Validity of high-school grades in predicting student success beyond the freshman year. University of California, Berkeley, Center for Studies in Higher Education. Research and Occasional Paper Series. https://eric.ed.gov/?&id=ED502858
Heckman, J. J., & Kautz, T. (2012). Hard evidence on soft skills. Labour Economics, 19(4), 451–464.
Jackson, C. K. (2018). What do test scores miss? The importance of teacher effects on non-test score outcomes. Journal of Political Economy, 126(5) (pp. 2072-107). https://www.scholars.northwestern.edu/en/publications/what-do-test-scores-miss-the-importance-of-teacher-effects-on-non
Kautz, T., & Zanoni, W. (2018). Measurement and development of non-cognitive skills in adolescence: Evidence from Chicago Public Schools and the OneGoal Program [Unpublished manuscript]. Mathematica Policy Research.
O’Conner, R., De Feyter, J., Carr, A., Luo, J. L., & Romm, H. (2017). A review of the literature on social and emotional learning for students ages 3–8: Characteristics of effective social and emotional learning programs (Part 1 of 4) (REL 2017–245). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic. https://eric.ed.gov/?&id=ED572721
SRI International. (2018). Promoting grit, tenacity, and perseverance: Critical factors for success in the 21st century. https://www.sri.com/wp-content/uploads/pdf/promoting-grit-tenacity-and-perseverance-critical-factors-success-21st-century.pdf
West, M. R., Kraft, M. A., Finn, A. S., Martin, R. E., Duckworth, A. L., Gabrieli, C. F. O., & Gabrieli, J. D. E. (2016). Promise and paradox: Measuring students’ non-cognitive skills and the impact of schooling. Educational Evaluation and Policy Analysis, 38(1), 148–170. https://eric.ed.gov/?&id=EJ1089915
Cross-posted from the REL Mid-Atlantic website.
