No-Deadlines Synthetic Control and Exploratory Outcomes Analysis and Recommendations for a Future Rigorous Evaluation

We find that a synthetic control method is a promising approach for examining the impacts of NDL for programs with enough appropriate counterfactual programs (in other words, programs in directorates in which only a subset of programs switched to NDL and for which a weighted average of outcomes for non-NDL programs looks like NDL programs in the years before NDL implementation). We conduct an exploratory outcomes analysis of two NDL programs in the Directorate for Geosciences (GEO) (out of 12 NDL programs in our sample) and find that NDL substantially reduced the number of proposals received (by about 120 to 130 proposals). We also see some evidence that NDL may have increased the amount of requested funding, decreased the number of reviewers, and decreased the percentage of proposals rated by reviewers to be fair or underperforming for these two NDL programs. These findings are exploratory and not generalizable to other programs or directorates.

We also discuss how considering different analytic decisions and resolving certain data issues could improve a future rigorous evaluation of an NDL approach. For example, future research is needed to determine the advantages and disadvantages of conducting the analysis at different time intervals (for example, annual versus quarterly), how to treat programs that switch to NDL in the middle of a fiscal year, and the extent to which principal investigators (PIs) may anticipate NDL. This analysis could also be improved by better understanding potential data quality issues for variables used in determining NDL status, improving program linkages across time, more closely examining programs that implement NDL for only a subset of proposals, and refining the process for choosing comparison programs.