To P or Not to P? Understanding Statistical Significance and Its Role in Developing Evidence for Policymaking

Policymakers, program operators, and researchers often depend on statistically significant findings to identify what works in public policy and programs. It is important, however, for both the policy and research communities to have the clearest possible understanding of what statistical significance means and when it matters.

Last year, the American Statistical Association (ASA) issued a policy statement about p-values, the most widely used metric for demonstrating statistical significance. The statement addresses the ongoing concern that p-values and statistical significance are often misunderstood by researchers and decision makers, and it highlights the need for continued conversations, particularly among the policymakers and practitioners who rely on evidence to inform their programmatic and funding decisions.

On March 1, from 3:00– 4:30 p.m., Mathematica’s Center for Improving Research Evidence (CIRE) convened a panel of experts to discuss this issue from the perspectives of researchers, policymakers, practitioners, and funders. The event, “To P or Not to P? Understanding Statistical Significance and Its Role in Developing Evidence for Policymaking,” was held at Mathematica’s Washington, DC, conference center and via live webinar. Immediately after the forum, a networking reception took place in Mathematica’s 12th floor lobby.

Our panel of experts included:

• Ann Person, Mathematica
• Allen Schirm, Mathematica (moderator)
• Ronald Wasserstein, American Statistical Association
• Stuart Buck, Arnold Foundation
• Timothy Day, Center for Medicare and Medicaid Innovation
• Molly Irwin, U.S. Department of Labor