Since its launch in 2006, the Alliance for a Green Revolution in Africa (AGRA) has been working to increase the yields and incomes of 30 million smallholder farming households in 11 African countries. In 2017, AGRA refreshed its strategy to move away from siloed interventions toward a more integrated implementation approach across three thematic areas in each focus country: (1) policy and state capability, (2) systems development, and (3) strategic partnerships. This new five-year strategy for 2017–2022 was launched with financial support from the Partnership for Inclusive Agricultural Transformation in Africa (PIATA), an alliance between the Bill & Melinda Gates Foundation; the United Kingdom’s Foreign, Commonwealth & Development Office; the Rockefeller Foundation; and the United States Agency for International Development.
Literature on the potential for PIATA’s strategy to drive agricultural transformation on the continent is cautiously optimistic, while raising concerns about some key elements of its approach. To contribute to public discourse about PIATA’s strategy and role and to help inform the future phase of PIATA, the Bill & Melinda Gates Foundation commissioned Mathematica to conduct an independent, retrospective evaluation of PIATA’s five-year strategy in 2021.
To identify an appropriate and feasible methodological approach to the evaluation, we considered three key factors: (1) the complexity of the PIATA strategy, (2) its implementation focus on market systems change, and (3) the retrospective nature of the evaluation.
First, the evaluation needed to accommodate PIATA’s complexity. A growing body of literature defines key features of complex programs; PIATA’s strategy exhibits nearly all of them. For example, not only does it span many countries, but within each country, multiple grants are made to a broad range of organizations—from national governments; to nongovernmental organizations; to small, medium, and large agribusinesses—each working at different levels of complex agricultural market systems.
The second factor was PIATA’s focus on catalyzing market system change to achieve farmer-level outcomes. PIATA does not always provide direct services to farmers, but instead works through market system actors to strengthen these systems in ways that would ultimately benefit farmers. Thus, the evaluation design needed to consider that PIATA’s targeted outcomes would be affected by a range of factors, including but not exclusively or directly linked to PIATA’s efforts.
Finally, the retrospective nature of the evaluation informed our design. Therefore, we would need to rely primarily on existing data, including AGRA’s programming and survey data, and secondary data sets, available from the 11 focus countries.
As evaluators embrace the complexity of large-scale programs and strategies in their work, systems and complexity-informed evaluation designs are becoming more common. These types of evaluations provide extremely important inputs for policy discussions on how complex initiatives can evolve in response to emerging evidence and adapt to improve their interventions.
Based on these considerations, we designed a theory-based, mixed-methods evaluation using contribution analysis as the overarching analytical approach, with a nested quasi-experimental design (QED). We designed the QED to answer questions about whether and to what extent farmer-level outcomes changed during the five-year strategy, using AGRA’s data and other secondary data. For example, using AGRA’s farmer-level data, we deployed a matched-comparison approach across three countries. We drew on the Living Standards Measurement Study–Integrated Surveys on Agriculture household survey project in Tanzania, the only country where surveys were administered before and after the launch of PIATA-supported initiatives, and where we could credibly identify subnational variation in the intensity of AGRA’s efforts within countries from AGRA’s programming. This process enabled us to study changes over time in areas that were relatively more affected by the national-level interventions supported by PIATA and to tease apart the additional effect of AGRA’s work. It also enabled us to account for national or regional trends and context-specific factors (such as agroecological conditions) that can directly affect such agricultural indicators. We similarly drew on subnational annual statistics on agricultural productivity, from the ministries of agriculture where we could obtain these data, to look at these underlying trends more closely and to assess the possible contribution of AGRA’s national-level interventions.
Contribution analysis helped us uncover the influence of PIATA’s complex programming on observed changes in market system-level and farmer-level outcomes and to answer the “hows” and “whys” of what drove program effectiveness. It also helped us understand the influence of AGRA’s overlapping and often national-level interventions on policy changes and private-sector engagement and investment in agriculture. Theory-based approaches such as contribution analysis are used when many factors affect the outcomes of interest, and these approaches are increasingly recognized for their usefulness in evaluating complex efforts to change market systems. Through our analysis, we drew on all data streams—quantitative and qualitative, secondary and primary—to develop a series of narratives to support contributory claims according to PIATA’s theory of change. These narratives assessed the degree to which changes in key outcomes and conditions in PIATA’s theory of change held true, determined how PIATA’s strategy contributed to the changes, and assessed the sustainability of the PIATA-influenced changes.
What we found
PIATA was successful in catalyzing key policy reforms, mobilizing public- and private-sector investment, and reaching farmers with extension and seeds. Its support for national governments enacting and implementing policy reforms helped incentivize private-sector engagement in the production and delivery of improved seeds in several African countries. In addition, PIATA-championed agricultural investment projects and strategic partnerships successfully used government, donor, and private investment to provide services to smallholder farmers, particularly in Ghana, Tanzania, and Kenya. PIATA also increased the supply of certified maize seeds through direct support to seed companies and enhanced linkages between input supply actors, particularly in Rwanda, Ethiopia, Ghana, and Nigeria, while expanding farmers’ access to extension services across several countries through its village-based advisor model.
PIATA’s impact on farmer productivity, sales, and food security was mixed. Of the six countries for which suitable data were available, we found evidence suggesting that PIATA improved maize yields in Ethiopia, Nigeria, and Ghana. However, we found that PIATA’s efforts in Tanzania, Burkina Faso, and Kenya had no discernible impact on maize yields. In addition, farmers’ maize sales increased in only one of the three countries for which data were available, and we found no positive effects on food security. These mixed results likely reflect remaining farmer constraints in access to affordable inputs and output markets, as well as low per-farmer investment levels. In addition, the effects of PIATA’s successes on key policy reforms and investment mobilization during this five-year strategy would not yet have impacted farmers.
Next steps: Prioritizing impact and learning
Our evaluation offered several insights that can help strengthen AGRA’s efforts to improve farmers’ productivity and income in its next phase of the PIATA strategy. PIATA’s mixed results on farmer yields and sales suggest that the next phase of the PIATA strategy needs to focus more on understanding and addressing farmers’ binding constraints, such as input financing and access to output markets. To have a lasting impact, it should scale up interventions that incentivize market system actors to offer these services and exit out of interventions that can be sustained only with external funding. The success of its policy-focused work will be more sustainable with a more intentional approach to enhancing longer-term public capacity in policy development, even if initially focused on countries with strong political will for policy reforms. We hope that AGRA finds the insights covered in our report valuable when working with the PIATA partnership to design its next five-year strategy and increase its impact.
In addition to highlighting areas where PIATA’s approach worked well, and areas that need improvement, our evaluation highlighted critical gaps in PIATA’s internal monitoring and evaluation (M&E) processes. Taking steps to become a more learning-focused organization with less burdensome grantee reporting and more rigorous, targeted farmer surveys will be a necessary component of the next strategy. The next phase of PIATA should consider convening its leadership, technical, and M&E staff to set a focused learning agenda that informs data needs to more rigorously measure PIATA’s farmer impact. A learning-focused M&E system would enable PIATA to better track its progress, drive adaptation based on evidence, and ultimately, increase its impact.
