At the Walton Family Foundation, we have worked to strengthen our evaluation and learning processes over the past five years. It has helped us when other philanthropic leaders have shared their challenges and failures. That’s inspired us to do the same.
So in the spirit of improving together, this is a story about how our foundation evaluated a failed strategy and used the findings and input from community leaders to alter our investments in the Arkansas and Mississippi Delta. Board and staff remained committed to the communities and people in this region. But as a result of an intensive, sometimes difficult assessment process, we have decided to try helping in a different way.
The Delta was once a flourishing nursery of agriculture. But in the latter half of the twentieth century, the bloom withered and the area began to struggle. The Walton Family Foundation has been involved for three decades in trying to help it revive.
Two counties in particular captured our attention: Phillips and Coahoma. Their poverty rates were among the most severe in the country. We felt strongly we had to do something, so in 2003 we adopted a strategy similar to the Hewlett Foundation’s Neighborhood Improvement Initiative, the Annie E. Casey Foundation’s Rebuilding Communities Initiative, and other examples of “comprehensive community change.” The idea was that transformative efforts are best achieved by pulling many social and economic levers at once. It was the first time the Walton Family Foundation tried to address poverty in this broad way, rather than through discrete projects.
We spent $52 million on the effort between 2003 and 2013. We provided our operational support through an intermediary charged with coordinating the area’s development. Southern Bancorp Community Partners was already a presence in the community and agreed to convene and steer new efforts, help grantees request funds, and coordinate reporting of results.
The first task was to build a plan based on wide community involvement. Hundreds of residents contributed to a blueprint that would guide grantmaking. Our giving included KIPP charter schools, promotion of the King Biscuit Blues Festival, a civil leadership program, construction of Civil War monuments and forts, and investments in downtown revitalization. The collective impact of these grants, we hoped, would be a whole greater than the sum of its parts.
To measure progress, we tracked everything from the number of residents leaving the Delta to poverty rates to tourism dollars. We tracked student outcomes in academic performance, college readiness, and college acceptance.
It wasn’t long before we began to see shortfalls. A year after launching the project, we asked an outside evaluator to track macroeconomic indicators, assess our intermediary’s performance, interview community stakeholders, and conduct a quality-of-life survey, among other tasks.
The more closely we observed our project, the clearer it became that we weren’t making progress on our goal. The poverty rate and unemployment ticked up. The population continued to decline. Even comparing our counties to similar areas in the Delta to account for national economic trends, we saw no improvement at scale.
Our internal evaluation staff arrived at a somber insight: Our aims were too ambitious. Spreading $5 million per year across a diverse set of projects, however well-intended, was nowhere near enough to dig the region out of entrenched poverty. To add insult to injury, our intermediary began to struggle, making only three grants in one year. Because of leadership turnover, some local action committees met only once or twice.
What did we miss?
Our outside evaluator was fully immersed in the project, attending committee meetings, conducting interviews with residents, scanning local news and economic measures for pertinent information. The verdict was clear: we had catalyzed more positive interactions among community members, but the project was “achieving little in the way of substantive community and economic development.”
We wondered if initiatives like ours had run into similar obstacles and started combing through analyses of other efforts around the country. We saw that the Aspen Institute had reviewed 48 community-change initiatives implemented between 1990 and 2010 and had found that “place-based efforts have had difficulty stimulating broader economic development, as too many of the forces that drive economic activity are outside of the control of neighborhood actors.” This rang true with our experience.
Urban Ventures also added perspective in a similar review: “Too many change efforts have been ‘sold’ based on a well-articulated but utterly unrealistic set of results that bear no possible relationship to the actual program interventions made or the scale of resources invested.”
With dreary data in hand, we figured the best next step would be to ask our community what we could have done differently. So in January 2013 we began interviews and conducted a quality-of-life poll. Elected officials, public safety officers, business leaders, clergy, and teachers confirmed our evaluation that there had been some successful individual projects in our initiative but no wholesale transformation.
The poll responses also pointed out several items we had missed. One was the region’s staggeringly high crime rates. A concern for safety and private property drained productivity and inhibited growth and investment. Nine in ten residents ranked organized youth activities as an important community need and improved law enforcement as a top priority. Other than supporting the Boys and Girls Club of Phillips County, our $52 million initiative had not made any grants in teen activities or public safety.
On the bright side, KIPP was recognized as an academic success story. A once blighted area now featured a good playground. We found anecdotal evidence of increases in positive interracial communication and increased civic participation. Issues of common interest were making it to the public conversation.
But the economic trends and the public feedback raised difficult questions about the efficacy of our investments. The time to rethink was at hand.
How we told our board and moved forward
Admitting failure is vastly more difficult than claiming success. Our foundation had been investing in the Delta for decades, and staff and board members enjoyed long-term relationships with nonprofit and community members. Now we had to break the news that our involvement hadn’t moved the needle on our overarching goals of community-wide improvement.
Our evaluation staff began to present findings in formal and informal settings. We brought academic and community experts to the table. We found that if we gave our board multiple sources of information in varying forms, painting a thorough picture of the project’s impact—or lack thereof—our message was better received.
We came out of this season with a confirmed commitment to the region but eyes open to our own shortcomings. The idea that one organization and one plan could succeed where both the private and public sectors have failed was perhaps too audacious. At the very least, our theory of change was wrong. We know how to fund meaningful particular projects in the Delta. But, like others who’ve embarked on comprehensive community change efforts, we haven’t discovered the way to help transform an entire community.
So we ended this strategy. We committed, however, to maintain a high level of investment—about $50 million from now to 2020. We’re focusing on shorter-term goals like improving public safety and offering afterschool programs. We stopped our exclusive reliance on using an intermediary organization and instead are partnering closely with a consultant raised and currently living in our target counties. We’re doubling down on the pieces of our project that did yield results, like our investment in KIPP charter schools and the Teach For America Delta corps. We are also adding resources to improve public safety and to help junior and senior high-school students pursue higher education.
Evaluation is never easy, nor is it a walk in the park to ask hard questions about heart-tugging interventions. But when we see failure, we need to recognize it for what it is and be willing to change course. That was the tough decision we made while staying committed to our partners. That was the best thing for the people we aim to serve, and the only responsible course for our board and staff.
This article originally appeared in Philanthropy Roundtable.