Impact Evidence and Standards of Evidence: Part 1

The Trust and Evidence Balancing Act of Impact 

The quest for robust, defensible evidence of impact often seems daunting – either too complex or disappointingly resource-intensive. So, there’s an undeniable appeal in the simplicity of placing your trust in passionate individuals and organizations whose missions drive them forward. 

In this piece, we explore when it’s better to rely on the vigor and vision of these change-makers, and when the pursuit of concrete evidence is so vital that we must chase it, no matter the cost. How might funders balance between trust and evidence in the world of philanthropy and impact investing?

Trust vs. Verification 

I’ve been in plenty of conversations recently about this perennial debate in the “mission over money” world of impact investing and philanthropy in education. 

• Some centered around the contrast between the “trust people to do good” hands-off approach and (at the other end) a rigorously academic focus on randomized controlled trials 
• Others were concerned with linking an aspirational and laudable theory of change with real-world, short-term effectiveness. 

Admittedly, selecting and gathering evidence is considerably more complex in the world of learning (whether pre-school, adult, or anywhere in between) than in other areas of impact. In the climate and health sectors, it’s relatively straightforward to track empirical metrics. One can easily and empirically measure the parts per million of carbon in the atmosphere, or the number of individuals who have been vaccinated in a specific geography. However, in education, choosing the proper measures and the challenges of comparing contexts can often feel like a much more arduous task.

Finding a balance 

I think there’s a way through here. Evidence gathering is a continuum, not a binary choice. And even trust-based investment can be based on some level of assessment and evidence.  

Giving with “no strings” attached 

Some examples can be illustrative. McKenzie Scott’s recent approach of giving away large grants with “no strings” – for example, to Historically Black Colleges and Universities – is very trust-based. As far as I am aware from public information on these donations, no reporting of how the money is being used, to what effect, is required. Yet, this approach is both data-based and trust-based, with an implied threshold of evidence and judgement of risk. Whilst there is no guarantee that Ms. Scott’s donations will be put to good use, and she has no comeback if they are not, Howard University (to take one example) has a storied track record of educating Black students well, and providing increased opportunities for those on low income. Chances are, it will continue to do both, likely better for the money the institution has received. There is evidence, albeit retrospective. That is good enough for Ms. Scott and her advisors.

Giving with all strings attached 

Another approach: one grant funder with whom we recently worked asked for clear, academically referenced evidence around the theory of change and impact that a grantee was planning on delivering, before any money was approved for release. Arguably at the other end of the spectrum, this sought the highest level of evidence possible throughout the grant-giving process.      

So, what’s enough evidence for you? 

Each investor and funder’s definition of “enough” will be different. A “right-sized” approach needs to fit your values, ambitions, resources, and – in particular – timescales and relationships with risk.  

In part 2 of our series on Evidence and Standards of Evidence, we will examine several tools and frameworks that can help make the right decisions. We’ll explore how these tools and frameworks contend with and answer the challenges associated with collecting the types of evidence we need, including an interrogation of the delicate balance of determining what level of evidence is “enough” for you.  

If you’re grappling with these issues or similar ones, please don’t hesitate to get in touch with us – we’d love to discuss them with you. If you’re at ASU+GSV in San Diego in a few days, there’s no substitute for a conversation in real life. Otherwise, we’re happy to talk virtually.