Impact Measurement
.jpg)
Written by
Lee Crockford
April 2018
Last week, spur: delivered a workshop for 35 charities and other impact-focussed organisations from across South-East Queensland, in collaboration with BDO.
The session was designed as an introductory dive into the world of impact measurement — an area that a many organisations don’t do as effectively as they could or should.
This post is a summary of the key concepts around effective impact measurement that were explored through the workshop.
Depending on what resonates with you, this post is broken into sections so that you can easily jump into whatever’s most relevant:
Impact measurement is the ability to metric not just what you do as organisation, but to deeply understand the effects and outcomes of that work.
For example: If a charity builds one hundred drinking wells across remote communities, there’s an assumption that good is being done — and chances are the work is indeed good. However, what is the actual impact that the building of these wells creates? Some questions that immediately arise include:
These types of impact-focussed questions are inherently at the root of all charities, social enterprises and, we would argue, should be at the root of all businesses and government initiatives. Yet, these types of questions are all too often overlooked — or their answers assumed, rather than proven.
We have a few internal mantras at spur: (some of which you can see here) that help keep our work focused. One of these is: if you can’t measure it, then it didn’t happen.
At the risk of over generalising, we’ve found that organisations are typically:
What’s most important to remember here is outputs ≠ outcomes.
Side Bar: A common phenomenon which dovetails into this disconnect between an organisation’s outputs vs outcomes is a rise in “awareness campaigns”.
Think: Organisations that are dedicated to raising awareness for mental health or climate change or civil rights issues.
Yes, awareness may be an important step in creating social change, but it is always only just that — a step towards behavioural change. It is not behavioural change in and of itself. E.g. You might convince the entire population of the ills of plastic shopping bags, but if they don’t change their behaviour, then all of your effort is moot.
The argument for organisations to improve their impact metrics is often framed through the lens of charities justifying their funding requests.
As an organisation that has been commissioned by government to review funding to charities, we appreciate that this lens is legitimate. However, to frame metrics solely through this lens is to relegate the measurement of impact as an onerous, begrudging exercise in box ticking. Rather, impact metrics are deeply important for myriad reasons:
Six men suicide each and every single day in Australia.
Long before spur: was even spur:, we started as a single campaign called Soften the Fck Up.
STFU was a mental health campaign we devised for young men in our age-group for whom we believed exisiting mental health initiatives were failing to speak to authentically.
The mental health space is now inundated with initiatives and campaigns that speak to a whole range of demographics — and vary in effectiveness. But, back in 2011 when we launched STFU the mental health space was drastically different.
At the time, most campaigns were gender-neutral and heavily pushed clinical terms like depression and anxiety — terms we believed to be alienating to many. Soften the Fck Up was designed to be the antithesis of this: Young guys talking frankly about tough times they’d been through and how talking about it was the first step in taking action for their own positive mental health.
It went gang-busters: Tens of thousands of views, international coverage, celebrities and politicians propagating the message all helped propel us onto the map as a force in the mental health space: Not bad for a small bunch of volunteers and a budget of $500. My-oh-my were we proud of ourselves.
Well, for a short while.
Upon reflection we realised that despite all the coverage and views we received, we had no idea how effective the campaign was at actually getting men to take action for their mental health . One comment on our YouTube channel said:
“This video saved my life 6 months ago. My circumstances haven’t really changed, but my outlook has. So, thank you.
It’s an incredibly humbling comment, but at the end of the day it the only data point we have to speak to our effectiveness. We came to the sobering realisation that we didn’t have a single metric that demonstrated whether we had positively contributed to men’s mental health or a reduction in suicide.
We reached a cross-road: Should we even be in the mental health space if we can’t measure our impact? Or, if we do want to continue, how might we develop metrics that will ensure that our future campaigns will demonstrably make difference? How could we ever know that what we do today might affect someone’s life tomorrow, next year or in 5 years?
We went into a period of deep introspection and research. Not only can metrics be challenging to develop at the best of times, but when you’re working in a preventative space where others are working too, it becomes even more complex.
We mapped out the mental health space, demographics, the “user journey”, and what other organisations were doing (after all, if someone else is already doing it, then why duplicate it?). A common theme began to arise from the research: In order for men to take positive action for their mental health the first two steps are always “understanding / acknowledging how’re they’re actually feeling” and “talking / sharing how they’re feeling”. Although we might never be able to say that our campaigns direclty reduced the rate of suicide in Australia, we can embed the first critical step of mental health action into each of our campaigns: Men talking about how they’re feeling in a way we can measure.
This was transformational. It aligned the teams’ focus, it provided a unique differentiation in the mental health space, it’s broad enough to work across demographics and a range of campaigns, it’s concise and simple in a way volunteers can easily rally around, and provides a central metric to prove the effectiveness of our work.
spur:org now has evidence of over 110,000 instances of people talking sharing how they’re feeling. This metric isn’t the only metric we use, but it’s our core metric that allows us to know that we’re tangibly making a difference.
More specific examples of our impact metrics are used as case studies within the next section.
“Romeo, Romeo, wherefore art thou demonstrable social return on investment through clear and precise impact metrics?”
At spur: we utilise a range of frameworks as thinking tools. For the purposes of this blog, we wanted to delve into one we call the Romeo & Juliet Model.
(Note: We love a good framework. Yet we also believe that all frameworks should be treated as tools: A device to prompt thinking, and not a silver-bullet-solution.)
Remember the story of Romeo and Juliet? It was written by some guy with the name Shakespeare. Pretty obscure fella.
If you need a refresher: It’s the tale of two star-crossed lovers who long for each, yet end up dead. Kinda depressing.
You see, despite their best intentions: Poor communication, a lack of pragmatic strategie,s and less than ideal external pressures lead to their demise. It’s a pretty good parallel for how many organisations function and it forms the basis of our framework. This is now the part where you have to use you imagination as we’re heading into the land of metaphors:
Above we have Juliet in her tower. We equate her to the Vision of an organisation: She can see the big picture from up there. She has wonderful intentions. However, confined to the balcony her ability to “get things done” is pretty limited.
On the other hand is Romeo. He’s down in the weeds. He has the ability to create action and get the ball rolling. Yet, he can get distracted by the details.
The most important thing, however, is that dotted line in between them. How might be we connect them effectively? Both Romeo & Juliet serve important purposes and they compliment each other well, but they don’t represent the full picture of what’s needed.
When working with organisations we begin by asking them to articulate their “Juliet” and “Romeo” (ie; their vision and inputs) as these can usually be articulated pretty quickly and easily.
As an example we’ve outlined spur:org’s Romeo and Juliet below:
We then need to fill in the blanks. This is an imperfect, messy process where we work from the top down, bottom up, inside out, rejigging and remapping as we go. In an ideal world, we’d start at vision and lineally work our way down to inputs, but in reality this isn’t an effective way to brainstorm.
(Note: We often find that what people initially consider to be their vision or inputs are in fact, not. That’s a story for a another blog, but the asterisk for those playing along a home is that what you start off with as your vision and actions may very well change — and that’s OK.)
The way we fill in the blanks is to answer the following questions:
When working from Vision down: How might we achieve this?
When working from Inputs up: What is the result of this?
Again, here is spur:org’s example:
Regardless of the organisation or area of work, we typically find five tiers of impact emerge in this model, which we call Actions/Inputs, Enablers, Change, Improvement and Vision.
We then need to examine the possible metrics for each tier: Which of them can be measured? Which can be measured by your organisation? Which might be measured by someone else? What might this measurement look like?
For spur:org, it looks a little like this:
If you’re going to attempt the Romeo & Juliet Model youself, it should be noted that when spur: launched the strategic arm, we started with mapping it through the Romeo & Juliet Model. It took a team of people locked in a room for two days to agree on the final tiers and language used. So, the disclaimer here is that this isn’t a framework to be “perfected” in an hour by yourself. It is a deeply collaborative framework designed to challenge and inspire you to think about your impact more deeply.
It should also be noted that this model can be “nested”. We have an R&J model for spur: as a collection of organisations, we have an R&J for each arm of spur: and we have R&J models for each campaign, too.
They nest not in spite of each other, but rather form an interconnected mesh.
Lets really bring it down to the bare bones:
Many organisations talk about what they do — the actions they busy themselves in day-to-day. Some talk about the summary of those actions, but few effectively speak to the impact their work has on the world. This is frequently because the impact isn’t measured or comprehensively understood.
If you aren’t measuring your impact — how do you really know you’re achieving anything? Or doing the best you can to help those in need? Or getting the most funding you could?
Regardless of frameworks, approaches and methodologies, if organisations simply asked themselves more often “To what effect?” they would be able to articulate the impact they have far more clearly:
We build wells.
To what effect?
To provide clean water.
To what effect?
To reduce disease.
To what effect?
To improve wellbeing and life expectancy of remote communities.
Cool. Now, how do we measure this impact?
