What Ratios are telling us
JEREMY NICHOLLS 29TH MAY 2013
One of the common concerns raised about SROI is that reporting a ratio between value created and cost will lead people to inflate ratios.
Although the SROI Network’s position is that the ratio is only one way of presenting the analysis of social value and is of most use for internal management and board discussions of efficiency, it remains a commonly used reference point. We have now been collecting SROI reports since 2005 and have around 140 in the member’s area of our website. This is by no means a list of all SROI reports and only represents our best attempts at collating a combination of SROI reports available on line and SROI reports that have gone through assurance and can be made public.
Based on best fit of the ratios reporting, over the last 7 years there has been a small increase in the trend on the best fit line from a ratio of 5 to 1 to 5.9 to 1. This includes one reported ratio of 37 to 1. If this were excluded the increase is from 4.5 to 1 to 5.5 to 1. True, an increase but hardly a huge inflation.
Of course all reports on performance risk overstating results, and as soon as there is anything to compare to, there is a risk of inflation as we attempt to out-do the past. Performing better than the past is of course a good thing so long as the improvement is real. It could just be that the reason for the increase is because organisations are becoming more effective at creating social value. Like it or not, comparisons do drive human behaviour and this can be both positive and negative.
But let’s ignore the possibility of improved performance and focus on the risk of only ‘improving’ how social value is reported. Let’s also assume that no other evaluation has ever had to compromise between the evaluator’s perception of the evidence and the organisation’s perception, that poor results are always reported, that organisations are not able to hide information from external evaluators, that there is no tension and no disputed space in any process of summarising activity and reporting on that summary. I know, yes, ridiculous. But we have to make that assumption in order to control for all risks of inflated ratios, so that the argument that the increase in SROI is because SROI is more at risk of inflating reported results than any other approach where comparison is possible.
This is why SROI has a principle for this - ‘verify the result’.
This principle exists because we recognise that socially constructed information needs an independent check. The SROI Network offers two approaches to this principle; one is our report assurance process for SROI analysis and the other is a report review process for any social impact reports. Demand is steady and as high as we can easily manage.
On the best fit line the ratio for assured reports has fallen from 5.2 to 1 to 4.5 to 1.
This is an important result
It suggests that the assurance process, which is a bottom up, community driven approach to deciding whether judgements being made are reasonable, is slowly closing the range within which judgements are acceptable.
This is not dependent on any organisation’s claims of their internal quality assurance processes and it is independent of the employer consultant relationship and inherent pressures.
It belies the criticism of SROI that there will be inflation in reported ratios and shows why the principle of verification is so important.
I wonder how many of those criticising SROI are really aware of this principle and how many apply it to their own favoured approach. How have you verified the judgements that you MUST have made in summarising peoples experiences? SROI has never argued that this experience can be reduced to a number without other information, qualitative and quantitative information all play a critical part as well. But all these types of information are still summaries of people’s experience and summarising needs judgements.
Do you ever wonder that the reason not enough use is made of evaluations is because we don’t trust them? We don’t know whether, if we use the information to make a decision, we are more or less likely to make a better decision – and so we don’t use it.
I know it’s my favourite comparison but financial accounting and reporting only works because the information is verified. This may be prepared by a qualified accountant for a board report or audited by external auditors for a public report but there are appropriate verification processes in place. Would you invest in businesses that haven’t been audited? It is bad enough getting businesses to pay enough tax within audited accounts; imagine what would happen to tax receipts without audit. And this audit process isn’t something being applied in a consultancy relationship; it is applied by an independent structure which also registers auditors.
There are a few that do this in the world of social impact, there’s what we do and there is the Social Audit Network and there are also the listing platforms like NEXII and ASIAIIX. The Institute of Chartered Accountants of England and Wales can maintain independence because audit is a legal requirement. Even if you are audited by PwC, the audit manager has to be registered with ICAEW and has to use established audit standards.
The SROI has had very welcome support from the Hadley Trust for this not very sexy area.
But if we don’t put these processes in then the boom in interest in social impact reporting will fade. Unless it is possible to say ‘hold on you haven’t had as much of an impact as you are saying’ it will become a rubber stamp. The information won’t be used to make decisions. Careers will be made, resources spent, but it won’t make much difference.
Accept no alternatives.
Get your reports assured.
Using an independent process.
|
