Thursday 12 January 2012

Lies, damned lies, and statistics

Sadly, spin and distortion in an attempt to support a dubious parochial objective is a feature of pretty much everything these days, so nothing can be taken at face value any longer.  This just wastes a huge amount of time and effort arguing over the numbers rather than spending it on solving the problem.

A case in point is the suggestion that anything up to 70% of Atos WCAs are wrong.  Nobody quite says this, but many are happy that this might be the impression you walk away with, after all it proves without doubt that Atos is rubbish.

However it appears that only 9% of assessment decisions have been overturned at appeal – as recently verified by the independent fact-checker Fullfacts.org. (I have always found Fullfacts to be about as straight as anyone can be).

The REAL issue is therefore whether or not 9% is acceptable and if not what is?  It doesn’t sound a lot – less than 1 in 10 – but in such a sensitive area it is clearly not good enough, but no “errors” is unrealistic.
The sooner we do this, the sooner we can have a sensible discussion about how best to achieve the target (and by when) and establish a series of milestones to get us there.  This is no more than good business practice and I really cannot understand why this was not sorted out at the very beginning of this project.  The fact that it was not is inexcusable and even Harrington reports do not address this basic issue.  How can anyone possibly say that the current overall process is capable of achieving an acceptable error rate without knowing what it is?

The concept of KPIs etc has been much maligned but the one and only management cliché that I have found always holds true is “if you don’t measure it, you can’t manage it”.

[PS I do realise that WCA “errors” are not all of equal size and perhaps views would be different if they were just at the margins.]

2 comments:

Anonymous said...

You are right: even journalists sometimes mistakenly state that the WCA has a 40% error rate.

However, for me the real issue is that the WCA is a totally invalid test; it does not assess work capability in a real world but basic functional ability. As we all know, you can fail to meet the criteria for ESA and yet be unfit for work.

So in a sense there is little point in trying to improve the reliability of the assessment as long as the test itself remains invalid.

In my view all the statistics relating to the ESA/WCA are pretty meaningless. We simply don’t know how many of those who fail the WCA or whose appeals are rejected are actually fit for work; nor can we assume that those who don’t appeal are truly fit.

What I’d like to see is a sensible discussion about the design of the assessment, the structure of ESA, and the assumptions underlying the whole benefit reform – but that will never happen.

Tia Junior said...

I agree. Possibly the biggest omission is around work disciplines (http://tia-junior.blogspot.com/2012/01/welfare-reform-3-big-assumptions.html)
All these stats are valuable depending on the precise comparison you are trying to make, but they are usually quoted without due care and out of context, which just muddies the water and inhibits progress.
Objectively and comprehensively taking condition variability into account through a single 40 min session with someone who has limited clinical training & experience is an impossible proposition. Passing the result to an administrator for a decision hardly helps.
Oddly too, nobody has ever asked me what I’d like to do with regard to work. Such a discussion would go a long way to sifting out the wheat from the chaff as it were. My view is that the process should seek to establish agreement as soon as possible, because from then on everything is easy and appeals disappear. The current process being so surreptitious is confrontational and just generates conflict start to finish.