Skip to content

You are reading 1 of 2 free-access articles allowed for 30 days

Making the numbers work

Much of the best modern theory of health management systems comes from the well-known US physician Dr Donald Berwick. But one of Berwick’s strengths is his openness to a range of different influences. When Berwick, who is a trained paediatrician and a former Administrator of Centres for Medicare and Medicaid Services, became involved in health quality improvement and management, he took a seminar on statistics.

The seminar was run by the famous and influential statistician William Edwards Deming. In the course, Deming referred on numerous occasions to the work of another US statistician, Walter Shewhart, whose work relates to the distinction between normal and abnormal variations. Berwick immediately realised the value of Shewhart’s theory in relation to healthcare, but was surprised that he never heard of him.

“Don went to Deming’s seminar and afterwards he wrote about it and talked about it a lot,” internationally-renowned quality improvement expert Mr Lloyd Provost tells the Medical Independent (MI). Mr Provost, who lives in Austin, Texas, is a statistician and is a Fellow of the Institute of Healthcare Improvement. He was in Dublin recently, where he delivered a presentation at the RCPI on the best ways to use data to improve healthcare systems.

“Don said he couldn’t understand why Deming talked about 10 times a day about this guy Shewhart and the importance of him. He said, ‘how come I went through 12 years of medical school and this was never mentioned?’”

In 1991, Berwick wrote a paper titled A Consultation with Walter Shewhart. Mr Provost says it was the first piece about how Shewhart’s theory could be applied to healthcare. Yet it has been slow to catch on. According to Shewhart’s theory of variation, there are two main types of variation: Common causes, and special causes. Common causes are those inherent in the system over time that affect everyone working in the system and affect all outcomes of the system.

Mr Provost also criticised the use of ‘traffic-light’ grading systems, such as the HSE’s recent HealthStat system

In contrast, special causes are those causes not part of the system all the time and do not affect everyone, but arise because of specific circumstances. It is a more nuanced way of examining variation than classical statistics, Mr Provost says, yet many remain wedded to more traditional methods of data analysis.

“Most of my colleagues, people who are my good friends, do not understand this theory; they stick to classical statistics and think that way,” he explains.

“When Deming was alive, I worked with him and he would have meetings with statisticians and he would do his best to try and convince them. In the US, one of my friends, a bio-statistician, could make $300,000 (€273,000) working on traditional statistics for a drug company and I’m telling him ‘the statistics are rubbish, you need to do things a different way’. He says, ‘I’m doing fine, I’m getting recognised, I’m getting my pay cheque each month but I’m not going to do that’. We’re in a system that’s working there and letting go of that is a big deal.”

Deming predicted that Shewhart’s work would take 60 years to be accepted by the mainstream. Even though Shewhart was recognised in his lifetime, and was a Fellow of the American Statistical Association and the Royal Academies in the UK, Mr Provost does not believe his methods sit well with classical ones.

“You have to be understanding that I’m either thinking and doing things this way, or I’m doing the classical way. The classical methods have won out over time; they’ve been ingrained and they’re in our system. In the United States, our Food and Drug Administration requires you to do statistics this way and to think this way. Shewhart is not in there because not enough [people] know it.”

But, like Berwick, Mr Provost believes his work is extremely important for improving the quality of healthcare systems. In his talk at the RCPI, he focused attention on the need for the use of run charts in improvement projects. Rather than using confusing data tables, a run chart makes it easier to measure improvement through time in a simple but elegant way. While tables carry the same data, they do not easily reveal temporal variations, which is key to any quality improvement process.

The classical methods have won out over time; they’ve been ingrained and they’re in our system

Mr Provost also criticised the use of traffic-light grading systems (such as the HSE’s recent HealthStat system), which are frequently used by organisations to measure quality, on the basis that they are too crude, do not effectively monitor change and have no emphasis on prediction. The variations in the run chart, which could relate to anything from wait times for colonoscopies, to monthly revenue, often require further analysis. This is where the Shewhart chart comes in. The chart, which is also called a control chart, can show whether the variation is expected or is due to an exceptional cause.

“In other industries, not doing that means you have a bad computer that doesn’t work. In healthcare, it means that a patient dies,” Mr Provost says.

Implications

“Working in healthcare means you realise the implications of systems not working or not doing the right thing. It is an industry; if there is any place where we have to be using the best knowledge we have, it should be here.”

He recommends that health managers should move from a tabular view of data to one where each measure is displayed on an appropriate Shewhart chart, and that all Shewhart charts should be put on the same page in order for them to view the whole of the system. This will allow managers to more accurately assess progress of changes in the system, become aware of system relationships, appreciate both dynamic and detail complexity within the organisation and predict the performance of the system.

Although Shewhart is still neglected, there are still some areas of good practice where his methods are employed.

“In England, they have used this theory for a while but they are usually in the background, a professional statistician that is setting-up a system. There is a system called Dr Foster that looks at data from all the NHS hospitals and gives feedback; they use Shewhart’s method to look at the variation. But as a leader, you would never know that. All of a sudden you get a notice that one of your surgeons appears out of line for some reason; you don’t need to know how they concluded that. What you do need to believe is that it works.”

Mr Provost adds that more and more papers in the journal BMJ Quality and Safety are now using Shewhart’s methods.

“Ten years ago, there wouldn’t be a single Shewhart chart in the issue; if you pick up the current issue, at least half the papers would have a Shewart chart in it. It is starting to be used. We are publishing good-quality improvement work and it is starting to be something that is becoming more standard. One of the problems, and I was at meetings like this earlier in the week, is that at least half of those are bad; they do an incorrect job on the charts. That is because we don’t have reviewers yet. So we have to build up a whole system where the reviewers can recognise that this is the proper use of the charts. But at least there is a big move in the direction there.”

Benefit

The HSE’s National Director of Quality and Safety, Mr Philip Crowley, was one of the attendees at Mr Provost’s presentation. He said the HSE would benefit greatly from using the methods Mr Provost described, as it would help managers to stop reacting to minor shifts in data and to start examining the changes that are truly significant.

“I think overall, the HSE has massive amounts of data,” Mr Crowley tells this newspaper.

“What we really need to get better at is changing that data into intelligence — something we can really use. With a lot of our data, we look at a point in time, what is happening today. The take-home message from here, something we have started to do, but are forced to do now, is that we need to start looking at that data over a long period of time and see if the variations we see in data all the time in performance are things that affect patients.

“Is it significant or is it the natural variation in a complex healthcare system that you would see anyway? If it is just that, then what we should focus on, which I’m trying to lead on, is how do we shift that norm; how do we improve that norm overall and shift the graph?

“We take very seriously what he said today. We have the capacity to look at data the way he describes it and we are going to explore doing that.”

Leave a Comment

You must be logged in to post a comment.

Scroll To Top