The Emperor Still Has No Clothes: no evidence for ITIL

We still call ITIL "best" and we still put up business cases for millions of dollars to implement it, yet there has never been any empirical scientific research to show that ITIL does better than, say, astrology as a framework for IT processes.

"The Emperor has no clothes. Where is the evidence for ITIL?" is the second-most-viewed entry on this blog. Now an updated and revised version has been published as an ITSM Watch article.

ITIL is big business worldwide, perhaps the biggest game in the IT town right now. But there is still no systematic rigorous evidence of its effectiveness.

Analysts often publish “research” which involves asking managers:

  • How successful they were in implementing something
  • How much money they returned to the business after spending the organisation’s funds

Not only do the participants in analyst surveys often have a vested interest in painting a rosy picture, but the methodology is entirely subjective (“On a scale of 1 to 5, how brilliant were you…”) and totally unscientific:

  • There is no control group to compare to.
  • There is no blind sampling to remove researcher bias.
  • There is often no random sampling: respondents self-select by agreeing to respond.
  • There is no peer review.

Often there may appear to have been actual research by the analysts but close examination reveals it is all hearsay.

Everything the IT Skeptic has seen to date involves asking people what they plan to do or how well they did. It is all anecdotal, self selecting, self-serving, skewed bilge: amusing, even useful in some aplications, but not evidence.

Comments

No evidence of ITIL... in solitary

When I joint the world of IT and ITIL through companies as Quint Wellington Redwood, HP, Pink Elephant (now Pink Roccade) and other international experiences, I was amazed to learn that even best practices seem to live their own life, in isolation and through the eyes of a few. Coming from a logistical and manufacturing background it was obvious to me that one half can't make a whole and that contribution to a collective can't be measured in positive figures before the negative impact is spelled out. And even there you go wrong when you try to pin-point the negative result to a standard or a best practice.
Look at the ingredients of a nice pie. Each individual ingredient is tasty, but combining all ingredients together might give you a bad taste after all. Not even speaking of the processes involved when preparing, baking or even serving the pie... low-fat, sugar-free... How do you measure your result? Exactly, by the taster or the audience. And they will NOT categorically tell you that a specific ingredient destroys the expected result.
ITIL in that respect is just an ingredient in the process of coming to a good taste. And before starting to give ITIL a performance result in isolation, we first have to look at the tasters and their specific requirements. From that perspective we start to learn how we can measure our success and hopefully come to a single performance indicator on an ITIL contributor, next to the processes of HR, finance, strategy formulation, architecture, procurement and even Gartner. They all make up the ingredients towards a possible success and all of them can screw up the taste when not correctly mixed or balanced.

Happy Cooking!

How to cloth the emperor

I'm not surprised that there is little "scientific" evidence of the value of ITIL (or ISO 20000). Very few studies about organizations have a solid "scientific" basis. Action research produces some interesting and suggestive organizational results, but very little that "proves" one approach is superior to all others. It's not even clear to me how one would formulate a "scientific" hypothesis about ITIL. ITIL as opposed to what? And how to filter out contextual factors that might influence the "scientific" outcome.

In my mind, it comes down to two related questions: Is there demonstrable value in being disciplined in the delivery of IT services? And if there is, is there demonstrable value in using disciplines that are common across multiple organizations? Let me address the two questions in order.

Anyone who has spent any time in IT has seen what can happen when service delivery discipline is weak. There are very real costs when services go down or are otherwise disrupted. Some of those costs have even been measured. In many (most?) organizations some additional service delivery discipline can be justified based on avoiding the cost of service disruption. In the vast majority of cases, it makes economic sense to move from level 1 to level 2 (using a CMM/COBIT scale). In many cases, it is justified to move from level 2 to level 3. But, in my experience, moving to level 4 is often not always justified. And level 5 ...

The same comments would apply to maturity of each of the typical 10 ITIL processes. Some additional discipline is often justified, but not a blind striving for level 5 (nirvana).

The next question has to do with whether it is cost justified to use ITIL as the source for such additional discipline as may be deemed to be warranted. In the case of a single organization, it would be hard to "prove" that ITIL is the best way to go. It's always possible, especially over the short term, to develop a local discipline that better exploits existing local strengths. But that doesn't address the problem of work flowing between organizations that apply different service management disciplines. ITIL is the de facto common source for such discipline.

How does it add up? Don't expect to ever find "proof" that one organizational approach is superior to all others. Apply additional discipline only when it can be justified. And look to ITIL if you expect to be cooperating with other organizations.

Aside: The question of ROI (and VOI - Value of Investment) for ITSM is one of three themes that will be the focus for the 2007 itSMF Canada National Conference (http://itsmf.ca/2007). We may not "scientifically" answer all ROI questions about ITSM, but we will be asking them throughout the conferences. (Disclamer: I'm Conference Chair and may not be the most impartial commentator on our conference.)

ITILNoooooooooooooooooooo

ITIL is a bandwagon for people to jump on. It promises much but the gains are very little, in my experience, and OGC are putting themselves in a pretty awkward position.

Anyway, the British government pushes this "de facto" standard for IT and yet look at a prime example of Connecting For Health, 12Bn UKP spent and just a tiny part of the project implemented, over budget (if there was even one set) and later than late could ever be. Key suppliers dropped out and costs run wild. Where's the work package agreements, the SLA's, the OLA's and the rest of the checkpoint meetings and decision points? And the world wants to follow the same processes? Why?

ITIL is a big moneyspinner and, on the course I did of around 30 students, not a single person failed. It's a big club with no substance for the boys (and girls) who want a new evangalistic pseudo-religion in the workplace.

if developers were as disciplined as operators are slowy becomin

I do of course agree with much of what you say, but I'm not sure you can nail Connecting For Health on ITIL's door. Perhaps if developers were as disciplined as operators are slowy becoming (again, we were there twenty years ago)....

Ah, but I disagree...

But I disagree in a nice way :)

If you take ITIL at the level it's being sold at, which is pretty much an IT installation and maintenance nirvana then CfH can be nailed, in my meagre opinion, on ITIL's door. I daresay Prince2 has to take some (ok, a lot, more than a lot even) blame as well but these are both government backed systems and they are both used for CfH.

Personally speaking, I have seen a really atrocious implementation of ITIL and the Service Desk still doesn't follow it's own basic ITIL processes over two years down the line. Staffing has increased by 300% (and still increasing!) and yet incident completions are down on pre-Service Desk figures. All I see is a bloating of the IT department with no apparent, or even discussed, benefits. Our 'customers' aren't happy and as a taxpayer I'm not happy with CfH.

I may be a bit bitter over this with my experiences, and I apologise for that, but I still question the take up of ITIL given it's track record in UK business and government developments. What frustrates me is that there is clear evidence, to my mind, that ITIL doesn't work in some circumstances because people don't have a clue about common sense and are totally myopic in how they apply ITIL - Or as in my case, the implementers are too full of their own egos to see what they are really doing.

I recently read that ITIL should be treated, financially, as a core part of the business say like a PC or a UPS, an essential part of the business. So, where you may have to produce a return of investment on some specific item to improve the business with your PC, you don't because it's essential. I can see why they want to do that with ITIL - The thing is a money sponge!

Sorry, rambled a bit... Bur I do feel better for that :)

CfH and ITIL

CfH looks like the mother of all badly run projects, but I am not sure this is can be laid at ITIL's door - as you say, the Service Desk doesn't follow its own basic processes after two years. Surely that's not ITIL's fault its the people who have implemented it.

From what I have read CfH was flawed from the start, the customer (end users) were not consulted on requirements, vendors were allowed to push products based on what they could deliver NOT what was requireed, and the whole system was in many cases unnecessary - a centralized medical system that would allow a doctor in Middlesborough to see patient records in London ???? In the rare case that this may be necessary, we have the system in place - its called email !
(ok rant over)

I think there are valid comments about ITIL being a money-spinner BUT in my 20 years in IT, I do feel it brings the 'common sense' approach to running IT services.

When people talk about a framework they often make the mistake of being blinkered. The whole point of ITIL is 'this is the best practice, ie in an ideal world Incident Management links to Change Management, links to Release Management' and ITIL shows you HOW this should work, but if your organization cannot be moulded to work that way for any number of reasons then you don't HAVE to do it. There is no 'ITIL COMPLIANCE' award.
You dont fail !

To me the 'best practices' offered are pretty difficult to argue with, the point is understanding how it should work and either adopt the model OR be aware of how to best fit your structure if you can't.

If you have a clean slate fine but life is not always like that and you have to adapt rather than adopt.

ITIL is a framework and should be used as such - it doesn't get you over, lack of management buy-in, internal politics, vested interests, badly run projects etc etc.
The tool is not the problem

Good Luck.

statistical inference from selected groups

Thankyou for that very thoughtful input, Robert (and welcome to the site)

I agree with all but one point; that business results cannot be measured scientifically. In another life I do a lot of research on one aspect of business, and there is a big body of academic research around the outcomes from styles of management, strategies, methodologies ...

Closer to home, Carnegie Mellon has done some real measurement of CMM.

It is no harder than trying to measure the effectiveness of drugs on people. One can do statistical inference from selected groups

I look forward to what comes of the conference. Good luck with that.

No evidence for ....

Being a skeptic isn't a bad attitude. Actually, the points you make are excellent in refuting the kind of nonsense that is happening around proof for the success of ITIL. But does it have anything to do with ITIL? This is both true for salespeople trying to sell their ITIL consultancy and products as for those that claim that there isn't any evidence for ITIL's success. ITIL isn't a thing to implement and solve a standard set of issues. Or in other words "ITIL" cannot be implemented! ITIL is a set of books describing good ideas to improve certain elements of service provision that are potentially not working well in some organisation. It is like a travel guide on Rome. If you go to Rome for a weekend you would be crazy to do all the things in a travel guide. Therefore it is nonsense to claim that a 75% of all trips to Rome where travellers used a particular travel guide are made with 50% reduction in costs. And it is even more rediculous to claim that this resulted in a successfull/nice/rewarding trip. The problem with ITIL is that it cannot be put in numbers in such a way that it means anything for another organisation. This is of course a problem for salespeople. Not for me...

I wish people would approach ITIL as a guidebook

Hi Maarten, and welcome

very good point! I wish people would approach ITIL that way

The reality is however that ITIL is positioned as a "framework" and perceived by just about everyone in IT as something that you systematically do to revise your IT processes. ISO20000 just cements that attitude.

So they don't dip into it like a guidebook. They try to visit every temple and ruin and monument and cafe and guesthouse in town.

So since most ITIL projects do systematically implement all of ITIL, there ought to be a measurable difference to justify the six or seven figure bill for doing so. if there isn't then they shouldn't have done it. As I said ages ago on this blog: if I choose to renovate my house and overcapitalise as a result, it is my money and I can do what I like. But if I choose to renovate IT, I have a responsibility to those whose money it is to show that the value of their investment went up by more as a result. And if they are any sort of managers, they ought to be asking for evidence that it will before I start ripping up perfectly good carpets to polish the floors.

ITIL vs PMBOK

Perhaps it is my Project Management background, I concur with both of you. Because of my background (and some time as an Apprentice Chef) I had always approached the "Books" as an examble of "Best Practice" in PM Land the view is that the PMBOK is the collected knowledge of the profession. If you pass the Project Management Professional exam - a whole lot lighter than the ITIL Managers Certification - this person is in possession of the reasonable knowledge you would expect a PM professional to know. This all sounds a little like tort law in my country - essentially Duty of Care.

The ITIL books are not like the "Guide to the Project Mangagement Body of Knowledge" the Guide gives an overview of project management knowledge areas but provides little guidance on how to apply them - for that you need to acquire the multiple volumes that actually make up the Body of Knowldege. At no place in the Guide or the PMBoK do they prescribe a Methodolgy - they just tell you the ingredients of Project Management.

The ITIL books tell you how your cake should look but doesn't tell you the ingredients or the method to make it. This silence or gap, in my view, the cause for the difficulties and confusion surrounding ITIL. On all of my ITIL training the common question was "How do I implement this?" the common response was "run a project baseline where you are, then define the end state and build a workplan to get you to the new steady state." and I enjoy being told to suck eggs too.

I would put my project manager's hat on and ask the difficult questions like "What is the Business Imperiative? What happens if I do nothing? How much will it cost? Which process do I do first?" You would be deafend by the silence and then the class would move on.

In one of the sessions we had the "develop a business case for ITIL question", I develop the type of Business Case I was used to writing to get things over the line in a large commercial organisation and got less than 50% because it was not a Business Case in the ITIL world. That was when I knew I was back at uni - depsite knowing what would actually fly, give the instructors and examiners what they want.

Enough of this, I will write an entry in here on the real gold out there and that is the importance and criticality of work instructions to engender the organisational culture change.

Looking in her eyes I felt as near to Icarus as to the city before me as to the death of the sun. (The Long Road of the Junkmailer, Patrick Holland, 2006)

Syndicate content