The IT Skeptic - Comments for "The Emperor has no clothes. Where is the evidence for ITIL?"

Not halo effect

Charles T. Betz — Mon, 08 Dec 2008 00:58:42 +0000

I question your scholarship, and believe your own “confirmation bias” is evident in your selective quotes.

Bringing Rosenzweig into the argument was a case of a hammer seeking a nail. He is not critiquing operations research in his work; he is going after bigger fish (like Tom Peters) who are attempting to explain organizational financial performance – a much more dubious enterprise.

“This is classic halo-effect. Why were these organizations "recognized" for their "operational excellence"? Is this the hypothesis? If so, it is already contaminated…
the hypothesis (and test questions) emerged/changed as the data took shape? After the study began? I don't think this is quite in keeping with the scientific method…. the survey responses are shaping the measures of performance?”

You seem to be deliberately overlooking the fact that the survey was executed in two parts. First, they established the questions. That is where the iterative interview method was applied. The actual survey data gathering was not part of this first phase. When they got to the second phase of surveying the 341 organizations, the questions were fixed in form. This is the second time I have pointed this out.

Every survey needs a starting point. You may ask, “how did they identify the 11”? Provided with an answer, you could go on to question the assumptions behind that. And so on, endlessly denying the antecedent. They had to bootstrap the thing somehow.

”… the survey responses are defining change success? How is confirmation bias kept from contaminating the data?”

Your own anti-ITPI confirmation bias is showing through again here. While it is true that change success rate is also a dependent variable, asking the respondents how they define change success actually reduces confirmation bias. It decomposes the general assertion of “change success” into three less ambiguous factors:

(1) functional success (did what it was intended to do),
(2) timing success (done when it was supposed to be), and
(3) adherence to build procedure (done how it was supposed to be).

They established that the top performers consistently used the more rigorous definition.

”Why is a particular practice linked to performance? What is the evidence-based logic? …Or is performance based on the fact that managers “believe” it works, attributed to measures they have in common with other managers? Are there any tests for refutation or are they all simply based on confirmation. The scientific method requires that these questions be asked"

Refutation is implied by non-correlation. One particularly disappointing aspect of the study for me personally was that it did not support using a CMDB to assess downstream impact as predictive of improved performance (there was a weak correlation, but it didn’t “make the cut”).

The very act of interviews/surveys shape results. People perform better when they know they are being observed. Ask any attorney. How is this kept from influencing the conclusions? For example, data before the practices were implemented, control group, etc.

The Hawthorne effect is not applicable in this case, since there weren’t people in white coats with clipboards running around these companies’ operations distracting people. Summary indices of performance were gathered through one-time surveys.

Control data is notoriously difficult to come by in any but the hardest sciences. It’s an Achilles heel for operations research as well as for much of economics and social sciences. All one can do is accept the limitations. In ITSM, it would be tantamount to asking some company, “Can you stop doing Incident and Change Management for a while”? At this point, these practices are culturally embedded in IT, and no-one is going to abstain from them in organizations of a certain size. If that is your critique, you are essentially saying that no research is possible in this area.

Very disconcerting is the lack of visibility on the variables *before* the time period covered by the data on the performance that these practices presumably predict. By using words like “predict”, the study is arguing causal order. How do we know?

Again, a combination of bootstrap problem and law of available data. And, don’t agree with your semantics. The word “predict” is a more tentative assertion than the word “cause.”

Should I go on? There's more...

I’m quite curious if you actually have something. Please continue. And I still don’t know what you meant by saying the work is not credible because it is based on inductive reasoning.

This whole debate continued here:

http://www.itskeptic.org/node/376

Charles T. Betz
http://www.erp4it.com

Are actionable service catalogs the silver bullet...?

ianclayton — Fri, 28 Nov 2008 04:47:23 +0000

With all due respect to Kevin, Gene and George, the Visible Ops approach is similar to Lean. It sets laudable goals of finding and eliminating waste and improving overall efficiency. It drives the process through problem then change and sets up a continuous (not continual) mechanism. I wish more IT folks would check Lean out - the success I have had in 2008 with my Lean Service Management program has been significant. Anyway, I wanted to speak to the catalog comment.

A word to the wise for all these ITSM projects that have set out to build a laundry list of artifacts and then replace their 'processes' one by one - STOP. The humble service request may well be your key to success. In many sites it represents 80% of workload drivers. Package everything that might consume IT resources in a request, including incidents. No need to separate them. Now we have a single prioritization and workload management option.

Next, interview your customers and slowly and carefully match their activities to the requests (I can explain how but not here). Build catalogs based upon what customers/user DO. Focus on efficient processing of service requests, and specific ones that cause problems or are resource intensive.

Best of all - employ workflow management tools to manage and automate these requests. Connect a portal, a catalog and request management and you are actually doing Lean Service Management with all the pull, flow and Lean Consumption/Provision buzzwords in play - oh I think some folks call than an 'actionable service catalog'.

If you are attending any practitioner training - ask them how to do this - if they can't tell you over the phone, or don't know - save your money.

Where do you start - learn how to do problem management for real - don't reference ITIL - it sucks - use a healthcare view... Understand how to define a problem and its impact and translate it into an opportunity for improvement. Oh I am back at Lean again.

If you want to discuss this (Lean) more - drop me a line... ian@servicemanagement101.com

Visible Ops

david.stucky — Thu, 27 Nov 2008 06:57:19 +0000

Those interested in measurement of ITIL's effectiveness should check out The Visible Ops Handbook or, more broadly, the work of the IT Process Institute (http://www.itpi.org). They're not vendor aligned, have done excellent work on benchmarking, assessing the benefits of good practice, etc.

Some of ITIL's claims are more easily cashed out than others. Visible Ops focuses largely on Change, Config, Incident, and Problem Management. The link between process and business value is pretty straightforward in those realms, primarly because every incident ticket costs money and the processes mentioned have a very direct ability to reduce the number of incidents which arise or to reduce time/effort required for resolution.

Other ITIL claims are more difficult to cash out. Personally, I've done work with organizations on Service Catalog and Portfolio Management which have easily demonstrable financial return, primarly because we work directly with financials to guide our efforts. I'm not aware of any academic research reflecting such work. Similarly, Capacity Management can be readily tied to financial efficiencies associated with improved technology acquisition practices.

likely to show a significant return

skeptic — Thu, 28 Aug 2008 21:48:00 +0000

Agreed many busines cases are dodgy. But there is some good data around quality management, at least in manufacturing. It may be that IT is inherently hard to measure, but if so I think that reflects on our lack of sophistication in our techniques more than fundamentals of the industry.

I do believe IT ideas are measurable, at least to the level of "likely to show a significant return". i don't think we even know that much about ITIL yet.

Different from any other "business cases"

Keith Hamburger — Wed, 27 Aug 2008 22:13:58 +0000

But how are the ITIL business cases any different from the thousands of other business cases presented by, and to, MBAs to justify all of the other management systems out there?

Do you really think it can be clearly quantified that Six Sigma is far better than TQM (or vice versa)?

Do you think the numbers presented to show ROI on a quality control initiative are really any less blue sky than the same attempt for ITIL or any other ITSM initiative?

Those sort of things are used to justify to the bean counters that an improvement will help. If improvement is needed I don't think it really matters how "real" the numbers are. If improvement isn't really needed hopefully someone will call bovine feces on the salesman who's trying to pass on something that's not likely to show a significant return.

the world is full of people who want to believe

skeptic — Wed, 27 Aug 2008 21:28:41 +0000

Hi Keith and welcome. thanks for contributing!

I agree with all you say. ITIL's value probably can never be measured to the final satisfaction of all (but we CAN measure much better than we do, which is why I applaud the proposed new itSMFI Journal).

My issue is that business cases are floated that quote all sorts of Crap Factoids about the positive returns of ITIL. And ITIL is sold as this magic pixie dust that makes IT work better.

People like you know better. many don't. the world is full of people who want to believe. See the scene from Life of Brian [Brian of Nazareth not Brian Johnson] with the followers of the sandal vs the gourd. Or many other scenes from that classic film ("How shall we **** off oh Master?").

I'm a bit worried at those who would take Real ITSM seriously.

Value of ITIL probably impossible to measure

Keith Hamburger — Wed, 27 Aug 2008 18:13:50 +0000

Being a religious agnostic I have a hard time taking anything "on faith" but, as others have mentioned, one probably has to do so with most all management systems. This reminds me of the discussions as to what the "best" quality management system is, of which there is probably no answer.

The value of ITIL, and other management systems, is most likely impossible to measure and compare to other options. However, it is unlikely that there is truly any significant difference between one management system and another. The value comes from having a framework to guide your direction for improvement. Most likely any well designed framework will do.

To give an example from a project I'm currently working on, I'm working with the corporate HQ of a company that has grown through acquisitions and divestitures. Currently corporate is managing two organizations, each hundreds of miles from each other and hundreds of miles from corporate, that were acquired while all other segments of the company are gone. One organization had its own internal IT while the other had contracted IT that was brought "in-house" following the acquisition.

There are no standard processes between each the organizations and with corporate. There are no standard SLAs and ticket handling control is so lax that any that did exist would be meaningless. Our task is to design processes for service operations that are consistent for the entire organization while trying to minimize the disruption to the organization.

While not considered a full ITIL implementation we are using ITIL as our guideline to developing these processes with an eye to a potential expansion of ITIL to the entire IT organization in the future. This means that our processes, while not necessarily complete, have to be consistent with the ITIL framework.

Is this an ITIL implementation when considering ROI?
If this fails to have a positive ROI in terms of $$$ does that make the project a failure? Or, are there other considerations and values that must be considered?
Is it really possible to consider all of the factors that this impacts when calculating the return on the effort?

And, as mentioned in another previous comment, the primary driver of this effort is the need to upgrade their service desk software to a new version before support for the old version is discontinued. In such a case is it possible to seperate the return from the upgrade of tools from the better implementation of processes?

I would have to say that it is impossible to really determine the value of the effort in any really hard numbers. It just has to be taken on faith that putting more structure to the organization provides a return compared to the lack of structure currently in place. And, given that, it is just as impossible to compare one framework with another to determine which is better.

If an organization has a solid structure in place it will be far harder to show a positive return from changing that structure to one that is more ITIL oriented. If an organization has minimal structure, however, ITIL is a fine set of guidelines to use in implementing a structure and will almost certainly prove to be of benefit.

Just my $.20 (inflation, you know)

Why we may NEVER have evidence of ITIL

skeptic — Fri, 22 Aug 2008 05:58:41 +0000

From "Diarmid" on the itSMFI forum:

Firstly, there is the issue of defining what you mean by "the implementation of ITIL". Technically it would be better to say "application" rather than "implementation". But the more serious point is how you define an organization as having applied ITIL. Do you evaluate their service management against some yardstick that says that they have definitely embraced ITIL in all (or most of) its glory. Or do you accept them if they "talk ITIL" and claim to be using it?

Secondly, how do you distinguish between organizations that have "discovered" ITIL and used it to radically improve their service management and those that have discovered ITIL, looked at it, realized that basically that is what they already do and then embraced the ITIL terminology because they like it?

Thirdly, there is the chicken and egg situation. It could be argued that most organizations will move toward ITIL practices when when they are being driven by change all around them. So they know they want to change and they see that ITIL is a good way to go. At most you could say that ITIL was one enabler for the change, but quite likely ITIL will have been no more than the direction that their change took. It is also the case that organizational change will be going on anyway and in some organizations is always going on and is often superficial. This can muddy the waters.

Finally, you will find practical difficulties in obtaining much useful data from more than a few organizations and you may have difficulty in determining how "typical" your sample is.

For the outcome of your research to have any value these are some of the issues that you will need to deal with.

It is my view that ITIL is essentially about process and control and can therefore be accommodated within a wide range of organizational models. Thus organizational change would rarely be essential although sometimes a logical consequence.

Working out just fine

JamesFinister — Sat, 09 Aug 2008 11:42:43 +0000

Amongst all the ITIL hype a message that gets forgotten is that in most organisations the true value of using ITIL is not the mythical "become a world class service management organisation by the end of the year" but to bring a way of working that is broken back under control to deliver a consistent service that is fit for people to use, and that they can trust. It is unexciting and dull and doesn't make front page news when you achieve it - but that's just fine and how it should be.

In theory the more broken your way of working is to start with the more benefit ITIL should deliver.

Evidence

sporty — Sat, 09 Aug 2008 07:51:00 +0000

Oooh I don't know

I use ITIL in my infrastructure and its working out just fine. What kind of evidence do you need?

[link removed as suspected link-spam, sorry]

fair enough

JoePearson — Thu, 22 May 2008 07:23:40 +0000

*sigh* OK, that's fair. It would add something to the value of reading the discussion though ... I can interpret and understand better the arguments by yourself or Charles Betz, in the context of all your other writings. Not to discount anything as bias, but to get a better feel for where the writer is coming from. We are all human after all.

Anonymity

skeptic — Thu, 22 May 2008 00:25:19 +0000

if it is who I think it is, they have a good reason not to. Anonymity is acceptable on this blog, though I'm happier if you tell me privately who you are and why. But ultimately it is a personal decision.

Could "Visitor", who's

JoePearson — Wed, 21 May 2008 18:46:45 +0000

Could "Visitor", who's posted opposing the value of the ITPI research, identify him/herself? I assume the posts are all one person.

Analysts produce crap

skeptic — Wed, 09 Apr 2008 21:45:11 +0000

It is obviously time I published this post that I have been working on

My concern is more

Visitor — Wed, 09 Apr 2008 16:32:40 +0000

My concern is more fundamental. Research prompts beliefs which prompt action. If research is done improperly then it produces half-truths that are partly right at times but flawed enough to get organizations in trouble. Because it arrives in a cloak of science and objectivity, these half-truths get repeated time again and again.

"Process excellence is vital."
"The best organizations have the best people."
"Productivity through people."

When half-truths are treated as the whole truth, they cause severe damage to careers, companies, and employee quality of life.

Management is hard. It is difficult to lead and manage organizations; there are no simple, easy formulas. It requires devotion to learning the craft. Including better ways of thinking about business knowledge.

Applies to a good bit of IT

Visitor — Wed, 09 Apr 2008 13:55:34 +0000

Sounds like they were describing quite a few IT management practices. They quickest way to spot an IT model or practice based on pseudo-science is to simply ask, "How do you know?"

It's a great form of entertainment.

A look at the 62-page report

Visitor — Wed, 09 Apr 2008 13:46:20 +0000

A look at the 62-page report reveals:

Page 11: "We started with interviews of 11 organizations recognized for their data center operational excellence."

This is classic halo-effect. Why were these organizations "recognized" for their "operational excellence"? Is this the hypothesis? If so, it is already contaminated.

Page 11: “Interview data collection and analysis proceeded iteratively with early interviews being more open-ended, and later interviews being directed by five emerging themes of common practice.”

So the hypothesis (and test questions) emerged/changed as the data took shape? After the study began? I don't think this is quite in keeping with the scientific method.

Page 11: “We also asked about key measures they used to gauge success in these areas.” Page 11.

So the survey responses are shaping the measures of performance? How is false attribution mitigated?

Page 21: (Actual survey question) “Which of the following best describes your definition of ‘change success rate’”?

So the survey responses are defining change success? How is confirmation bias kept from contaminating the data?

Why is a particular practice linked to performance? What is the evidence-based logic?

Or is performance based on the fact that managers “believe” it works, attributed to measures they have in common with other managers? Are there any tests for refutation or are they all simply based on confirmation. The scientific method requires that these questions be asked.

Should I go on? There's more...

Pseudo Science

JamesFinister — Wed, 09 Apr 2008 05:57:50 +0000

I went to an academic conference the other day on the subject of philosophy's role in exposing pseudo-science. Various criteria put forward for deciding something falls into this dubious category included:

- Anachronistic ideas that have already proven unworkable
- Appeal to myths
- A grab bag approach to evidence
- Irrefutable hypotheses
- Argument from spurious similarity
- Refusal to revise in the light of criticism
(source:Radner)

- Escape hatch hypotheses
- No self correction
- Focus on confirmation not refutation
- Excessive relience on anecdotal evidence
- No peer review
- Absence of connectivity with other knowledge
- Use of impressive sounding jargon
- Failure to specify settings under which claims do not hold good
(source:Lilienfeld)

Thank goodness ITIL could never be accused of any of those things!

My concerns with research

skeptic — Wed, 09 Apr 2008 03:40:00 +0000

Hmm.. good question. All that.

My concerns are that research is

commissioned to prove a point, like cancer research paid for by the tobacco industry but with less observers ready to scream "foul"
created as a revenue generating exercise, therefore the results need to be useful, attention getting and self-serving (grow the market)
often anecdotal and opinion-based
Often asked of the wrong person: "How brilliant were you..." "Did you make the right decision to..." "what ROI have you had from your spending..."
lacking transparency (and hence impossible to reproduce): what was the methodology? what questions were actually asked? how was the sample derived? what controls were there (generally none)? what were the raw results?
lacking peer review ('cept blogs like this). Where are the professional journals and conferecnes with real review boards?

Funny I just finished writing a post on that very topic which will show up on the blog here soon.

Fair enough

Charles T. Betz — Wed, 09 Apr 2008 03:13:27 +0000

Agree 100%, as stated previously, that this should be openly published. But I won't criticize anyone's business model until I have walked a mile in their moccasins.

Does your discomfort with "research" in the ITSM field center around the fact that so much of it is survey-based, by necessity, with all the abuses that surveys are prone to? Or does it center around the fact that most of it is directly industry funded? Is it method, or sponsorship, or both? Or something else entirely?

Charles T. Betz
http://www.erp4it.com