The IT Skeptic - Comments for "ITIL's biggest hole: where is the meta-lifecycle?"

Detailed ITPI response

Charles T. Betz — Wed, 02 Jul 2008 02:42:22 +0000

Regression is a process which tests if variation in a dependent variable can be predicted through knowledge of the independent variables.

Methods vary depending based on the relationship (e.g. stochastic or deterministic). This relates to the nature of social science as compared to physical science. For example, you can compute the velocity of an object given known conditions.

This is not the case with systems based on human behavior. Change management, for example, is affected by many human variables. Worse, independent variables, such as other processes (e.g. Incident management), or skill/experience/staffing levels, workload, complexity of tasks, time of day, day of week, business cycles, and so on, are seldom in equilibrium. The degree to which these variables impact performance will always remain imprecise; “predictions” will not hold up well.

This is exactly the point of research, to attempt to sort through such multiplicities of factors (they surveyed on 57 variables) and determine “what really matters.” Variables suspected of not being in “equilibrium” need to be controlled for. This is all part and parcel of the method.

Your critique could be applied generally to all uses of regression, and in fact all attempts to discern causality, in complex domains.

I don’t know why you are reluctant to admit this, other than the fact that it aligns you rather firmly with strict positivist views of science that tend to be disdainful of all social sciences (including psychology, economics, management theory, and operations research).

I have confidence that ITPI's statisticians understand these basic issues, which are covered in introductory statistical courses.

The same with Change Management. For example, performance enacting 5 daily RFCs will differ from a day with 50000 RFCs, and not in a linear way.

What do you mean by “performance”? Changes not rolled back? Business value derived from the changes? What is your source for this assertion? Do you have a data set showing 50000 RFCs a day?

BTW, the largest organizations I have encountered, with IT budgets in the low billions US, might process on the order of ~10000 RFCs a month. You’re proposing an example two orders of magnitude larger. Have you worked in IT shops with budgets in the hundreds of billions? I do not think there are any IT organizations that large.

Linear regression does not work unless the variables are transformed with nonlinear techniques.
We have not yet established that the correlations are nonlinear.

Causation in human behavior is very often complex.
Of course. But are you saying understanding human behavior is impossible? That would be a philosophical statement, not an empirical one.

Performance of Change Management can be affected by things having nothing to do with the CM process (e.g. Incident Management). Linear regression works well when the impact of one independent variable has on the dependent variable, controlling for the presence of the other independent variables. This is fine as long as the independent variables are not strongly correlated to one another. If they are related (positively or negatively), then the results breakdown. It is difficult to sort out the effects of the independent variables on the dependent variable. If the problem is solved by deleting the independent variable, then the result is a seriously biased.

I question how thoroughly you have read the ITPI material. If you want to frame a critique here, you need to address their use of factor analysis, “a data reduction technique used to represent multiple observed random variables in terms of fewer derived or synthetic random variables called factors” (p.50); and their use of stepwise linear regression, in which “variables are automatically added or removed from the set to eventually develop the set of predictive variables that best accounts for the greatest variance in the dependent variable” (p. 55).

Statistical inference: A regression represents a hypothesis of how some performance can be explained.

Regression per se does not seek to explain or hypothesize. It simply identifies relationships. It is up to the social scientist to hypothesize explanations for the relationships.

Therefore, the model requires testing in order to determine whether or not its predictive ability is significantly better that what would have been obtained by chance or guessing (without knowledge of the independent variables). Where's ITPI's tests? Moreover, where is it's hypothesis?

The tests are clearly embedded in the method. They didn’t assert prediction unless the correlation was statistically significant. The null hypothesis is that “nothing matters,” that none of the practices make any difference. Therefore, ITPI’s hypothesis would seem to be that “something matters.” This is so obvious and trivial that yes, they probably assumed that the intelligent and charitable reader would infer it.

In personal correspondence, ITPI director Kurt Milne states:

“the formal hypothesis – (which we don’t write up in the report) is something like:

• “H: There exists a subset of change, config and release practices that impact IT operating performance to a greater extent than other change, config and release practices.
• “H0: There does not exist a subset of change, config and release practices that impact IT operating performance to a greater extent than other change, config and release practices. Any difference in performance is a matter of chance.

“When we do the analysis – it is definitely geared toward testing a hypothesis. We follow an analysis plan that we write before we collect data (to avoid exploratory research) to run the tests. We just don’t write all that up – because our target audience (IT decision maker) doesn’t want or need all the rigor of that documentation.”

Having now obtained some of this supporting evidence, it appears to me that ITPI has credible statisticians in their employ, using standard and accepted practices which account for the issues you raise here.

Normal Distribution - The use of regression assumes a normal distribution in all the variables. If the data is not normally distributed, you cannot make the statistical tests assumed in the model.
Milne also states:

“… we did run standard distribution tests and the stats guys were confident that we have a good distribution on independent variables.”

ITSM variables appear to lend themselves to power distribution laws rather than Gaussian distribution. I’d love for either you or ITPI to explain otherwise.

A categorical statement about “ITSM variables” is dubious. There are probably several dozen major data entities in the ITSM space at a conceptual level, each with potentially hundreds of individual attributes, representing all the various metric domains (categorical, ordinal, cardinal, real), and each variable described by a distinct distribution (assuming one could accumulate the data).

For example, duration of incidents might be a normal distribution, while financial impact of incidents probably does follow power law (relatively few incidents contribute to the majority of loss, certainly true in my experience). Chris Verhoef has done excellent work on IT portfolios, rigorously demonstrating that they actually follow lifetime distributions (neither normal nor power law.) See “Quantitative IT Portfolio Management,” Science of Computer Programming 45 (2002), 1-96.

ITPI’s statisticians, per Milne, have observed this empirically: “Some (like derived “top half count” are very normalized. Others have various distributions – one tailed, two tailed etc. My stats guys assured me that the regression analysis works with different shapes – if independent variables are normal.”

And there are others like heteroskedasticity or ensuring the absence of autocorrelation or path dependence, and so on... I'd explain but this debate is getting rather boring.

There are many terms and concepts one might pull out of statistical texts, and yes, describing them is tedious when any interested reader can easily look them up. The question is, does ITPI have access to professional statisticians who understand these issues? You seem to be making a poorly substantiated claim that their efforts are amateurish.

By Visitor (not verified) at Sat, 2008-04-12 16:03 reply
Concession
It occurs to me, a bit too late, this isn’t really about a critical analysis of ITPI’s research. This is about wanting something to be true; needing it to be true.

It’s similar to the phenomenon of those who need a certain candidate to win the presidency. Critical thinking left town a long time ago. Once the mind is made up, no amount of logic can effect change. Facts are subject to the interpretation of desire.

At the very least, there have been serious doubts placed at the feet of the ITPI research. Here’s my concession: If after all this you still believe the research is valid, go for it. Promote it, proselytize it, hand it to your boss, give it to your co-workers. I won’t object.

Unless I’m in the room.

By Visitor (not verified) at Sat, 2008-04-12 16:50 reply

You've not succeeded in raising serious doubts. And your parting shot is ad-hominem, obviously. Let me propose a counter-interpretation: you have some personal interest running counter to ITPI, perhaps because you work for a competing organization, or due to industry alliances. This would also explain your anonymity.

Alternative explanation #2: you jumped to ill considered conclusions regarding ITPI, and after actually obtaining their research and studying it, you have begun to realize the depth of your error and the difficulty of sustaining your case.

I think, pace Occam, either is a simpler explanation than psychoanalyzing me.

For myself, I admit no interest (beyond collegial) in the success or failure of ITPI, and I deny any emotional attachment. As an end consumer of the ITIL/ITSM intellectual enterprise, my primary concern is “what are the terms of the debate”? If mainstream operations research and statistical theory is insufficient to understand IT Service Management, then we are truly at sea with no compass.

Even Taleb admits that linear regression is appropriate in psychological survey research (page 245, The Black Swan), and the ITPI survey research is not far removed. I for one am not ready to abandon well established methods just because of the existence of black swans.

Charles T. Betz
http://www.erp4it.com

ISO SPICE & 15504

John Worthington aka MySvcMon — Tue, 01 Jul 2008 13:29:21 +0000

I'd be very interested to know more about any plans for ISO 20K; particularly regarding product compliance. I have not seen anything in that regard. I am aware that ISO has dropped the word 'software' out of the 15504 standard; it is now a 'process assessment' standard and work is being completed for ITSM based on ITIL V3 (that is my belief).

ISO 15504 also 'harmonizing' (see the SEI website) CMM, CMMI, etc. What I've been looking for is the 'official' V3 assessment guidance which I believe will be based on 15504.

Perhaps now that APMG is charging for 'complimentary guidance' we'll see some self assessment tools based on 15504 for V3 along the lines of what CobiT's doing?

Would be nice to get a clearer picture of what we can expect. If it includes product compliance (CMDB interoperability anyone?) that sounds like a pretty big bombshell.

Giving the big gorillas that kind of lever doesn't sound like a good thing to me, but an official statement on assessment --- with a maturity model that can enable incremental adoption --- would be very nice.

By the way, OGC did recently release a new book "Building an ITIL based Service Management Department.

John M. Worthington
MyServiceMonitor, LLC

MOF

Bordewijk — Fri, 09 May 2008 10:08:04 +0000

...and if I may add: MOF v4 (recently published: http://www.microsoft.com/mof). The MOF Team Model is still a very useful and open standard for implementing a role/team structure for all other ITSM initiatives. Don't be fooled by the "M" in its name: it is universally applicable. See the Global Best Practices book for guidance!

Maarten Bordewijk
Getronics PinkRoccade

the beat of the cultish drums

skeptic — Thu, 08 May 2008 22:48:09 +0000

I hear the beat of the cultish drums again: "There is only one true BOK and that BOK is ITIL. Hear ye all the truth and bury your false BOKs."

There are many paths to the top of the mountain. itSMF itself is responsible for two: ITIL and ITSM Library. Then there are SMBOK, ISM/FSM, FITS ...

Get your white hood off and get off my lawn.

Scared of a wider conversation or competition?

Ian Clayton — Thu, 08 May 2008 22:32:29 +0000

Have you read the SMBOK? Do you view SMBOK as an ITIL 'competitor'? If so, then I will suggest this is exactly the mindset we need to sweep away. ITIL can only get better through more open dialog and perhaps a competitor or two!

SMBOK and many other similar sources of good ideas are not designed to be 'competitive'. take for example the latest publication from Jan van Bon - ITSM Global Best Practices - an amazing effort to compile thoughts on how to 'do' service management. Just published, this book ads 620 pages of interesting concepts and ideas to the 675 in SMBOK. Add other books such as those written by Gronroos on Service Management & Marketing and you"ll discover a whole wealth of invaluable resources.

These publications, like ITIL, are attempts to contribute to the conversation we are all party to on service management. SMBOK attempts to understand what is out there and how to codify service management and good references as it relates to service provider organizations, services and key roles. It presents links to a vast source of pre-existing materials using a common framework and language. This is contrary to the traditional closed, exclusive approach of ITIL, and as clearly represented in your blog entry.

The SMBOK is popular because it is openly extensible, invites professional opinion and input, and actually includes ITIL as well as any other book or set of books that speak to service management. SMBOK is universal in its approach in that it is written to service management principles already in place within most businesses. It is largely IT agnostic. There in lies some excitement - a service management body of knowledge that spans both the IT and business 'perspectives' - and is in customer/business compatible language.

Why not allow open discussion on what service management is, what it is not, as well as the best and good practices to help folks achieve some degree of professionalism and success at service management? We need INCLUSIVE strategies not exclusive and many sources of knowledge from which we can all learn and mature.

As for it being out of print - thats true - sold out, it also needed a more mature and experienced publishing strategy and publisher. Small US clique - thats also true in part - not so much a clique as early adopters who are open to good sources of help and who are prepared to take it forward.

As for burying - lets leave that to customer opinion. Like ITIL, I suspect you"ll find out very soon the SMBOK, or rather is latest incarnation, is here to stay.... many feel thats a good thing...

SMBOK - does anyone use it?

Visitor — Thu, 08 May 2008 15:37:07 +0000

If SMBOK is so good, then how come the book is out of print, and no-one seems to use it outside a small US clique?

ITIL V3 is here to stay, and the rest of the known world outside of the US seem to be taking to it.

So lets bury SMBOK once and for all.

Nope

JamesFinister — Tue, 15 Apr 2008 20:25:36 +0000

I don't think the Ludic Fallacy is what is in play here. I do remember the black swan concept from my undergraduate philosophy days.

I can iamgine a platonic dialogue about risk.

sure

Visitor — Mon, 14 Apr 2008 15:24:58 +0000

If there is a specific area you have a question on, feel free to send a message to visitingskeptic@gmail.com. I don't want to weigh down the thread too badly.

Interesting!!

avallesalas — Mon, 14 Apr 2008 07:43:52 +0000

>>I'd explain but this debate is getting rather boring.

Hey visitor!! This seemed to be really interesting. May be its boring for you, but not for me because you are adding some science and maths here. Please could you explain?

:-) Thanks!
Antonio

close, but no free lunch

JamesFinister — Sun, 13 Apr 2008 14:26:56 +0000

Again, sort of, but not quite. Thanks by the way to everyone contributing to this line of thought.

Denial is clearly a symptom of the behaviour I'm thinking about, but is it an explanation?

Let me posit an idea. Most of us with a sceptical predisposition...no hold it, maybe that is the answer....I believe some research has been done on how teenagers assess risk. The answer. of course, is not very well, which comes as no surpise to me as the stepfather of three of them. Teenagers will tend to favour the high risk option even when presented with the evidence, because in the nicest possibly sense they are narcistic, and think the stats don't apply to them because they are special, but most grow out of it. Perhaps senior managers don't? Is the answer that a senior manager thinks "This is risky for everybody else, but not for me"

Meanwhile I will suggest a solution to a client whilst kjeeping my fingers croissed behind my back bedcause I know in the horrid real world my advice will only work if the planets align.

Denial?

Visitor — Sun, 13 Apr 2008 13:22:33 +0000

Denial?

Perhaps a varient on it?

JamesFinister — Sun, 13 Apr 2008 09:06:36 +0000

I've often used cognitive dissonance as a concept in my lecturing days, and another useful related concept is that of change blindness. I'm not quite sure though that it conveys the precise thinking pattern amongst senior management that I'm thinking of, though I am sure that it is an important contributing factor.

I suspect coginitve dissonance cuts in quite late in the day, when managers try to hold on to a low risk world view despite the existence of contrary eveidence of things going wrong. The root of the problemn seems to start much earlier, when managers "switch off" when it comes to risk and not to understand what they are being told.

Well... it has a name

avallesalas — Sat, 12 Apr 2008 23:15:38 +0000

it is called Cognitive Dissonance.. people want it to be true. :-)

Very reasonable research.

Visitor — Sat, 12 Apr 2008 18:26:01 +0000

Very reasonable research. The data was taken for what it was and the conclusions did not overreach.

I particularly liked this caveat:
"This survey suffers from the fundamental bias that all surveys suffer from: it doesn't capture the opinions of people who aren't willing to fill out a survey."

Poetry.

Concession

Visitor — Sat, 12 Apr 2008 16:50:24 +0000

It occurs to me, a bit too late, this isn’t really about a critical analysis of ITPI’s research. This is about wanting something to be true; needing it to be true.

At the very least, there have been serious doubts placed at the feet of the ITPI research. Here’s my concession: If after all this you still believe the research is valid, go for it. Promote it, proselytize it, hand it to your boss, give it to your co-workers. I won’t object.

Unless I’m in the room.

Not yet it doesn't

JamesFinister — Sat, 12 Apr 2008 16:31:09 +0000

No, the current version of the standard doesn't have a maturity model, but a draft Part Four of the standard does exist, which will provide a maturity model of some form. Currently it is based on SPICE.

Term of art? So we are back

Visitor — Sat, 12 Apr 2008 16:03:36 +0000

Term of art? So we are back to attacking things other than the idea. Some would argue ERP4IT is not a term of art. But that argument is equally silly since “term of art” is wholly subjective and based on this proclivity for confirmation bias rather than evidence.

I will succumb to my own confirmation bias and point out that "Ludic Fallacy" is quite topical on Wall Street (to describe the sub prime mess and such), the UN (to describe the lead up to the Iraq war), the IMF and a whole community of statisticians seeking to evolve their profession. Philosophers have generally ignored it because, well, it’s a very old idea going back to John Stuart Mills. IT managers can certainly benefit from a little of this knowledge.

I'm not against the social sciences; only science done improperly. I'm not against skeptical bias; I'm against bias for confirmation. I'm not against research; I'm against being a sucker.

For example, I’m not saying linear regression is rarely done on human behavior. I’m saying it is often misused, as in the case of ITPI's analysis:

Regression is a process which tests if variation in a dependent variable can be predicted through knowledge of the independent variables. Methods vary depending based on the relationship (e.g. stochastic or deterministic). This relates to the nature of social science as compared to physical science. For example, you can compute the velocity of an object given known conditions.

For linear regression to work, certain assumptions must be met.
- Linearity: assumes the variables in question have a constant proportionality in the relationship. When driving speed doubles, the distance needed to brake increases fourfold. It is not a linear relationship. It is curvilinear. The same with Change Management. For example, performance enacting 5 daily RFCs will differ from a day with 50000 RFCs, and not in a linear way. Linear regression does not work unless the variables are transformed with nonlinear techniques.

- No Multi-collinearity : Causation in human behavior is very often complex. Performance of Change Management can be affected by things having nothing to do with the CM process (e.g. Incident Management). Linear regression works well when the impact of one independent variable has on the dependent variable, controlling for the presence of the other independent variables. This is fine as long as the independent variables are not strongly correlated to one another. If they are related (positively or negatively), then the results breakdown. It is difficult to sort out the effects of the independent variables on the dependent variable. If the problem is solved by deleting the independent variable, then the result is a seriously biased.

- Statistical inference: A regression represents a hypothesis of how some performance can be explained. Therefore, the model requires testing in order to determine whether or not its predictive ability is significantly better that what would have been obtained by chance or guessing (without knowledge of the independent variables). Where's ITPI's tests? Moreover, where is it's hypothesis?

- Normal Distribution - The use of regression assumes a normal distribution in all the variables. If the data is not normally distributed, you cannot make the statistical tests assumed in the model. ITSM variables appear to lend themselves to power distribution laws rather than Gaussian distribution. I’d love for either you or ITPI to explain otherwise.

And there are others like heteroskedasticity or ensuring the absence of autocorrelation or path dependence, and so on... I'd explain but this debate is getting rather boring.

This may sound complicated, but its not. But it does take hard work.

Not a term of art

Charles T. Betz — Sat, 12 Apr 2008 04:41:09 +0000

You can split hairs about the date it was coined, but the point is that it is not a generally recognized term of art or philosophy, and throwing it gratuitously into the debate makes bystanders feel like they "live under a rock," i.e. out of touch and unqualified to debate.

By the way, quite a catchall of social dysfunctions attributable to the Ludic Fallacy you have there. Don't you think equating ITPI with the current credit mess is a bit over the top?

Glad to see you aren't writing off the social sciences wholesale. We'll return to that in a bit. Are you saying that applying linear regression to human behavior is rarely done?

On another topic, I would be interested in anyone's opinion on this survey, run by a noted Agile Methods practitioner:

http://www.ambysoft.com/surveys/dataManagementAugust2006.html

Charles T. Betz
http://www.erp4it.com

Very few people have seen the IT Skeptic and ANYONE else

skeptic — Sat, 12 Apr 2008 01:33:54 +0000

Very few people have seen the IT Skeptic and ANYONE else in a room together. You get your chance in Holland at the end of this month

Have you noticed no one has

Visitor — Fri, 11 Apr 2008 19:03:32 +0000

Have you noticed no one has ever seen Taleb or The ITSkeptic in the same room at the same time? Their skeptical styles bear a remarkable similarity. :-)

The IT Skeptic - Comments for "ITIL&#039;s biggest hole: where is the meta-lifecycle?"

Detailed ITPI response

ISO SPICE & 15504

MOF

the beat of the cultish drums

Scared of a wider conversation or competition?

SMBOK - does anyone use it?

Nope

sure

Interesting!!

close, but no free lunch

Denial?

Perhaps a varient on it?

Well... it has a name

Very reasonable research.

Concession

Not yet it doesn't

Term of art? So we are back

Not a term of art

Very few people have seen the IT Skeptic and ANYONE else

Have you noticed no one has

The IT Skeptic - Comments for "ITIL's biggest hole: where is the meta-lifecycle?"