IT geeks resistant to change (management): buffalo in a china shop
What is it with IT geeks' inability to accept the need for change control? Here is a beautiful example of how they just don't get it.
The OGC Change Log is a wonderful resource - I should have spent more time browsing it. Take a look at issue 142:
Issue: 142, (Closed) for ITIL v3 Glossary and Acronyms (Created:10-01-08) (Updated:25-02-09)
Request to add the following definition to the glossary: Repair: A change that is considered low risk and follows a defined procedure that is performed to return hardware or infrastructure configuration item to proper operation after a hardware componant failed, creating a change in the operational state of the hardware or infrastructure configuration item. For example, the “repair” of a server with a failed hard drive, is performed by “changing” the failed hard drive with a new one of like model or an equivalent one that is sized to be the same. Within the server’s system internal hardware configuration, the hard drive serial number will change from the old drive to the new one. This is in actuality a “change” that does not affect the established normal functional operation of the server that was “repaired”. If the drive serial number attributes are being tracked as part of the server CI attributes within the CMDB, then the change of the hard drive will be reflected in the change
Two recent experiences of mine:
Tech goes out to a major branch office to change one failed hardrive in the site server. For reasons I neither recall nor care, server gets taken out just on shift change when all staff want to do their timesheets. It takes two days to fully restore function.
Tech who is working late drops into server room in another building on his way home to swap a failed harddrive in the SAN. For reasons I neither etc etc he takes out the entire SAN. The only SAN, there is no failover. The entire organisation is disabled for several days.
They just don't get it. Any change is a change, and it is only a standard change if it is demonstrably quickly recoverable. Any change - no matter how minor - should be discussed with as many peers as possible before being performed. Every disaster involves more than one cause: you need to know what contributing factors are already present that will turn your routine little disk swap into a major calamity.
This is all part of the same crap about setting IT techs free to innovate. IT geeks say "You must learn to trust us. We are intelligent sensitive creatures. We must be allowed to run free, like the buffalo." Unfortunately boys (and it's usually boys who talk this way), IT isn't a wide open plain, it's a china shop of fragile systems. Put the nose-ring on.
Made in New Zealand 
Comments
Actually, there is a reason...
Science has discovered that there is biological reason for resistance to change. Activating patterns of learned action originate from the primitive brain stem. For example, tying a tie or your shoes. Once you learn a task, it takes virtually no glucose (e.g., thinking) to "replay" it. We often call this "muscle" memory (it's not really memory stored in muscles though.) When this type of action occurs the brain is rewarded with the feel-good endorphin serotonin. Thinking (that is, not using "muscle memory" but rather the modern parts of the brain burns loads more calories and and this the brain is counter-incented to think and learn with the "pain hormone" cortisol. Bottom line, when people say they hate change and learning makes their heads hurt they are correct. You can see it on a brain scan.
Simply said, the pleasure is the opposite of change, quite literally and at a very basic level. Change of anything -- from shaving before the shower to after -- or asking someone if you can turn off the SAN -- is not something the brain makes easy or pleasant. This fact is often left out of organizational change, culture and climate efforts. It has only been known for a short time, and perhaps making people understand this fact about our shared human nature and how our bodies really work can factor into improving things.
Then again maybe not.
Excellent Entertainment
I have to suspect that this is entirely fake and bogus as it is so enormously entertaining. All the way from OGC change log to poultry mammaries.
Even in the relative backwater of UK educational technical support guys like this have long since learned to keep their deluded rantings to themselves. I'm going to use this thread as a teaching point - hope you don't mind.
Thank you IT Skeptic.
Alex Jones
Thanks El Reg
It looks like the PFY from those BOFH stories made it all the way from The Register. Congrats to The IT Skeptic for landing such a great series of IT rants. I bet they cost a $ or 2. I look forward to the next eposiode.
it's real
For those mystified, PFY from BOFH is Pimply-Faced Youth from Bastard Operator From Hell, a VERY funny series: if you haven't read it, recommended.
I swear, Daran and DemneMaol are real, unpaid, self-motivated contributors. And of course nobody can make Ian do anything. So this is a legit unstaged unscripted discussion. Amazing innit?
Overhead?
Working for a major outsourcing company, I urge all IT department to take this approach and grind down in minor details. Our sales staff will be contacting your IT Governance board shortly. ITIL is already commonly associated with bureaucracy and delays, and blocking even the most minor change won't make it any friends. (Of course, internally we use many aspects of ITIL, but our sales weasels like to invent their own name(tm), so we can usually avoid the taint, while still putting all the checkboxes in 'requirements bingo'.)
stifling your genius
Oh poor Daran, is the mean old bureaucracy "grinding down" your technical freedon, stifling your genius, taking away your opportunities to demonstrate your swift, deft, masculine command of technology? get over it man.
If it is a standard change, go ahead. That's the point of standard change - it is demonstrated that somebody has already thought about what you are about to do.
if it is not a standard change then your potential for error is higher and you have a professional responsibility to check that what you are about to do - no matter how apparently minor - is not going to make you look an idiot, and incidentally ruin the day of tens, hundreds or thousands of other people, on a scale somewhere from minor annoyance to death. In both the cases I cited, the latter was a possibility - we're not talking paper shufflers here.
if you haven't twigged yet that what you do, every minor thing, affects the lives of others (and the profitability of your outsourcing employer) in grossly amplified ways, and you are professionally responsible for that, its time you did something else. "Shit happens" is no longer an acceptable explanation in IT.
Whoa, did I hit a sore nerve
Whoa, did I hit a sore nerve or what? Read a newspaper, watch television: in real life, shit does happen. So it was in this case: disk fails (shit one), failure cascade as techie replaces disk (shit two, and swapping a hot swappable disk may well have been a pre-approved minor change), organization needs two days to recover from error (shit three). And in case you are not paying attention, the only failure both avoidable and major is the organization which needs two days to recover (unless this is the agreed SLA, in which case, why did you bring it up as an example?). If the techie didn't use a forklift to clear out the server room, failure of a disk cabinet or server should be within the expected disaster recovery parameters and therefore not result in two days downtime.
Of course, the relevant manager will likely have blamed the techie as a scapegoat for his failure to organize his department as agreed in the SLA, and a very good chance that general management has swallowed the story due to lack of insight in the world of IT. And, the conclusion of the management may well be that they need less techies, and more bureaucrats, excuse me, change managers.
In normal language a policy of
is generally refered to as 'cover your ass'. Any organization that depends on a 'cover your ass' policy to get work done - human nature being what it is - will be needing external parties to get any actual work done. As stated, my employer is happy to take up that role.
People keep telling me that driving my car is affecting the lives of billions through global warming; to be honest, I'm still not 'twigging' that. The majority of my work is because 'shit happens'. I don't pretend that if the documents prescribe that shit doesn't happen that it won't. I look for it, I prepare for it, I test for it, I route around it and I get good results. And if my project brief requires me to get everything signed off, it will be signed off; especially as - again human nature - the CAB will not stand in the way of it's own management when bonuses are on the line.
critical mass
In both cases the error was eminently preventable if someone had just said "I'm planning to swap this disk" which is what change management is all about
I'm running off a combination of poor memory and only passing interest in things technical but it was something like this:
Case 1: the server had been previously swapped but the config process/system wasn't good enough to tell you that (yet). Although there are normally two servers as failovers to each other for reasons I forget this site temporarily had one. He had the wrong type of harddrive. because it was a remote location the field guy asked to do it while he was out there was not a server expert. Plenty of people knew what server was really at that site, just not the one who rang the field guy
Case 2: being a cash-strapped public entity the money for the redundant failover SAN had never eventuated. The business had insisted the system go live without it and had signed off on the risk (but never understood the implications of course until it all went dark for two days). Being chronically short of spare parts (that $$ again) someone had "borrowed" two drives out of the SAN to fix failed drives in a smaller SAN knowing that it would cope with one or two out and that the replacements were on order. It can't cope with three out.
Every disaster has at least two causes (watch Air Crash Investigations to learn that lesson). If you announce a change in advance you have a good chance someone will know about another pre-existing cause and stop you achieving critical mass.
These systems are delicate. We have to stop charging about. It is simple professionalism
There is a difference
There is a difference between 'stop charging about', and 'get sign-off from the entire company before moving'. Endless waves of communication are not the way to go, as critical messages get swamped by the noise, and people set up filters and block lists in an attempt to get work done. In my experience all work in progress has / should have an owner, so everyone should be able to either quickly get (or be redirected to) the needed information / sign-off / expertise, or equally quickly discover that the situation is not under control (no or unavailable owner) and extra care is required.
Change is a mechanism that forces people to stop and think
You're right, I shouldn't have said "as many peers as possible" - it was an exaggeration (they happen now and then round here) - I should have said "all the right people".
Change control is a mechanism that forces people to stop and think. it ensures they have checked with the owner, that they have a backout plan when it all goes pear-shaped, that they do it at the right time for minimum impact on service, and - one we haven't touched on yet - that they won't inadvertently screw something else going on at the same time. Change control is our friend; it stops us hurting people and it stops us getting egg on face.
I'm a big fan of Standard Change. Much of the bad rap Change Control gets is because change managers get normal change control in place and stop there. they don't go on to then implement Standard Change
the quote that set this whole thread off said "A change that is considered low risk and follows a defined procedure..." and wanted to call it Repair. Now either they have no knowledge of what standard change is and want to reinvent it, or they consider standard change STILL too bureaucratic. What really got me going was they then used disk swaps as a good example. As I have described above, two big disasters I've seen in recent years both came from disk swaps and were both preventable with a little communication (and/or enough money to not compromise redundancy), and both clearly demonstrated to me that server-level or SAN-level disk swaps are not automatically low risk.
For any change, you need to talk to all the right people, not just mutter over the partition "just gonna go swap the disk in Alpha". When I said "as many as possible" what I meant was: think again - is there someone else who might have the crucial bit of info on why you shouldn't do this?
Skeptic is wrong again
Skeptic - cop on. You're dealing with professionals when you're dealing with IT. ITIL is bureaucracy of the highest order. You haven't a clue about IT, that's why you embrace ITIL. The technical guy knows what he's doing. If he didn't then he was hired for a job he couldn't do - HR problem and hiring manager. If he is sent to do something he doesn't know how to do it's his managers fault. Given he is still hired and sent to do a job - the assumption must be that he knows what to do. Let him do it. CHANGE MANAGEMENT? TOTAL TOSS OUT OF YOUR FEEBLE MIND. Daran is right and you know it. If there is no appropriate RTOs or RPOs and a system is down for 2 days then the failing is with the guy who made the agreement - probably a SALES problem - or the SERVICES TEAM who cannot live up to the agreement - not the techie. What are you flapping about? Did you not tell the customer you need 2 days to recover? Did you tell him something else and then not bother to put the appropriate measures in place to meet the SLA? Why then was there a problem? Did you send the wrong techie? Did you train him as part of your services team to ensure he could do the job? Maybe there was a different problem entirely when he got onsite. Do you have a process in place to handle that? "For reasons I neither etc. etc" - typical - this is why you have no clue. You are disconnected from reality and in an ivory tower. As far as I'm concerned stay there. You're probably less damage there, and definitely out of the way of real work. Keep fiddling with your ITIL stick. Leave the real work to people who have REAL jobs and make a difference. You don't. You are a pointlessly babbling nitwit who latched onto a fluffy little cloud called ITIL and hide in it. When's the last time you actually had any positive impact on IT? Give us a reference from a technical team. Useless. Your talk is drivel because you are not qualified to comment on technology and YOU KNOW THAT.
Just my opinion.
Regards,
DemneMaol
A real eye opener!
Wow!
<"You haven't a clue about IT, that's why you embrace ITIL" >
- the same is true for business and customer who don't care a **** about having a clue about IT . Of course, we "techies" should not care about them!
<"The technical guy knows what he's doing">
- and others are all morons. Oh yeah including the paying customer. Leave the techie alone!!!
<"If he didn't, then he was hired for a job he couldn't do - HR problem and hiring manager. If he is sent to do something he doesn't know how to do it's his managers fault. Given he is still hired and sent to do a job - the assumption must be that he knows what to do. Let him do it.">
I like that approach. I think you are right! I have second thoughts now - ITIL is all bull shit. We should go for techies! Blame all others except .... "techie".
Didn't you know? techies and their freedom is all important - who cares about down time? business impact? we have the "Service team" to blame for that. Don't touch the techie (who don't belong to the service team - you know!)
<"Did you train him as part of your services team to ensure he could do the job?" >
Oh, I thought we have established that the technical guy knows what he's doing!!! Oh sorry the blame has to go to some one else... right!
"Maybe there was a different problem entirely when he got onsite. Do you have a process in place to handle that?"
- yeah there was an apparent problem of the techie 'believeing' "he knows what he's doing". and the process is what (I believe) skep was talking about - CHANGE MANAGEMENT!! (oh sorry you meant some "technical process"?)
To summarize - for benefit of all others : all are wrong and useless and morons.. - except a special, untouchable and brillant group of people called "techies" who don't form part of the so-called "service team"!
Got you!
To demonstrate my understanding in a remote analogy: in our village, landlords who depend on their coconut farms and revenue from it, are suffering because of the "professional" climbers (not sure what else to call them) of the coconut trees (oh yeah - it really requires some skills, jokes aside!) put up and stick to their terms and conditions. If they say 2 days, it is 2 days. If they say only so many coconuts were there, go with that. The owners of the farm haven't got any other way, as they "don't have a clue about the coconut tree climbing". Now I understand, they should leave it to those professionals and keep quiet. They know what they are doing!
Dear Skep, take the advice, dont try to get into ivoray tower that techies have been building and enjoying all the while please!
And of course don't hide behind ITIL (be a little skeptic on ITIL for a change!! :-))
Vinod
Do chickens have tits?
Its rare that folks make themselves look like a complete tit on these blogs. Some of us have to struggle over many responses, I'll admit my share of attempts. But you have managed to do this with one blog entry. Congratulations, if the Olympics awarded medals for the 4 x 100 tit race you would win by a mile.
And you did it anonymously, which in my book makes you even more of a tit.
ITIL is not a bureaucracy. Its a set of books. Its like a gun. Guns don't kill people, people do.
Skep doesn't embrace ITIL. He is one of the few who have rocked their boat.
Skep's too much of a gentleman to be drawn into too much of this - tonight I'm not. I think you are full of bollox and up to a con trick trying to bait for some reason only you really care about. And.. I think you are a tit. A complete and utter tit with a capital 'T'.
Its a pity Skep has to put all this effort and money in to run a site like this so a TIT like you can prove that...
If you are going to personally attack someone - at least have the balls to say it in person. Chicken.... Do chickens have tits?
the free-range-IT culture
It's cool. I like the model that all cultural change is bereavement, which I learnt from Troy Dumoulin: applying the classic cycle of denial, anger, sadness, acceptance...
We can watch those who cling to the free-range-IT culture going through it. For ten years we've whittled away at their homelands, denied them their traditional hunting, shut down their rituals, taken away their totems. Of course there is a backlash. It is hard for them to accept they can't wander all over the servers whenever they like. No longer can they wear their root domain passwords as a badge of office. They have to dress in qualifications instead of parading naked with nothing but a CV to cover them. They live inside fences, behind locked doors, instead of sheltering in flimsy hovels made from Perl and html. You have to give them time to adapt, to get the dirt out from between their toes, to eat in a cafeteria instead of cold pizza off a keyboard, to not kill the plants in their cubicles. Sadly, some of them never will learn civilised ways...
thanks
'You are a pointlessly babbling nitwit who latched onto a fluffy little cloud called ITIL and hide in it." Woohoo, check out the dying cries of the old guard. This is such a classic example I begin to suspect you are faking it just to make me happy. if so, thanks.
Minor is as minor does
Anyone who has been around IT for more than a few years will have their own war stories of the "minor" change that brought an organisation to its knees*. The point very often is that techies decide it is minor because of the work that is involved, not the consequences of getting it wrong. Perhaps Rob should have said "no matter how apparently minor"
However, our mutual get out of jail free card is "follows a defined procedure" though I would add "that is itself subject to change control"
So a techie can change a hard drive if there is pre-existing approval that that particular drive can be swapped out in service hours. Possibly they might have to agree it with the service manager as well, who could know of exceptional reasons why it shouldn't be done today.
Trust me, when the techie gets the decision wrong the repercussions are not nice**, commercially or personally.
James
*Where do I begin? "There was this time we changed the phones in the offices and no one realised they wouldn't fit the acoustic couplers." Is that showing my age?
Fast forward 20 years and it becomes "We moved to DECT phones but no one realised you couldn't fit the PSUs into adjacent power sockets because they were too big"
** I ended up having to draft an apology to the UK's Parliament for having apparently misled them about London's crime figures when an OLE change that was supposed to allow a job to accept multiple tapes ended up with the same tape being read twice.