An outage on the IT Skeptic website and why technology hates me

I hate technology. Really. I work with it every day, but I feel about it the same way that sanitation workers feel about their medium (or the way I HOPE they feel about it). I use technology when I have to, to do what I want to do. But I hate it. And it knows it and it hates me back. Take for example the weekend's outage on this blog.

First of all, sorry about that folks. Sorry for the interruption to service.

I can't contract out of accountability, so this is not an excuse, nor an attempt to weasel out of blame. But the technology picked its moment and attacked.

For most of the past year, I've been working at home. The first time I travel 10,000km from home, the VERY DAY I'm in transit, the following happens:

Dear WestHost Clients,

On Saturday, February 20th at approximately 2:20 p.m. MDT, we experienced some hardware failure as a result of an annual fire system inspection at our Data Center (DC). An inadvertent release of Inergen was triggered in the DC environment. The release was triggered by an actuator that was not removed as required in the DC's pre-test checklist. As such, the entire DC was impacted at some level, however not all servers were affected. This error was not the result of a mistake by WestHost employees, our hardware or systems. The mistake was triggered by an error from a vendor. With that said, we take responsibility and are doing everything we can to restore all systems completely.

There were multiple servers impacted by this outage; some of the servers have had damaged hardware replaced while other servers need to be restored from backups. A large majority of the servers that were affected have since had all services restored and are back online. We still have a few remaining servers that are experiencing problems. We are doing everything we can to restore those remaining servers as quickly as possible.

We apologize for any inconvenience or frustration this outage has caused you. We are committed to providing you with the highest level of service possible. We will be providing more information on your server status throughout the day. Thank you for your patience while we strive to improve our services. Please contact us if we can be of assistance.

Sincerely,

WestHost Management

Just like me they can't contract out of accountability, but hey! these things happen. As I will continue to remind readers, complex systems are inherently broken, and every now and then the broken bits align. On the other hand that doesn't rule out negligence. If there was negligence in this case, I say to that guy that this is a bad time to be unemployed.

But wait! there's more! I'm at the Pink Elephant ITSM Conference in Las Vegas (thank-you guys for the opportunity). Didn't bring a laptop. My PDA wouldn't connect with the Bellagio's wireless. Nor would a number of other folk's devices so it's nothing I did. See how the technology gangs up on me? (Incidentally I am writing this on a laptop loaned from Chris Dancy, the famous ServiceSphere, the maestro of service management social media. Thanks dude. Again. And thanks to Martin from Pink for lending me one yesterday.)

And not only are my websites down but so too are my mail servers. And gmail doesn't tell me it can't get to them. So I don't know. (Thank-you all the people who emailed or tweeted me to warn me - I really appreciate your support.)

Not only didn't I know about the outage but my wife didn't know I was safely in Vegas, because I emailed her to tell her so. She uses the company email server too. (I'm broke. Global cellphone roaming is one of the last remaining great telco scams - I refuse to participate).

I hate technology and it's mutual. (Chris is nervous about lending me his PC now).

Another reason I have this aversion to technology is to try to restore some balance to IT. Technology seduces IT people, and leads them astray, makes then think it's more important than process and people. Take for example Steve Chambers who made an attack on ITIL (and on this conference I'm at, and me personally) recently. Sometimes I thing Steve is too dependent on shock value in his posts. You can read his post if you want (so you get the idea, it's called "The ITIL believers are massing, Pink with embarrassment?"). My comment was:

Folks know I’m no fan of ITIL (and especially not of CMDB – ITSM should have stayed out of technology) so I agree with some of what you say. I take exception to your opinion that I have nothing to say myself in this space [because "Rob England, The IT Skeptic, is the biggest offender here and a self-professed technophobe. Yet he’s preaching to IT staff how to run themselves?!"].

Technology is important to ICT like trucks are important to transport companies. Can’t do it without them but if all you have is mechanics and drivers you’ll go broke. Process is and should be technology agnostic. The mapping comes in the work procedures which MOF does well and ITIL doesn’t try. (At a process level MOF and ITIL are of course very similar and if you think MOF offers more you are confusing process with procedure)

I studied Operations Management for my degree [before the 25 years I've spent working with IT technology] – I could tell a transport company quite a bit about how to arrange their logistics whilst knowing jack shit about how to drive a truck or fix a truck engine.

Mechanics have been the high priests of IT for way too long. It’s not about the technology any more. The reason ITIl got a following in the first place was because techoes with no process break things too often, don’t look after users properly, and have no way of engaging with customers. If techoes could work this out for themselves there would be no ITSM consultants. That’s why we keep you in the back rooms now :D

I could give you some great SAN horror stories – that doesn’t mean the idea of SANs is a bad one (maybe). Likewise ITSM and ITIL – it’s not the theory that’s the problem. Sometimes it is the consultants. As often as not it is the failure of cultural change, of which a consultant can only do and drive so much.

So if you think technology is the means and end of IT, how 'bout you stay in the back room and leave the rest of us to worry about improving IT and meeting customer needs.

Comments

Embrace technology but hate stupid people without processes

“On the other hand I'm not sure what provision a low-revenue-earning one-man organisation would make for alternate support. or DR.”
I almost wrote: go to the cloud, of course.
B,ut, somebody wrote an interesting post recently about the threats related to cloud computing:
“The top seven issues with Cloud Computing”
So, it is dangerous, too.
However, I still vote to Google an not to Westhost (or whatever) if I have to trust in their operations practices, technology and preparedness for an outage.

Secondary comment: you wrote you hated technology. Don’t do that. In this case (too, as usually) not the technology caused the issue but stupid people. Our job is to make people smarter than this. Technology has no brain, no feelings, doesn’t follow processes and doesn’t need endorsement or punishment. People do.
Don’t forget your mantra even in desperate situations: people first , than processes and only in the end of the line, technology.

The google cloud

Hi Ivan,

One of the issues was a communication failure between the hosting company and Rob. I guess that would not have been better with google....

You should think about the mantra as well, you propose a technology solution to something you yourself called a people issue. But yes, maybe googles people are better than Westhosts and their processes are better, but who knows what they are better for (google? you? the itskeptic?).

Even in such a case you will need a local "last-line-support".

Regards,
Marc
http://buzina.wordpress.com

I soooo get this post

Not only is this my job as an Incident/Problem/Crisis Manager to deal with this type of stuff, I am also a victim of it on more occasions than I choose to imagine. Its fine and dandy when I break something at home and get yelled at by my users there, but to have to manage our enterprise environment when something breaks and try to wrap some sense around it all is just maddening. But hey, I love technology in a sadist type of way.

Wish I went off to the Pink Elephant, but no funding from the Company for that this year, maybe next.

Good stuff as usual sir,

Rik

self-flagellation

hey Rik, long time no hear. Yes there is some kind of self-flagellation involved in working in IT isn't there?

Murphy

Hi Skep,

So you made the mistake of believing in technology? If you are the "final and last level of support" for the itskeptic.org service, you should have either a replacement available (and a proper means of communicating with your provider, them sending an email is not a proper communication service, they will not know if it reaches you) or you should have had your equipement ready. Or you just took a entrepreneurial risk which tends to drop you into the cacky sometimes.

And yes, technology has the intersting capability of sensing when a service is most vulnarable to failures and it will undoubtedly fail at that exact moment. That is a special case of the law that was discovered in 1949 by Edward A. Murphy jr.

http://buzina.wordpress.com

what provision

Perhaps the cobbler's children have no shoes. On the other hand I'm not sure what provision a low-revenue-earning one-man organisation would make for alternate support. or DR. Nor what communication mechanism a budget hosting service would use to their many thousand customers, other than email and web notices. Open to suggestions...?

Suggestions

Hi Rob,

Well, alternative support could be from similar low-revenue on-man shows, by ways of a gentlemans agreement. Combining strengths can help. As to a better communication method, yep sending the information about a server down to mail adresses located on the very servers can be avoided (I once received my mail password in that exact way, per mail to the address I was going to get!).

This is not to bicker about your quality of service (hey, your site has always been up when I tried!), but to talk about improving in difficult situations. So grab an alternate mail address & server from the large supply of free hosters, maybe use a service which sends a text message to your cellphone (roaming texting, is still rediculously expensive per byte but due to its size limit probably within the budget if used exclusively for this purpose) when it receives an email and request your hosting service to send this kind of info to both your mail adresses. Should work, especially after such a down.

So you had a major incident, now do a review, record a problem and try to reduce the impact, risk off re-occurance and find means to mitigate the risk.

You posted here, which is your medium for exchanging info and tapping into a large pool of knowlege, so you probably did...

Btw. Technology is the means, not the end to IT. But if it becomes, it is usually the end of IT ;-)

Regards,
Marc Buzina
http://buzina.wordpress.com

P.S.: If technology is inherently broken, why does it allways frustrate us when it breaks down? Maybe we should do something to reduce the impact upon our feelings....

Wooo spooky

OK now I'm getting paranoid about technology. I write a diatribe about how i hate technology and what happens? That post and only that post has a technical problem and is inaccessible to unregistered users. Et tu Drupal?

Syndicate content