The IT Skeptic - Comments for "DevOps and traditional ITSM - why DevOps won't change the world any time soon"

Don't drink and drive

John Worthington aka MySvcMon — Fri, 02 Sep 2011 01:20:41 +0000

love the post... also liked very much Jez's Continuous Delivery post very much!

Truth be told, whether you're on a DevOps or ITIL binge really doesn't matter.... it's really all the same Kool-Aid. If DevOps results in both Dev and Ops working together to understand and capture end-to-end service dependencies then by all means let's get the party started! Sure beats the finger-pointing that seems to go on right now (in which both Dev and Ops lose). This paragraph said it all to me:

"Because operations personnel get to see the application from early on in its development, they can feed their requirements (such as monitoring capabilities) into the development process. Because developers get to see the system in production, they can run realistic load tests from early on and get rapid feedback on whether the architecture of the application will support its cross-functional requirements. This early and iterative release process also exerts a strong pressure to ensure that the system is production ready throughout its lifecycle."

While establishing these Change, Config and Release processes --- and achieving real service monitoring intelligence --- is not easy, the concept of continuous delivery is right on... and much of it right from the Good Books.

My hope is that posts like this can help each of us reach a better understanding of the other (Dev...Ops) and share a few laughs along the way.

So let's go have that beer, but let's not drink and drive!

John M. Worthington
Third Sky, Inc.

The DevOps Binge...

John Worthington aka MySvcMon — Thu, 01 Sep 2011 22:03:14 +0000

Been missin' you Skep. I need to make a mental note to read this post in full detail, but this is what came to mind… can't wait for the ITIL/DevOps post!!

I was somewhere around Change Management when the ITIL took hold. I remember saying something like, "this virtualization and cloud computing stuff is awesome…" and suddenly there was yelling and screaming all around us, and Dev was screaming: "let us drive! We can get there faster!!" No point in mentioning the huge pothole right in front of us, the poor bastards will see it soon enough. Our organization had the full compliment of ITIL, PMI, COBIT, ITSMBOK, et al. Not that we really needed or used all of this, but we were definitely into Best Practice. The only thing that worried me was DevOps. There is nothing more helpless and irresponsible and depraved than an IT shop in the depths of a DevOps binge. And I knew we'd get to this pretty soon.

John M. Worthington
Third Sky, Inc.

still in the playpen

skeptic — Tue, 30 Aug 2011 16:35:24 +0000

Yeah it's coming, but right now Agile and DevOps are still encapsulated. And I'm starting to think they will be for a long time - limited to those systems where high rates of change are competitive and hence worth the risk (see comments above)

To continue the analogy: they're still in the high-sided play-pen with all sharp objects moved out of reach :)

Dealing with the kids

Ian Mitchell — Tue, 30 Aug 2011 13:31:14 +0000

This is an excellent post, and each of the points you make is quite defensible. However the time at which the car keys can be handed over may not be as far off as you think.

Firstly, DSDM is behaving in a responsible manner; the beard is trimmed and he's wearing a suit. Secondly, it is quite possible to encapsulate XP or Scrum crusties within a PRINCE2 work package. I have done this myself on a number of occasions.

agile change

skeptic — Tue, 30 Aug 2011 02:39:00 +0000

Jez, I don't recall seeing your ITIL change management in an agile fashion post. It is a superb post, everyone should read it, especially bigots at both ends of the spectrum. It lays out precisely how ITIL Change doesn't have to get in the way.

But how telling that the very first comment you got was in the "this'll never work with those ops assholes" style.

And the next one from Peter Brookes was on "how do you have continuous training and awareness of end-users and support?" It is all very well to change software at lightning speed but you can't change process at the same pace and you certainly can't change people that fast. Peter's point is:
When someone calls the service desk to say they don't understand the new feature, how the heck do the service desk know what they are talking about?
When someone calls to make a new kind of request, likewise how does anyone know how to action that?

Your other post I don't agree with so much. i think "if you don't do all your strategic development in agile you won't be around in 10-20 years" is melodramatic. Well actually its not because the majority of organisations wont be around in 20 years anyway :)
If extremely high rates of change in a system is a strategic essential in order to compete, then you better be considering high rates of change. But that is true of only a small percentage of all "strategic" systems which are in turn a small percentage of all systems - most are utility. I maintain my position that agile development and agile ops are niche techniques. Most of the time stability wins.

but going back to your first post, i am still in favour of the concepts in any case. Change Mgmt will only ever achieve successful cultural acceptance if it minimises pain and maximises efficiency, which is all about Standard Change and easy CAB and all the "agile change" you describe :)

Creating resilient systems

Jez Humble — Mon, 29 Aug 2011 23:23:50 +0000

I accept that there is a deeply ingrained difference in mindset between developers and operations - a combination of a (lack of) experience and the fact that dev is measured by throughput and ops by stability - and I would never allow devs carte blanche access to prod. In fact, I don't think anybody should ever be logging directly into prod except in an emergency.

The problem is that the barrier that has been put up has, in many cases, actually made things worse overall in terms of the ability of organizations to manage risk. This isn't the fault of ITIL and COBIT, but it is the fault of many of the people implementing ITIL and COBIT, and many of the people auditing these implementations who come from an accountancy background. I like reading your blog because, as you say, you are pretty spot on in your attacks on ITIL zealotry. I actually think ITIL and agile can co-exist - I blogged on doing [http://continuousdelivery.com/2010/11/continuous-delivery-and-itil-change-management ITIL change management in an agile fashion] - and I co-authored an article in the August 2011 edition of the Cutter IT journal which discusses at a high level how they can work together.

You're quite correct to say that unwinding 30 years of IT process evolution is madness. It needs to be evolutionary change, backed by evidence. Devs need to prove they can create production-ready software and reduce the risk of releases before ops should begin to trust them and give them more access to environments. However in order to do that, devs need help from ops - the ability to spin up production-like environments for testing purposes, for example - there is a long list. Devs also need to understand what happens in ops, which is why they should be doing L3 support, rotating through ops, and carrying pagers. But fundamentally I believe - and I have seen evidence that shows - that stability and increasing the rate of change are not a zero-sum game, and that in fact increasing the rate of change can actually improve the stability of production systems, *if you are disciplined about your delivery process*.

I do take issue with your claim that "most DevOps practitioners know more about dev than they do about ops, and the minority who do know ops think that it is all about tools and automation". Automation and tools are definitely part of the story, but I've always said that progress with automation should happen incrementally as part of a programme of continuous improvement rather than as an end in itself, and that tools are never the right place to start, although of course they are important to enabling any automation effort.

As you say though, the proof will be in the pudding. Like you, I am not advocating that people drop everything and go agile / devops - but [http://continuousdelivery.com/2011/01/strategic-vs-utility-services/ for strategic projects under active development], it definitely does make sense. If enterprises don't start adopting agile for these kinds of projects, many of them are not going to be around in 10-20 years' time.

Silos

Information Technology Indoctrination Library — Mon, 29 Aug 2011 23:00:13 +0000

You write, "Not only are silos inevitable in any organization with more than say 100 IT staff, but silos are not a bad thing when managed well."

Recognition of the division of labor at last!

Some context. In my former place of employment, three of six directors were moved under directors that the VP hired. Our titles were changed from director to manager. This wasn't a disciplinary action: there was no notation in anyone's personnel record. It was a "consolidation". The DevOpportunists had triumphed.

Exhibit A. My new manager said he did not know what my department (research computing) did, but call it a "little silo" despite admitting he didn't know what it was.

Exhibit B. We were required to attend many more astronomically dull change management meetings. Directors and managers of all departments had to attend to the ephemera of Windowlogical systems administurbation and superficial reports on the cleaning of digital bed pans, whether this was relevant to their job. Not to do so would encourage "silos".

Exhibit C. Job descriptions were changed to generic civil service titles, for "flexibility", we were informed. Another move to break down "silos". My job description no longer reflected my education, interest and abilities--I said so at my exit interview.

Exhibit D. We were told that we needed to work with others. Research computing had collaborated on some technically challenging projects with this director's group, but they were discounted. "Maybe you did, but I would fault you for not immediately consulting with everyone else before changing anything. You solve your own problems. By not immediately consulting with everyone else in the department, you're limited by what you know.” That we were the only group with the requisite experience wasn't the point. The perceived failure to consult with others first, no matter their level of interest, availability and expertise, was a "cultural" problem.

After a few repetitions of this mind-numbingly boring drivel, I decided that I was wasting my time in IT and that I would no longer support anyone else's research. Those days were gone. Masterful inactivity in not a panacea: while it could extend the range of fools one could suffer gladly, it cannot do so indefinitely. I would do my own research, or get out of academia altogether. I quickly obtained a research position elsewhere, and handed in my resignation.

In fact, I had waited too long to get out. Once research computing falls in the hands of IT, it ceases to be research computing, if it is subject to the same kinds of risk management appropriate for service, as opposed to research. Development is somewhere in between--tensions are inevitable since managers want to expand the domain of their responsibility. The principal-agent problem again.

---

Pressed for justification, a Change Manager can lose his equanimity. ”Change, change, change, change, change! That’s what we’re about!” It was an uncharacteristic outburst for a Christ-like figure with his own nimbus.

I have been critical of the absence of citations in these discussions. Let be provide a few on the subject of change.

“We believe from molecular evidence that fig wasps and fig trees have been evolving together for over 60 million years,” says Dr. Compton. “Now we have fossil confirmation that gets us a bit closer to that date. Although we often think of the world as constantly changing, what this fossil gives us is an example of something remaining unchanged for tens of millions of years — something which in biology we call ‘stasis’.”

— Science Daily (June 16, 2010), World’s Oldest Fig Wasp Fossil Proves That If It Works, Don’t Change It.

But innovation, like “change,” has no inherent value. It can be bad … as well as good.

— Joseph Stiglitz, Capitaiist Fools

You don’t need a Nobel Prize to state the obvious, but sometimes you need a Nobel Laureate to endorse it.

Love the tone

Information Technology Indoctrination Library — Mon, 29 Aug 2011 21:06:16 +0000

My this is patronizing: "My friend.. take a deep breath. Change is hard, especially when on the outset it tends to disrupt something you represent .. very well. My advice is "Lean into it"... My guess in a year from now it won't seem as bad as it does now. Remember cloud?"

I fail to discern any reference to the literature in these remarks. I see yet another manifestation of the principle-agent problem: managers attempting to control each other's behavior. Repulsive. State your argument, if you have one. Cite the literature. Show us the data. Access to and control of data processing systems is one thing; if only capturing system and time logs amounted to the analytical prowess of a doctorate in sociology.

Make a sociologically substantive statement, "my friend." State the definitions of "culture" and "behavior" as these are understood in various IT methodologies. If these are deficient, improve them. Define those terms in a way that can be measured. Compare this with the "culture" and "behavior" of control groups versus some methodologically enlightened groups.

cloud and devops

skeptic — Mon, 29 Aug 2011 17:59:00 +0000

PS. You talk about Cloud as if it is a solved problem, old hat. No its not. There are still huge legal, business, logistic and management issues around Cloud. Just like DevOps the jury is still out.

You guys no doubt think I'm closed-minded. I already recommend cloud--hosted servers and SaaS apps to clients where appropriate but that doesnt mean I wholeheartedly embrace Cloud, see it as the panacea, or regard it with anything less than the deepest suspicion until time proves it generally safe.

So it is with DevOps. I can see me recommending it in certain circumstances, but in general I see dangers to watch for, potential failure points. Only the tests of time will address those.

talking to each other

skeptic — Mon, 29 Aug 2011 17:48:00 +0000

John, I expect an enlightened view from you, and I like your writings on the topic.

But even with your exposition of CAMS or CALMS I still struggle to get a consistent view of what DevOps actually is. Like any new movement it is whatever followers define it to be.

And many of them define it to be a way to overcome evil change control and release processes. Holding up worst-case scenarios of bureaucratic change control as Jez does as a justification for rejecting processes is as bad as my characterization of dev techs. Change management without Standard Change is inexcusable - if only ITIL made that explicit. And I'm a huge fan of people actually talking to each other... but so is any good manager.

Even more folk, including yourself, define it as a cultural movement to get dev and ops to work more closely together. I'm all for that. See my Dead Cat Syndrome. I fail to see why we need Automation or agile cross-disciplinary teams to achieve that. And our course Lean, Measurement and Sharing have always been essentials of ITSM.

So the only unique aspects of DevOps I can identify are a passion for ops automation, a belief that agile scales, and a desire to tunnel through traditional IT production controls. Those are where we disagree.

Incidentally my new improvement methodology Tipu advocates agile methods for process change. It is only software change where agile makes me nervous. I believe it is a primary function of IT Management (I don't like "Ops" - too narrow and techie) to be calming influence, to promote stability, risk control and maintenance of big picture and ling term considerations. Rapidly changing software with reduced controls over architecture, continuity, securoty, integrity etc does not help. I'm all for agile development up to pilot phase. I'm happy for dev to run up infrastructure for dev and test in an automated fashion. I'm cool with level3 techs having controlled access to prod to restart machines. But beyond that I think a risk-averse culture is not only appropriate, it is professional responsibility.

Now it may be in years to come that the anti-ops factions fall away from DevOps, and Agile becomes proven to be safe, and deployment and control can be automated in cost effective and low risk manner. That will be a happy day.

Until then i'll go on promoting standard change, operational readiness (dev/ops collaboration), evolutionary process improvement, and most of all cultural change programmes, all within existing ITSM frameworks.

The same old argument...

John Willis — Mon, 29 Aug 2011 16:23:12 +0000

Rob,

Since you are new to this debate I'll give you the benefit of doubt. However, your arguments has been used for over a year now and each time it is used it is debunked and the authors tend to come around (remember cloud?).

Short and sweet...

1) Devops s not about Cowboys. There are cowboys in every disciple; however, cowboys are not welcome in true devops implementations.

2) Agile dones't scale... See Jez's comment... Absolutely silly and border line ignorant..

3) Devops wants to create a race of super human systems operations folk. Not even close. See CALMS (Culture, Automation, Lean and Sharing). That's what defines #devops.

4) Operations is relegated to hacks and not useful. Wrong bunch dude, pick a fight with the NoOps guys. Operations as a strategic weapon is one of the core fundamentals of Devops.

5) Devops is tool centric. CALMS starts with culture. We don't even talk about automation until behavior is understood. Yes we want to automate everything; however, we know that not everything can be automated. What's wrong with that ideal?

My friend.. take a deep breath. Change is hard, especially when on the outset it tends to disrupt something you represent .. very well. My advice is "Lean into it"... My guess in a year from now it won't seem as bad as it does now. Remember cloud?

moderate DevOps

skeptic — Mon, 29 Aug 2011 15:22:00 +0000

So if you mix in moderate DevOps circles, what do you make of all the assumptions I cite? If I accept that I'm being over-defensive and DevOps are not the barbarians at the gate wanting to undo all the good work (which I don't yet), then let's get past the cultural clash issues: convince me it will work in a traditional moderate-to-large scale legacy-IT setting. I say it won't scale.

And convince me there is some difference from the way we do IT now other than improvements which good ITSM is striving for anyway. Some visions of DevOps sound like the status quo but with a sprinkling of new automation toys. In that case DevOps is equally attacking a strawman "failed IT" which doesn't relate to actual experience.

doesn't reflect the attitudes

Zac Stevens — Mon, 29 Aug 2011 10:41:00 +0000

"I don't doubt that some of the core thinkers of DevOps such as yourself are Ops-sympathetic or even ex-Ops. But DevOps as a movement still has a hefty element of SmashOps, which drives that polarisation you speak of."

I'm really baffled by this - the DevOps strawman you attack in this article doesn't reflect the attitudes of any of the self-identified DevOps crowd that I've spoken with. Straw polls at meet-ups generally find 50+% of attendees working in operations (I'm one of them).

The "NoOps" crowd are derided by the DevOps folks as entirely missing the point, for exactly the reasons you mention. I don't see credible calls to minimise or eliminate Operations (unless you're giving far too much credence to Forrester), nor for system administrators et al to have an active role in writing application code. A team of specialists collaborating effectively can create better outcomes than a team of generalists sharing only shallow experience with the various skills required.

I hope that your forthcoming "DevOps and ITIL" article will focus a little more on those facets (or interpretations) of DevOps that you think are compatible or shared with ITIL, and the ITSM view of the world. Dismissing all of DevOps on account of a few loony interpretations is no less counterproductive than dismissing all of ITIL because some organisations make change management their business prevention process. Well-run ITSM may be the foundation on which the DevOps culture can flourish, and wouldn't it be great if a bunch more people were actively working to increase the number of organisations that are doing it right?

I'll be looking through your archives as you've clearly got a lot of valuable experience to learn from, and I don't imagine the audience you had in mind for this piece is the crowd who develop a nervous twitch at the very mention of ITSM.

Thanks for writing it, in any event.

Zac

an awful lot of proving

skeptic — Mon, 29 Aug 2011 06:42:48 +0000

Hey I'm an active proponent of cultural change - it is my primary focus these days. And I'm all for a big part of that being better communication between groups.

Likewise I'm also a big believer in ITIL-done-properly, and this blog has a long track record of attacking ITIL zealots and ITIL-the-cult.

Especially i agree that testing, change control, release, and deployment should be as slick and painless as possible. But no more. Controls are there for a reason.

So I'm on page with some of DevOps' primary goals.

I don't doubt that some of the core thinkers of DevOps such as yourself are Ops-sympathetic or even ex-Ops. But DevOps as a movement still has a hefty element of SmashOps, which drives that polarisation you speak of.

Dev are from Mars and Ops are from Venus. there is a huge cultural disconnect and I for one don't believe it is artificial. As one comment I read put it: "the gap between dev and ops didn't used to be there. It was put there for a reason". Right now it seems to me most DevOps practitioners know more about dev than they do about ops, and the minority who do know ops think that it is all about tools and automation. To me Ops is about managing and stewarding (and protecting) the business's IT services to provide service delivery to agreed standards, and adopting new services with maximum value and minimum risk to the organisation. Not many puppets and chefs in that definition.

Right now DevOps is a good idea finding its way. The suggestion that I might propose to most of my clients that they unwind thirty years of IT process evolution and open the gates to their precious production systems today on the basis that it seems a good idea is just laughable. I'm watching with interest but there is an awful lot of proving to be done first.

DevOps and Agile are risky

Jez Humble — Mon, 29 Aug 2011 05:13:56 +0000

Part of the reason I think DevOps is kept so nebulous is so that you have to engage with the actual arguments rather than using ad hominem attacks, although I admit that they can be a lot of fun.

However, the truth is that a bunch of us - including several of the people who co-authored the agile manifesto - actually come from a background working in huge IT shops where we perpetually saw big IT projects fail miserably - both in terms of value delivered, and in terms of scalability, availability, security, business continuity etc.

There are a bunch of criticisms I have of your post, but my main one is against your statement that "Agile and DevOps are risky"

Aaargh!

Agile is not risky! Cowboy coding is risky! I thought that we got over the whole Agile = cowboy coding thing ten years ago. Apparently not. Agile requires high levels of discipline and is focussed on "satisfy[ing] the customer through early and continuous delivery of valuable software" (first principle). High quality, scalability, availability and the rest should be part of the definition of "valuable" if that's what's important to the customer. If you're really doing agile, you're using empirical data to identify and manage risks - for example by ensuring that cross-functional characteristics of your software are validated continuously from early on in the delivery process, rather than waiting until right at the end of the project to get a production-like environment to validate them in. Agile - done properly - is all about effective risk management.

However implementing segregation of duties by not allowing dev and ops to communicate, and implementing a change management process where somebody has to send an Excel spreadsheet to somebody in another country who has no idea how to evaluate the risk of the change - both implementations of ITIL & COBIT that I have seen in real life - are really, really terrible ways to manage risk.

Personally, I think that agile and devops done right actually have a lot in common with ITIL and COBIT done right. Throwing bricks at the other tribes, while entertaining, is exactly the problem that devops is trying to solve.

Your point about process is also way off base. Many of us are discussing process and trying to catalogue good practices (I co-authored a book which is an attempt to do exactly this). We talk a lot about measurement, quality control, improvement and many of the other things you mention. Yes, agile says "individuals and interactions over processes and tools", but that doesn't mean that process isn't valuable, just that you can't compensate for having crappy people by imposing a bunch of process.

However this leads me to where I think you are actually dead on target: "Agile and DevOps philosophies are predicated on an unrealistically rosy view of the standard of people available". Agile is definitely glass-half-full in this respect, and one of the conversations I have a lot with my Chinese colleagues is how the hell they're going to be able to get teams of tens of thousands of people to effectively adopt agile methodologies. Agile depends on having enough responsible adults to enforce the discipline necessary to deliver high-quality software. However, your criticism doesn't just apply to agile - it's true of *any* process. Good luck if you think any process implementation is going to make a bunch of Wallys magically deliver high performance, scalable, maintainable software (I am pretty sure you don't think this, but it's a corollary of your statements that "you can't manage large groups without [process]" and that it's impossible to find enough good people to scale agile).

To say that "You cannot use the 'empowered professionals' model on a large scale across the industry. You need accountability, formal controls, and performance management" is a false dichotomy. Believe me, I - and all sensible agilistas - are all for accountability, controls, and performance management. What I am against is when the controls are so rigid that team is unable to engage in continuous improvement because they're not allowed to make any changes to their process.

Finally, I'm totally in agreement with your statement that when you have a cross-functional team responsible for managing a service throughout its lifecycle (I dislike the term "noops" because the team should include ops people), that the SLA should state that this team "is responsible for operations and support, and accountable for availability, continuity and integrity". I am all for devs carrying pagers, performing L3 support, triaging critical incidents, and rotating through ops team, because it's the only way for them to learn how to write production-ready software.

The devops conferences I and my colleagues have attended in Europe - where the movement started - were overwhelmingly attended by folks from ops background - and much of the focus of the movement is finding ways to make devs accountable for what they do. It's a movement that cuts both ways. The tragedy is that many devs see it as ops folk bashing them, and many ops people see it as devs taking a pop at them - exactly a symptom of the problem the movement is trying to address.

The IT Skeptic - Comments for "DevOps and traditional ITSM - why DevOps won&#039;t change the world any time soon"