Controlling Application Development
This post has been podcast
How does IT Operations get control of "the other half"? How do we stop them chucking dead cats over the wall for us to own? It is pretty straightforward.
We have recently been discussing Application Management, and there have been several comments about getting those developers under control.
IMHO, it is pretty simple to get some basic control.
- Define a "production readiness" standard (does ITIL offer any help at all here?)
- Get it included in the architecture: in design specs, in requirements gathering, in deliverables, in required infrastructure. The Architects usually go along with this readily - they are mostly non-partisan on the Dev/Prod divide.
- Get CIO mandate for: "no Operations signoff on production readiness, no production change". This is the hard bit but it is easy to make a business case for this: cost of failures and degraded service, cost of manual support processes, cost of re-working apps to make them supportable
- Push in to dev project-startup meetings and make it known this is how things are from now on - nicely. Then help.
- Try to get as much as possible retrofitted into in-flight projects - lots of coaching and sleeves-up help required here. Be seen to be helping.
- Get a "head on a pike": make a project late by refusing signoff. The more fuss and debate the better. Whether you win or lose this fight in the end does not matter. Project managers like a smooth ride. They'll take note and get readiness engineered in to their other projects.
As one comment said, it is all about communication. This is a People issue not a Process one. (Nor a technology one: "we need better lifecycle tools"). Go along to project meetings, especially in the early days and especially those projects that make you nervous.
- Explain why we need this stuff to make a project into a successful service.
- Explain how much of it is design thought and documented information and only a little of it is actual engineering in the system.
- Explain how Operations wants to work with the project team to help design it and document it (then deliver on that promise).
- Explain what will happen if it isn't there. Tell the story of the project that didn't play nicely.
Sound straightforward? Correct? What did I miss?
Made in New Zealand 
Comments
A few other steps
When we introduced the service assurance role at AXA to oversee the service transition process there were a few other key points that made it successful.
We had a mixed team that betweent them had a combibation of skills that gave us credibility with Apps - a mix of service management, security, audit, development and operations.
We built on existing strong relationships with other stakeholders, including the technical architects (I already sat on the architecture review board) our outsourced supplier, capacity management, the testers, and very obviously, the business.
Building on that last point, we made it clear we ran the process on behalf of the business. the decision to go live was always a business decision based on our input and assessment of risk.
Risk dominated our approach. We identified the key points in the development lifecycle where we needed to interact with projects. We adjusted how much involvement we had with a project, and the criteria for allowing it in to production based on the risk attached to the project. We didn't have the resource availble to sit on top of every single project.
We made sure we provided a service to projects that they recognised as such - and generated a demand for our involvement that exceeded our capacity. Most welcomed was our help in clarifying NFR specifications, and in dveloping a programme of testing for service.
James
Wonderful practical feedback from the real world
Wonderful practical feedback from the real world. thankyou James. key points for me:
"the decision to go live was always a business decision". Quite right. The go/no-go on the prod RFC is the business's based on prod readiness assessment/signoff from Ops.
"we provided a service to projects". yes, more than just bogging in and helping it needs to be positioned as a service.
"NFR"? NFL is Not Very Likely so NFR is...?
Thanks skep
Maggie Kneller deserves a special mention as the then Head of Service Delivery for giving me a free hand to design and implement the approach, and also Ken Goff whose ideas I creatively enlarged upon.
Re-reading what I wrote reminds me the effort we put into avoiding a confrontational approach, and there were times (particular kudos to my then colleague Kevin) when this enabled us to make difficult calls that projects were not in a fit state, but we did so in good time, and always with advance warning of our concerns before the final review meeting to give the project a chance to address them.
NFRs - Non Functional Requirements - which we ensured were joined up with key components of our SLAs. We ensured these were, if I remember our mantra,"complete, clear, and coherent". Personally I found discussing with the test team how we were going to test conformance with the NFRs very useful as a quality check.
Oh Skeptic, if only reality was so simple...
Asking nicely and making impassioned presentations that spell out the benefits should work in a perfect world, but sadly the scenarios that planetheidi spelled out in the previous comment ARE the reality. In fact, I would say that this is the case at far more organizations than it is not the case. Not only is this area the focus of the dev2ops.org blog, but it is an area we deal with on a constant basis in a professional capacity.
In my opinion, the problems trace back to these sad, but true, conditions:
1. Traditionally, the value placed on software creation is greater than the value placed on operating the software. The entrenched thinking is that you make your money by creating better software with more features. Developers are your fighter jocks and functions like QA or Operations are just things you need to keep your fighter jocks in the air. The past 30 years of technology has been dominated by this thinking and it is dug in deep. Want proof... Where do IT organizations stockpile their best talent? What part of the organization provides the best path for advancement? The best and the brightest CS grads overwhelmingly submit their resumes for what kind of jobs? Development. Development. Development.
2. Very few businesses recognize that they really are in the software operations business. Ask a company (even a "web" or ".com" company) to give you a short list of core competencies that will enable them to win in their space... very few will list "software operations" anywhere near the top. "Software operations" always seems to be the afterthought.
3. When push comes to shove, the business will demand that corners be cut and releases pushed by brute force if need be. It is only human nature that Development will point the finger at Operations and Operations will be viewed as the obstructionist if they do anything other than run the traditional all hands on deck fire drill to get it done.
4. Willingness by the business to spend money and time on this problem is often low do the entrenched thinking I mentioned in points 1 and 2. When times are good it is out of site and out of mind and its hard to scare them into caring. When the alarms are going off, all they care about is that you use brute force and get it down. As soon as the emergency is fixed, willingness to spend money and cycles goes back to the back burner.
5. Developers and Operations folks just see the world differently. They use different vocabulary. Just look how radically different the world looks through their natural choices in tooling. Even when their are working to solve the "handoff" problem they probably don't even agree on what the problem really is.
All of these factors add up to a far more complex situation with compounded problems that are far more difficult to solve than your original post gives credit.
One last thought... tools do matter and are incredibly useful when trying to tackle the development to operations handoff. Tools provide a way to formalize the contract between development and operations (i.e. no getting around what a "good" release is), provide efficiency/automation that gets the business off your back, and provides the business with clear visibility into the deployment process.
-Damon Edwards
http://dev2ops.org
http://controltier.com
Robust debate
Hi Damon
Welcome and thanks for a thoughtful contribution. First of many we hope :-D
I somewhat agree.
1: traditionally yes. Today, it is changing. Some of the developers still have prestige - e.g. BAs, solutions architects - but programmers and testers are just workers. yes the grads go there but then i don't tjhink grads are seen as the most valued employees. I think there is a distinct path into Operations for some experienced people, and Service Delivery as a discipline is gathering prestige. Who saves face? Who saves money? Finally, I think the work has been done in Development and it is pretty orderly. Much attention is on Operations now, from CIO down.
2. Disagree strongly. Any organisation that depends on IT understands the strategic importance of reliable service delivery, whether they be ISP, .com or a bank.
3: the developers struggle with the same problem: cut short the testing, go live with a few known problems, pull that out until next version...
4: once there has been a painful experience, many organisations are willing to do some hard work to prevent it recurring. Newspapers, shareholders and government all help here :-D
5: yup, agreed. I once had a client who wanted to use Tortoise open source software versioning as their Release Management tool, and jira for Change Management. Shudder. That's why it is all about communciation and culture change. Hard stuff.
6: Complex? No more complex than anything else ITIL takes on and see how easy they make everything look. (ooh there's a blog post in that)
7: a tool without procedures is only a tool. Procedures without buy-in are only paper. it's a people problem.
A tool defines nothing: people agreeing to use the tool defines the interface/contract. the same conversation can be had over a piece of paper.
Automation often gets the business ON your back when the automation project eats resources and distracts from fixing the systemic problems.
Tools are the 10% icing on the cake.
Thanks for joining in. We love robust debate around here.
Ideal vs. Observed
Hi Skeptic,
I actually personally agree with almost everything you "disagreed" with. :)
The point of my post was to outline the shocking attitudes that I find to be all to common. You'd think organizations learn from their mistakes... but sadly few do. You'd think organizations pay more than lip service to a "commitment to operational excellence"... but sadly few do. When push comes to shove, few are willing to take on the (political and budgetary) pain that comes with true change. As an industry, we should be holding each other to a much higher standard that is based on measurable results.
Regarding your point about tools... You are right that tools are worthless without well defined processes and contracts between groups. I would never disagree with that. What I was trying to say is that, in my experience, tools are the best way to FORMALIZE and DEFEND those well architected processes. When push comes to shove, written processes are the first to go out the window when operations goes into firedrill mode. Its the processes that have been formalized into code or tooling that are the last to be circumvented and the easiest to identify in a "where did it all go wrong" postmortem. Again, this isn't theory or a "what should happen"... this is just observation at a variety of organizations. Your mileage may vary.
-Damon Edwards
http://dev2ops.org
http://controltier.com
The role of tools
One thing I love about having been involved with ITIL over a long time is how my opinions change. I spent years pushing the message that you shouldn't chose or implement a tool until you knew what you were going to do with it. Yet a couple of times in recent years I've followed Damon's suggestion and used the tool to formalise the process and it has worked really well. After all a service level built into workflow is much more powerful than a service level stuck in a shelf-ware SLA. Then the last two Remedy implementations I've come across have both remimded me why you need to decide how you want to exploit the tool first!
Harris Kern library & ...
The library I turn to for the classic point of view on production acceptance is not ITIL, but the lesser known Harris Kern books, http://www.ecinst.com/, in particular the IT Production Services volume.
The fundamental challenge for operations folks trying to get leverage against developers: developers can play the "business card" more easily.
"Can you believe it, those data center people are delaying YOUR application just because we haven't documented it! Oh and there is this silly monitoring requirement they have that we don't agree with, that doesn't have anything to do with those sexy new screens your users are champing at the bit to use..."
I am not sure we are going to get over such issues unless the traditional dev/ops organization split is attacked at a basic level. Where do the "service managers" sit? I would put them in a third organization with a strong matrix to both dev and ops.
Etc.
Charles T. Betz
http://www.erp4it.com
Who owns the "service"?
ctb said Where do the "service managers" sit? I would put them in a third organization with a strong matrix to both dev and ops.
I agree with that - strongly. The only variation I'd make is that service ownership - especially the go-live decision, the "just saying no" - shouldn't really be another division at the same level as dev and ops - it should be the CIO him/herself. Or, since that's too much work for one person, an arm of "the office of the CIO".
No "get a mandate from the CIO" - it's the CIO's job.
And (reading further down the comments to James Finister's) it's not the CIO's authority - it's the business's. The office-of-the-CIO gets approval from the business. Or else the business says "NFR" ("Not Ready" :) ).
Protecting the innocent
Sometimes you have to make sure that the CIO is making the right decision, whilst making sure it is still their decision - based on the right information and the right perspective on risk. It is often senior management who have agreed to the de-scoping of operational requirements during the project life-cycle whho have to face up to the results of their decision at go life. I'm sure most of us have come across the service where contingency planning, DR and resilience have all been sacrificed to stay within budget.
Has anyone worked out where the various functions mentioned in the Service Transition book should/could sit effectively?
and why can they be sacrificed?
Yup, I've been in situations where those things have been sacrificed. Or done with lip service only.
Why does the CIO let this happen? Is it because there's no perceived consequence? "Keeping it running is Operations' job, I've delegated ... abdicated it to them." This could be improved with more standards and a clearer definition of what the actual service is. - This could go two ways: more governance (see recent thread!) or having the customer be more responsible in buying/receiving the service.
Or it could just be because people discount a future risk (a possible disaster) compared to a current one (increased cost and delay). We can't change human nature.
Service Strategy vs. Managing IT as an Investment
P.S. I Have ECI's Managing IT as an Investment
by Ken Moskowitz and Harris Kern. I must finish reading it, but I like the look of it. I'd be interested in a comparison with ITIL V3's Service Strategy
from anyone more learned than me (which I imagine doesn't rule out anyone reading this blog). It seems a little more... um ... grounded, but that is just an impression.
professionalism in defending the service levels
As below in response to Damon, I think developers have got away with this in the past, but these days end users are as likely to want a smooth transition, customers to see better service for their money, and management are sick of developer cowboys.
I believe exhibiting a little professionalism in defending the service levels goes down well with most audiences. Only for so long, but you can't win them all. As I said below, even if you eventually get steamrolled the people who matter are the PMs. They don't like roadblocks even if they eventually crunch them into speed humps. Next time they plan ahead for a smooth transition.
Do this wrong and it can be career suicide
Getting the CIO mandate, even with a business case is still hard. (assuming you can frame the case in a way the CIO will accept).
What about the number of companies that are culturally developer-led (aka software producers)? Tough sell there. Add the push to get new products in the pipeline (for either internal or external customers). And then consider the amount of respect that Operations usually gets ("Shuddup, just make it work.")
I'm not saying this is a bad path to take. In fact, it may be the only reasonable way to effect lasting change for the better. But it does require a lotta grease, lotta support and a good amount of soft skills that are hard to find in an IT shop. And it helps if conditions are right in the organization you're in. Often, they're not. Yes, I am a skeptic too.
And it may require self-sacrifice. Lemme explain. The types of Ops managers who I've seen succeed at this are generally viewed as the organizational a-hole. The ones who will say "no, this is not up to standard". The ones who say "This will cost my people time, go back and do it right". The ones who stand up for quality and say the emperor has no clothes. I've seen them win, after a long hard battle. But their reward for success was usually the boot. And then the C-levels hire in someone more permissive, "who can work with people" and "get things done instead of saying no". Of course, the new permissive Ops manager gets to take advantage of the clean new processes and infrastructure to build upon. And what do you know, everything is great with the organization. Except for the old Operations manager.
As you say, "nicely and then help" It may work, but it'll cost you too. So much that just this one initiative may spend all of your political capital and all of your resources. Which also may be bad for your career because you may have other critical things on your plate.
All in all, this is very tough. Yes, it's all about People and Communication. Lots and lots and lots.
So yes, straightforward. But in some ways, it's also like telling us to become rich by buying low and selling high. For a lot of IT folks, it's going to be touch sticky business getting buy-in, keeping it on track and still not derailing your current mission. And remember, traditional IT geeks are notoriously bad at business/personal interactions.
training a Development puppy
It's a bit like training a puppy [sorry folks it is all puppy analogies round here for a while]. You just gotta keep saying "No" and putting them on the newspaper.
And you can't train them until after they have made a mess so that they make the connection. The ideal time to initiaite production acceptance is right after Development's failure to provide a ready system has cost money, reputation, property or - god forbid - lives. Music to my ears is a Board or CEO saying "This must never be allowed to happen again". So don't make it about what it has cost "my people", make it about what it has cost "us".
As for the career, it is the same as introducing Chnage Management. it is all stick and little carrot. There are few friends to be made. So my preferred model is hire a contractor in for 3-6 months to do the dirty work, then they can leave and take much of the negative baggage with them. It's worth the money [said the consultant].