Six Tactics for ITSM to Deal with Agile

As far as mainstream ITSM is concerned the primary impact of Agile development method is on our carefully constructed production environment. Agile has the potential to undo decades of work building protection for the assets for which we are accountable. It's our job to control risk as well as enable value. Sometimes it seems Agile ignores the former in pursuit of the latter. Here are six tactics to help deal with that.

In a recent online chat* with Dave van Herpen and Rod Weir, Dave asked about how ITSM should deal with Agile. I suggested six tactics:

1) Ring-fence the Agile production target

Some business unit has adopted Agile methods (usually Marketing or Communications or HR) over the protests of IT Operations? Fine. If they want to adopt an elevated risk profile that is their decision. But it is your job to maintain the risk profile of the corporate data and applications until instructed otherwise by the governors.

So try to limit the damage by separating out - ring-fencing - their production target.

  • Any data it updates should belong to their system, not be a shared resource of the organisation
  • Any code they modify should belong to their system, not be a shared resource of the organisation
  • Put them outside your security perimeter, or at least partitioned off
  • All interfaces to corporate data or systems should be via the same interfaces used by any external agency, and treated the same way

Give them their own little world to play in - and screw up - at their discretion.

2) Get a waiver

And if you can't achieve tactic #1, then get a written waiver signed by the business owner of the group wanting Agile, which agrees that you have explained the elevated risk to the corporate assets from rapid repeated change with reduced controls, and that business unit accepts all responsibility for loss or damage caused beyond their system.

3)Governance

Any team adopting Agile methods is asking for an elevated risk profile. This ought to be an informed business decision taken at the highest levels as a matter of policy, but almost always isn't.

Sometime I'll blog more broadly about the modern taste for increased risk in the name of agility or efficiency. I'd have thought recent economic events put paid to that but apparently not. You can get away with higher risk but only for so long. Repeated risk taking (say for example in high-frequency Agile releases) is only asking for probability to have its way. You can get away with it but not for ever. Everyone in IT should watch Aircrash Investigation as a job requirement. I had this conversation recently: I'm happy to visit Manila, but I'd be mad to live there for years on end.

So do what you can to escalate the issue of Agile risk to the governance levels of the organisation, with as much accompanying advice as possible so that governors or senior management can make an informed decision. If the decision is to adopt the higher risk profile, then our job is to manage that risk as best we can. We don't need to obstruct - we can collaborate within an organisational decision to go for it. that is the basis of the following tactics:

4) Give them standard change

All organisations that have change control should have standard change. if you don't have standard changes and you face complaints and resistance over change processes, it serves you right. I tell all my clients standard change is mandatory from day one in order to facilitate cultural acceptance of any new change management regime.

If you don't have standard change and Agile is coming and you can't ring-fence it, you have a major problem. If you do have standard change, then you only have a serious problem :) Work with the Agile implementers to find as many procedures as possible that can be categorised as standard change. This is of course an ongoing process: after they've hurled a few releases out there, you can start assessing the procedure for eligibility as a standard change. And then you can work with them to adapt their procedures to meet eligibility.

This isn't the place for a lesson in standard change, but in a nutshell a standard change is a type of change that is pre-approved. They still need to record the change but they don't need to seek permission every time they do it, just once in advance for all future occurrences of that type. The key elements (for me) are:

  • Standard change types should be assessed on risk not scale of change. Big change can be standardised if it is low risk, and small changes aren't automatically standard ones until proven to be low risk
  • Repeatability
  • Fully documented and tested, especially including back-out/recovery procedure: this is the "standard" bit.
  • Most of all: unambiguously defined. Before you open the standard change door, it has to be agreed and clear what constitutes that type of change... and what doesn't.

5) Automation

Automation of Agile change is a large chunk of what DevOps is about, as I understand it (along with some hippy-dippy stuff about how we can all be one happy family without specialisation or tribal divisions). Yes DevOps helps minimise the risk of Agile change, but it is not a silver bullet. It doesn't circumvent the need for the previous three tactics - it only reduces the risk somewhat. Three issues:

  1. Automation doesn't prevent errors, it just changes their type from many small ones to fewer more catastrophic ones.
  2. Automation is often inappropriate . Automation is expensive. We in IT always underestimate the TCO of automating our own processes.
  3. GIGO. Automation is just a tool. No tool fixes process, practices, roles or attitudes.

6) Outreach: have an Operational Readiness programme

I've talked a lot about Dead Cat Syndrome, and the need for an Operational Readiness practice that bridges the gap between development and operations/production. A key principle of that practice is that the ultimate accountability is with the owners of production, and so it is Operations' job to seek out developers and talk to them not vice versa. Operations should have an outreach programme which offers advice, guidance and tools to help projects get systems operationally ready, baked in from day one of the project.

Project teams - including Agile teams - should know what they are in for and what is required to get stuff live long before the day comes.


As always, I'm no expert on this stuff. These six tactics are suggestions which will hopefully stimulate some healthy debate. Over to you...


* It was a "Tipu Talk", wide ranging ITSM chat loosely related to my Tipu improvement method. I've since dropped the Tipu Talks but may soon revive them as "IT SkepTalk". Let me know if you'd like to see that.

Comments

no accountability without authority

David Moskowitz made a good point to me on twitter:
This isn't about Agile per se. It's about any attempt to wrest away the controls we have put in place to protect the corporates assets that we have been entrusted with. It's our job dammit. We in operations are paid to protect the existing value of information and technology, as much as we are paid to empower new value from it. Developers lose sight of that balance.

Balance

The balance is being lost on both sides.

Yes, developers don't see the need for operational stability, but Operations doesn't see that a business standing still is a dead business.

IT can't be the stick in the mud that drowns the business when the tide comes in, all in the name of operational stability.

The Agile and DevOps movements are an attempt to respond to the ever-increasing pace of business and economic innovation. They have the right of it, just as much as those of us who adhere to ITIL/ITSM philosophies.

We had better have a good, long look at what is happening over in Agile-land and attempt to move in that direction. We do not need to risk our operations, but we do need to figure out how to be more responsive to the needs of our business partners.

Or... we will find ourselves outsourced.

Everyone holds up the

Everyone holds up the bogieman threat of outsourcing. I think the world is slowly maturing to understand that outsourcing is no cure for dysfunction.
And if tactics 4,5 and 6 aren't Operations being responsive, I dunno what is.

If we are going to accommodate Agile we certainly are going to risk our operations, or more precisely we aRe going to risk what operations has stewardship of: the organisation's informational assets. Increased volatility and decreased planning and testing must increase risk. It is fundamental. All I ask is that that risk taking be an informed decision by the governors, not by a bunch of idealistic code hackers and a service owner in a business unit.

There is no wall

Some to many to all controls, depending on the specifics of the business, are outdated. Defending the fort from the watchtowers when the walls have been removed by the king. Ask the king for renewed instructions.

the fort walls are still there.

To continue your analogy, the wall has not gone. Most organisations still have a stable and secure production environment. Small wild hordes have managed to pull a few stones out. Certainly they are plotting more concerted assaults in some companies, but mostly no.
And they are certainly not acting under instructions of the king. The governors in many organisations are fiddling in a remote tower, oblivious to the risk or the needs of IT. Hence my other analogy, the failing parent.
I tire of this apocalyptic view that all the foreseeable consequences of SoMoClAg* have already happened: no they bloody haven't. The effects are just starting to show and there is an awful lot we can do before (and if) the world changes. Your personal in-hand IT experience may have been transformed but don't confuse that with the business IT condition, which is much slower to change because of my favourite point: we own the existing legacy assets, which contain value and must return that value on their investment. I'm amazed the number of people who don't get that simple business principle.

*I got SoMoCl off ITSMWP, I think it is Matt Hopper's. SOcial, MObile, CLoud. I just added the "Ag".

new castles without walls

In some companies, there really is no wall. Never has been. There is no Dev and Ops, there is not even a DevOps. There is Continuous Deployment with Test in Production. Yep, there is no stable production environment anymore. And the customers are happy.

Some companies are moving towards this.

And in others - in most, I would agree with you - the wall is still there. Cloud is making the transition to wall-less world somewhat easier, a long list of 'traditional' IT assets can be replaced with a shorter one, of which a big part is "I don't really care as long as it works" type of assets.

Configuration Management is less and less about the attributes of the hardware used, while more and more about auto-discovery and relationships. One might argue that this is exactly what CM was intended to be, in relation to asset management.

I find that a lot of the "ITIL does not work in the new era" discussion is based on these examples. The % of companies living and breathing Agile is perhaps 0.001% and I might be mistaken a thousand-fold. But they are there and they have shown it can work. On the orders of the king, i.e. the Business. Which is sort of an obsolete distinction as everything is Business there. Except maybe for the ... no, everything.

One of our tasks, as I see it, is to ease the transition, while accepting that not all that has been was the preferred end state, but rather a step in between.

the other 99.999%

Sorry but I address this blog to the other 99.999% of organisations. The future is all very exciting but we have a job to do right now. Attempting to structure the other millions of sites to have open walls right now just because a tiny tiny minority have decided to take that risk is irresponsible in the extreme.

We appear to be talking about different assets. IT technical people need to wake up to the fact that all our technology is not the real asset: the corproate information and the corparate ability to interact with that information in transactions - those are the only assets that matter. Cloud is just twiddling with the underlying layer we build to look after the data and enable the transactions. it has little significance for the assets that matter.

The number of nines are diminishing

Like Kaimar, I have seen these new models working. I have also worked in a corporation where IT wanted to control the corporate information. The problem was that the information was not where the corporate IT wanted it to be.

If a new model is significantly more efficient, it will win. The new support number is 1 million customers per one support person.

Aale

a matter of policy

As I keep saying I don't mind if the new models win. But the competition should be judged by the governors of the organisation as a matter of policy. It is not some business unit's or dev team's call.

we can still learn

Both of the real assets you mention are taken care of - and much more so - in the new model. Cloud is not about the technology (well, it is, but this is just one side of it) but about the way the reliability, availability and consistency of *real* assets can be increased. Agile is not about technology but about how to meet the business needs in a more flexible manner while the data is protected and the ability to interact with it is kept at a high level.

I'm not saying all companies should move to the new model - it is not mature enough yet. But - it probably makes sense to take a look at the new model and think about why it works for those using it. And then take some of its elements and apply it to the *traditional* model, which for many consultants is ITIL + Cobit + various other sources of good information. Agile and Devops provide concepts that can, after a proper carwash, be added to this list.

automation

Automation in today's continuous deployment / agile / devops world is not "just a tool". It is a mindset, adopted by all parties involved, supported by tools. At least if done properly. Manual meddling with the prod environment is an exception.

The same way as ITSM is not just a tool, but if it is treated so, the results will be less than impressive.

We shouldn't approach Agile too defensive

Thanks for this essential blog post, Rob.The reason this topic is on my mind is two-fold. First, I see lots of organizations struggle with aligning their operations to agile development teams. Second, I strongly believe we (whether we are responsible for operations, or IT management, or application management) can benefit greatly from agile methodologies and principles. I see your tactics 4, 5 and 6 appeal to the required, more intensive collaboration between operations and development. I very much agree with this need. We can no longer treat dev and ops as two separate entities. I believe we even should envision shared agile teams across development AND (application) operations and maintenance. I think visual management tools and techniques ( eg. Kanban) can be of significant value in achieving this collaboration.

Your first 3 tactics, however, are too defensive in my view. They origin from a lack of trust in the quality of the IT deliverables resulting from the agile teams. Ring-fencing and getting waivers are only bandaids. Instead we should learn from the best practices which have succesfully increased the quality of the agile products. Key is to focus on customer requirements (product owner), but also include operations' insights from the early start of agile projects, so essential requirements from "our" perspective (stability, performance, maintainability, etc) are included in the design and eventual products. After all, it may be crucial for operations to enable fast delivery and short time-to-market, it is still our job as well to keep the lights on......

bridging the gap

Dunbar's Number tells us you can't have one group of humans in a large organisation: they will always self-organise into groups. There will always be divisions. We can't all be one big happy peace-love-dope family. In most communities we take advantage of this to get people organised into specialist sub-groups: companies, departments, teams. We leverage it. Trying to get all of IT into one Agile-based team is not going to work above a certain size. You won't be able to find the multi-skilled people, and you won't be able to hold the team together. "You guys work on the code, we'll look after the infrastructure" is only weeks away from reverting to the status quo.

hence Operatinal Readiness as an essential practice: it bridges the gap. And it does it in a formalised managed manner.

Human nature being what it is, I see no problem with being defensive as well as collaborative. There is no place for blind trust or assuming the best in people in any effective risk management.

Syndicate content