IT operations must beware inappropriate automation
Being technical folk, we are far to quick to leap to technical solutions to non-technical problems. Automating operations tasks is one of those cases.
[update 2019, I've learned a lot since 2011:
The business case for automation often only looks at the local optimum, like this post does.
but if it is one if the last pieces of the puzzle to completely automate a workflow, then the value is great. we should look at the impact on the whole value stream not just the task.
Conversely, it's equally important to look at TCO over time.
And risk of a key automator leaving.
and most of all, automation can negatively impact agility. Agility is not about how fast you work. it's about how fast you change how you work.]
.
I've talked about automation before:
There is another principle we must always bear in mind with automation: only some transactions are suited. You should only automate that which is
- repeatable and strictly clearly defined
- frequent (or very expensive). It costs a lot to automate. there need to be enough of them to provide an ROI
- stable. It costs a lot to change automation, just like it costs a lot to change any production software-based system. If the rules or procedure are always changing then people flex and change faster and more cheaply than automation
In the operations context, how many tasks fit those three criteria? Not many, and certainly not all that i've seen IT staff try to automate. We are way too optimistic in IT about automating operations - some might say cavalier.
Testing software - whether it be automation or anything else - far exceeds the cost of coding and (often? usually/ always?) exceeds the cost of planning and design. This is frequently overlooked by IT Operations staff automating IT systems, who think that banging another rule or script into Tivoli, Patrol or Unicenter is a few hours work. Likewise Service Desk tool admins adding workflow.
if the business is automating a core process, then the system is usually properly design and built.. and tested. If security staff want to automate access provisioning, someone bangs up a "self-documenting" Perl script, give it a quick unit test on the bench, then release it into the wild.
Our operations tools are production systems just like the ones we are stewards of - there's another concept I struggle to get across to some ops staff - and need to be subject to the same justification, planning, specification, design, testing and release as any other.
Made in New Zealand 
Comments
great quotes on automation
"If you automate the wrong process you get the wrong result faster" ITSMWP Episode 63
Variations:
“If you automate a mess, you get an automated mess.”
– Rod Michael
"Automation applied to an inefficient operation will magnify the inefficiency. "
- Bill Gates
Isaac Asimov
In "Prelude to Foundation" there are two bits of history that are recounted that speak to this:
1) When the humanoid robots become too powerful and we must destroy them all (more because it's funny)
2) When Hari visits the heatsinks (a thermal power source on another planet), they talk about how hot it is, and maybe it should just be automated.
The counterpoint by the manager of the heatsink is perfect for this. He says that automation will fail, probably because of some computer malfunction, or physical break, so we must have people here or it could take up to 15 minutes to get a person to the correct location. Then, he makes the part that applies: people fail, automation fails, but they rarely fail synchronously.
Provide a good balance (which I believe summarizes at least part of your article).
Oh yes they do
Matt,
I think you'll find the combination of the human factor with the machine factor is more common than you might think, and the consequences possibly worse than putting trlience on one or the other. Think Homer in a powerstation.
James Finister
www.tcs.com
http://coreitsm.blogspot.com/
coverage
It's true major failures happen when the broken bits coincide - people and automation - but equally I agree that people and systems cover and correct each other too.
Regarding the topic of Automation
Hi Rob,
I think I agree with some of your statements but not all.
First, experience has shown that there are two consistent forms of Automation:
1) The automation of human labor
2) Automation to further improve existing automation (i.e. multiple automated processes can be re-engineered to be improved, even further)
(We can argue that there's a 3rd that is a combination of the two but, fundamentally, it's the above two.)
Both of these happen regularly and often, throughout enterprises in all industries, globally.
Given the above, and I believe to some of your points, there really are criteria for what should or shouldn't be automated, in a given enterprise. For example:
- Priority of Work and the Organization
- Repeatability
- Ease of Automation
- Supportability & Maintainability of Such Automation
- Internal Skill Sets
- Cost of Automation
- Value to the Enterprise (Not always a tangible and measurable financial return. Sometimes it's a "soft" return.)
- etc.
When it comes to automation, there are usually three types or patterns that occur:
A) Complex (requiring formal Product & Release Management disciplines to be applied to somewhat complex programming. FYI: Not ITIL Release Management but what developers call Release Management)
B) Moder via Workflow/BPM tools (a mix of some programming and configuration)
C) Simple (usually the type of automation that revolves around "scripting" instead of more complex programming)
I've also learned that "who" does the automation is very important. In most enterprises, the most technically capable resources that can be leveraged to perform automation are usually highly trained, highly experienced and very "current" Software Developers/Designers and Engineers. Such resources are also empowered to write code, regularly.
There are also risks associated with your enterprise's Outsourcing strategy. Software Code products, such as Programs and Scripts are the types of Assets that often require regular review and improvement. As a result, you need Skill Sets and Resources with such Skill Sets that are available to perform such functions. If your enterprise Outsources the resources and the skill sets required to create and manage such constructs, it will be far more expensive and complex, in the long run, to support and change such constructs, as needed, in the future. Most enterprises that are successful at automating usually invest in keeping such resources internal to their enterprise, so that they don' t have to retrain, from the ground up, every time someone has to go back in to review and/or modify existing automation solutions.
From my own experiences and regarding the strategy of "automating" (and this is not to say that there are others out there), the most successful companies I've seen perform automation usually had a general principle that was professed from the top down: "If you have the skills and the power to automate, using the tools and and technologies that have been sanctioned by the enterprise, and if automating specific tasks frees you and/or others up to perform other work, then automate, automate, automate! Don't wait for reviews. Don't wait for permission. Don't wait for funding. Just do it. The worst case is that what you automate can be undone and we can revert back to original state." Such enterprises have the attitude of "Automate it hard! Automate it fast! Automate it often!" These types of enterprises not only automate their Core Business Functions but they also automate their Core IT Functions, such as those associated with IT Deployment. They get that IT is way too expensive and work hard to drive IT costs down, through the automation of everyday IT activities.
I've also found that people who don't automate for a living, doing it day in and day out, every day, often lack the understanding of how to best automate, when to automate, why something should or shouldn't be automated, what to use to facilitate best automation, and where to best target automation. In my experience these are skills that most IT people do not have. And, the way many enterprises set up Dev & Engineering vs. Support, it's rare that IT Support / Ops teams are best equipped to do so (Again, this is noting that there are always exceptions).
It's my opinion that with the advent of Outsourcing, most enterprises that have tried to outsource their development and/or engineering functions are often outsourcing and eliminating their most innovative and most empowered IT resources... Those that get how to most effectively automate the right things, the right ways. Such enterprises usually realize this after it's too late.
It's also my opinion that automation is the foundation for IT innovation as well as for Business Productivity. If you're not automating, you're not innovating. And, in these modern times, if you're not innovating you're not competing. If you're not competing, you might as well give up because your competition definitely is... and they will eat you for lunch.
Anyhow, I hope this adds value.
My Best,
Frank
The International Foundation for Information Technology (IF4IT)
Open IT Best Practices
Lack of rigor
Automation consists of more than stringing together multiple logical units of work, often utilizing a dependency tree as in good old "make." Let's ignore the blossom of failure scenarios introduced by unattended automation where literally no one is paying attention.
IT is so proud of its ability to verify correct functioning of a code chunklet under ideal conditions. Unit testing is all the rage.
The next level is the ability of code to recognize that it's in trouble then raise an exception to the invoking system. Only high end code is fully validated for this functionality.
The next level is for the invoking system to handle all exceptions correctly, making the appropriate adjustments, backing out any partial processing, retrying the process or even raising an exception to the higher invoking system (often a human being). Precious few systems are validated to perform this function correctly under all conditions.
Instead, we find code chunklets which each work just fine "in the lab" and string them together with a script then declare that we have automated something.
It's no wonder that things fall apart.
Automation
You make some important points.
I'd say that the cost of testing is high, as with other costs being high, often because of poor design. Following the Service Design Package method allows for a proper holistic approach to be taken to the design of services. In the case of automation of, for example, standard changes, rather than design each automated change, design a process and tools to produce automated changes - design the design process. Get that right, and the actual production of new processes become more reliable, less costly and time consuming to test and has fewer defects in operation.
Because this is often not done, and, rather, things are automated piecemeal, you end up with incompatible tools as well - which adds to the cost of training and increases the risk of improper use.
Overdesigning the design process
Peter, with all due respect, there have been notable failures in the philosophy that "if we just have a better design process we'll get better quality systems." Essentially, that's the root of waterfall thinking. New processes, if they have any value, tend to present unexpected difficulties. You cannot have a master, meta-process that controls for that fact, a reality that goes back to Turing's proof of the Halting Problem.
We need a faster, more adaptable process that accepts the fact that product design is always risky, uncertain, and best handled iteratively. That's why I am an unfailing advocate of the DevOps movement.
Charles T. Betz
http://www.erp4it.com