A new concept goes into over-hype: Agile
The latest buzz in IT is of course Agile, and its bastard spawn DevOps. I've written before about how the change is becoming the steady state and stability the exception; and how the old mainframe-centric concepts of change control will have to adapt. I'm even confident that concepts from agile will play an important part in that. But nothing in that warrants the frenzied hype around agile right now. And most of all, nothing in that warrants letting the IT cowboys out of the corral.
[updated : I did of course change my mind a few years later,
http://www.itskeptic.org/kamu
or maybe the hype just died down.... Um, no. Agile is still overhyped, but the value is real. I'm a DevOps devotee and an Agile transformation consultant www.tealunicorn.com.
But back then I said.....]
Just because small teams of cool-tool developers can knock up websites with an agile approach has little relevance - yet - to monolithic corporate operational systems. Agile developers remind me of those testosterone idiots who fizz about on jet-skis on a Saturday afternoon in sheltered bays and wonder why ships need lifeboats and harbour pilots.
As Chris Clarke, VP of Product Management and Strategy at Collabnet, an international supplier of agile development and project management tools, put it, team based adoption of agile practices is easy and that has happened. Now the hard part begins. That’s the adoption of agile for large projects with distributed teams working on complex systems.
(BTW I was highly amused by this other quote from the same article
Robert Holler CEO of VersionOne, maker of another widely used suite of agile project management tools, said, “It will be a long implementation cycle..."
WTF?)
Agile is struggling to step up to large-scale development. To suggest that its concepts should be allowed to subvert all the stability and control that we have fought for decades to recover is madness. We had proper controls once in the mainframe days, then the distributed computing cowboys blew it all away . The introduction of proper Change Management and other ITIL disciplines has slowly clawed back some semblance of stability, of management of availability.
The whole world in the 21st Century is leaping around in a demented frenzy of fads and stampedes, whether it be shiny black phones or postal flight attendants or actionable service catalogues. IT needs to carefully and gradually evolve to a more flexible mode of operation. Ignore all the pundits bleating their "change or die" messages. If you are still running some MVS/COBOL or VAX/VMS or even Windows 2000, you know how fast the world really changes. The vendor-and-analyst industry (it is all one) survives on stampeding you - don't fall for it.
Attendees at the itSMF Australia conference in a couple of weeks can hear my keynote on the Twenty-Teens. One concept I introduce there is that we will gradually lose control of much in the environment that we traditionally had corralled, from servers to developers. We need to grow the controls we have around the environment to contain potential damage:
- service management to ensure what gets delivered maintains levels of utility and warranty, and to detect when it doesn't
- governance to set the tone, the rules, and the boundaries, and oversight to shoot anyone who goes outside them
- assurance: security to keep the riffraff out, audit of what's going on in the moshpit, maintaining necessary compliance, and driving professionalism into the now-empowered community. And risk - most of all risk management
Most of that stuff is pathetically immature in the IT industry. Broadly embracing agile concepts now would be like lowering the drinking age to 16 without having a police force (or social workers or hospitals). It is way too soon for the feral techs to be dancing around bonfires whooping up their victory over sensible IT. Break out the riot shields and batons and get 'em back to their desks... for now.
Made in New Zealand 
Comments
The Agile Dilemma
Skep,
It is really quite interesting to read what you wrote back in August 2010 on agile, then reflect on something I wrote back in 2004 https://docs.google.com/Doc?docid=0AbdkV3VbtwJFZGR3NHFxa180d2RmbXpo&hl=e... .
Overall I could not agree more with you in regards to maturity of IT and the Hype that was (IS) agile. Don't get me wrong Agile is a great idea and very effective when done well. The interesting thing for me though is Agile does not mean less control, in fact more control, it's the same as agile means less documentation - Nope it doesn't.
What is most interesting for me though, is the situation does not seem to have changed in 6-7 years.
It is human behaviour at it's best.
Cheers
Andrew
It is pure hype - Old wine ...
You are 100% right. The whole Agile is a hype - just like SOA. I quote my own comments on another article on Agile.
It makes you feel a little dizzy when we read "What we are seeing is that there is a lot of confusion in the development community around what exactly agile is, and, of course, what it’s not." (Michael Blechar Gartner) Why is this so? This is typical of all methodologies emerged in the last decade to qualify them with confusion even in the very basic definition. (SOA is anothere xample). If there is no definition it only means the concept itself is not clear. Then all the rubbish which follows will be hazy too - and it is. One should ask: why is it difficult for anyone to even define Agile or SOA when a common man has no problem even understanding the basics of the Theory of Relativity or the Hydrogen Atom Model or the duality of light as a particle and wave (the ultimate paradox)?
Gartner's Matt Hotle is right "-The shift to agile development is more about culture than it is about process.". Then it follows that this cultural change can be brought about in any 'old style' project also.
The comparison to waterfall has always been exaggerated since there indeed was never any pure waterfall approach. Always there had been prototyping and some iterations with the end-user even without any Agile.
In many Agile projects at the end of the day one does not see any difference in the actual code develepment process, tools or methodologies. Well one can always classify the morning 'stand-up' meetings and SCRUM boards as a big change!
namespace collision
One problem is the word 'Agile' is ill-defined and nebulous while being a word that most people believe they understand intuitively.
I think it shows some bias on your part to focus on those quotes from an article where the main message is how Agile is successfully moving into the enterprise.
Agile is not one thing. At Agile's best, rapid delivery is enabled by disciplined adherence to making the impact of change known and predictably controlled. On one end of the spectrum, there are rigorous technical practices and commitments to delivering value, on the other end, there can be a lot of promises made while asking very little in return. If the Agile that gets adopted stops at having standups, not writing documentation and delivering something every two weeks come hell or high water, then plan for disaster.
devops is the same. There is a body of knowledge and a community of practice based on solving real problems. Will devops be misunderstood and misapplied? Inevitably... but the core is aligned with your points:
With the exception of 'shoot anyone', I consider this 100% aligned with the goals of devops. When someone goes outside the governance, 'why' and 'how' they went outside can be an opportunity to understand where the governance has failed the organization. At least give them a fair trial, as it might be the rules that need to be shot.
Small frequent changes are much easier to manage and reason about, but that doesn't make any sense to pursue until there are considerable metrics being gathered and the ability to correlate changes with those metrics. The real measures are the ratio of incident per change, the mean time to detect incidents, the mean time to recover and the overall impact. In many cases, heavy handed approaches to change management have a negative impact on those metrics and only provides the illusion of stability. It's just a game of Jenga that takes longer because no one actually moves.
I don't expect you to be able to read every book that comes out, nor can I, but I do think you might take the opportunity to expose yourself to the knowledge, experience and practice before you dismiss devops out of hand. That effort will only improve the potential dialogue and likely be reciprocated.
I would highly recommend
I would highly recommend that you all read 'Agile Principles Unleashed' as it explains proven approaches for achieving real productivity gains in any organisation!!!
The most compelling argument in favour of trialling Agile approaches, according to Jamie Lynn Cooke, is the fact that it costs the organisation very little to get started. All you need is one project that is small enough to influence, but important enough that its success will be meaningful to the organisation. It could be a scheduled event or even a customer service activity. Commit to trialling Agile approaches on this project for three months and monitor the progress.
You can get your copy of this publication here:- http://bit.ly/9HyVeh
part of the future
With respect the "small enough to influence" troubles me. i don't doubt Agile works in a small environment, and I don't doubt it works in a standalone environment like building websites.
It may even work in large monolithic environments where systems need to be sociability tested against integrated systems, or where systems are three or five tier architectures, or very-high-throughput OLTP systems, but I'd like to see lots of evidence. Just because one passionate group of gurus can make it work once doesn't mean it is ready for mainstream IT.
And I'll be watching closely for the longterm value. What are the ongoing maintenance costs of an agile-developed system? What is the release failure rate and the resulting business costs? etc
I say again Agile is part of the future. And part of the present. That's less than the hype engine would have it
Agile is Definitely Not Hype
Absolutely agree that Agile software development practices are easier to implement on a flexible technology platform than on a mainframe. Also agree that the industry has not done a sufficient job of collecting empirical evidence to support Agile success stories. But to say that Agile is "hype" - or that it cannot be achieved in organizations that adhere to strict change control and governance frameworks - is extremely short-sighted.
The "hype" that is Agile is actually based on proven approaches that have been used in the IT industry since the mid-1990s, with some elements of lean software development having their origin in lean manufactuing practices documented by Henry Ford in the 1920s (http://amzn.to/bVm3qZ). Large and highly reputable organizations have not only been successful in using Agile practices for their internal product development, e.g. Microsoft (http://bit.ly/acahs1), Yahoo! (http://bit.ly/cLLtJ8); some, such as BT Innovate & Design, have even mandated the use of Agile for all development activities across the organization (http://bit.ly/di2qHY). The fact that these organizations' self-published Agile success stories do not have the impact of a full-fledged Gartner report does not diminish the proven effectiveness of these practices.
Equally, to consider Agile practices as the exclusive domain of "rogue developers" building websites in uncontrolled environments shows a clear misunderstanding of the underlying principles that drive these practices. Scrum, for example, is a popular Agile method that is based on:
- Prioritizing development work to continually focus the team on the highest business value activities
- Directly involving stakeholders throughout the development process to ensure (as much as possible) that deliverables align with their expectations
- Using face-to-face communication (not piles of documentation) as the primary way of sharing information between the development team and the stakeholders.
- Regularly reviewing and adjusting work to ensure that the development team is producing high business value outcomes that meet stakeholders' expectations.
There is nothing that stops an organization from using highly-managed change control procedures to populate and order the list of priority work used in each Scrum iteration (i.e., "sprint"). There is also nothing stopping organizations from directly involving stakeholders in producing deliverables that adhere to strict project governance frameworks.
The bottom line is that you cannot rely on regimented procedures alone to address the historical failures within the IT industry. The structure and accountability provided by governance frameworks need to be coupled with an organizational shift in focus from producing working software to delivering high business value outcomes. And the delivery of relevant, valuable outcomes is at the heart of all Agile practices.
P.S. BTW, the reference to "small enough to influence" was a guideline for how people can first introduce Agile within their organizations, not a limitation on where Agile can be most effective. :-)
what is left
Hmmm I look forward to finishing the book to see what is left that is actually "agile" once one has applied highly-managed change control procedures and strict project governance frameworks
Evil waterfall dies
ITIL Change Control is not the enemy. In practice it is a 2 week lead time thing. Agile development can and does deal with it, it's not a bad thing to have a couple weeks "cooling off" anyways.
The enemy is the formalized, evil waterfall concept of "stage containment"; that you should never revisit requirements once you are in design, and you should never revisit design once you are in construction. I was taught this pernicious concept at Accenture (who I suspect have disowned it by now). We as an industry are gradually purging the last die-hard remnants of this discredited philosophy from PMOs (Project Management Offices) around the world, but there are still some holdouts clinging to their heavyweight templates and methods. They must and will either adapt or die...
These PMOs do not exist in a vacuum and cannot cling to these strict, dysfunctional frameworks if both business sponsors and leading development teams are simply telling them "enough." IT risk professionals, perhaps their former allies, are becoming well aware that evil waterfall is associated with many very large software catastrophes....
("Evil waterfall" as distinguished from "good waterfall" as proposed by Royce, who fully recognized its iterative nature.)
Charles T. Betz
http://www.erp4it.com
It's the hype
Thanks Andrew. It's the hype I'm going after, not the concepts themselves. As I said i don't doubt they are the future. We need to get the message out (contrary to what one reads in the chattering press of the internet) that it isn't the present for the majority of sites, who don't have the controls or metrics to safely let it loose. i think we all agree on that.
I'm reading the book. Expect a review, and so far not a positive one.
Don't believe the hype
From that perspective, the hype is dangerous, but such is the nature of technology and process adoption.
The continuous delivery book is only one slice of what people do. There are some good ideas there, but it is largely focused on application deployment and really only touches at all on web centric operations.
I'm all for peer review and critique.
I'd love to get your feedback on the points in my interview here (doesn't start for about 20 minutes)
[http://devopscafe.libsyn.com/index.php?post_id=632136 Andrew Shafer on DevOps Cafe]
I'd also like to hear what you think of the ideas discussed in these panels.
[http://www.infoq.com/presentations/your-mileage-may-vary Your Mileage May Vary]
[http://www.infoq.com/presentations/infrastructure-as-code Infrastructure As Code]
[http://www.infoq.com/presentations/DevOps-outside-Web-Operations DevOps Outside WebOps]
[http://www.infoq.com/presentations/changing-culture-to-enable-DevOps Culture and DevOps]
Bottom line, I don't think IT or service management are solved problems and anyone one who claims otherwise is probably about to ask you how much you are willing to pay.
20th Century Analog Boy
I don't own an iPod (my i-Mate - which had an "i" in its name long before Apple started making iCandy - will play audio but I've never figured out how and I long since lost the earpiece). I do have a mono headphone but I seldom use it. I can't listen to something and simultaneously do anything more complex than drive a car - and that's only on a good day. So the only audio I ever listen to is the occasional Rolling Stones CD in the car and Dancy's ITSM Weekly on my funny old headset. And I'm weeks behind with that. I'm a 20th Century Analog Boy: I can read about a thousand times faster than I can listen. Point me to text please :)
you might want to be more cautious
Rob,
Before you opine further on this I suggest you read Continuous Delivery by Jez Humble and David Farley. dev2ops is a maturing set of practices endorsed by prominent software engineer practitioners and recognized thought leaders in the field. You'll not be taken seriously if you don't have a reasoned and careful critique. Which is not what I see here.
Agile is proven, plain and simple. It's been evolving for many years and I think has solved the large project problem. Dev2Ops is doing exactly what you assert is necessary, with its rigorous use of test first development and automated deployment and rollback. It's an extremely well controlled approach, with an admirable objective of reducing cycle time from idea to value-add use of new functionality while simultaneously preserving system stability.
Construing it as an attack on ITSM good practices is simply a mischaracterization.
You'll attract quite a food fight here on this topic; perhaps you're just trolling...?
Charles T. Betz
http://www.erp4it.com
P.S. The "buzz" around Agile was actually news around the year 2001 or so.
There's buzz, then there's buzz...
Rob said, "the latest buzz" around Agile. Yes, the buzz started almost 10 years ago, but it has never been louder tnan it is today. I am certain Agile-buzz is old news for someone with your IT-background. But for many folks, they are only now hearing the buzz - largely because the hive has grown quite large and the busy Agile bees have never been angrier.
Steve Romero, IT Governance Evangelist
http://community.ca.com/blogs/theitgovernanceevangelist/
a real buzz
Precisely Steve. A few geeks getting in a tizzy is not a real buzz. Vendors slapping it all over their brochures and CEOs reading about it in magazines is a buzz. people starting to mash the word up with other words - "Agile ITSM" - is a buzz.
A real buzz is when it has shifted into over-hype.
Next we'll hear ITIL was a buzz in the 1990s, or web services, or... Most things have been around for a decade before the wider community decides they're new and exciting
Trolling
Troll? Me?
I have the book. I may get time to read it. Average turnaround right now is about 6 months :(
I'm prepared to hear that agile is maturing. Is the typical IT shop (as i see them) ready to set it loose? No ****ing way.
It's the future, sure. I'm disputing that we are ready for its widespread adoption now. For you guys who work for level-5 rocket-shops, maybe. For the other 95%, nah.