Reflections on DevOps Enterprise conference

Recently I had the privilege to attend the DevOps Enterprise conference in San Francisco, hashtag #DOES14. Videos here.

That conference was put together by a number of people, but perhaps we owe it to this guy more than anyone: Gene Kim

(Here's me trying to persuade Gene to visit New Zealand, a win-win for both Gene and NZ).

As Glenn O'Donnell said, we will look back at this conference and say "I was there". It was seminal. To the great unwashed masses of IT (represented here by me) DevOps has up to now been a fringe thing. As the DevOps world says, it has been something for the "unicorns": esoteric mythical organisations like Google and Etsy and Netflix. They have do their own langauage and do weird shit. Clearly different planet. DOES14 laid the unicorn myth to rest.

DOES14 showed us that this stuff is real. Real organisations (what the DevOps folk call "horses") doing real stuff in what I call Real IT (COBOL on a mainframe, FGS!).

The conference also showed it is still early days for DevOps. Most of those "real" organisations are still only doing DevOps in one team, or one aspect of DevOps (DevOps is a philosophy not a formula, you can do all sorts of subsets of it). As I said to Gene, it seems to me they were horses with one spangly blue leg.

The conference also showed that DevOps is rocking, growing at an astonishing pace. Remember how Agile was 10 years ago? A weird idea one team upstairs in Marketing was doing? That's DevOps. Only DevOps won't take 10 years. It'll be here tomorrow.

The vibe of the conference was evangelical. The crowd was positive and sharing and happy. Congratulations to all creating this conference, this philosophy, this movement. Sadly I think it will never be quite so wonderful again, as the idealism meets hard reality, as the rough ride of the Hype Cycle kicks in, as the community fragments, [as the vendors poison the well: already happening "buy your DevOps here!"] so I'm profoundly grateful I was there to experience it. Thanks Gene and everyone else who created, contributed, and took part in DOES14.

===============================


Here are my rough notes of the key points I took away from DOES14:

Making IT humane.

Working smarter to reduce the impact on individuals’ private lives.
e.g. sure we need to deploy some systems in the weekend but why not automate most of it “one person working for hours instead of three people working all weekend”
And if deployments are more dependable then we are more willing to deploy say in a weekday evening.
Sometimes what you are asking if your staff is just unacceptable.

Increase dependability of release:
automation of deployment reduces human error
those deploying have practiced it many times before - one team from dev to prod
continuous integration reduces differences between dev and prod environments, reducing suprprises
shorter cycles and faster feedback improve quality
smaller releases reduce risk
fast cycle means faster correction of any errors
those deploying understand the system, can resolve issues faster
automated testing and migration means emergency fixes are more reliable, less likely to make it worse

Enabling people to do their best

less bureaucracy
learning, improvement kata
iterate fast
making people talk to each other business-dev-ops
willing to fail: noble failure
blameless culture so long as we make it better, moving from blame to trust
provide predictability for downstream teams
practice all the time
leadership support “air cover”
optimise for speed not for cost
team led, not top down
inspect processes often, ensure they are not creating friction
“go to sea”: take risk, be prepared to fail, reward it = learning
http://www.youtube.com/watch?v=502ILHjX9EE

Principles:

shift left
tighten loops
automate everything
instrument/dashboard everything, telemetry services
talent, recruit and enable
no assholes
reduce queues
if it hurts do it more
repetition is the fastest way to mastery
make work visible
reduce WIP
operations engineering: ops as toolsmiths
smaller, faster, more often
data driven decisions
scale down: train and decentralise
DevOps is not defined: it is values and culture
dog-fooding: use your own stuff

Continuous integration: code is compiled and tested “as soon as” it changes - “immediate” feedback
Continuous delivery: code is “always” ready to deploy on demand
Continuous deployment: code is released “as soon as” it is ready

Horses and unicorns (and elephants)

There are many horses at DOES14: Target, GE, Barclays, Disney, Dept Homeland Security, Nordstrom, Macys, Telstra…
But:
all are doing DevOps in a subset of the organisation (some are now talking about widespread adoption)
it is usually web-related (not always: COBOL, firmware…)
it is often a subset of DevOps, not all aspects (as it should be: DevOps is a philosophy not a standard)
it is often in early stages, there is little maturity, may be proven now (just) but not yet generally-accepted
All report great results
Target.com
80 deploys a week, 250 commits, <10 incidents a month.
Infrastructure as code, serving : 30 teams 134 staff 8710 builds
internal devopsdays: 400 members
https://flipboard.com/section/%23make_awesome_happen-badgyM
reduce batch size 28 weeks -> 14 weeks -> 4 weeks
700 staff, 3000 servers. lead time 6 months -> 2 weeks. On demand self-provisioning for developers
Blackboard
devs deploy to prod
devs solve op issues in prod
GE Software Solutions Group
30% milestones hit -> 95%
now rolling out devops to 15,000 software practitioners in GE, sponsorship of Chairman
Nationwide
system of record, mainframe
450 IT staff, 42 lean/agile teams, 150 releases annually, $75M app dev
Chg Mgmt treats any Puppet-controlled change as a standard change
Macys
1700 manual tests every 10 days -> 100,000 tests/day
Redhat
internal systems, are a horse
halve cycle time, double frequency
Barclays
22,000 techs. DevOps team 160, rolling out to 1500

Miscellany

My tweets:
- @dfwbeebe I like that. Sometimes devops is a unicorn, sometimes it is a donkey in a party hat
- DevOps has jumped the unicorn. It's not about becoming a unicorn, don't aspire, it is about applying unicorn ideas to the real world
- RT @SoberBuildEng: I claimed we had reached "Peak Unicorn" in my @devopsdaysChi talk.

End every presentation with a slide "here's what I don't know how to do, or I need help on" - Gene Kim
Question of the year: do you have to hit rock bottom to get executive buy-in?

“Vocabulary engineers”: say agile do waterfall
Separation of Duties is not mandated by any(?) standard, it is only recommended. Auditors can be happy with DevOps automation controls.
Toolmaking can be treated as capex
ERP systems are “challenging” for DevOps
DevOps change takes a long time: years
Quantify technical and process debt
Define “good” DevOps