What we've learnt and what we want to put in practice from recent conferences. A summary by Neil Frawley, Sam Thwaites and Shaw Innes about a trip to the 2016 DevOps Enterprise Summit in San Francisco.
Presented at the January 2017 Brisbane DevOps Meetup Group
7. Scale
Employee numbers:
• Average around 5000 engineers
• Microsoft 60,000. SAP 40,000 IT staff
• Target USA on-board 70,000 new staff for
Christmas period
• Walmart have 2.3 million staff worldwide
8. Scale
General stats:
• Target USA 6TB logs per day, 87,000 TPS for one
service
• Walmart 550-600 million transactions per day
• HPE 30,000 servers
16. People
• Learning culture, empowerment, collaboration,
continuous improvement
• “Being a farmer, not a hunter” (talent growth)
• “Don’t waste a crisis” (catalyst for change)
17. My Key Takeaways
Validation
• Scale – it could be so much harder, it will work
• Open Source – greater adoption, collaboration
• Pipelines – extending to Infrastructure as Code
• People – create the right conditions
36. Random Stuff
• 25,000 stores
• 190,000 employees
• 2000+ new stores in 2016
(5+ per day)
37. My Key Takeaways
• Enjoy It, take it all in
• Try to visit other companies
• Remember - they are all just like us
Editor's Notes
Speakers such as:
Gene Kim co-author of the Phoenix Project
Jez Humble of Continuous Delivery fame
John Willis co-author of the new DevOps Handbook (along with Gene and Jez)
Scale of American companies blew away all misconceptions about DevOps only working for unicorns, start-ups, or non regulated companies.
Employee counts for IT:
HPE 7000, Nationwide 5000, Capital One 5000+, All State 6000
Lots of companies both using Open Source software and contributing back to Open Source projects.
Target: Spinnaker didn’t have OpenStack integration so Target in conjunction with Netflix, Google built that driver.
Capital One: wrote own tool Hygieia – Dashboard for delivery pipeline
Netflix: uses Open Source to attract people
Many talks focussed on pipelines, building them & using them.
Codification, Automation, Version Control, use CI/CD for EVERYTHING
Moving from approval to audit. From Change approval to Change record.
(HPE talk ‘Scaling DevOps across through Business Unit Acquisitions, Mergers, and, Spin-offs’, DOES16 San Francisco - DevOps: Who Does What?, https://www.youtube.com/watch?v=tybMo_AN4rY)
Build compliance into your pipeline (Adobe talk ‘Compliance Help for the Agile Enterprise’)
Changes are only made through the pipeline (HPE talk ‘Scaling DevOps across through Business Unit Acquisitions, Mergers, and, Spin-offs’)
Topo from CapitalOne’s 16 pipeline gates
Reference: DOES16 San Francisco - DevOps at Capital One: Focusing on Pipeline and Measurement, https://www.youtube.com/watch?v=6Q0mtVnnthQ
People was a re-occurring theme right throughout the whole conference.
Learning culture, empowerment, continuous improvement (no end state)
Being a farmer not a hunter. Talent recruitment vs Talent growth. “You can’t make ‘your plants’ grow you as the farmer create the best conditions for growth”
Reference: DOES16 San Francisco - The Talent You Need Is Already Inside Your Company, https://www.youtube.com/watch?v=xQ6r-b_Tm5g
New Zealand Ministry of Social Development: big waterfall project failed, within 2 weeks committed to a 6 week release cycle, just used people and process no new tools. “Didn’t waste the crisis”
Reference: DOES SFO 2016 - David Habershon - Ministry of Social Development New Zealand, http://www.slideshare.net/ITRevolution/does-sfo-16-david-habershon-ministry-of-social-development-new-Zealand (could not find the video)
So what will I put into practice from the conference? Well for me it was more about validation around the things we are already doing.
On Scale it made me realise it could be so much harder in a larger company. Moving a large ship takes more time and more effort. It was validation that it will work. Big companies were talking about how it was working for them, but had the same challenges.
On Open Source it validated the moves I’ve seen towards the greater adoption of open source such as Chef, Packer, Vagrant and seeing open source as just another way of encouraging collaboration.
On Pipelines it validated our Infrastructure as Code initiative as the next natural progression of extending our pipeline to include ALL THE THINGS.
On People it made me think more about creating the right conditions and encouragement to allow people to grow and change. Without the people this can’t work.
Other talks I would recommend (in no particular order):
DOES16 San Francisco - DevOps At Target: Year 3, https://www.youtube.com/watch?v=1FMktLCYukQ
DOES16 San Francisco - Top 10 Ways to Fail at DevOps, https://www.youtube.com/watch?v=zQKPOL3d0ao
DOES16 San Francisco - SAP’s DevOps journey: from building an app to building a cloud, https://www.youtube.com/watch?v=55xQizPra7w
DOES16 San Francisco - Banking on DevOps, https://www.youtube.com/watch?v=xgSkva_Eq5s
DOES16 San Francisco - Making the Middle Great Again, https://www.youtube.com/watch?v=zMsi5zQ9wp8
Introduce Dojo
A place which practice happens
Transformational goal
How to upskill large groups of people
Possibly distributed teams
Competing goals
Target constantly challenge themselves to move faster
in 2015 they announced their concept of DOJO
Adam Jacobs from Chef talks about building practice through repetition
Dojo is a space where experts take up residence
Support up to 8 or 9 teams
Teams are expected to take the learning back
Challenges – 30+ day leveling-up experience, agile, devops, etc (2 day sprints)
Flashbuilds – 1-3 day problem solving, feature building
Open Labs – 90-minute sessions, twice weekly for Q&A
Reference: DOES15 - Heather Mickman & Ross Clanton - (Re)building an Engineering Culture: DevOps at Target, https://www.youtube.com/watch?v=7s-VbB1fG5o
http://www.slideshare.net/ITRevolution/does15-heather-mickman-ross-clanton-rebuilding-an-engineering-culture-devops-at-target
Target, CapitalOne, KeyBank, AllState
Other’s mentioned similar approaches
Growing backlog, but not headcount
Visited Target to learn
Formed Core Teams
Teams had Product Owner, Scrummaster, Expert Engineers
Called for volunteers to round out team
Worked in 8 week Cadence
Reference: DOES16 San Francisco - Utilizing Distributed Dojos to Transform Our Workforce, https://www.youtube.com/watch?v=Yy7dNXKgEsE
http://www.slideshare.net/ITRevolution/does-sfo-2016-aimee-bechtle-utilizing-distributed-dojos-to-transform-a-workforce
Dealing with rolling outages
Large complex environment
Started with DevOps Ninjas putting individuals into teams
Created a learning Center (Like Target Dojo)
Reference: DOES16 San Francisco - Banking on DevOps, https://www.youtube.com/watch?v=xgSkva_Eq5s
Reference: DOES16 San Francisco - Utilizing Distributed Dojos to Transform Our Workforce, https://www.youtube.com/watch?v=Yy7dNXKgEsE
http://www.slideshare.net/ITRevolution/does-sfo-2016-aimee-bechtle-utilizing-distributed-dojos-to-transform-a-workforce
What we’ve been doing for the last couple of years
Ninjas
Scaling this is going to be hard
Challenge with more teams asking for our time
Office Hours or DevOps Dojo time
So many companies talked about products, not projects