3. Fear of the bus
Docs and Devops
Heidi Waterhouse | @wiredferret
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22. Fear of the bus
Docs and Devops
Heidi Waterhouse | @wiredferret
Editor's Notes
Thesis -- most ops and devops people are running flat-out to keep up with their work and don't feel that they have the time to talk to someone about their rapidly-changing docs or write them themselves.
That means a lot of mission-critical information gets stored in volatile memory or the human equivalent, or a wiki.
But if something happens to disrupt your power you lose the pointers, and even if the data still exists, it's effectively gone.
For a long time, IT has had the concept of "getting hit by or bus". The one pointer to the location of crucial information, like HOW TO MAKE A BUILD is gone, and there's no one who knows how to do the entire process as it's supposed to be done this week.
Now that I've given you all the shivers, what should you do about this?
Well, the obvious best answer is to hire a documentation person to chase your devops team around and pester them for information. Now that we've all stopped laughing, let's talk about what could realistically get done.
Rotate the build duty
If you're a serious figure skater, you sometimes skate the wrong direction so your legs don't develop unevenly. We have to change things up so that we maintain flexibility.
Your team also needs to change things up. It's gonna give them sympathy for the other person's roles, and it's going to develop their muscles in useful ways. And also, they're going to complain about it. So do the skaters.
Newsflash (muppet news?). We work in corporate America. No one is going to be here forever, and you won't always get to keep the most senior person. You MUST be able to replace a missing person and keep going.
Everyone can touch the docs
As a user documentation writer I want to control the information, the horizontal, and the vertical, so it stays correct and pretty.
As a DEVOPS writer. I do not give a …. Fig. The data moves too fast, and it's too internal for me to care about how it looks. No one cares as long as someone has documented the data you need in a place you can find it.
Basalmiq is my favorite interface design tool because it lets you show things as unfinished looking sketches. Psychologically, people are much more likely to make changes to that, rather than something that is polished smooth. You want your devops team to feel like making changes is trivially easy.
If what you care about is making sure things get documented somewhere, use one big page. It's ugly, and it goes against every design principle, but who cares? It's findable, searchable, everyone can have a single bookmark, and no one has excuses.
Use one canonical location, no matter how big it gets. Just push older stuff down to the bottom as you refine the method. This is messy and ugly, but remarkably effective, as long as you don't have any concerns about access or security
Use a flagging system so you can search on important bits of data and concatenate them together. I heard about a team that uses a unique emoji every time someone says something in Slack that should get put in the build book.
If you have enough people, use a buddy system to make sure there isn't a single point of failure. Everyone should have a backup who can actually replicate their skills AND knowledge.
Every team has their own idiosyncratic ways of communicating. It doesn't matter. What does matter is figuring out what your team is doing and putting documents in their existing path.
How do you test that you're doing it right?
Some of you familiar with the concept of the chaos monkey. The idea is that you select a person on the team, have them hand over their work phone, and take a day off. This is especially nice if they had a nasty on-call weekend. Who wouldn't enjoy a day off that was actually….off? Block them on slack and email, don't let anyone call their home phone. They're not here.
It will be hard. But let me tell you, the trial disaster is not going to be as bad as the real disaster. You may not be able to build today, but when your key person comes back, you'll know what you don't know.
The point of fire drills is to teach everyone to evacuate a building safely. The point of a devops drill is pretty much the same.
Handle a set of stressful actions when the world is not actually on fire, when people aren't actually emotionally compromised. If you can't remember where Maria keeps the build files when she's out for the day, you sure as heck won't remember when she's in the hospital and you're waiting to hear if she's ok.
Devops docs save time, money, and having to post pictures of the Hindenberg where your uptime stats used to be.