Transcript from 5/20/2015 talk for LaunchCoderGirl

1. [Slide: Blue Screen Of Death, Fitbit error, BBC 500 error]
Let's start with a quick survey. How many of you have had a program crash on your laptop? Who's had
an app crash on their phone? Now put your hand up if you had an app crash and then you tried to make
it crash again. You may be QA people.
We go by a lot of names. Quality Assurance, Quality Engineer, Tester. At Asynchrony, we're called
Quality Advocates. Quality is everyone's job on the team, and we're the point person. We're
the conscience, we're the scout, we're the bridge between dev-speak and customer, we're the most evil
hacker your software will probably encounter this side of a public release, and we're the dumbest new
user seeing it for the first time. Good QA people are the best ally a developer has, we make them look
good and they rely on us to put their work through the wringer. We don't do it because we're mean, or
to prove they did it wrong, we do it to make the app better.
[Slide: QA walks into a bar]
For instance, one of my teams routinely puts code in their web app to prevent the user from pasting
large chunks of text in. Why? Because if you get the plain text public domain edition of Tolstoy's "War
and Peace", you've got yourself three megabytes of text, which is about five hundred thousand words.
The first time I dumped that into a field on a web app and hit submit, the web service call rejected it and
the app got stuck. So they fixed that and I tried it again, and this time the database rejected it because it
couldn’t fit that much text into the field it was aiming at. So they fixed it again. Then I put War and
Peace directly into the database and had the app try and pull it out, which was another set of bugs.
So what do Agile QAs bring to a project? Let me show you three different scenarios. The first project ran
with no QA at all for the first 12 weeks. Then they got QA, and the five developers spent two weeks on
nothing but fixing the bugs, and at the end of the project, we got this:
[Slide: Case Tracker story type graph]
This is a chart showing what kind of things we worked on, and nearly a third of our time was spent fixing
bugs. User stories are the features, tech debt is stuff you have to fix but it doesn't provide a new feature
to the user, technical stories are things you have to do that aren't related to a user story, like fixing
memory leaks or database queries, design is wiring in the new mock-up from the UX team.
We defined a defect/bug as anything that was in the main branch of the code. The main branch is
supposed to be something you can release at any point, you never put bad code into the main branch.
So this was not good. Really not good. And it’s not that this team was full of sloppy coders, these are
smart people who write good code. But developers aren’t good at testing their own stuff. No-one wants
to kick their own baby. That’s what we’re there for.
The next project is one that's had a QA for the whole lifetime of the project, and we’re still adding new
features
[Slide: TMS story type graph]

2. So this is better, the bug rate is down to about 10%, so the team can spend more time working on
features. Ten percent of the time, developers were fixing bugs instead of writing new features. This is a
piece of software that's been released to the client several times, it's been in the field for well over a
year. That's a much better graph, but we can do better than that.
[Slide: Combinator story type graph]
This is from a project where QA was on the team from day one, and I was writing automated user
acceptance tests in the first week, using C# and NUnit inside Visual Studio. This app was installed at the
client site, and in a year of use by a company of over 2000 people, it's not had a single bug reported.
We're pretty proud of this one. We repeated this process on the next two projects with similar stats and
it's a model that's spreading. What it requires is a QA that can write code .
A good QA who can read and write code is an invaluable person on a team. Imagine you're testing a car,
but the hood has been welded shut. You can drive the car, you can see the dials, you can mess with the
wheels, but if you could only lift that hood, there's a whole world of testing opportunities that you've
never seen before. As a QA, if you can code, you understand how the app is put together, what it
depends on, and you know where to shove a screwdriver into the works to cause an error. You know
where the weak spots are. You also understand it when your team starts talking about something and
they haven't taken into account problems like internationalisation. Or accessibility. Or security,
encryption, network load, performance, database access on a shared cluster, any one of a hundred
things.
So what’s the difference between a developer and a coding QA? Not a huge amount, but the biggest
difference is we’re much more rare. Asynchrony has three, and those three are training another six, and
five people are teaching themselves using Code School and books. That's fourteen people out of a group
of twenty four. I think QA has the best job in the house. Developers are nice people who want to build
things that work. QA gets to play, and burn things down, and make a mess, and we don’t even have to
fix the problems, just find them and fight for the ones that will cause the users the most pain.
Who saw the movie Tron? The original one and Tron Legacy both had the same line: “I fight for the
Users!” That’s what QA does. We work with the users in mind. Smart ones, not so smart ones, malicious
ones, ones who speak other languages, ones who are colour-blind, ones who can’t use a keyboard, ones
who can’t use a mouse, partially-sighted ones, chronically busy ones, we fight for all the users.
What do we do on an Agile team?
We start working on day one of the first iteration, even before a line of code has been written. That’s
when we start thinking about how we’re going to test the application, how the team will work together,
what the definition of a “bug” is for this project. We might do some investigation and learn a new tool
we’ll need to know, or pull together some requirements for what we’ll need from the developers. One
thing my team lead builds for me is a data loader to put several thousand records into the database so I
can test under more realistic production conditions.
And it’s where we start thinking about roles. Who plays D&D? You should recognize this:

3. [Slide: D&D alignment chart]
We cycle through these roles all the time. Up in the corner, the Lawful Good square represents all the
“happy path” tests. These are the tests that the application does what it’s supposed to under ideal
conditions. Developers spend most of their time thinking about the Lawful Good users.
Chaotic Neutral is a fun role to play. There is a tool for Android developers called the Monkey. Monkey
will click everything clickable in the user interface, and in the Android system, as fast as it can, over and
over until something breaks or you get to the end of the test. Monkey has no rhyme or reason, it’s
totally random.
For Lawful Evil, you’re staying within the bounds of the application, but playing everything as a worst
case scenario. They give you an address field? You paste a photo of a kitten into the zip code field.
You’re testing a mobile app? You step into a Faraday Cage which cuts off all network signal, or you call
the test phone from your cell phone.
Chaotic Evil is where you really get to play. Anything goes here. Go into the database and put in
deliberately bad data, pull the network cable out of the test machine, poke at the API with a sharp stick
and see what squeaks. Change your system language, run something to make the network really slow
and unreliable, intercept the network traffic and tweak it to introduce errors, run a piece of JavaScript
that hits the submit button seven times in a row as fast as possible and see what gets through to the
database.
There are no limits but time and your imagination. A good QA has this sense of diabolical creative glee at
putting an app through the wringer. When we’re done with it, it will be stronger and ready to face the
users.
[Slide: Little Bobby Tables]
This is what's called a SQL Injection attack. If you don't sanitize your inputs by treating them as a block of
text and stripping out stuff that would look to the database like a command to run, bad things can
happen. This is from a web cartoon called xkcd, which doesn't stand for anything, but has a lot of geek
humour. You can also inject JavaScript to mess with a web app, or PHP script.
What makes a good QA?
QA is the one job where you are paid to tell the truth. You never sugar-coat it, you never tell it like it
isn’t, and your team is depending on you to be honest with them. How many jobs can you say that
about? We would never do this:
[Slide: Dilbert performance test failure]
The most important thing that a good QA does is ask questions. Lots of questions, even when we know
the answer. Especially when you know the answer will be “I have no idea”, or “I didn’t think of that” or
“Oh crap!” Those are the good questions. Sometimes we play dumb to get our team to explain things to
us. When you’re breaking things down to tell someone, you get it straight in your own head, and
sometimes that process shakes loose something you forgot to include, or a flaw in your logic, or a

4. requirement that was never added but should have been there, like the ability to print something in a
particular format. We ask a lot of questions, and we listen to everything.
[Slide: Stuff QA says]
If we don’t ask, no-one else will. We fight for the users, and the client, and our teams. Our QA
apprentices have two jobs: Listen, and ask questions about everything you don’t understand. There are
no dumb questions, because the right dumb question will make your team think about things
differently. We just don’t know which question is the right one is until we’ve asked all of them.
QAs are detail-oriented people. We see the painting that’s crooked on the wall, the typo in the help
files, the colour that's one shade off the rest of the screen system. We are creative in finding every
possible way to stress a system, bust a database, and blow out a web request. It’s a good day when we
find a bug, or a glitch, or a missed acceptance criteria. Those things are like buried treasure, and the
more effort it takes to track down a bug, the bigger the treasure. QAs are fearless. We’ll switch the
language on our systems to Mandarin to see how the app responds to switching locale. We’ll delete the
database out from under a running application. We’ll put something in the app that blows up the web
service, never makes it to the database, and overloads the server error-reporting tool. That was a good
day.
The tools in our arsenal are many and varied. One of my favorites is War and Peace. Another is a tool
which can throw thousands of HTTP requests at a server to simulate a crowd of users hitting the app all
at once. There’s a tool that lets you spy on the network traffic going in and out of your machine. There
are tools that can make your network connection slow and unreliable. Firebug is a tool for the Firefox
browser that lets you inject JavaScript into a page, check on variables, and see the inner workings of a
site. And then there’s these:
[Slide: Foreign text]
From top to bottom, that’s Arabic at the top, which is a non-Roman alphabet that you read from right to
left, not left to right. There aren’t any characters in common with English. Next up is Cyrillic, which is
used in Russia. Some letters are the same, some are very different. And finally we have Greek, a third set
of unique characters. If you’re testing an application that will be used in a non-English speaking country,
you need to test in those languages. Right-to-left text looks odd when it’s on a left-to-right layout. The
German work for “Cancel” is “Stornieren”, and that’s not going to fit on a button designed for the word
Cancel.
A good QA is always learning. If you always test in a certain way, you’ll find a certain set of bugs. And
when those bugs are fixed, you won’t find others until you change up your process. One way to do that
is to call in another QA and hand them our app. No explanations, just hand it over and watch what they
do with it. Every QA is different. We all see different things. For example, can anyone see what’s wrong
with this app?
[Slide: LDI app]

5. We found a glitch in the layout that only affected Motorola devices running Android 4.1.2. Didn’t see
this on the Android emulator, didn’t see it on the HTC phones, or the Samsung ones.
[Slide: LDI app with red circles]
Right under the C of account, a piece of the header is shifted down by one pixel. Not all the way across,
it bumps back up further along. “Please move it three pixels to the left” is a valid thing for QA to ask for,
if those three pixels make the app look more polished, more professional.
Asynchrony QA apprentices
I mentioned QA apprentices earlier. In April of 2014 we hired someone with no software background at
all. He worked side by side with me every day for three months and worked hard. He was the sole QA on
that team for about 9 months before it merged with another team. Right now, I’d rate him as one of our
better QAs and he’s still learning. He’s also teaching two other QAs that have joined that team.
When we hire an apprentice, they are assigned to a senior QA as their mentor. They work side by side,
the senior explains their process, and the apprentice listens and asks questions. When we hire an
apprentice we’re looking for people with a strong desire to learn, someone with curiosity about how
things work and how to break them, someone that can think on their feet, someone that’s not afraid to
ask for help and ask for people to explain things until they get it. We look for people that want to be part
of a team. At Asynchrony, the team succeeds together, and fails together.
We discovered that having an apprentice around was good for the team, and good for the person
mentoring them. When you have to explain why you do everything you normally do by reflex, it makes
you pause, review how you work, and spot the gaps in your process. The job of an apprentice is to listen
and ask questions. The job of the mentor is to answer all the questions, and teach their apprentice how
to be a good QA.
Asynchrony is currently training our fourth QA apprentice. She’s on an iOS team with one of our coding
QAs learning how to test an iPad app that lives in an ecosystem of connected applications.
Wrap up
I hope I've given you some idea of the life of a QA in a software company. As a QA, you develop a
superpower you can't turn off. There are typos everywhere. Software may just break around you. And
this stuff is funny to us. We giggle when we see the mall display screen that's showing an error, because
someone didn't test it fully. We critique every app on our phones and try to break things just for fun. We
see broken software, it's everywhere. And somewhere behind the software you use every day there was
a QA.
[Slide: I see broken software]