Research and testing
•Download as PPTX, PDF•
2 likes•154 views
Presentation done for members of FRII in Stockholm on the 14th of February 2017.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (15)
Featured
Featured (20)
Research and testing
- 2. ¿ A/B ? 2
- 3. 3 Step 1 Research Step 2 Define Action Step 4 Fix Step 3 Prioritize Reference: Peep Laja, Conversion XL CONVERSION FRAMEWORK
- 4. STEP 1: RESEARCH FIND WHAT REALLY MATTERS 4
- 5. RESEARCH: WHERE IS THE PROBLEM? Technical analysis - Cross Browser - Cross Device - Screen resolutions - Speed 5 Tools: GA, Viewport Resizer, Browser Stack
- 6. RESEARCH: WHERE IS THE PROBLEM? Technical analysis - Cross Browser - Cross Device - Screen resolutions - Speed 6 Tools: GA, Viewport Resizer, Browser Stack
- 7. RESEARCH: WHERE IS THE PROBLEM? Technical analysis - Cross Browser - Cross Device - Screen resolutions - Speed - Click maps, scroll maps - User session video replay - Form analytics Web Analytics - What are people doing? - How is each feature performing? - Identify leaks 7 Tools: GA, Viewport Resizer, Browser Stack Tools: GA, Klipfolio oe any other tool to plot data
- 8. RESEARCH: WHERE IS THE PROBLEM? Technical analysis - Cross Browser - Cross Device - Screen resolutions - Speed - Click maps, scroll maps - User session video replay - Form analytics Web Analytics - What are people doing? - How is each feature performing? - Identify leaks 8 Tools: GA, Viewport Resizer, Browser Stack Tools: GA, Klipfolio oe any other tool to plot data
- 9. RESEARCH: WHERE IS THE PROBLEM? Technical analysis - Cross Browser - Cross Device - Screen resolutions - Speed Mouse tracking analysis - Click maps, scroll maps - User session video replay - Form analytics Web analysis - What are people doing? - How is each feature performing? - Identify leaks 9 Tools: GA, Viewport Resizer, Browser Stack Tools: GA, Klipfolio Tools: HotJar, Inspectlet, SessionCam
- 10. RESEARCH: WHERE IS THE PROBLEM? Technical analysis - Cross Browser - Cross Device - Screen resolutions - Speed Mouse tracking analysis - Click maps, scroll maps - User session video replay - Form analytics Web analysis - What are people doing? - How is each feature performing? - Identify leaks 10 Tools: GA, Viewport Resizer, Browser Stack Tools: GA, Klipfolio Tools: HotJar, Inspectlet, SessionCam
- 11. RESEARCH: WHY IS THERE A PROBLEM? Qualitative research - On-page surveys - Email surveys - Interviews 11 Tools: HotJar, Qualarroo
- 12. RESEARCH: WHY IS THERE A PROBLEM? Qualitative research - On-page surveys - Email surveys - Interviews User testing - Identify usability & clarity issues 12 Tools: HotJar, Qualarroo Tools: In person! Or… Invision, Usabilityhub, Lookback
- 13. RESEARCH: WHY IS THERE A PROBLEM? Qualitative research - On-page surveys - Email surveys - Interviews User testing - Identify usability & clarity issues 13 Tools: HotJar, Qualarroo Tools: In person! Or… Invision, Usabilityhub, Lookback
- 14. RESEARCH: WHY IS THERE A PROBLEM? Qualitative research - Email surveys - On-page surveys - Interviews User testing - Identify usability & clarity issues Heuristic analysis - Experts’ assessment - Relevancy, clarity, value, friction, distraction 14 Tools: HotJar, Qualarroo Tools: In person! Or… Invision, Usabilityhub, Lookback
- 15. STEP 2: DEFINE ACTION 15 Only these two “require” A/B Test Author: Craig Sullivan, Conversion XL Instrumentation - Needs to set up more analytics to understand Investigate - More info needed Just do it (JFDI) - Low brainer, low effort Test - “Obvious” Simpler solution Hypothesize - No clear singe solution, test plans
- 16. STEP 3: PRIORITIZE - How easy is it to fix it? - Opportunity score 16
- 17. STEP 4: FIX FIX & RESTART ALL OVER AGAIN 17
- 18. BE HUMBLE <3 • Your personal opinion does not matter – you are not the giver • You don’t know what will work – if you did, you would have already made more money • There are no magical templates – what works for one website does not work for another 18
- 19. THANKS! 19 Aline Rauh Müller Produktägare UNICEF aline.muller@unicef.se
Editor's Notes
- We have all been advised to do A/B tests. magical formula A/B Tests are quite amazing but there some issues. Problem: traffic: Most of us do not have the traffic that it takes, and without reaching statistic significance, you cant be sure of your results, they can just be random, its bad science.. So I want to go through ways to improve websites without depending so much on ab tests. Another problem: using great tools like GA for reporting only, and not for decision making. We will also talk about how to make decisions based on these tools. We will do it by going through a framework I use at Unicef based on the work of the consultancy Conversion UX.
- Research, where you find what problems you have. Define what you are going to do with each single problem. Prioritize the problems. Fix them. Start Again. Very Continuous improvement process.
- An investigation of what problems we have and what we need to fix them. That is: find out what is that really matters. It is very easy to spend too much time on things that do not really matter. You sometime spend too much time on copy of one page or redesign of another without even knowing there is have a huge leak in another page, the idea is to be as strategical as possible. It is not like you go on GA and get insights. Good research starts with having solid questions to be answered. So now we are going through 6 research methods, this will be the biggest part o the presentation.
- First we have 3 research methods that point out where the problem is. Technical analysis: a low hanging fruit and super important. So what you are going to do is pretty much segment each of your conversion according to device, and then according to browser, and also by browser version to identify possible technical and design problems of your site. What is good about it is that it is super easy to quantify: you have a browser converting 20% less than the average, you got the average ticket of transactions from that browser, you know directly how much time is ok for your developers to fix the problem without losing money. Screen resolutions is also an important one, there is a lot of Google Chrome extensions that let you simulate your visit in different screen resolutions.
- First we have 3 research methods that point out where the problem is. Technical analysis: a low hanging fruit and super important. So what you are going to do is pretty much segment each of your conversion according to device, and then according to browser, and also by browser version to identify possible technical and design problems of your site. What is good about it is that it is super easy to quantify: you have a browser converting 20% less than the average, you got the average ticket of transactions from that browser, you know directly how much time is ok for your developers to fix the problem without losing money. Screen resolutions is also an important one, there is a lot of Google Chrome extensions that let you simulate your visit in different screen resolutions.
- The second method to find where you have problems is web analytics. Web analytics: That means visiting your BFF GA. Remember this is for conversion, so things like number of visitors, which city they are in, how much time they spend, don’t mean anything here. You want to actually investigate user behaviour and understand: what are people doing on your website? Where is the website leaking money? What is the performance of every feature you have? Super important to make sure everything is set up correctly, that GA goals havent been updated in a long time, so if you are unsure, make a GA audit first. If you have questions coming up often, you can create custom reports. 404.
- The second method to find where you have problems is web analytics. Web analytics: That means visiting your BFF GA. Remember this is for conversion, so things like number of visitors, which city they are in, how much time they spend, don’t mean anything here. You want to actually investigate user behaviour and understand: what are people doing on your website? Where is the website leaking money? What is the performance of every feature you have? Super important to make sure everything is set up correctly, that GA goals havent been updated in a long time, so if you are unsure, make a GA audit first. If you have questions coming up often, you can create custom reports. 404.
- The third and last method to find where you have problems is mouse tracking analysis. - Mouse tracking analytics. So this allows you to see how people are interacting within a single page. I think the most common tool for this is HotJar. Make sure to have 2,000 views on your page before checking out results. So you got click maps, showing where people are clicking. You might see things like people clicking on things that are not links, in which case you have to either make them become links or change their design so they do not look like links and don’t create frustration. Scroll maps, showing how far down people scroll. Of course the longer your page is the more people you will lose on the way. The scroll maps will show you with you prioritze your content well and also if you might need better design cues so people understand how you separate content, finish one session and start another. Watching users actually interact with your website can be super valuable, specially if you have specific questions to answer. So as an example, this last xmas campaign we pushed for our Gåvoshop, and I had quite a lot of questions about the checkout process there that’s is very long. So during this campaign period I set Hotjar to only record sessions of users who have started on the checkout process so I can see how they behave. Form analytics: which fields cause most error messages? Which fields are most left empty at first even though required? So, when you are looking for where you have problems, you have these three different ways to go around and find them.
- The third and last method to find where you have problems is mouse tracking analysis. - Mouse tracking analytics. So this allows you to see how people are interacting within a single page. I think the most common tool for this is HotJar. Make sure to have 2,000 views on your page before checking out results. So you got click maps, showing where people are clicking. You might see things like people clicking on things that are not links, in which case you have to either make them become links or change their design so they do not look like links and don’t create frustration. Scroll maps, showing how far down people scroll. Of course the longer your page is the more people you will lose on the way. The scroll maps will show you with you prioritze your content well and also if you might need better design cues so people understand how you separate content, finish one session and start another. Watching users actually interact with your website can be super valuable, specially if you have specific questions to answer. So as an example, this last xmas campaign we pushed for our Gåvoshop, and I had quite a lot of questions about the checkout process there that’s is very long. So during this campaign period I set Hotjar to only record sessions of users who have started on the checkout process so I can see how they behave. Form analytics: which fields cause most error messages? Which fields are most left empty at first even though required? So, when you are looking for where you have problems, you have these three different ways to go around and find them.
- Next we have 3 research alternatives that look for why is this problem actually happening? Qualitative sureys: Once again, we are talking about conversion, so ask for things that you can actually act upon. You ask people questions through on-site surveys. You can easily set them up with hotjar and make specific rules to what page and who is going to see this question. Remember that you should ask things that you can actually act on. So some examples: at Unicef.se the only thing you need to create an account to is to start your own fundraising. I was getting a lot people creating accounts but not creating fundraising, which was making my conversion funnel look bad, so I started asking people just after submitting registration why did they register for? And I got a huge range of things people were expecting to be able to do, showing we needed to communicate better. Another thing is email surveys, and that is something I actually do quite regularly. For example, this last xmas I had some questions for our Corporate donors. I wanted to know if the flow was fine, and if they had used the banners we offer in our thank you page (I could see that the banner page itself had really good visitors from the thank you page). So I wrote them all directly from my personal email, presenting myself and asking them 4 questions and a few days later I knew what was not visually fine with them. And if someone brings something relevant I can just keep the email conversation going.
- The Second method to find out where you have problems is User testing: Observing real people using and interacting with your website, paying attention to what they say and experience. You can give people specific tasks like ”find how to become a volunteer”, ”create a fundraise” and see if they make it on their own. It is quite common that we keep adding and adding things to a website or changing where things are located, so this makes sure people actually understand your page. A comment in brackets here: I personally like to use this mostly when you are doing big changes on page and for new product development, that is you create clickable mockups and user test them before actually doing any line of code. This can save you a lot of money.
- The Second method to find out where you have problems is User testing: Observing real people using and interacting with your website, paying attention to what they say and experience. You can give people specific tasks like ”find how to become a volunteer”, ”create a fundraise” and see if they make it on their own. It is quite common that we keep adding and adding things to a website or changing where things are located, so this makes sure people actually understand your page. A comment in brackets here: I personally like to use this mostly when you are doing big changes on page and for new product development, that is you create clickable mockups and user test them before actually doing any line of code. This can save you a lot of money.
- The third and last method to find where you have problems that we are presenting is Heuristic Analysis. Heuristic analysis: it is normally the first step that many take but I bring it up as last because we need to take care of the definition of what an expert is. Heuristic analysis is a group of experts (so not alone) sit together and go through your pages in a structured and focused manner. The best thing about it is that it is fast. The outcome is areas of interest that require further investigation from the other research alternatives. Do it in groups. It is not because I visit a lot of websites that I understand how to make it better to our donors. Just like it is not because I go to the supermarket a lot I know how to better design and organize one. when we say ”experts” here we mean ”optimizers, ux designers”. They will go through the pages and taken each page’s goal, will evaluate it according to how well it accomplish its goal. So, when you are investrigating the reason behind your problems, you have these three diffeerent ways to go around it. So now you got your research done and duzend of problems. What do you do?
- This list can easily get to 50 items. So after you’ve done your research and have a list of problems, you can put each one of the problems in one of these buckets that say how you are going to solve problem. Instrumentation: it means we need to improve the analytics reporting. It can include fixing, adding or improving tag or event handling on the analytics configuration. Example: you have different links for a visitor to reach his cart but dont have different events for them so you cant understand the flow fully, so instrumentation here would be to set up those separated events. Investigate: You might need to do some extra testing, like checking specific browsers. Example: You have a lower conversion in Safari with a good number of users on it, so you need check cross-browser compatibility or mobile UX issues.You can just go to browserstack.com JDI: Easy to identify issues, low effort. Bugs, Broken design in certain screen resolutions, people cannot read because the font is too small. Test: More obvious opportunities go. Example: you have a form where you see that 50% of users who get error messages when trying to submit it get the same error message. So you can test simpligying the form around that field. Hypothesize: Something is not working well but there is not really a clear single solution. Brainstorm hypothesis, create test plans. Note the plural here: it is not a tactical one off, it is a process of improvement. If your organization is not very analytical and data driven, you will probably get a lot of Instrumentation and Investigate issues in the beginning, and that is great, because it means you will start setting up proper analytics to answer to your questions. Note: if you get a lot of things under the Hypothesize and Test fields and you dont have enough traffic to A/B test, I normally do two things: I either go back to step one (Research) User Testing and do Usability tests with some users, like I will send some donors that I talk with an specific link and ask for feedback, and I will just change the website directly and closely monitor what happens. Of course you will have a lot less people in your usability test then in a failed a/b test, but it will give you more depth since you have the participant’s feedback. So now that you have everything categorized, what do you do? You prioritize. Why is it better to then not do a/b test and go for user testing and changing website? So yes, you have 5 different ways to categorize your problems in terms of how to solve them.
- Since we cannot do everything at once… we need to prioritize. Two things to take into consideration is: How easy is it to fix it? How much time it takes, are there any risks. Opportunity score: What lift will you get in your conversions, which is unfortunately always somewhat subjective. Score each of them 1-5. You can also give them stars 1 to 5. Once the prioritization is done, you can then create a structure.
- So finally, the moment we all waited, the time when you can go and start checking all those boxes of stuff to do!
- Here is a final note when testing: be humble, no one knows what is going to work, otherwise you wouldn’t need to test and follow up, be open minded.