Rise of the Robots: How well can machines measure UX? | Search Leeds 2019 | Andy Duke, Stickyeyes
•Download as PPTX, PDF•
1 like•801 views
As UX becomes an increasingly important online metric, Andy will look at how well machines (including Google) can actually measure the experience of a user. He’ll discuss the advantages, but also the dangers of relying on (easy to measure) factors like bounce rate or time on site. And he’ll also share his tips on getting the best of both worlds – machine measured UX with harder to measure human UX insights. We imagine there may also be quite a few Terminator references! Presenter: Andy Duke, Head of Design and Development, Stickyeyes
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Rise of the Robots: How well can machines measure UX? | Search Leeds 2019 | Andy Duke, Stickyeyes
Similar to Rise of the Robots: How well can machines measure UX? | Search Leeds 2019 | Andy Duke, Stickyeyes (20)
More from Stickyeyes
More from Stickyeyes (20)
Recently uploaded
Recently uploaded (20)
Rise of the Robots: How well can machines measure UX? | Search Leeds 2019 | Andy Duke, Stickyeyes
- 1. Rise of the Robots: How well can machines measure UX? AndyDuke @andyduke @stickyeyes
- 2. One big happy family… SEOs and UX teams should realize that they both focus on delivering a great experience... In fact, SEO and UX are two sides of the same coin. Because search engine algorithms have evolved based on user experience, marketers have learned to use SEO and UX together Chris Giarratana Search Engine Journal
- 3. GREAT UX = GREAT SEO
- 4. How do you measure it?
- 6. How to really measure UX? behavioural OR attitudinal OR qualitative OR quantitative OR HEART framework (happiness - engagement - adoption - retention - task success)?!
- 7. UX Hygiene Factors Basic UX Measures Super-dooper tricky complex measures 1 2 3 UX measures
- 11. UX Best Practise Playbook Travel Playbook
- 12. Clear CTA above the fold UX Hygiene Factors
- 14. UX Best Practise Playbook Travel Playbook
- 15. Use Professional Imagery UX Hygiene factor
- 17. UX Hygiene Factors • Link elements not too close together • Defined Viewport Area on mobile • Legible font sizes • Clear CTA above the fold • Contextual Breadcrumb • Have descriptive CTAs • Professional imagery • Clear benefit-oriented value prop above the fold • Don’t use full page interstitials • Have a value prop at every point in the journey • No drop-downs for inputs with < 4 options • Include CC for Video
- 21. Average Page Depth Basic UX Measures
- 23. Conversion Rate Basic UX Measures • Sales Conversion • Newsletter Sign-up • Submit Enquiry • Download Trial • Watch Full Launch Video
- 24. ?
- 25. UX basic measures • Pagespeed • First contentful paint speed • Bounce rate • Site search usage levels • Exit rate • Page views • Average session duration • Pages per session • Returning Visitors • Back button usage • Form completion rate • Page / scroll depth • Reading age • Form field context / help
- 26. Super Dooper Tricky Complex Measures
- 27. Effective Visual Hierarchy Super-Dooper Tricky Complex Measure…
- 29. Page Flow Super-Dooper Tricky Complex Measures…
- 30. But it’s not just about collecting data…
- 31. Super Dooper Tricky Complex Measures • Sentiment (NPS score) • Social engagement • Navigability • Effective visual hierarchy • Design consistency • Social sentiment • User Flow • Form error rates • Task level satisfaction • Focal points (eye tracking) • First click • Perceived complexity • Design consistency level • Scanability
- 32. UX measures UX Hygiene Factors Basic UX Measures Super-dooper tricky complex measures 1 2 3
Editor's Notes
- Hey everyone I’m Andy Duke, I’m Head of design and development at Stickyeyes I’m going to talk a little about the importance of UX and how you can measure it with machines - also just to keep everyone awake there are 20 movie robot references hidden throughout the talk
- So the talk begins, like all good wannabe ted talks, with a quote... QUOTE ABOUT SEO AND US BEING THE SAME So on the face of things this makes sense right - the job of a search engine is to give people the most relevant results - to give them what they want. And a good user experience, is in essence, an experience that gives users what they want
- So it’s all hunky dory right - perfect alignment of the stars - great UX = great SEO. Make sure your user experience is on point and SEO will be improved, sales will rise, targets will be smashed - everybody gets a huge bonus and a yacht ….only there is a bit of a problem with over simplifying things
- On the one hand you can measure your SEO success relatively easily (this is where I start getting pelted with eggs by some of the people in the room) - what I mean is the success metric is an obvious one. Position 1 is good - position 41 is not so good. But when we try and do the same with UX it gets a little tricky. User Experience is a very human concept - it’s woolly and varied and undefinable. Just like SEO it’s made up of hundreds of tiny little component parts (UI design and form field labels and CTA styles and icon choices) but when you get measuring it - it’s not so easy
- But here’s where it gets interesting - google are pretty open about the fact that they want user experience to be a key ranking factor. Its their goal to give you the results that offer the best experience based on what you’re searching for. So how the hell are they doing this? Well it can only be one of 2 things They have an army of 8 million cloned humans stashed away in a bond villain style lair hidden in the arctic circle and they look at every single page on the internet and review each one individually using their human brain and measure the UX Or - slightly less excitingly - they are using machines to review and measure the user experience of every single page on the internet :) / Will
- So let's assume - disappointingly - it’s machines and not bond villain clones that are measuring site UX - how are they doing it - what are they able to measure programmatically - and can we do the same? Now there is a hundred different ways you can cut up and categorize all the thousands of different ways you can measure UX. You’ve got behavioural vs attitudinal - OR - qualitative vs quantitative - OR you’ve got the HEART framework (happiness - engagement - adoption - retention - task success). But for purposes of this we’ll split it like this
- Hygiene Factors Basic Measures Super-dooper tricky complex measures UX hygiene stuff is the measures that are almost a checklist - your site either is or isn’t (easy for robots to check). The Basic measures are a little more complicated but not by much, they’re standard simple measures (also pretty easy for robots to check). And then the snappily titles ‘super complex tricky measures’ are where things get a little more complicated for the robots to take over - so let’s take a look at a few examples
- UX Hygiene stuff Some great examples of this type of stuff are to be found in google’s own mobile friendliness tool. If we jump in, we can see some great examples of UX hygiene factors like - Is the text big enough to read? or Are clickable elements far enough apart? Of course if the text isn't big enough to read on mobile you are not offering a good user experience. Of course if you buttons are too close together that people with big fat sausage fingers keep tapping on the wrong when and getting frustrated then - you are not offering a good user experience
- So here’s the million dollar question - can a robot measure this for you? Yes they can indeed and already do as most of you will already know the google mobile friendliness check is already there and ready to do this for you - so lets chalk up one to the robots :)
- Some other great examples of hygiene factors for your User experience can be found in the sector specific google UX best practices playbooks If we jump into the google pdf that covers the travel sector we can see under homepage they list out ‘Clear CTA above the fold’ Again - it makes sense - for travel websites and most other websites a clear logical CTA to continue the user journey can be considered best practise for a good experience.
- They even include some handy example of travel sites that are flying the flag for UX – here are their CTA proudly above the fold
- So can a robot measure this for you? It’s a little trickier but yes - with the write script you can programmatically determine if a site has a CTA located towards the top of the page that most users would see on their initial view - we’ll give the robots half a point on that one
- But when we move on and pick another for the results page recommendations you see other factors like “Use Professional imagery” Which on the face of it seem pretty simple doesn’t it
- It’s the difference between this and this You can't fault it from a UX perspective - absolutely, professional looking imagery on your travel website is the difference between this and this It’s an easy enough hygiene factor to measure - is it professional looking or not - yes or no
- But where does this leave the robots? As a human it’s pretty easy to say which one offers the best user experience - but how does the robot fair? Now you can argue that the latest AI coupled with image recognition software is pretty impressive at the moment but I’m gonna say not on this one? And that’s where an over reliance on programmatically derived metrics starts to fall down - when things become a little more subjective, to truly measure the user experience of your site you need to truly understand your site - which for now at least needs a human brain
- So those are a couple of examples of UX hygiene factors - now what about if we more on and look at some of the basic measures Now if a hygiene factors is a yes or a no - a basic measure is more on a scale
- Perhaps the best example of this is again one you can get easily from a tool courtesy of our friends at google - pagespeed Arguable one of the most critical factors that influences the user experience of your site, if it’s super slow and makes users sit and wait for it to load - its frustrating - it’s delivering a bad user experience The tool from google is fantastically simple to use and gives you tons of data not just the speed score – other more meaningful elements like first contentful paint which is how long it takes for part of the page to begin to show
- Perfect one for the robots too - the google pagespeed tool (or the lighthouse audit toolkit that it’s now part of) gives you a super neat score out of a hundred, it gives you so much context it even colour codes the score for you too just to really judge you So that’s a great example of a basic measure that you can measure programmatically - but what about when there isn’t a nice neat score out of a hundred
- Let’s jump into google analytics now (other analytics packages are available) Now obviously GA is a wealth of information that you can use to measure the effectiveness of your UX - if we grab an example from the engagement metrics that GA tracks like average page depth. So it’s the number of pages viewed in a single session - it’s a nice simple basic measure - easy peasy for a robot to measure for me but again - kind of like the ‘professional photography’ hygiene factor it only becomes useful with context. You can immediately get meaning from your page speed score - 99 out of 100 - great 15 out of 100 - yikes. But if we look at something like page depth, there is no implied grading with that score - it just tells it like it is - your average number of pages viewed is 3. But for your specific site is 3 showing that you have good user experience or a bad one? You need context
- This is where obviously a human comes in handy - but let’s not write the robots off yet. If you track a basic measure like average page depth over time you can start to see movement and all of a sudden you can tell people start looking at twice as many pages as they were last month - that related content block you added is giving users a better experience.
- If we step outside of googles various tools for a moment and look at what is widely seen as the most significant basic measure of UX - conversion rate Now although it can give you a nice percentage figure to measure - conversation rate isn't always an easy one to define in the first place let alone measure For ecommerce it’s easy enough - of the users that came to the site, how many were bought something but what if you don’t sell anything what if you're a charity and your main goal is awareness of an important issue, is there even a conversion point to measure. The answer is usually yes - it might not be a purchase but your site should always have a primary goal even if it’s just users signing up to a mailing list or submitting an inquiry.
- The good news for the robots here is if it’s an action that takes place on a website it’s something that you can track - either through your own internal systems or through event tracking with something like google analytics it’s another one to chalk up for the UX measuring robots. Except conversion (like a lot of other basic measures) is tricky in that it’s not necessarily a UX metric at all. It’s usually a huge part of it but you can't make the argue that conversion is a 100% UX metric - it’s impacted by so many other factors - price - brand awareness - market conditions. You could have the greatest user experience ever crafted in the history of the internet but if your CEO has just been publicly convicted of fraud and murdering puppies your conversion rate is going to tank - equally if you are giving away free Ferraris you can have a dreadful UX but I’ll still muscle my way through for my Ferrari :) Conversion rate is certainly a measure to use but more as an overriding guide to overall site success rather than as something 100% directly tied to website UX
- Right so far - we’ve got some decent UX hygiene factors that we can track easily and some basic measures that as long as they’re looked at over time can also help our robots measure UX. Super dooper tricky complex measures But what about the Super-dooper tricky complex measures - the measures that are more conceptual and tougher to define like how brand sentiment or perception is supported by the UX or complex interrelated measures like user flow between landing page and exit page
- If we take one of the more conceptual UX design focused measures - visual hierarchy - the measure that your site should present the user with an effective visual hierarchy. You can’t argue with it as a measure for good UX, studies have shown that sites with a clear visual hierarchy perform better in readability studies and often perform better in terms of task or sale conversion. But what does it actually mean - as soon as something related to the visual design of the page we’re right back where we were with the professional travel photography is subjective. Sure you can boil it down and make it into some hygiene factors - clearly defined H1 titles etc and that’s easy enough to check with a robot, but when it’s something as super -dooper tricky complex like ‘visual hierarchy’ who is the say what exact pixel size of title paired with what font colour or style and padding or use of iconography or images goes into making giving the best experience from a visual hierarchy.
- So is this where we say goodbye to the robots - well yes to a large extent yes and this is where the dangers of over reliance on following metrics come in with UX.
- Even measures that you see within google analytics still fall into the super dooper tricky complex category - page flow is a great example GA is fantastic for visualising this complex data point showing commonality in how users navigate through your site highlighting common landing pages and exit pages, giving you a sense of which pages are common exit pages, pages that users navigate through to reach their desired content. But again while being able to have this type of data visualisation automatically generated for you is fantastic and hugely useful in understanding your users it isn’t, in itself, a UX measure it might tell you that a certain page is the most common exit page but it won’t tell you why that’s a problem if it is indeed a problem at all.
- So while you can use C3-PO to visualise data and track your easy to follow metrics like conversion rate or time on site - don't think that puts you on top of the UX of your site. An overreliance of auto generated metrics is of little help when things go wrong When your conversion rate falls through the floor an auto generated percentage rarely tells you exactly why. Now in some cases, page speed as a great example - there are tools that do help you solve the problem as well as highlighting it but this isn’t the case for everything.
- If we go back to my incredibly scientific classification method you can see that your basics can largely be monitored and benchmarked programmatically, basic measures can, for the large part, be followed on a trend level but might need humans to give the trends meaning and context. And when you get to the super dooper tricky complex measures well them it’s people power all the way Robots Hygiene Factors Robots Basic Measures Humans Super-dooper tricky complex measures
- So to wrap it all up, there is - always - the temptation to boil everything down to numbers easy to digest - percentages - pluses and minuses - but in reality user experience is a combination of a thousand different things and the optimum mix of those things is different for every website in every sector for every user. So that’s why I don’t think robots will take over from UX practitioners just yet - but I do think (and it looks as though google agrees) that benchmarking the user experience of your site is largely possible through robots… with just a smidge of human intervention. But just to end by bringing it back around to search - if google are trying to factor in more and more UX elements and they’re using machines to measure it - it can’t help to keep on top of the measures that we know are easiest to track with machines.
- Now obviously I’ve only touched on just a handful of UX measures there - I think if I covered them all one by one we would still be here on Sunday night – there is hundred of them possibly even thousands To give you some quick insight into how we do this in real life at Stickyeyes without giving all our trade secrets away. We do utilize all the automated methods we can to help give a broad UX benchmark across hundreds of hygiene factors weighted based on significance - we also use the basic measures but we then layer in human UX knowledge still to ensure everything has context For everything beyond that it’s about the human touch - whether that’s lab based usability testing, session analysis with tools like hotjar or just a simple expert review from a qualified UX Designer - we won’t be replacing our UX team with robots anytime soon.
- If you have any questions feel free to grab me after and if you have a desire to check out the slides again I think we’ll be throwing them up on the Stickyeyes site Thank for listening everyone - cheers