Mobile Testing Isn't Just About Screen Size
•
3 likes•1,595 views
This document discusses the differences between mobile and web testing. Mobile testing is more complex due to the many device types, screen sizes, operating systems and versions in use. Usability and performance are especially important for mobile. The future of mobile testing includes challenges from more devices, operating systems, form factors and new technologies. The author recommends testing on both real devices and simulators, considering battery life, operating system updates and automation approaches.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to Mobile Testing Isn't Just About Screen Size
Similar to Mobile Testing Isn't Just About Screen Size (20)
Recently uploaded
Recently uploaded (20)
Mobile Testing Isn't Just About Screen Size
- 1. Mobile Testing, That’s Just a Smaller Screen, Right? Stephen Janaway © Stephen Janaway 2014
- 2. Who am I? Stephen Janaway • 14 years experience in mobile software testing. • Mobile devices. • Mobile networks. • Mobile applications. • Have worked for companies such as Ericsson, Motorola and Nokia. • Work for the NET-A-PORTER GROUP. • www.stephenjanaway.co.uk • @stephenjanaway © Stephen Janaway 2014
- 3. www.testinginthepub.com @testinginthepub Search for “Testing In The Pub’” on iTunes © Stephen Janaway 2014
- 4. Today • An overview of mobile. • Why is mobile different? • What makes mobile testing different? • The future of mobile testing. • Your mobile testing future. © Stephen Janaway 2014
- 5. Mobile devices are • Smart Phones • Feature Phones • Tablets • E-readers • Big screens, small screens, touch screens and keypads. © Stephen Janaway 2014
- 6. And now… • Fitness devices. • Smart watches. • Wearable devices. • Beacons. • Household gadgets. © Stephen Janaway 2014
- 8. Mobile devices are much more • Alarm Clocks • Maps • Link to loved ones • Hand held shops • Tickets • Music Players • Books • Games Players • Cameras • …… • These are key experiences. © Stephen Janaway 2014
- 9. My alarm didn’t go off…
- 10. I’m lost
- 11. WHY IS MOBILE DIFFERENT? © Stephen Janaway 2014
- 12. MOBILE USERS ARE MORE EMOTIONALLY ATTACHED
- 13. MOBILE USERS ARE NOT SECOND CLASS CITIZENS
- 14. © Stephen Janaway 2014 ?
- 15. BUT QUALITY IS CURRENTLY STILL NOT GREAT… AND USERS INCREASINGLY DON’T TOLERATE THIS
- 16. MOBILE USERS CAN LEAVE BAD FEEDBACK MUCH MORE QUICKLY
- 17. What Makes Mobile Testing Different? • Many platforms. • Many OS versions in the field. • Many different hardware configurations. • Devices with limited memory and processing capability. • Limited battery life. • Network connectivity. • Application stores. • Low maturity of automation solutions. © Stephen Janaway 2014
- 18. Usability Matters • Usability is more important in the mobile world: – Smaller screen. – More difficult to interact with and typically used one handed.
- 19. Brand Consistency Matters • Think about what a mobile user would want. • Something familiar. • Taking advantage of mobile functionality: – Location. – Mobile search. – M-commerce. • A design that can be viewed on a small screen and in bright light. © Stephen Janaway 2014
- 20. Mobile Introduces a Whole New Language © Stephen Janaway 2014
- 21. Mobile Introduces New Scenarios • Download and install. • Un-install. • Upgrade. • Rotation. • Interaction with Sensors. • Location. • External media. © Stephen Janaway 2014
- 22. Mobiles Are Complicated • While your application runs, the phone is: – Talking to the network – Working out where you are – Maybe receiving calls, texts, Facebook and Twitter updates. • What happens when you have your application running, and then get a call, and a text, whilst uploading video to YouTube? Test for it and find out.
- 23. Performance Matters • Limited processor speeds. • Limited on-board memory. • Typically not turned off very often. • Don’t forget - most apps are just clients to web services. Use your web performance testing skills. © Stephen Janaway 2014
- 24. Data Matters • Check for excessive data consumption. • Using on device data measuring tools. • Using a proxy. © Stephen Janaway 2014
- 25. The Mobile World © Stephen Janaway 2014
- 26. Subscriptions vs. People © Stephen Janaway 2014
- 27. Different OS’s © Stephen Janaway 2014
- 28. Different Mobile Browsers © Stephen Janaway 2014
- 29. The Future Of Mobile Testing © Stephen Janaway 2014
- 30. More Devices = More Testing (?) © Stephen Janaway 2014
- 31. OS’s © Stephen Janaway 2014 Source: gs.statcounter.com
- 32. Android versions Source: developer.android.com
- 33. Screen Sizes © Stephen Janaway 2014
- 34. Deciding What to Test • Know your predicated customers (not your current ones). • Find out what OS updates are coming up. • Get as much data on device usage that you can: – In app analytics. – Web searches. – Similar apps. © Stephen Janaway 2014
- 35. Options • Don’t use them at all – use simulators or user agents. • Buy all or some of them. • Borrow all or some of them. • Rent all or some of them. • Use a cloud based service (Perfecto Mobile, DeviceAnywhere). © Stephen Janaway 2014
- 36. My Recommendation • Buy 2 from the top, 2 from the middle and 2 from the bottom. • Understand what your customers use or will use. • Test first on simulators not only on simulators. • Use cloud services if needs be, to fill any gaps. • Never just use browser emulation for mobile websites. © Stephen Janaway 2014
- 37. Open Device Labs • Borrow devices, normally for free. • http://opendevicelab.com • Open Device Labs = shared community pools of internet connected devices for testing purposes. © Stephen Janaway 2014
- 38. Making More Use of Your Users © Stephen Janaway 2014
- 39. Dog fooding/ Alpha/ Beta • Get an internal user group going. • Distribute app via TestFlight, TestFairy or similar. • Alpha/ Beta test via app stores. © Stephen Janaway 2014
- 40. More Financial Transactions Will Happen On Mobile © Stephen Janaway 2014
- 41. Security • Big recent rise in the amount of malware targeted at mobile devices. © Stephen Janaway 2014
- 42. © Stephen Janaway 2014
- 43. Perception of Security © Stephen Janaway 2014
- 44. Smartphones Everywhere = Your App Everywhere © Stephen Janaway 2014
- 45. Accessibility • How readable is the app/ website with a large font? • Is there voice activation / control? • Are there text alternatives to images? • Tools: – TalkBack (Android) – Voice Over (iOS) © Stephen Janaway 2014
- 46. Voice Activation • Siri – iOS • Google Now – Android • Also on Windows Phone and Blackberry. • Is the app voice activated? • What happens when you activate voice with the app running? © Stephen Janaway 2014
- 47. Your Mobile Testing Future You Can Avoid The Common Mobile Testing Mistakes © Stephen Janaway 2014
- 48. Only Testing On Simulators © Stephen Janaway 2014
- 49. Thinking a Mobile Device Lab Is a One Time Investment © Stephen Janaway 2014
- 50. Staying In The Office © Stephen Janaway 2014
- 51. Don’t Just Test Here…
- 52. On Here, and At Here…
- 53. Out Here…
- 54. In Here…
- 55. Forgetting About Battery Life © Stephen Janaway 2014
- 56. Battery Life © Stephen Janaway 2014
- 57. Ignoring OS Updates © Stephen Janaway 2014
- 58. OS Updates • Always regression test your app or website before an OS update. • Get on the developer program for early OS versions. • Users update more quickly on iOS than Android. • Carriers often control update cycles. • Decide beforehand how many OS version you will/ can support. © Stephen Janaway 2014
- 59. Thinking Mobile Test Automation Is Like Web Test Automation © Stephen Janaway 2014
- 60. Automation Suggestions • Determine if you require cross platform support. • Avoid tools that use embedded servers or agents – they alter the production code. • Look for tools with active online communities. • Consider the future. © Stephen Janaway 2014
- 61. My Picks • Appium: – Cross platform. – Well supported user community. – About to be more closely integrated in Selenium 3. • iOS Driver: – Well supported user community. – About to be more closely integrated in Selenium 3. • Calabash: – Well supported user community. – Heavily used at the BBC. © Stephen Janaway 2014
- 62. So……
- 63. Mobile Test Strategy – Top 5 Tips 1. Understand the potential user: – Which platforms and devices will they use? – How will they use the app or website? 2. Understand what OS versions are being used and when new ones are coming. 3. Test using simulators, real devices and users. 4. Understand the cost of automation: – It is where desktop automation was 6 years ago. – Best combined with extensive unit and integration testing. 5. Use an exploratory approach to manual testing: – Mobile device projects are typically short. © Stephen Janaway 2014
- 64. “Be proud, your work will soon be in the hands of millions”
- 65. Stephen Janaway Questions? www.stephenjanaway.co.uk @stephenjanaway www.testinginthepub.com
Editor's Notes
- Hi everyone. I’m Stephen and I’m going to talk about Mobile Testing.
- I’ve been in the industry for 14 years. I have a website and blog Follow me on Twitter.
- I also curate a regular podcast all about software testing. You can get more details at www.testinginthepub.com.
- Here’s what I’m going to be talking about.
- They can also be medical devices, smart readers, hand held scanners, and much more. Basically anything with limited memory, portable, preferably with a data connection.
- Android Wear - http://googleblog.blogspot.co.uk/2014/03/sharing-whats-up-our-sleeve-android.html Android FIT APIs Wearable camera’s NEST thermostats
- Remember when we had to carry all this stuff about? Now it’s just in one device. It’s taken less than 30 years to get to this point. Most people are carrying around a small mobile computer in their pocket.
- It’s much more than just devices Imagine if something went wrong…
- You are late for that important meeting because your phone alarm clock does not work. IPHONE EXAMPLE
- You are lost due to a problem with your phone’s navigation software. IOS7 MAPS EXAMPLE. OK – so of course you could use a map but that won’t actually direct you. And maybe you don’t even have maps anymore.
- Actual Hardware – drop it, get it wet, stuff is removable Consumer electronics – fast paced, device jealousy.
- To people they are much more. Users are more emotionally attached to their mobiles than their PCs. PERCEPTION OF QUALITY Often a person’s phone is the first thing that they look at when they wake up. In fact it wakes them up!! HTC advert and personal experience. People classify themselves by what device they carry. It becomes part of them. The apps they use become status symbols. Think about users perceptions of quality on each device.
- MOBILE USERS WANT A MOBILE OPTIMISED EXPERIENCE BUT THEY DON’T WANT LESS….
- I JUST WANT TO POST A COMMENT. HEAR ME!! Mobile Web Mobile sales have already overtaken desktop sales, and mobile Internet usage is predicted to overtake desktop internet usage by 2014. It is only logical that mobile search will overtake desktop search at some point in the near future as well. Since 67 percent of users claim they are more likely to purchase from a mobile-friendly website
- Mobile users have far higher expectations of mobile apps when compared to other software like browser applications. The majority of reports and surveys state that nearly 80% of users delete an app after using it for the first time! The top four reasons for deletion are always bad design and usability, loading time and crashes immediately after installation. PERCEPTION OF REQUIRED QUALITY IS HIGHER Nearly 60% of users will delete an app that requires registration, while more than half of users expect an app to launch in under two seconds. If the app needs more time it gets deleted. Again more than half of users experience crashes the very first time they start an app.
- Apps stores enable this. Do you download apps with consistently bad review and 1 star? Often lots of other apps to choose from. YOUR BRAND IS YOUR BRAND, NO MATTER WHICH PLATFORM!!! Find out what your users want and understand more about your users via apps stores. So what’s important? Testing, at all levels of app design!! SO LET’S LOOK IN MORE DETAIL AT TESTING AN APPLICATION.
- What makes mobile testing different? We will touch on these much more in the rest of the presentation.
- So, as testers, what do we need to understand. Firstly...usability. Conduct real user tests. Observe users using the product. Make friends with your UX people. Use user profiles. Screen is smaller and used one handed. TEST LIKE THE USER DOES!!!! In app analytics. Website analytics. User persona’s. Similar applications.
- So what are you looking for when you test for mobile? Consistent brand Engagement - People typically have more than one device. They get distracted.
- You need to learn a whole new language when testing mobile.
- Think about things like screen rotation, as well.
- Mobile devices are low power, multi-tasking, lower performance and memory. Stress them. Test for the interactions.
- Q to the class – what should we think about? Typically two processors – main plus GPU. Max 3GB RAM. 16/32GB ROM. Memory leaks, etc.
- Users hate you wasting their data allowances. Test for it.
- Being able to test effectively for mobile means understanding the mobile world in detail.
- The number of mobile cellular subscriptions is about to equal the number of people. Last time I looked it was - 7,259,491
- Android is king. Note Series 40 which MS just killed.
- Interesting thing here is the leader is the Android browser worldwide NOT Chrome as shipped on most recent Android devices. AOSP… UC Browser is a 3rd party browser which uses compression technology.
- So here’s what I think we’ll be doing in the future of mobile testing.
- Here’s the challenge – how to ensure testing covers enough of them? How do you keep on top? More wearables means more devices to get. More complicated test labs. More interactions between devices. More difficult test scenarios. IT MEANS USERS ARE BECOMING MULTI DEVICE AND THEREFORE MORE DISTRACTED AND MORE TIME POOR. Apps need to really grab the attention.
- Tricky – lots of versions in live.
- And it’s also about screen sizes. And carrier software.
- Subscribe to the relevant updates/ seed notices from platform manufacturers Make friends with UX
- So what should you do? Not just simulators: They don’t fully emulate the device (some are better than others). They are not running on low performance hardware. Cannot use on real network connections. Security is normally permissive. Just be aware that cloud based services have some issues with latency, and try them out first to make sure you are happy using them.
- A great idea. Why not start one?
- As testers – start taking control of this.
- Users still don’t trust mobile as much as desktop for commerce. Browse on mobile, buy on desktop. But this is slowly changing with things like Paypal and google wallet. APPLE PAY. Barcode scanning, contactless payments, london tube, location based use cases. Security is becoming more and more important. What does this mean for testing?
- Test for security issues. Think about things like browser security, cookies, etc. Security settings. App permissions.
- It’s your companies brand, no matter which platform. As testers we need to ensure companies are taking security on mobile much more seriously. User and credit card data. Stored on device or server? Stored encrypted? Stored on SD or SIM card? Removed when the app is uninstalled? Security of channel. Wifi sniffing. Man-in-the-middle. User left logged in. Can the app be de-compiled? MobiSec and OWASP.
- And it’s not only real security issues. Perception of security. Application permissions. Watch out for these. Are they appropriate? This is a flashlight app (malicious example but sometimes it’s just due to lazy coding). TEST FOR THIS BEFORE LAUNCH. Users perception of security (affects mobile web too).
- What does this mean for testing? Localisation Different character sets Right left languages Is the app available in different languages? What happens if you change languages? What happens if you change keyboard language? Watch out for right-to-left languages – a frequent source of UI bugs.
- Wearables drive this.
- So – for the future why not make a commitment to avoid some of the common Mobile Testing Gotcha’s?
- Mistake number 1 As we mentioned before: They don’t fully emulate the device (some are better than others). They are not running on low performance hardware. Cannot use on real network connections. Security is normally permissive.
- Mistake number 2 A mobile device lab is not just for Christmas! This is consumer electronics. You need to keep it updated regularly. Think about open device labs as well.
- Mistake number 3 Users are not in your office!!!
- Don’t just test in perfect Wifi and perfect light. Test in low signal conditions. Opensignal.com is brilliant for this.
- Test on the train. Overloaded networks. – test in busy areas. Waterloo station is the rush hour for example Test for swapping between 2G/ 3G/ Wifi. Speed.
- Deadspots Mention testers knowing where all the deadspots in Nokia offices are. Find a walking route with poor connections and record it using a fitness app.
- Mistake number 4!!
- Applications with poor battery life are hated by users. Moves being a good example. Some users just blame the device. Battery life – typically a problem with apps that use advertising incorrectly (Android especially). Very, very important. Typical scenarios: Normal Low battery Battery drain time Affect of poor/ no network connectivity. Does the app keep trying to connect?
- Mistake number 5 Ignoring OS updates or thinking that they don’t affect your app.
- iOS and Android (to Google Play devices), happen quickly. For the rest carriers control things and their acceptance testing adds significant delay. Always test before the update. Get on the developer programs and get early versions of the OS. But beware – they are not always the final functionality so be sure to also regression test in live after official update.
- Mistake number 6 Mobile test automation is like desktop test automation 6 years ago. You may want to consider pushing for more unit and integration tests before spending time on mobile UI automation.
- So – if I could sum up a mobile test strategy in 5 top tips, what would it be?
- And I’ll leave you with a quote from someone I used to work with in Nokia, which sums up why I think mobile and mobile testing is cool.
- As a project manager once said to me in Nokia: “Be proud, your work will soon be in the hands of millions”. No pressure on us testers, right?
- Please ask any questions. REMEMBER TO REPEAT THE QUESTION BACK TO THE AUDIENCE BEFORE ANSWERING. I’m about for the rest of the conference so please come and have a chat. Or you can contact me through my website or via Twitter.