Mobile games testing, crowd-sourcing testing of mobile games. Game analytics: tools, metrics.

1. The Way to Production:
Game Testing & Analitics
Lecture 04
2016-2017
Mobile Games Development & Promotion, Maksimenkova
Olga, FCS, SSI
1

2. Today
Testing
• What to test in games?
• How to test?
• Features for mobile games?
Statistics
• Metrics
• Tools
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
2

3. Game testing
Testing and QA
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
3

4. Beta testing
• Closed (internal) beta
• Open (public/external) beta
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
4

5. What should be tested in a game?
• Balance (balance testing)
• Fairness, a gameplay is fair to both the human and AI player alike
• Difficulty balance
• Levels balance
• Maps, weapons,….
• The guidelines for balancing a game see in (The many faces of testing: the
game life cycle, Game Development Essentials: Game Testing & QA by L. Levy,
J. Novak)
• Hardware compatibility (compatibility testing)
• Video subsystem
• Audio subsystem
• Controllers (input/output devices)
• Network…
• See a sample checklist at bitbar (http://bitbar.com/mobile-game-testing-
part-4-test-for-connectivity/)
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
5

6. What should be tested in a game?
• Compliance testing
• Meeting standards, specifications, requirements
• Tolerance
• Warnings
• Copyrights, ….
• Internationalization (Localization Testing)
• Funniness (Playtesting)
• Usability (usability testing)
• UI/UX
• Tfortesting. Testing make things perfect, Test Cases For Games Apps /
Checklist for Games Apps
(https://tfortesting.wordpress.com/2012/10/04/test-cases-for-
games-apps-checklist-for-games-apps/)
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
6

7. Testing techniques
• Ad-Hoc
• Finding bugs by playing.
• Automated
• Let computer do repetitive work for you
• Functionality
• Make sure that game functions properly well according to your expectations
• Functionality requirements see in (Race to the Finish Line: elite bug hunting,
Game Development Essentials: Game Testing & QA by L. Levy, J. Novak)
• Progression
• Seeking possibly interruptions in a gameplay
• Regression
• Checking the results of old bugs fixing
• Mobile game testing see at bitbar (http://bitbar.com/mobile-game-
testing-the-importance-and-difference-from-app-testing/)
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
7

8. Study
• Find and compare definitions of crowdsourcing
• Please avoid using only Wikipedia
• What is the difference between crowdsourcing and outsourcing?
• What is crowdsourcing testing?
• Give a list of features with descriptions and links
• Explore the types of software testing, compare them with game
testing.
• Give a table of comparison
• Use all above and sources from the References to build testing
strategies of your game.
• Describe stages, game testing scenarios
• Prepare testing artifacts like checklists, testing requirements document,
software testing plan, game testing plan
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
8

9. Crowdsourced testing
• The Definitive Guide to Crowdsourced Testing (for Testers and
Companies) (http://www.softwaretestinghelp.com/guide-to-
crowdsourced-testing/)
• Traditional Testing vs. Crowdsource Testing vs. Freelance Testing
(https://www.qualitestgroup.com/blog/test-methodologies-and-
philosophies/traditional-testing-vs-crowdsource-testing-vs-freelance-
testing/)
• Outsource Vs. Crowdsource Testing
(http://www.base36.com/2013/05/outsource-vs-crowdsource-
testing/)
• Crowdsourced Testing: A New Way to Do Mobile App Testing
(https://dzone.com/articles/crowdsourced-testing-a-new-way-to-do-
mobile-app-te)
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
9

10. Game analytics
Metrics and Statistics
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
10

11. Basic metrics
• Relationship between User and a Game
• DAU (Daily Active Users)
• The number of unique users that start at least one session in a game on any day.
• DAU may be called a measure of “stickiness” of a game
• Sessions
• Sessions number is increased every time user opens a game.
• Brings an information as a ratio with DAU
• DAU/MAU
• MAU is Mothly Active Users
• DAU/MAU ratio may show how well a game retains users
• Retention
• Churn
• How many users who downloaded and may be set up a game are no longer playing?
• Money
• Conversion Rate
• measures the percentage of unique users that have made a purchase out of the total number of users
during that time period.
• ARPDAU (Average Revenue Per Daily Active User)
• Allows to measure how a game performs.
• Useful if tracked before and after user completes scenario/campaign/level.
• ARPPU (Average Revenue Per Paying User)
• ARPDAU only for paid users
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
11

12. In-Game Metrics
• Source-Sink-Flow
• Source means primary and secondary currency
• Sink the ways of spending these types of currency
• Flow defines the balance between gaining and spending currency
• Start-Fail-Complete (progression metrics)
• Start is about starting new levels. It measures the number of times a player
starts a new level.
• Fail measures the number of times when user started but did not complete
level.
• Complete measures the number of times when user started and complete
level
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
12

13. Reference Study (Metrics)
• Study the following links:
• 15 Metrics All Game Developers Should Know by Heart
(http://www.gameanalytics.com/blog/metrics-all-game-developers-should-
know.html)
• Glossary of Metrics
(https://unity3d.com/ru/learn/tutorials/topics/analytics/glossary-metrics)
• The Game Life Cycle & Game Analytics: What metrics matter when?
(http://www.slideshare.net/TomSente/casualconnect2012-honeytracks-
game-lifecycle-kpis)
• How to calculate Mobile Game Retention and LTV using Google Analytics
(http://measureofluck.com/calculate-retention-ltv-using-google-analytics/)
• Complete the list below with new metrics and their descriptions
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
13

14. Reference Study (Analytics Tools)
• Study the following links:
• What is the best analytics tool for online mobile game?
(https://www.quora.com/What-is-the-best-analytics-tool-for-online-mobile-game)
• Analytics that empower you (http://www.gameanalytics.com/features)
• Help your players level up their game
(https://www.helpshift.com/gaming/?utm_source=google&utm_medium=cpc&utm_ca
mpaign=Mobile+Gaming&utm_term=%2Bmobile%20%2Bgame%20%2Banalytics&mt=
b&ad=108252874846&gclid=CjwKEAiA94nCBRDxismumrL83icSJAAeeETQxdtomAPDzvB
EfxGEUImyKbnvLFbzY5AFNHvdJ-HFORoC_yzw_wcB)
• The ten best free analytics tools for mobile app developers: from Google Analytics to
Fabric, Firebase, Flurry and DevMate (http://www.techworld.com/picture-
gallery/apps/eight-best-free-analytics-tools-for-mobile-app-developers-3636407/)
• Prepare a list of analytics tools for mobile games, what metrics they allow
to measure.
• Select one tool suitable for the genre and monetizing strategy of your
game-project.
• Study it carefully and justify its application in your project.
• Describe scenarios with screenshots of how will you measure your game metrics.
• Give the examples of metrics for your game.
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
14

15. References (Testing)
• L. Levy, J. Novak, Game Development Essentials: Game QA & Testing (2010).
• C.P. Schultz, R.D. Bryant, Game Testing: All in One (2011)
• J. Hoberg, Difference between software testing and game testing
(http://www.gamasutra.com/blogs/JohanHoberg/20140721/221444/Differe
nces_between_Software_Testing_and_Game_Testing.php)
• J. Carrera, Test Planning and Execution in a Mobile Development Project
using SCRUM (https://www.scribd.com/document/53989828/Test-Planning-
and-Execution-in-a-Mobile-Game-Development-Project-using-SCRUM)
• Game testing methodology (https://www.scribd.com/doc/6555758/Game-
Testing-Methodology)
• 17 Strategies for End to End Mobile Testing on both iOS and Android
(https://www.ymedialabs.com/17-strategies-for-end-to-end-mobile-testing-
on-both-ios-and-android/)
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
15

16. References (Metrics)
• Mobile Game Metrics: Why Focus on a Single KPI Can Be Dangerous
(https://sweetpricing.com/blog/2016/08/mobile-game-metrics/)
• Tracking the metrics that matter in social and mobile gaming
(http://www.develop-online.net/opinions/tracking-the-metrics-that-matter-in-
social-and-mobile-gaming/0196581)
• A comprehensive list of metrics for free-to-play games
(http://www.gamesbrief.com/2014/05/a-comprehensive-list-of-metrics-for-free-
to-play-games/)
• Free to play game metrics for beginners
(http://www.gdcvault.com/play/1015473/Free-to-Play-Game-Metrics)
• 9 Mobile App KPIs to Know (http://mashable.com/2013/09/04/mobile-app-
metrics/#OlOMPY9cGkq1)
• Mobile Game Analytics Example (https://calq.io/docs/resources/track/mobile-
game)
• New mobile game statistics every game publisher should know in 2016
(https://www.surveymonkey.com/business/intelligence/mobile-game-statistics/)
• Retention and LTV as Core Metrics to Measure Mobile Game Performance
(http://www.gamasutra.com/blogs/IgorMatrofailo/20151221/262214/Retention
_and_LTV_as_Core_Metrics_to_Measure_Mobile_Game_Performance.php)
2016-2017
Mobile Games Development & Promotion,
Maksimenkova Olga, FCS, SSI
16