This document provides an overview of various techniques for automating the testing of mobile apps, including unit testing and instrumentation. It discusses using Xcode's Instruments tool to monitor reference counting, battery usage, network activity, and crash reporting. Specific instrumentation techniques covered include the Automation instrument, Calabash, Appium, and UIAutomation. The document also provides tips for testing build/version numbers, deploying to devices, and integrating with DevOps tools like Xcode Server.

1. Automating the Gaps of
Unit Testing Mobile Apps
Effectively Utilizing Xcode’s Instruments

2. @ggeoffreggeoffre.com
20+ years of experience in the development, marketing, sales,
and leadership of computer software development
A C T S

3. Preparing your QA team for mobile testing
http://www.slideshare.net/ggeoffre

4. Where did the term "bug"
come from?
66 years ago on
September 9th
1947, operators of
the Mark II Aiken
Relay Computer
being tested at
Harvard University,
found something
curious trapped
between points at
Relay #70, Panel
F.

5. Bugs not Defects!

6. Bugs not Defects!

7. Bugs not Defects!

8. Bugs not Defects!

9. Bugs not Defects!
Outside the scope of most acceptance or unit testing
Tend to design around them not to them
Defensive programming techniques employed
More likely hardware based than software based
Environment plays a major role discovering them
Magnified greatly with mobile based development

10. Controlled Environments

11. Today’s Topics
Reference Counting
Battery Usage
Network Monitoring
Communicating Build and Version Numbers
Xcode Server Initial Setup and Bot Creation
Test Deployment Options to Devices
Crash and User Analytics Reporting
DevOps
UITesting
Instruments
Automation Instrument
Calabash Appium
UIAutomation XCTest

13. Xcode ▶︎Open Developer Tool ▶︎Instruments

14. Reference Counting

15. [Objective-C]
alloc/init
retain
copy
release
autorelease

16. Properties and @IBOutlets
atomic
nonatomic
strong
weak
retain
assign
unsafe_unretained
copy
readonly
readwrite

17. Automatic Reference Counting
Edit => Convert => To Objective-C ARC
Project => Build Settings

18. Swift
When to use ? and when to use !
When to use “if let”?
When to use as? and when to use as

19. WWDC 2015
Optimizing Swift Performance
https://developer.apple.com/videos/play/wwdc2015-409/

20. WWDC 2016
Understanding Swift Performance
https://developer.apple.com/videos/play/wwdc2016/416/

21. Xcode 8 InstrumentsI

22. Leaks
The Leaks profiling template uses the Allocations
and Leaks instruments to measure general
memory usage in your app and check for leaks—
memory that has been allocated to objects that
are no longer referenced and reachable.
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingLeakedMemory.html
iOS Developer Library - Instruments User Guide

23. Leaks
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingLeakedMemory.html
iOS Developer Library - Instruments User Guide

24. Allocations
The Allocations profiling template uses the
Allocations and VM Tracker instruments to
measure general and virtual memory usage in
your app. However, to track down abandoned
memory that’s been allocated but isn’t needed
again, you’ll focus strictly on the Allocations
instrument. This instrument measures heap
memory usage and tracks allocations, including
specific object allocations by class.
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingAbandonedMemory.html
iOS Developer Library - Instruments User Guide

25. Allocations
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/FindingAbandonedMemory.html
iOS Developer Library - Instruments User Guide

26. Zombies
The Zombies profiling template uses the
Allocations instrument to measure general
memory usage in your app, with a focus on the
detection of overreleased “zombie” objects—that
is, objects that are called after they’ve been
released and no longer exist.
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/EradicatingZombies.html
iOS Developer Library - Instruments User Guide

27. Zombies
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/EradicatingZombies.html
iOS Developer Library - Instruments User Guide

28. Energy Savvy Users
Battery Life is very important to
users
iOS Now Displays Apps in
order of Battery Usage
Be sure to test in Low Power
Mode

29. Energy Gauge

30. Energy Diagnostics
Settings > Developer >
Instruments
Enable Energy Recording
Start Recording
Xcode > Instruments
New Energy Log
File > Import Legged Data
from Device

31. Energy Diagnostics

32. WWDC 2015
Debugging Energy Issues
https://developer.apple.com/videos/play/wwdc2015-708/

33. Data Savvy Users
Data caps are real
iOS displays cellular data
usage and allows users to turn
on/off apps
Be sure to test with Network
Conditioner

34. Network Conditioner
Settings > Developer >
Network Link Conditioner
Enable
Choose a Profile

35. Network Instrument

36. Network Monitoring

37. Network Monitoring
Use iTunes to find your UDID
UDID

38. Network Monitoring
Use Xcode Devices Window to find your UDID
UDID

39. Network Monitoring
rvictl -s <UDID>
ifconfig rvi0
tcpdump -s 0 -A 'tcp dst port 80 and
(tcp[((tcp[12:1] & 0xf0) >> 2):4] =
0x504f5354)'
...
rvictl -x <UDID>

40. HTTPS Server Trust Evaluation
Technical Note TN2232
App Transport Security Exceptions
<key>NSAppTransportSecurity</key>
<dict>
<key>NSExceptionDomains</key>
<dict>
<key>yourdomain.com</key>
<dict>
<!--Include to allow subdomains-->
<key>NSIncludesSubdomains</key>
<true/>
<!--Include to allow HTTP requests-->
<key>NSTemporaryExceptionAllowsInsecureHTTPLoads</key>
<true/>
<!--Include to specify minimum TLS version-->
<key>NSTemporaryExceptionMinimumTLSVersion</key>
<string>TLSv1.1</string>
</dict>
</dict>
</dict>

41. macOS Sierra Apps
WireShark
tastycocoabytes.comwireshark.org
Cocoa Packet Analyzer

42. WWDC 2015
Your App and Next Generation Networks
https://developer.apple.com/videos/play/wwdc2015-719/

43. Asynchronous Programming
Concurrent code execution on multicore
hardware
Grand Centeral Dispatch
Blocks
Closures
UIView Animations
NSNotifications
Local and Remote Push Notifications

44. Blocks [Objective-C]
dispatch_async(/* a queue /*, ^{
/* your code here */
});
dispatch_get_main_queue()
dispatch_get_global_queue(
DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)

45. dispatch_async(/* a queue /*, {
/* your code here */
})
Closures Swift
dispatch_get_main_queue()
dispatch_get_global_queue(
DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)

46. func perform_on_main(block: dispatch_block_t?)
{
dispatch_async(
dispatch_get_main_queue(), block)
}
func perform_in_background(block: dispatch_block_t?)
{
dispatch_async(
dispatch_get_global_queue(
DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), block)
}
Closures Swift

47. perform_on_main {
/* your code here */
}
perform_in_background {
/* your code here */
}
Closures Swift

48. [UIView animateWithDuration:1.0
delay: 1.0
options: UIViewAnimationOptionCurveEaseIn
animations:^{
/* your code here */
} completion:^(BOOL finished){
/* your code here */
}];
[Objective-C]UIView Animation

49. UIView Animation
UIView.animateWithDuration(0.7,
delay: 1.0,
options: .CurveEaseIn,
animations: {
/* your code here */
}, completion: { finished in
/* your code here */
})
Swift

50. NSNotifications
[[NSNotificationCenter defaultCenter]
addObserver:self
selector:@selector(someMethod:)
name:@"SOME_NOTIFICATION"
object:nil
];
[[NSNotificationCenter defaultCenter]
postNotificationName:@"SOME_NOTIFICATION"
object:self
];
[[NSNotificationCenter defaultCenter]
removeObserver:self
];
[Objective-C]

51. NSNotifications
NSNotificationCenter.defaultCenter()
.addObserver(self,
selector:"someMethod",
name:"SOME_NOTIFICATION",
object:nil)
NSNotificationCenter.defaultCenter()
.postNotificationName("SOME_NOTIFICATION",
object: self)
NSNotificationCenter.defaultCenter()
.removeObserver(self)
Swift

52. WWDC 2015
Building Responsive and Efficient Apps with GCD
https://developer.apple.com/videos/play/wwdc2015-718/

53. WWDC 2015
Designing with Animation
https://developer.apple.com/videos/play/wwdc2015-803/

54. WWDC 2011
Using Local And Push Notifications
https://developer.apple.com/videos/play/wwdc2011-517/

55. WWDC 2015
What's New in Notifications
https://developer.apple.com/videos/play/wwdc2015-720/

56. WWDC 2016
Advances in UIKit Animations and Transitions
https://developer.apple.com/videos/play/wwdc2016/216/

57. WWDC 2016
Thread Sanitizer and Static Analysis
https://developer.apple.com/videos/play/wwdc2016/412/

58. Testing the User Interface

59. iOS Human Interface Guidelines
https://developer.apple.com/ios/human-interface-
guidelines/overview/design-principles/

60. UIKit Catalog (iOS)
https://developer.apple.com/library/content/samplecode/UICatalo
g

61. Accessibility for Developers
https://developer.apple.com/accessibility/

62. Accessibility Audit
Xcode => Open Developer Tool => Accessibility Inspector

63. Accessibility Attributes
name Derived from the
accessibility label.
value The current value of the
control, for example, the text in a
text field.
elements Any child elements
contained within the current
element, for example, the cells in
a table view.
parent The element that contains
the current element.
The name property is one of four properties of these
elements that can be very useful in your test scripts.
Select and Set

64. Accessibility Inspector
Xcode => Open Developer Tool => Accessibility Inspector
Accessibility Label

65. WWDC 2015
iOS Accessibility
https://developer.apple.com/videos/play/wwdc2015-201/

66. Auditing Your Apps for Accessibility
https://developer.apple.com/videos/play/wwdc2016/407/
WWDC 2016

67. AUTOMATION
Behavior Driven Development
Acceptance Tests
JavaScript
Tool Based (Command Line Execution)
Instruments

68. AUTOMATION
Install Xcode from the App Store
https://itunes.apple.com/us/app/xcode/id497799835?mt=12
Xcode => Open Developer Tool => Instruments
Choose “Automation”
Select the target app to test with
Record/Run

69. AUTOMATION
Accessibility Label

70. TuneupJS.org
Tune-up is a collection of JavaScript utilities that
builds upon and improves the UIAutomation library
provided by Apple for testing iOS applications via
Instruments (get it? "tune-up"? Instruments? get
it?).
http://www.tuneupjs.org

71. T XCUIApplication()
Behavior Driven Development
Acceptance Tests
Swift or Objective-C (XCTest)
Tool Based, Project Test Target, (Command Line
Execution)
Xcode, Xcode Server Bot

72. T XCUIApplication()
Install Xcode from the App Store
https://itunes.apple.com/us/app/xcode/id497799835?mt=12
Create a new Project
Product => Test
Record

73. T XCUIApplication()
let app = XCUIApplication()
let someTextField = app.textFields["MyCustomLabel"]
Find Things
someTextField.tap()
someTextField.typeText("qwerty")
Do Some Stuff
let expectaion = self.expectationWithDescription("Something")
expectaion.fulfill()
self.waitForExpectationsWithTimeout(5.0, handler: nil)
Set Expectations

74. T XCUIApplication()
Xcode

75. T XCUIApplication()
xcodebuild test
-project SampleProject.xcodeproj
-scheme SampleProject
-destination
'platform=iOS Simulator,
name=iPhone 6s'
Command Line

76. Behavior Driven Development
Acceptance Tests
Ruby/Cucumber/Bash
Command Line Scripting
RubyMine (optional)

77. Clone the iso-smoke-test-app repo…
https://github.com/calabash/ios-smoke-test-app
Install the calabash sandbox… (setup Ruby)
https://github.com/calabash/install
Build a .app for the Simulator…
Run your first cucumber test:

79. Scenario: Touch OK button on alert
When I touch the show alert button
Then I see an alert
Then I see the "Smoke Test!" alert
And I wait for all animations to stop
And I can dismiss the alert with the OK button
When(/^I touch the show alert button$/) do
wait_for_element_exists("view marked:'show alert'")
touch("view marked:'show alert'")
end
Features
Steps
Ruby
Gherkin

80. Behavior Driven Development
Acceptance Tests
Earl Grey

81. WWDC 2015
UI Testing in Xcode
https://developer.apple.com/videos/play/wwdc2015-406/

82. Managing Simulators
Visible and Meaningful Version
Numbers
Xcode Server Bots for CI
iTunes Connect Crash Analytics
Flurry User Analytics and Reports
Fabric Crash Analytics Support
HockeyApp QA and Beta Testing
Some DevOps Tips

83. Manage Simulators
Window => Devices

84. Add a new Run
Script Phase
Reset All Simulators
osascript -e 'tell application "iOS Simulator" to quit'
osascript -e 'tell application "Simulator" to quit'
xcrun simctl erase all
Project => Target => Build Phases

85. Run Script Git Version Info
Build Version Number
buildVersion=$(/usr/libexec/PlistBuddy -c "Print CFBundleShortVersionString”
“${PROJECT_DIR}/${INFOPLIST_FILE}")
buildNumber=$(git rev-parse --verify HEAD | cut -c1-5)
buildString=$buildVersion' ('$buildNumber')'
/usr/libexec/PlistBuddy -c "Set PreferenceSpecifiers:1:DefaultValue $buildString”
"${SRCROOT}/Settings.bundle/Root.plist"
/usr/libexec/PlistBuddy -c "Set :CFBundleVersion $buildNumber"
"${PROJECT_DIR}/${INFOPLIST_FILE}"

86. Build Version Number

87. File => New => Settings Bundle
Build Version Number

88. Settings Bundle Root.strings
Build Version Number
/* A single strings file, whose title is specified in your preferences schema. The strings
files provide the localized content to display to the user for each of your preferences. */
"Group" = "SampleProject";
"Name" = "Version";

89. Settings Bundle Root.plist
Build Version Number
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict>
<key>PreferenceSpecifiers</key>
<array>
<dict>
<key>Title</key>
<string>About</string>
<key>Type</key>
<string>PSGroupSpecifier</string>
</dict>
<dict>
<key>DefaultValue</key>
<string>1.0</string>
<key>Key</key>
<string>version_preference</string>
<key>Title</key>
<string>Version</string>
<key>Type</key>
<string>PSTitleValueSpecifier</string>
</dict>
</array>
<key>StringsTable</key>
<string>Root</string>
</dict></plist>

90. Install OS X Server and Xcode on the target Mac
Enable the Xcode service (administrator account)
Add the remote repository for the project
Create a “BOT” to build and test the project
View the results in the browser
Setup Xcode Server

91. Enable Xcode Server

92. Create Bot for Project

93. WebView Screen Saver
GitHub / liquidx / webviewscreensaver

94. WWDC 2016
Advanced Testing and Continuous Integration
https://developer.apple.com/videos/play/wwdc2016/409/

95. Crash Analytics
Technical Note TN2151
Understanding and Analyzing Application Crash Reports
https://developer.apple.com/library/content/technotes/tn2151/_index.html
App Distribution Guide - Analyzing Crash Reports
https://developer.apple.com/library/content/documentation/IDEs/Conceptual/
AppDistributionGuide/AnalyzingCrashReports/AnalyzingCrashReports.html

96. WWDC 2015
Getting the Most out of App Analytics
https://developer.apple.com/videos/play/wwdc2015/303/

97. Flurry
Measure, track and analyze app performance, user
acquisition and activity with Flurry Analytics. Get a deep
understanding of app performance metrics and everything
your users are doing.
Create a Yahoo account and get your API Keys
Add the Flurry framework to your project
Integrate Flurry Analytics into your App
https://developer.yahoo.com/analytics/

98. Fabric hooks deeply into your existing build process and
automatically upload your dSYMs so you don’t have to. Now
you can sleep well, knowing you’ll get the crash reports you
need to keep your app stable and your users happy.
Create and confirming your Fabric account
Select the project you want to integrate Fabric into
Add the Fabric framework to your project
Fabric
https://developer.apple.com/testflight/

99. Distribute beta versions, collect live crash reports, get
feedback from real users and analyze test coverage. Bring
DevOps to your apps by integrating with your existing build
and work item management.
Create an Account to Manage Teams and Apps
Uploading a Build for an App
Set Up and Invite Testers
HockeyApp
http://hockeyapp.net/features/

100. TestFlight Beta Testing makes it easy to invite users to test
your iOS, watchOS, and tvOS apps before you release them
on the App Store. You can invite up to 2,000 testers using
just their email address.
Create an App Store iTunes Connect Record
Uploading a Build for an App
Set Up and Invite Testers
TestFlight
https://developer.apple.com/testflight/

101. ggeoffre

102. ggeoffre
slideshare.net
http://www.slideshare.net/ggeoffre