Mobile is now facing problems that have been existed in web for many years: fragmentation. In mobile, it is supporting the many phone and tablet models, operating system and versions, and languages. Calabash is an open source technology for functional testing of Android and iOS apps. It combines advanced automation technology with the Cucumber tool. Tests are written in a natural language of the business domain, and the cucumber tool can execute tests on Android and iOS devices or simulators.
Automating Google Workspace (GWS) & more with Apps Script
Calabash, an open-source automated testing technology for native mobile, by Karl Krukow
1. Introducing Calabash
automated functional testing for mobile native apps
Karl Krukow,
karl@lesspainful.com,
LessPainful
CodeMotion 2012, Rome
1
fredag den 23. marts 12
3. About me
• PhD, Computer Science, University of Aarhus, 2006.
2
fredag den 23. marts 12
4. About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java
enterprise, last two years on iOS.
2
fredag den 23. marts 12
5. About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java
enterprise, last two years on iOS.
• Co-owner, iOS responsible at
2
fredag den 23. marts 12
6. About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java
enterprise, last two years on iOS.
• Co-owner, iOS responsible at
• Spare-time Hickey & Clojure fan-boy!
2
fredag den 23. marts 12
8. Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
3
fredag den 23. marts 12
9. Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
• Introduce Calabash
• Focus on iOS only
3
fredag den 23. marts 12
10. Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
• Introduce Calabash
• Focus on iOS only
• Live Demo:
• Calabash iOS
• LessPainful: test service and device cloud
3
fredag den 23. marts 12
12. Functional testing
• Functional and Acceptance tests
• Actual app, as opposed to an isolated component
• Often based on use-cases written in natural language (for domain)
• Visual appearance of screens matter! (Design guidelines, branding,..)
• As realistic an environment as practically possible
4
fredag den 23. marts 12
13. Functional testing
• Functional and Acceptance tests
• Actual app, as opposed to an isolated component
• Often based on use-cases written in natural language (for domain)
• Visual appearance of screens matter! (Design guidelines, branding,..)
• As realistic an environment as practically possible
• For mobile apps, in particular
• Many devices, screens, OS versions, languages
• often a manual process: repetitive, expensive
• regressions: why should I test that again?
4
fredag den 23. marts 12
15. Functional testing
desiderata
• Minimize distance between use cases and actual test
code (DSLs?).
• Expressive and efficient to write.
• Extensible
• High-level, declarative (robustness against “minor” UI
changes).
• Support testing in realistic environments (multiple real
devices, on multiple OS versions, languages).
• Support Continuous integration.
5
fredag den 23. marts 12
17. • Cucumber provides
• a framework for writing software specifications
• a tool for executing those specifications
• Specifications are written in a business readable
language that is close to natural language.
• Extremely popular tool for test and specs of
web applications.
• http://cukes.info/
fredag den 23. marts 12
18. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
19. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
20. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
21. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
22. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
23. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
24. Step Definitions
• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Feature Step definitions
fredag den 23. marts 12
25. Step Definitions
• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Feature Step definitions
Scenario: Add test user
When I touch the Add User button
...
fredag den 23. marts 12
26. Step Definitions
• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Feature Step definitions
Scenario: Add test user When /^I touch the Add User button$/ do
When I touch the Add User button btn_txt = 'Add user'
... touch("button text:#{btn_txt}")
end
fredag den 23. marts 12
27. Calabash
• One interface: Cucumber, for Android and iOS tests.
• Predefined and custom steps (Ruby + soon: JVM).
• Reuse of tests across platform possible.
• Run on physical devices and simulators.
• Support for hybrid apps (embedded webviews)
• Free, open source with commercial add-ons:
• Run tests in a device cloud using the LessPainful service,
• we can write and run your tests,
• setting up continuous integration, consulting, training, support.
10
fredag den 23. marts 12
28. Architecture iOS
iOS App
--------------------
Calabash library
fredag den 23. marts 12
29. Architecture iOS
features
iOS App
--------------------
Cucumber Calabash library
(ruby api)
Test report
(cucumber)
fredag den 23. marts 12
30. LessPainful
Test Execution in the Cloud
• Execute Calabash tests
concurrently on many
devices, OS’es, languages.
• Authentic: Not jailbroken, iOS
and Android devices, rotation.
• Visual test reports.
• Comparison across models
and operating systems.
• Continuous integration.
fredag den 23. marts 12
34. Calabash iOS:
more detail
15
fredag den 23. marts 12
35. Calabash: more detail
• Very easy to get started for iOS developers/QAs.
• Declarative query language for finding views.
• Based on UISpec, but simplified and extended. (New
Implementation, EPL licensed).
• Touch synthesis, supports (multitouch) gestures.
Extensible. (pan, swipe, pinch, your own)
• Full power of Ruby programming language for test
logic
• Interactive, exploratory development experience.
• Can use device accessibility for identifying views.
16
fredag den 23. marts 12
41. Questions?
Making app testing less painful...
Please contact us with any questions:
contact@lesspainful.com
karl@lesspainful.com - iOS
jonas@lesspainful.com - Android
http://www.lesspainful.com
fredag den 23. marts 12