This document discusses integrating Ranorex automated functional testing with NeoLoad load testing. It begins with introducing the companies and tools, then discusses how performance impacts user experience. Automated functional testing benefits like speed and coverage are reviewed. While functional and load testing are typically separate, combining them allows validating critical workflows under real load and finding issues earlier. A workflow is proposed where automated tests and exploratory testing are augmented with load testing. A live demonstration then follows.
2. Welcome to the Ranorex-Neotys Webinar
Christian Breitwieser, Ranorex
Innovation Manager
Henrik Rexed, Neotys
Technical Evangelist
3. Agenda
• Quick Company and Tool Intro
• Performance Matters in User Experience
• Benefits of Automated Functional Testing
• Benefits of Combining Automated Functional and Load Testing
• Live Demonstration
4. • Functional test automation for desktop, web and mobile
applications
• Cross-technology end-to-end testing
• Tool for testers and developers
21. Continuous Performance Validation Solution
Pre-production
System-wide Testing
Production
Proactive Monitoring
Development
Continuous
Integration &
Component Testing
22. Continuous Performance Validation Solution
Pre-production
System-wide Testing
Production
Proactive Monitoring
Development
Continuous
Integration &
Component Testing
23. Continuous Performance Validation Solution
Pre-production
System-wide Testing
Production
Proactive Monitoring
Development
Continuous
Integration &
Component Testing
24. Agenda
• Quick Company and Tool Intro
• Performance Matters in User Experience
• Benefits of Automated Functional Testing
• Benefits of Combining Automated Functional and Load Testing
• Live Demonstration
25. User Experience is Crucial
• 57% of users use their mobile devices to access applications
• 40% of users move towards a competitor after a bad user
experience
• 44-61% of users share their feelings on social networks
30. Agenda
• Quick Company and Tool Intro
• Performance Matters in User Experience
• Benefits of Automated Functional Testing
• Benefits of Combining Automated Functional and Load Testing
• Live Demonstration
34. Manual & Integration Testing
Manual testing
• Time-consuming
• Error-prone
• Cannot „run“ overnight
Integration testing
• No black box
• Hard to cross technology
borders
• Good coding skill required
35. Benefits of Automated Functional Testing
• Quick feedback
• Parallel test execution
• Scalable
• Codeless functional testing
• Increased test coverage
36. Numbers & Facts
TomTom – Case Study:
• Testing time: from 10 days down to 4h
• Over 90% of time saved with Ranorex automated tests
Siemens – Case Study:
• Due to the use of C#, new test cases can be implemented more
effectively
• Only 60% of the time needed with Ranorex compared to previously
used tool
37. Agenda
• Quick Company and Tool Intro
• Performance Matters in User Experience
• Benefits of Automated Functional Testing
• Benefits of Combining Automated Functional and Load Testing
• Live Demonstration
38. Separated Functional & Performance Testing
• Functional testing ensures functionality
… but not when the AUT is under load
• Load/Performance testing ensures a fluent user experience
… but does not guarantee a functional UI
39. Ranorex Customer Experience Survey
No Idea
Yes No
54% 37%
9%
Do you combine Ranorex test automation
with non-functional tests?
In addition to functional test automation
you perform…
4%
19%
22%
27%
31%
33%
Other
Load Testing
Database Testing
Web-Service-Testing
Performance Testing
Unit Testing
40. Functional – Performance Tests
Golden Rule: Automated performance testing should not take place if functional regression tests failed.
Performance tests require that the application or part of the
system is:
• Operational: No systematic errors or exceptions
• Representative : Enough data available in the test
environment to determine the behavior under load conditions
41. How Can We Increase Risk Coverage?
100
test
cases
40 test cases
covering 80% of the risk
not necessarily selected for performance testing
We prioritize test cases based on risk. These test cases are validated through :
- Automated regression testing
- Exploratory testing
42. Validate Critical Business Case Under Load
This guarantees:
• The stability of the crucial use cases under real usage conditions
• High user experience
• The identification of potential bottle-necks, even across
technology borders
46. Agenda
• Quick Company and Tool Intro
• Performance Matters in User Experience
• Benefits of Automated Functional Testing
• Benefits of Combining Automated Functional and Load Testing
• Live Demonstration
47. Further Resources
Combining Automated Functional and Load Testing
http://www.ranorex.com/blog/combining-functional-and-load-testing
How to Combine Ranorex and NeoLoad Tests
http://www.ranorex.com/blog/combine-ranorex-and-neoload-tests
48. Questions?
Henrik Rexed: henrik.rexed@neotys.com
Download NeoLoad Free Edition
Visit our neotys.com
Follow me on Twitter @hrexed
Christian Breitwieser: cbreitwieser@ranorex.com
Download free Ranorex trial
Visit www.ranorex.com
Follow us on Twitter @Ranorex
Editor's Notes
Our mission: help organizations deliver FASTER applications FASTER through continuous performance validation
{click}
Offices around the globe {click}
Over 1600 customers
Nearly every industry
Big name organizations trust Neotys
[Transition]
Choose Neotys because of focus on key performance challenges they face…
{click}
Our mission: help organizations deliver FASTER applications FASTER through continuous performance validation
{click}
Offices around the globe {click}
Over 1600 customers
Nearly every industry
Big name organizations trust Neotys
[Transition]
Choose Neotys because of focus on key performance challenges they face…
{click}
Our mission: help organizations deliver FASTER applications FASTER through continuous performance validation
{click}
Offices around the globe {click}
Over 1600 customers
Nearly every industry
Big name organizations trust Neotys
[Transition]
Choose Neotys because of focus on key performance challenges they face…
{click}
3 categories of challenges:
User Demands
Application Development Process Revolutions
Technology Transitions
{click}
[User Demands]
High expectations for performance
As fast as google search
Mobile as fast as desktop
The results of bad performance:
Lost revenue
Lose users to competitor apps
Brand damage
Lost productivity for internal apps
FASTER APPLICATIONS
{click}
[Process Revolutions]
Agile
DevOps
Continuous Integration and Continuous Delivery
Force testing to become more automated
QA being squeezed out
QA needs to keep up with development
DELIVER FASTER
{click}
[Technology Transitions]
More and more interfaces:
Web
Mobile
APIs
IoT
New technologies:
WebSocket
Docker
HTML5
HTTP/2
Need to test whatever technologies developers send your way
[Transition]
And this is why we developed our Continuous Performance Validation solution…
{click}
3 categories of challenges:
User Demands
Application Development Process Revolutions
Technology Transitions
{click}
[User Demands]
High expectations for performance
As fast as google search
Mobile as fast as desktop
The results of bad performance:
Lost revenue
Lose users to competitor apps
Brand damage
Lost productivity for internal apps
FASTER APPLICATIONS
{click}
[Process Revolutions]
Agile
DevOps
Continuous Integration and Continuous Delivery
Force testing to become more automated
QA being squeezed out
QA needs to keep up with development
DELIVER FASTER
{click}
[Technology Transitions]
More and more interfaces:
Web
Mobile
APIs
IoT
New technologies:
WebSocket
Docker
HTML5
HTTP/2
Need to test whatever technologies developers send your way
[Transition]
And this is why we developed our Continuous Performance Validation solution…
{click}
3 categories of challenges:
User Demands
Application Development Process Revolutions
Technology Transitions
{click}
[User Demands]
High expectations for performance
As fast as google search
Mobile as fast as desktop
The results of bad performance:
Lost revenue
Lose users to competitor apps
Brand damage
Lost productivity for internal apps
FASTER APPLICATIONS
{click}
[Process Revolutions]
Agile
DevOps
Continuous Integration and Continuous Delivery
Force testing to become more automated
QA being squeezed out
QA needs to keep up with development
DELIVER FASTER
{click}
[Technology Transitions]
More and more interfaces:
Web
Mobile
APIs
IoT
New technologies:
WebSocket
Docker
HTML5
HTTP/2
Need to test whatever technologies developers send your way
[Transition]
And this is why we developed our Continuous Performance Validation solution…
{click}
3 categories of challenges:
User Demands
Application Development Process Revolutions
Technology Transitions
{click}
[User Demands]
High expectations for performance
As fast as google search
Mobile as fast as desktop
The results of bad performance:
Lost revenue
Lose users to competitor apps
Brand damage
Lost productivity for internal apps
FASTER APPLICATIONS
{click}
[Process Revolutions]
Agile
DevOps
Continuous Integration and Continuous Delivery
Force testing to become more automated
QA being squeezed out
QA needs to keep up with development
DELIVER FASTER
{click}
[Technology Transitions]
More and more interfaces:
Web
Mobile
APIs
IoT
New technologies:
WebSocket
Docker
HTML5
HTTP/2
Need to test whatever technologies developers send your way
[Transition]
And this is why we developed our Continuous Performance Validation solution…
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
2main products:
NeoLoad for Load and Performance Testing
NeoSense for 24x7 Performance Monitoring
{click}
Tightly integrated to deliver a Continuous Performance Validation Solution
{click}
[Development]
Validate the performance of individual components
API or web services level
{click}
NeoLoad integrates with Continuous Integration Servers like Jenkins and Bamboo
Automated performance tests on every build against SLAs
Regression Testing for performance
[Pre-production]
Test designs from development can be passed on to pre-production
Build more realistic business use cases
In pre-production, system-wide load testing
Complex user paths
Mobile device and web browser simulation
Network Virtualization
High volume cloud load generation
{click}
Integrate with functional testing tools to capture end user experience under load
APM tools for deep dive diagnostics
{click}
Results passed back to Dev for fixes
{click}
[Production]
Same user used by NeoSense in production to proactively monitor
SLA = email and SMS notifications go out immediately
{click}
Send notifications to 3rd party dashboards and IT search tools
Integrate with APM tools for RUM and Code-level diagnostics
{click}
dashboards and reports can be shared with Dev and pre-prod to fix issues quickly
{click}
Automated functional testing:
Exploratory testing: Identify issues not covered by automation
Load testing: Validate the stability/availability of the application under load
Exploratory testing under load: Validate crucial business cases and user experience under load
Automated functional testing:
Automated functional testing under load: Validate the stability/availability of the application under load
Exploratory testing: Identify issues not covered by test automation
Exploratory testing under load: Validate crucial business case and the user experience under load