Performance tests are not only an important instrument for understanding a system and its runtime environment. It is also essential in order to check stability and scalability – non-functional requirements that might be decisive for success. But won't my cloud hosting service scale for me as long as I can afford it? Yes, but… It only operates and scales resources. It won't automatically make your system fast, stable and scalable. This talk shows how such and comparable questions can be clarified with performance tests and how DevOps teams benefit from regular test practise.
6. Performance
The ability of a system to execute one task
within a defined dimension
= efficiency of a system
↪ 1 server manages 250 rps with p99 500ms.
10. What makes me recognize that
I have a performance problem?
http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
12. What makes me recognize that
I have a scaling problem?
http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
13. If your system is fast for single users
but slow under load.
22. “In software engineering,
performance testing is in general,
a testing practice performed to
determine how a system performs
in terms of responsiveness and stability
under a particular workload.”
https://en.wikipedia.org/wiki/Software_performance_testing
– Wikipedia
24. Performance Tests
• Collection of
non-functional test methods
• Not to be defined with
absolute selectivity
• Different goals and perspectives
Load Testing
Stress Testing
Spike Testing
Soak Testing
Endurance Testing
Resilience Testing
Configuration Testing
Scalability Testing
25. Performance Tests?
• Not only for testing releases
• Evaluation of…
• technology,
• proof of concepts,
• troubleshooting and debugging
31. Challenges
• organizational obstacles, e.g. buy-in
(management & team)
• testing goal & scope
• visibility in your own systems
• The right performance test tool for the job
– (usually) no tooling problem
• start! 😉 👍😁
33. Steps to Perf Tests™
• bring together the stakeholders + development
🤓 operations
🐜 QA
👷 product
💰 management
😱 marketing
34. Steps to Perf Tests™
Kick off for 15 Min
– All Stakeholders Heads Up
STATE
We’ll NOW create a performance testing culture to make sure that we can
hold our promise to the business/customer to be fast and available under
different amount of usage/users/transactions.
We so match performance and availability requirements to keep the
business save.
ASK
Do you all agree?
STATE
OK, we start with testing a very easy use case to raise a baseline of numbers
about performance and availability. This friday, it’ll take a 2 hrs timebox.
+ development
🤓 operations
🐜 QA
👷 product
💰 management
😱 marketing
35.
36. Steps to Perf Tests™
Test Case
• Create a first, simple Test Case
• Q: Test Data?
• Q: How much traffic?
37. Steps to Perf Tests™
Test Case
• Ask Analytics
for the hour of the highest
number of users in a week.
• Analytics:
We have 10.000 visits in this
hour. The Users visits flower-list,
one flower detail.
30% check out and payment.
38. Steps to Perf Tests™
Test Case
• Test Environment &
Load Generator Setup
• Since Infrastructure is code:
Just start your test environment!
40. Steps to Perf Tests™
Run Test
• 20 Minutes
• Invite ALL stakeholders!
• It’s about collaboration
41. Steps to Perf Tests™
Inspect Report
• Response Time?
• If errors occurred:
Do you see these in your
monitoring, too?
42. Steps to Perf Tests™
Meet and communicate to stakeholders:
• ONE Statement like
We aim for baseline numbers.
We tested flower-list and flower-detail.
We created 4x traffic of peak hour.
We saw 20% of errors.
But always responded in under 1000ms (p99).
• Ask
Are you happy?
• Ask
Whats’ next? Checkout?
+ development
🤓 operations
🐜 QA
👷 product
💰 management
😱 marketing
56. Defining goals
• Defining goals beforehand (why are we doing tests?)
• Important!
Always define goals (and requirements) from the specialist side ☝
• A goal can be very simple at first, e.g. “surviving Black Friday well” or
“we want to become 20% faster”.
57. Where to start with testing?
• defining System Under Test (SUT)
• finding the right scope
• perimeter/end-to-end tests
• components/service tests
58. System Under Test
• testing “from top” (end-to-end) or
“sideward” against single services
• for the compartment side, only
integrative tests are interesting
• testing single services for the
purpose of trouble shooting
(technical perspective)
59. Define requirements
• make assumptions for non-functional requirements (NFR)
• assistance via “baseline” performance tests and performance
budgets
• simple requirements might be: “deliver sites under 2sec”, “10k
checkouts per hour”, …
• also here: start simple, learn, improve, repeat!
• do sanity checks!
61. Testing together
• Especially before the first tests: involve all the stakeholders!
• GO!
• Present setup, assumptions and findings
• Check or rework scenario sanity
• Repeat!
69. Spike Testing
• How does the system react during
(extreme) peak loads?
• Are we reacting fast enough?
• Occasions: Marketing 😈, Reddit,
Hacker News, …, TV shows
70. Soak Testing
• How does the system behave under load during a long period of time?
• (very) long load test
• Are there any longtime effects? Memory leaks?
71. Configuration Testing
• How does the behavior change if the configuration modifies?
• Series of test executions
• Investigate the impact of the environment on your own system
72. …in the cloud
• Types of instances
• Auto scaling settings
• Throughput provisioning
• Ideal usage of services?
73. …and in general…
• Hypervisor
• Operating system
• Web & application server
• Software configuration
• Software dependencies