ЧТО? и КАК? Вы можете протестировать имея ограниченное количество времени?
Это вероятно два самых популярных вопроса в реальной жизни современного QA. Тестирование основанное на рисках один из подходов в тестировании, который позволит вам концентрироваться на том что действительно важно.
Во время доклада, мы с вами обсудим следующее:
Что же такое «риск-бейзд» тестирование на практике,: какие риски есть в тестировании реального проекта? Как их определить? Как их оценить?
Как внедрить «риск-бейзд» подход на вашем проекте?
Как вовлечь столь необходимых вам стейкхолдеров?
А дальше мы посмотрим как строить тестирование на основании рисков, как поддерживать его в последствии и что же делать с новыми рисками которые будут происходить с ходом эволюции вашего продукта
И последний в теме доклада, однако не последний по важности вопрос: как измерять качество продукта в "риск-бейзд" тестировании, какие метрики взять за основу?
Visit to a blind student's school🧑🦯🧑🦯(community medicine)
QA Fest 2017. Ирина Жилинская. Тестирование основанное на рисках в реальности: как с ним работать
1. Киев 2017Risk based testing in reality: How to make it works on your project?
Risk based testing in reality:
How to make it works on your project?
2. Киев 2017
Speaker Info
• 7 years in IT in roles BA & QA:
Financial Services Industry, Sport Industry
• ISTQB Trainer
Risk based testing in reality: How to make it works on your project?
IRYNA ZHYLINSKA
LEAD SOFTWARE TEST ENGINEER
EPAM
3. Киев 2017
Agenda:
is Risk-Based testing approach?
values it brings?
metrics it gives?
WHAT
HOW to implement Risk-based testing
on New Project
on existed (on-going Project)
pros and cons risk based approach launching on your Project
Risk based testing in reality: How to make it works on your project?
4. Киев 2017
WHAT: is Risk-Based testing approach?
Risk based testing in reality: How to make it works on your project?
KEY TERMINOLOGY
POSSIBLE NEGATIVE RISK
+ = Can we test
It?
OUTCOME
PRODUCT
Quality
PROJECT
YES
NO
5. Киев 2017
WHAT: is Risk-Based testing approach?
Risk based testing in reality: How to make it works on your project?
KEY TERMINOLOGY
LIKELYHOOD IMPACT Risk Priority Number
x
=
RBCS link Article: "Risk Based Testing: What It Is and How You Can Benefit" by Rex Black
6. Киев 2017
WHY: Risk Based approach values
… for any system infinite # of tests exists
…so you have to prioritize what is matter to
test
and you have to measure what you test
but you DO NOT have forever to test
Risk based testing in reality: How to make it works on your project?
7. Киев 2017
HOW: implement Risk-Based approach
Risk based testing in reality: How to make it works on your project?
New project On-going project
Just started
No history
Ongoing
> 2-3 years history
Practical Examples:
8. Киев 2017
HOW: Risk-Based testing process looks like?
Risk Identification
Risk Assessment
Risk Management
Risk Mitigation
Risk based testing in reality: How to make it works on your project?
OUTPUT
Quality RISKs List
RPN
Test Prioritization
Correct Effort allocation
Defined Scope
STEP
Execute
Report
Analyze
Allocate new Risks
Monitor and Control
9. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
[-] Poor requirements
[-] Lots of changes
[-] No past experience
[-] Busy Stakeholders
[-] No Trust
Goal: Quality Risks List
New project
Identify + Communicate
Present
Apply
Learn DOMAIN (!)
Google
Brainstorming
Establish standartized KB
Ask RIGHT questions in RIGHT place
Focus on YOUR Team affected Area
Identify + Establish + Communicated
[+] Business availability
[+] No defined Processes
[+] Time
10. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
Goal: Quality Risks List
Ongoing project
Find ‘the RIGHT Guy’
Investigate
Brainstorm and Identify
bottlenecks
Identify Risks
Prepare to present
[-] Low Business availability
[-] Documentation: poor/surplus
[-] Custom process
[+] Trust
[+] Domain expertise
[+] Available past experience
Define Implementation Plan
Present Solution
Prepare to implement
[?] Existed test cases
11. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
Ongoing projectNew project
Define Quality Risks Catalogue
Maintenance Plan for Risks
(Vision)
Risks Reviews
Define Implementation plan
• Collect data from the past
• Trace past data with identified Risks
• Define missed area
• Allocate time and Resourses
READY TO MOVE TO RISK ASSESMENT
Quality Risks Catalogue
12. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
RESULT = Risks Check List
System response > 20 ms under full load
Patches and releases won’t install correctly
Incorrect exchange rate for currency pair GBP:CNY
User failed to log in and authentication failed
User find interface too complex
Performance
Supportability
Functional
Functional
Usability
US-101
TT-10
US-5
US-65
NA
TC-15
TC-25
TC-35
TC-88
NA
yes
no
yes
yes
NA
User
Story
Test
Case
Is
Aut.
Note: configure your tools to pick all this data automatically
Link for Templates: RBCS: Generic Quality Risks List and General Quality Risk Categories
In case on-going project better to pick
up existed information from the begining
13. Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
[-] Not clear requirements
[-] Not fully defined expectation
[-] No past experience
Goal: assign Risk Priority Number (RPN)
New project
[+] Right people available
Provide presentation
Make sure that all understand WHAT Likelihood
and Impact mean
All understand the Measurement Scale
Explain the RULES
Execute Assessment
RIGHT people involved
Appropriate time allocated
Establish Plan how to deal with ‘open questions’
14. Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
Goal: assign Risk Priority Number (RPN)
Ongoing project
[-] Formed Standards of Priority Scale
[+] Available past experience
Analyse Past Project Expirience
Execute Assessment
People involved (!)
Appropriate time allocated
Establish Plan how to deal with ‘open
questions’
Provide presentation
Make sure that all understand WHAT Likelihood
and Impact mean
All understand the Measurement Scale
Explain the RULES
15. Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
RESULT = Risk Check List with RNP assessed
1. System response > 20 ms under full load
(performance)
2. Patches and releases won’t install correctly
(supportability)
3. Incorrect exchange rate for currency pair
GBP:CNY (functional)
4. User failed to log in and authentication failed
(functional)
5. User find interface too complex (usability)
3
4
2
4
4
Likelihood Impact Current
RPN
5
1
1
2
5
15
4
2
8
20
Scale
1 =Critical
2 = Major
3 = Medium
4 = Low
5 = Very Low
Past
RNP
12-14
5
2-3
6
0
Analysed expirience -> form RNP based on
existed data
16. Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
[1-5] Extensive Run ~ 50 % from total test design, implementation end execution effort
a large number of tests that are both broad and deep, exercising combinations and variations of
interesting conditions.
[6-10] Broad Run ~ 25-35 % from total test design implementation end execution effort
a medium number of tests that exercise many different interesting conditions
[11-15] Cursory Run ~ 5-15 % from total test design implementation end execution effort
a small number of tests that sample the most interesting conditions
[16-20] Opportunity ~ 5 % from total test design implementation end execution effort
Leverage other tests or activities to run a test or two of an interesting condition (small investment of time
and effort)
[21-25] Report bugs
MAKE sure that EVERY Stakeholder understand definitions
17. Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
RESULT = Extend of testing assigned
1. System response > 20 ms under full load
(performance)
2. Patches and releases won’t install correctly
(supportability)
3. Incorrect exchange rate for currency pair
GBP:CNY (functional)
4. User failed to log in and authentication failed
(functional)
5. User find interface too complex (usability)
RPN
15
4
2
8
20
Extend of
testing
C
E
E
B
O
TCs
Suite-15
Suite-18
Suite-1
Suite-2
CR-5
US-101
TT-10
US-5
US-65
NA
US
Issues
I-5, I-9
I-15
I-25
I-65
I-8
TRACING
yes
no
yes
yes
yes
Is
Aut.
Extend of testing
1-5 = [E] = Extensive
6-10 = [B] = Broad
11-15 = [C] = Cursory
15-20 = [O] = Opportunity
21-25 = [RP] = Report Bugs
18. Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
Goal: allocate test effort
[-] Mixed coverage
[-] Catalogue Maintenance problems
[-] Miss issues (RB, O)
[+] Keep test FOCUS on WHAT is METTER
[+] Limit test afford with RNP
1 test case covers ONLY 1 entity
Automate Risk Catalog
Establish Team Rules
Allocate time for Experience based testing
New project & Ongoing Project
How to build Report with JIRA: Atlassian Support/JIRA Software Cloud/JIRA Knowledge Base
Atlassian Marketplace: qTest for JIRA Test Management
19. Киев 2017
Risk-Based implementation: Risk Management
Risk based testing in reality: How to make it works on your project?
Goal: analyze, monitor and control
[-] Catalogue Maintenance
[-] Process Maintenance
[+] Time Constraint problem Resolved
(breadth-first and depth-first)
[+] Bug-fix efforts focused on high-risk
problems
[+] Now we known the Scope
Use Retro
Build and Analyze Test Report based on
Risks
Communicate the process
New project & Ongoing Project
20. Киев 2017
Risk-Based implementation: Metrics
Risk based testing in reality: How to makes it works on your project?
Goal: use metrics to improve the process
• % of executed test cases per Risk item
• % of failed/passed/unexecuted test cases per Risk item
New project & Ongoing Project
Test Execution metrics
% defects identified in 1 Risk item
% defect identified/fixed per one Risk item in one
iteration
% defect fount in the 1st – 2nd – 3rd Iterations –
compere
Defect metrics to establish Acceptance ‘Go’ ‘Not-Go’
Defect metrics
21. Киев 2017
POSSIBLE NEGATIVE RISK
+ =
LIKELYHOOD IMPACT RPN
x =
HOW: Risk-Based testing process looks like?
Risk
Identification
Risk
Assessment
Risk
Mitigation
Extend
of
testing
Extensive Broad Cursory
or or
Opportunity
or =
Risk
Management
Execute Assign New RisksFix Report
+ + +
22. Киев 2017
WHAT: values Risk Based approach brings?
Risk based testing in reality: How to make it works on your project?
team knows the focus and domain
“find the scary stuff first”
“pick the right tests out of the infinite cloud of possible tests”
“release when risk of delay balances risk of dissatisfaction”
“give up tests you worry about the least”
All these BENEFITS allows:
the test team to operate more efficiently and in a targeted fashion, especially in time-
constrained and/or resource-constrained situations
23. Киев 2017Risk based testing in reality: How to make it works on your project?
ira.zhylinska@gmail.com
Editor's Notes
Is Risk-Based testing approach?
Values it brings to both Project and Client
Possibility (Вероятность возникновение)
Степень негативности
Вероятность:
То что являет фактом – точно не риск это ограничении
what is impossible is not a риск и то что достоверно это ограничение
Ну например мне не нужно волноваться что я сейчас поднимусь и ударюсь головой о потолок – тк гравитация работает
или невозмножно не является риском– ( например гравитация она работает) не нужно волноваться совсем
Possibility (Вероятность возникновение)
Степень негативности
Вероятность:
То что являет фактом – точно не риск это ограничении
what is impossible is not a риск и то что достоверно это ограничение
Ну например мне не нужно волноваться что я сейчас поднимусь и ударюсь головой о потолок – тк гравитация работает
или невозмножно не является риском– ( например гравитация она работает) не нужно волноваться совсем
In risk-based testing, the results of Risk analysis will form key inputs into the test planning process as well as the test estimation process.
In risk-based testing, the results of Risk analysis will form key inputs into the test planning process as well as the test estimation process.
[+] Business availability
[+] No defined Processes значит что мы можем его настроить так как удобно нам
[+] Time – ВРЕМЯ вначале проекта у тестировщика есть ВРЕМЯ на то чтобы заниматься риск бейзд тестированием
Identify + Communicate
Present риск бейзд стратегию
Combine – комбинируейте подходы, покуда процесс еще не установлен его можно рисовать так как прваильно и необходимо
Learn - важно знание домена!!!
Если домен сложный – тогда вовлечь команду – раздробить скоуп на куски и презентовать друг другу кусками
Google – вся аналитика статистика все что может помочь при определении того как систему пользуют и что хотят от нее
Кто юзера например? Основные фейлы предыдущих систем,
Brainstorm
Ask RIGHT questions in RIGHT place
Внедриться в тылы БА для того чтобы учавствовать в груминге, если нет – получить доступ к сторям как можно раньше
+ сделать Жиру таким образом чтобы к сторям линковались десижены их договорится что все вопросы в десиженах БА отвечает
+ пока идет выяснение требований вики страничку где все будет документироваться именно под и для куа и риск бейзд
Find the RIGHT Guy человека/людей у которыъ есть доменные знания
Привести пример Репортинговой апп (без доков и срочным релизом)
Инвестигируйте проект (прежде чем менять нужно знать «где мы есть», «какие апроучи используем» «проблематики»)
Подготовьте решение для презентации и подготовьте план имплементации (пример Инвестмент проекта): стадия обсервации, подготовки, презентации и план итеративного внедрения Риск бейзд)
и важно в отличии от например проекта нового мы презентуем решение уже решение, которое решает проблемы (ботелнеки)
Collect data from the past
Trace past data with identified Risks
Проанализировать предыдущие кейсы + автомейшен и намапить все на текущий чек лист
Collect data from the past
Trace past data with identified Risks
Проанализировать предыдущие кейсы + автомейшен и намапить все на текущий чек лист
Collect data from the past
Trace past data with identified Risks
Проанализировать предыдущие кейсы + автомейшен и намапить все на текущий чек лист
Особое внимание на 2е вещи
Analyse Past Expirience -> form RNP based on past data
Два момента
1- хорошо включать в риск ассесмент
2- но лучше оставить для корректировки чтобы не сбивать с толку команду
extensive testing effort takes
broad testing effort takes ~ 25-35 % from total test design implementation end execution
Cursory takes ~ 5-15 % from total test design implementation end execution
takes ~ 5 % from total test design implementation end execution
[-] Mixed User Story coverage
(1 Test Case covers few functional entity)
What if 1 step failed?
1 test case covers ONLY 1 entity
Automate Risk Catalog as much “automated” as possible
Establish Team Rules to support Correct Reporting
Allocate time for Experience based testing
[-] Mixed User Story coverage
(1 Test Case covers few functional entity)
What if 1 step failed?
1 test case covers ONLY 1 entity
Automate Risk Catalog as much “automated” as possible
Establish Team Rules to support Correct Reporting
Allocate time for Experience based testing
[-] Inappropriate Maintenance = забыли промаркировать = потеряшки
Communicate the process всегда напоминайте и проверяйте
Is Risk-Based testing approach?
Values it brings to both Project and Client
Possibility – what is impossible is not a ris
То что достоверно это ограничение
или невозмножно не является риском– ( например гравитация она работает) не нужно волноваться совсем