Sharing my thoughts and cases about co-work with test and developemnt. Two big approaches.
One is Engineering approach (
1. Early testing education
2. Test design
3. Test code guide
4. Pair-testing, programming
5. Test-Automation),
Second is Strategic activities (
1. Test Strategy/Plan
2. Test analysis/report)
Also, I wanted to mention tester's various career paths.
Thank you.
Optimizing AI for immediate response in Smart CCTV
ย
When develpment met test(shift left testing)
1. When Dev met Test
(The future of software testing)
2016.01 by JungGun
home: genycho.blog.me
2. role : specialist
actor : domain expert
role : generalist
actor : test specialist
Developer
Tester
When dev met test
2015 remade in Software testing
3. This is aboutโฆ
Message 1: Co-Work (3Amigos)
Message 2: Shift-left-testing (early testing activities)
Several cases that test people co-worked with
development team closely
Not only about testing, but also
about dev quality
5. Target projects
Small, Research&Dev project (Trendy technology related)
Video, Data analystics, Network&Security, IoT based projects
Current quality level was low, but the impact was huge
Small(~12people)
Internal
funding
Flexible
requirements
Complex,
Trendy tech
Young,
Highly educated
Human resource
Testability๊ฐ
๋จ์ด์ง๋
Low quality concept
6. Test Operation organization
Flexible test resource managing (Shared Service) for small, multiple projects
Each person has one or two specialized domain area
Big Data VA, IoT Security, Network
Domain
study
Test
Strategy
Test
Education
Test
Design
Test
Scripting
Testware dev,
Automation
Test
Execution
Quality
Review
0.25 0.25 0.25 0.25 0.5 0.25 0.5 0.25
Project A Project B Project 1 Project 2 a project
TM
TE
TM
TE
TM
TEโป Shared Service
โป Practices & effort
7. Projects information (2014yr~2015yr)
Mobile authentication Security FIDO - external requests
Admin portal(re-usable
asset)
Web application Admin portal 6 external requests
projects technology Target domain
Dev size
(people)
challenges, difficulty, etc
MES bigdata analytics
Data
Analytics
Manufacture
Factory
5
Verifying statistics functions result and
the integration with big data platform
Crypt/DeCrypt algorithm
Security
(en/decryption)
Encryption/decrypti
on
3
Complex security algorithm,
Integration with Legacy system
Logistics demand
prediction engine
Data
Analytics
Logistics 4
complex prediction algorithm based of
rules
Integration with legacy system
IoT based smart
factory(Smart OHT)
IoT
Semiconductor
manufacturing
6
Understanding IoT(hardware+software)
technology and its Testability
political issues with customers
8. I. Introduction
II. Engineering activities
1. Early testing education
2. Test design
3. Test code guide
4. Pair-testing, programming
5. Test-Automation
III. Strategic activities
IV. Other activities
V. Conclusion
9. Customized quality education
To prevent defects, project-customized education in the early phase
( An example about outline )
- what test activities will be done and why
- Frequently occurred defects
based on similar doman, technology
- How to write (User) test scenario &
the basic of test design
- Intro about sample unit test code
with very early dev code
- Test Automation introduction with Jenkins
10. > Education: sharing the test plan
- what test activities will be done and why
- the whole test cycle, roles&responsibility
- be able to get feedback by sharing with dev
11. > Education: Test Design guide
- the difference with Unit & Integration testing
- the template of test scenario and guide
- basic test design technique
- practical guide with other input documents
12. > Education: Sharing old defects, cases
- defects and test cases based on the productโs domain, technology
[ defects(Web) ] [ Quality guide for API ]
API Development
Standards on quality
- Logging
- Exception handling &
error code
- API Spec guild
- Unit test checklists
13. I. Introduction
II. Engineering activities
1. Early testing education
2. Test design
3. Test code guide
4. Pair-testing, programming
5. Test-Automation
III. Strategic activities
IV. Other activities
V. Conclusion
14. What Test Design
- The thinking process to test effectively and efficiently.
- because the projects were mainly a kind of pure algorithm libray, they were so
complex and needed to design test cases well.
Black-box approach
- initial test approach was black-box which design tests with specification
- various documents โ requirements, algorithm description, Architecture doc etc
- because they were so vague or insufficient to understand the algorithm, we focused
on visualizing the logic to share the understanding with customers
White-box approach
- the real code was the best documents
- made the domain-specific dev-code checklist
- do offline reviews with dev code or unit-testing code (regular pair-programming)
โป Grey-box testing?
- basically do blackbox-testing while checking dev code
15. โป Wait!, Who is the best person do test design?
- Test Desing should be done by Testers!!!
- They can do anyhow, but do they know what custmer really wants?
Business
Analyst
Customer
Tester
โANALYSTS!!!โ
(pros): knows well the businees and dev both.
Need to be reviewed what they analyzed
(cons) : not know about testing technique
โTESTERS!!!โ
(pros) : test experts, experienced in many domain, focused on testing
(cons) : need to understand requirements fully
โCUSTOMERS!!!โ
(pros): best know about business
(cons): no time, no tes or IT
knowledge
Who shuld do
test design?
The answer depends on โTest Levelโ or โtypeโ,
but basically everyone should work together.
16. Step 1) Understanding with documents, interviews, everything as possible
Step 2) Sharing what understood and getting feedback
(Example) Analyzing old test scenarios
find questions to confirm
And test conditons
Make checklist, diagrams to share with dev
team
17. Step 1) Understanding with things โ documents, interviews
Step 2) Sharing what understood and getting feedback
(Customer)
Easy to understand
with diagrams
(Developers)
With the checklists, I was
able to self-define the detail
parts
(Product Manager)
Visual diagram can help
potential stakeholders to
understand our product
I can give the team
more detailed
requirements
18. >Test Design: Test Coverage reviews
- testers review test coverage (line&branch coverage) on real code
- if needed, they add more test cases or ask developers to do
TC add)
need to test server and client different
version check test
Coverage check)
The code is not reachable(double checking)
Need to delete
19. โป (Example) Graphical approach to review
If the documents are insufficient, dev code is the best documents.
First, draw any diagrams if it can be helpful, then review the logic, workflow with
various stakeholders
์์ค ์ฝ๋
๊ตฌ์กฐ-Process ์ฒ๋ฆฌ์ฌ๋ถ
๊ตฌ์กฐ-Log ์ฒ๋ฆฌ
๊ตฌ์กฐ-Exception ์ฒ๋ฆฌ
์ ๋ ฅ ํ๋ผ๋ฏธํฐ ์ฒดํฌ
input table ์๋ ๊ฒฝ์ฐ ์ฒ๋ฆฌ
output table ์๋ ๊ฒฝ์ฐ ์ฒ๋ฆฌ
ํด๋น ์ปฌ๋ผ๋ช ์ด ์๋ ๊ฒฝ์ฐ ์ฒ๋ฆฌ
๋ก์ง ์์ if ๋ฌธ ์กฐ๊ฑด ๋์ด
๊ณ์ฐ์์์ ์ ๋ ฅ๊ฐ ์ง์ ๋ฐ๋ผ ๋ถ์ or ๋ถ๋ชจ๊ฐ 0์ด ๋๋
๊ฒฝ์ฐ
์ฝ๋ ๋ด์ ํ ๋ณํ(๋ฌธ์ -> ์ซ์, ์ซ์-> ๋ฌธ์, Double-
>int ๋ฑ)
๋ก์ง ์ํ ์ค ์ค๋ฅ ๋ฐ์ ์ ์๋ฌ ํ ์ด๋ธ์ ๋ฃ์ด์ฃผ๊ณ ์๋์ง
Ex 1) the sequence of dev code 2) Code review checklist
(a) main structure
(b) logging
(c) exception handling
โฆ
(f) if conditions
โฆ
(j) calculation
(k) type casting
20. I. Introduction
II. Engineering activities
1. Early testing education
2. Test design
3. Test code guide
4. Pair-testing, programming
5. Test-Automation
III. Strategic activities
IV. Other activities
V. Conclusion
21. Customized Test Code guide
- To minimize the learning curve to write unit-test code
. language - Java, C/C++, JavaScript, Scala(Spark)
. Specific framwork based test code(Spring fr, MapReduce, Spark, Kuraโฆ)
Target Guide Type
Complex pure
algorithm
Various Test conditons and
cases
junit
C language
program
Tool research, Gtest pilot GoogleTest
Big Data
Data-driven testing using
Junit, MRUnit
MRUnit(MapRedu
ce),
ScalaTest(Spark)
Complex
prediction
algorithm
Unit/Intg test code with Junit,
CascadingUnit,
Various test cases
Cascading
IoT based
software unit-testing mocking
Hardware parts
Eclipse Kura
Framework
Web application
various test cases on various
layers
Spring
Framework
Target to
Test
(TC1) basic workflow
[ Basic Test Approach ]
(TC2) code branched
workflow
(TC3) specific business
workflow
(TC4) exception
workflow
โฆ
[ Projects ]
22. โป (Example) Structured Test code
- To easy-maintain, data-driven testing
- Naming, Comments guide
Test Code Structure
FaroServiceTestCase
- ํ ์คํธ ์ํ ์ ์คํ๋งํ๋ ์์ํฌ
์ ์ค์ ํ์ผ ๋ก๋ฉ
- ๋งค ํ ์คํธ ํ ํ ์คํธ ์ด์ ์ํ๋ก
DB ์ด๊ธฐํ
- ํน์ ์ฟผ๋ฆฌ๋ฌธ์ ์ง์ ์ํ ๊ฐ๋ฅํ
์ธํฐํ์ด์ค ์ ๊ณต ๋ฑ
SampleServiceTest
@Autowired๋ก ๊ฐ์ฒด ์ฃผ์ ํ
- ๊ธฐ๋ณธ์ ์ธ ํ ์คํธ
- ๋ค๋ฅธ ํ๋ฆ ํ ์คํธ
- ์๋ชป๋ ํค ๊ฐ ๋ฑ์ ํ ์คํธ
- ํ์์ ๋ ฅ ๋๋ฝ ๋ฑ์ ํ ์คํธ
โฆ
SampleTestData
- ํ ์คํธ์ ์ฌ์ฉ๋๋ ๋ฐ์ดํฐ๋ฅผ
๋ถ๋ฆฌํ์ฌ ๋ค๋ฅธ ํ ์คํธ์์๋
์ฐธ์กฐํ ์ ์๋๋ก ์ ์
(์ถ๊ฐ๋ก ์์ ๋๋ DB์์
๊ฐ์ ธ์ฌ ์ ์๊ฒ ๋ณด์ ๊ฐ๋ฅ)
CodeMngService
- ์ฝ๋ ์ ์ฒด ์กฐํ
- ์ฝ๋ ๊ฒ์
- ์ฝ๋ ๋ฑ๋ก, ์์ , ์ญ์
โฆ
AbsractTestData
- ํ ์คํธ ๋ฐ์ดํฐ๋ฅผ DB, ์์ ๋ฑ์ผ๋ก
๋ถํฐ ๊ด๋ฆฌํ ์ ์๋๋ก ์ง์ํ๋ ์ถ
์ ํด๋์ค
Spring configration
loading, common variables,
utils are in a super
Parent class
If the configuration(file) is
changed, only one class needs
to be changed
If the data is changed,
Only one is to change
Data that can be
Referred in other tests
are stored in a separate
class
23. I. Introduction
II. Engineering activities
1. Early testing education
2. Test design
3. Test code guide
4. Pair-testing, programming
5. Test-Automation
III. Strategic activities
IV. Other activities
V. Conclusion
24. Pair-Testing
- not fix bugs, but fix human mistakes
- developer+tester pair in a sprint testing
- developers learn what, how testers think and test
testers learn what the code intended and expected
- (Expected)
better communication
reduced bugs (defect prevention) by learning
testers can find more serious bugs
the productโs quality is better
(Not-Expected)
Developers and testers do not trust each other
complaint that pair-testing is useless and time-consuming job
disappointment about testers, testing
26. โป Test Code Pair-programming
- developers can learn how to write test code in the specific cases.
testers can know better about dev code
- Testers can talk about quality aspects in dev code level
Well, at first, letโs write the most basic case which
We can debug what we write.
after that, everytime you need to check your code or
Debug, add more tests cover those cases
In someway, we can check the test coverage, it will
help us if we should add more tests or not
Should I write test code for this part? That is
too much and difficult to synchโฆ
27. I. Introduction
II. Engineering activities
1. Early testing education
2. Test design
3. Test code guide
4. Pair-testing, programming
5. Test-Automation
III. Strategic activities
IV. Other activities
V. Conclusion
28. Test Automation plan
- project customized Test automation plan should be established
RDMBS
DAO implementations
DAO interfaces
Service implementations
Service interfaces
Other remote
interfaces
Data Access
Layer
Service
Layer
Presentation
Layer
Web UI Test
Controller
REST API
UI
55 โ
Controller Test
Service Test
DAO Test
RESTful API Test
โ
11
22
33
44 โ
Test
Level
Target Tools
Unit
Test
(Sprint)
DAO Junit
Service Junit
REST API SoapUI
Web UI Pair-testing
Sprint
Integration
Test
Internal
Workflow
Selenium
(+Sikuli)
Integration
External/Legacy
Selenium
/ Manual
Integration
Test
Internal/External
Workflow
Manual
User
Test
Internal/External
workflow
Manual
[Example: Test automation strategy ]
29. Test Automation Infra in early phase
Customer,
Other stakeholders
Product
Manager
Dev team
Shared & visualized
Test results Okay.
We are
coding&testing
My tests are ok.
Junit test reports
Test coverages
(Method,Line,Bran
ce, etc)
Code Inspection
(PMD,โฆ)
Jekins
(+SonarQube?)
[Example: Test automation ]
30. I. Introduction
II. Engineering activities
III. Strategic activities
1. Test Strategy/Plan
2. Test analysis/report
3. Other Professional services
IV. Other activities
V. Conclusion
31. Test Strategy
(Example 1) Client-Server application
- define โTest Levelsโ like (1)Unit testing, (2)Integration1(Client, Server each),
(3)Integration2(Client-Server) and each levelโs purpose, target, role, method
User Env
Unit
Testing
Intg
Testing 1
Intg
Testing 2
Acceptance
Testing
๊ฐ ๋ฉ์๋(๋๋ ๋จ์ ๊ธฐ๋ฅ)์ ๋ํด
๋ค์ํ ์์ธ ์ํฉ์ ๋ง๋ค์ด ๋ฉด๋ฐํ ํ ์คํธ
์๋ฒ, ํด๋ผ์ด์ธํธ ๊ฐ ์์ญ ์์์
์ฌ๋ฌ ๋จ์๊ธฐ๋ฅ๋ค์ ๋ฌถ์์ ๋ํด ๊ธฐ๋ฅ ํ๋ฆ์
๋ช ๊ฐ์ง๋ก ์ ์ํ๊ณ ํ ์คํธ
์๋ฒ - ํด๋ผ์ด์ธํธ ๊ฐ์ ์ํธ ์ฐ๋ ๊ด์ ์์
๊ธฐ๋ฅ ํ๋ฆ์ ๋ช ๊ฐ์ง๋ก ์ ์ํ๊ณ ํ ์คํธ
XXXeaXXXea
XeaXea
XXeaXXea
????
์ฌ์ฉ์ ๊ด์ (์ฌ์ ๋ถ)์์ ์ ํ์ ์ด๋ป๊ฒ
์ฌ์ฉํ ์ง ํ๋ฆ์ ์ ์ํ๊ณ ํ ์คํธ
Client Server
Mod A
Mod B
Mod C
Mod D
Mod E
Mod F
Unnit Test
Acceptance Testing
Intg 1
Intg 1
Intg 2
32. (Example 2) Cascading Framework considered Unit/Intg Testing
- define Unit / Integration testing cosidering targetโs architecture and functional roles
Unit Test
: targets one Pipe mainly
Integration Test
: targets workflows from Source to
Sink(or failed end)
33. (Example 3) Hardware-Software integratied product
- define various test approach in various dev phase
: Software level Unit/intg testing, Hardware-Software integrated testing, real
environment testing etc
Level Software testing On board testing Simulated Env Real Env
Description
Software Unit/Intg testing for
Java programs
Application interface testing.
Hardware, software integration
checking
Simulated, down-scaled
environment testing
Real Envronment
Methods
Junit With test scenarios With business scenarios With business scenarios
In charge
Dev team(testers) Dev team(testers) Dev team, Customers Customers/Dev team
์ดํ๋ฆฌ์ผ์ด์
์ ๋ณด
ํ์ผ
์ผ์,
ํ๋์จ์ด
34. I. Introduction
II. Engineering activities
III. Strategic activities
1. Test Strategy/Plan
2. Test analysis/report
3. Other Professional services
IV. Other activities
V. Conclusion
35. Test Result Report & Assets
Dev/Quality issues based on test result refine best practices, documents, guides
36. I. Introduction
II. Engineering activities
III. Strategic activities
1. Test Strategy/Plan
2. Test analysis/report
3. Other Professional aids
IV. Other activities
V. Conclusion
37. [ Refinement about Test requirements ]
- Functional, Non-Functional requirements
- Test Plan for Non-Functional testing
[ Customized test strategy ]
- Architecture level test plan
- Communications with several group(dev, customer,
etc)
[ Items ]
- Purpose, Levels
- Input/Output, Entry/Exit Criteria
- Test env
- Roles and Responsibility
- Schedule
Test related documents(Plan, Scenario, etc) template, guide and drafts
ํ ์คํธํ ์คํธํ ์คํธํ ์คํธ ๊ณํ์๊ณํ์๊ณํ์๊ณํ์
Standized documents template and guide
39. I. Introduction
II. Engineering activities
III. Strategic activities
IV. Other activities
1. 3 Amigos
2. Code review
3. Code Inspection(tool)
4. Standard Process advisor
V. Conclusion
40. Testing was a communication channel during the develoment
- readable, visible approach as possible to show and get feedback to customers
Customer
Ahha,
Graphical test case documents Complex workflow to real world
41. Publish readable TestCase Spec documents with TestCode comments
- in-house tool which parse testcode comments and publish test specification on web
โฆ
Customer
Readable, Real-time,
Synchronized Test
Specification
@author
@Desc
@Step
@Input
@Expected
@Name
@TC_ID
GenerateGenerate
Writing comments
Other developer Dev lead,
tester
42. I. Introduction
II. Engineering activities
III. Strategic activities
IV. Other activities
1. 3 Amigos
2. Code review
3. Code Inspection(tool)
4. Standard Process guide
V. Conclusion
43. Dev, Test code review (Online)
- Formal process for devcode review is done, testers mainly review testcode
- reviews testcaseโs purpose, assertion and also work with test-coverage by checking
which line or branch covered or not
- if necessary, devcode review perfomed(often needed)
Test code review example
- The test method name should be
test{target}_{prupose}_seq.
- Each test method should test one purpose.
- Copy&paste โthrows Exceptionโ to each test method.
We donโt need to exception handling (try,catch) in test
code
- If you want to test the specific Exception occur, just
type the exact exception name not general โExceptionโ.
And use โfailโ statement just after the line you expect for
code readers
- External resource(File..) should be referenced in relative
path not fixed path in your pc
Dev Code review example
- Not covered toString method in test code.
One more thin, Eh,.. Itโs better to return at least empty
String like โโ rather than nullโฆfor toString() methods. It
can occur NullPointerException if somenone uses and in
some case.
- This checking code is not covered in testing.
Well, you are double checking, so it looks impossible to
reach that code. Need to refactor.
- This several catch clauses are not reachableโฆ
In my opinion, you can replace higher exception class than
lower ones
- This method will return null, if somethingโs wrong, but
the callers are not prepared. It needs to be considered
44. Rules and Occurrence
Code Inspection tool drive from the beginning
- Code Inspection tool(PMD) setting in the beginning of the project
- Detailed code guide for Top 10 violations regularly
Top 10, Rule explanation and solution, alternatives
45. PMD, security, performance,
Official process, etc
Quality/Test
Shared Service
Dev Team
Shared SA
Fortify, white hacking
~~
Shared TA
PMDPMD
SecuritySecurity
PerformancePerformance
PMD Rule customizing
meeting
Security testing
meeting
Performance test
meeting
. How to measure web client response time(any tools)?
. How to test JavaScript?
. What is the official process to finish the project?
HelpDesk for other quality services
- As a quality member, introduce what to do, how to do in quality
- explain similar project cases and its pros/cons
46. I. Introduction
II. Engineering activities
III. Strategic activities
IV. Other activities
V. Conclusion
1. retrospectives
2. Final message
3. Career path in testing
47. Retrospective 1
person Good Bad
C With PMD, I could learn a lot about coding.
I could have experienced TDD
I failed TDD..
D Anyway finished, itโs good!!!
With test cases co-writing, I could learn my
code myself.
Too short deadline
Documents synchrozing
E Test code, PMD helped my coding skill Too short time schedule
G At first, I didnโt believe testers cannot be
helpful, but now we together worked for the
productโs quality. That is important
Tight time scheduling
Test Scenario planning was so good, it should
be happened in very early phase.
H
(PM)
Well, I have never thought co-working with testers in dev phase. Next time, this approach will
be basic.
โI thought they would do nothing because they knew nothing about algorithm,
but what they did was that helping me understanding my product wellโ
48. Retrospective 2
[ Pair-testing ]
(Good) Lots of bugs couldโve been found. Pair-testing in Web UI was much more effective than
test code programming. It was good to hear other projects, other developers cases
(Bad) 30 or 40min is too short, I want to do more.
[ Test Scenario(cases) co-review ]
(Good) Well, I learned how you approach. It was much easier to think how to test.
(Bad) We changed requirements and test cases, that contents are not shred.
[ testcode pair-programming ]
(Good) It was good to show how Junit test code is written in detail. It was so easy after checking what you made.
(Bad) I needed to be prepared. I didnโt know much about Junit itself
[ Other words? ]
(Good) Time-boxing for every activity (30min, 40min) was really good.
(Bad) Well, I think there should be sessions that we talk what we should develop and test together.
Only with us, developers, we say similar things with similar views. Letโs work together.
50. Testerโs Job shifting
DEV
TEST
CUST
OMER
Test doesnโt know
how much we are
Burdened (crying.)
Dev doesnโt know what
I want(Crying..)
No documents.
Everyoneโs busy (Crying..)
AS-IS
DEV TEST
CUST
OMER
TEST
Test repsents us to dev
(Thnx~)
Test came to help me
(smile~)
My job is helpful for dev(smile~)
Became active (real) member
To deliver the right product(smile~)
TO-BE
51. Various Test careerpaths
- as many as various work needed, as many as various job needed
methodolgi
st,
SA,TA,DA,โฆ
develo
per
PM
PL
Tool
engineer
Dev project
Tester Test Lead
Test Engineer Test Architect
Test Manager
Tester
TM
TL
Test project
Test Strategy
Education, guide
Manual, Automation
Support
52. Q & A
Do you think testers can write business test scenario?
Why testers care about Unit-testing?, It is the job for developers.
The activities are too many, do you mean we should do them all?
Basically, every member(including customers) should participate in
writing business test scenario and evolved during the whole project period.
Who is in charge to write the draft depends on projectโs conditions
Well, Unit-testing is the developerโs job. Thatโs right.
However as quality people, we can check anywhere the quality exists, and
give them feedback with our profession, quality.
Moreover we can learn much about product internally.
Of course not. The message is just delivering co-work with dev.
Each person(testers) has different background, skills, interests, I just wanted to
Introduce various activities and make them as one, each career path