How to Release Rock-solid RESTful APIs and Ice the Testing BackBlob

20140918 System Verification Associates © 2014 1
How to Release Rock-solid RESTful
APIs and Ice the Testing BackBlob
Unicom Next Generation Testing Conference
Chicago, September 18, 2014
Robert V. Binder
SystemVerification Associates
Enabling High Assurance http://sysverif.com

Overview
• Background
• Advanced API Verification
• Dataflow Testing Model
• Model-based Testing Demo
• The Testing Twofer
• Q&A

Discovery Analysis Design Verification Support
BACKGROUND

You are here …
Browser
HTTP Client
HTTP
Server
App
HTTP Client
App
SOAP Client
HTTP Client
Files
HTTP
Server
Service
SOAP Server
Service
SOAP Server
Files
SOAP API
REST API

Programmable Web’s Growing Roster

Google Trends: REST and SOAP
100
News Headline Occurrence, Monthly
SOAP API
REST API

So many APIs, so little time …
Why is this happening?

Challenges
• Usability
• Narrow developer focus
• Poor documentation
• Revenue prevention
• Assurance Fragmentation
• Functionality
• Security
• Performance
• Low reliability
• Ineffective testing
• Manual UI interaction
• Developer-centric, hand-
coded unit testing
• Wheel spinning
• High QA expense
• Low quality
All-aspect approach needed

ADVANCED API VERIFICATION

Discovery Sprint
• Survey and catalog
• API documentation
• Open and closed issues
• Social media views
• Codebase
• Usage logs
• Results
• Strategy
• Test environment spec
• Report card

Analysis Sprint
• Workflow
• Construct usage profile
• Scrutinize documentation
• Abstract data model
• Results
• Doc issues
• Gap analysis
• Revised strategy

Design Sprint
• Workflow
• Configure virtual lab
• Behavior/data models
• Traffic capture/parsers
• Instantiate adapters
• Results
• Stable test environment
• All-aspect test model
• Revised strategy

Verification Sprint
• Workflow
• Model checking
• Generate/run test suites
• Collect traffic logs
• Analyze coverage
• Results
• All test artifacts
• Test coverage report
• Final report
• Briefing

Support
• As needed
• Incremental design review
• Usage monitoring
• CI and regression testing
• Results
• Continuity
• Protect investment
• Continuous improvement

DATAFLOW TESTING MODEL

System Under Test
Service
Browser
HTTP Client
HTTP
Server
App
SOAP Client
SOAP Server
HTTP Client
Files
Service
HTTP
Server
SOAP Server
Files
App
HTTP Client
REST API

Test Configuration
Service
App
HTTP
Server
HTTP Client
Service
HTTP
Server
Generated
Test Code
Test Model
REST API

REST = Methods + Resources + Parameters
Service
App HTTP
ServerHTTP Client
HTTP
Server Service
HTTP methods:
GET, PUT, POST, DELETE …
HTTP resources (URI):
http://foo.com/titles
HTTP returned payload, JSON format:
{"firstName": "Bob",
"lastName": "Binder",
"books": [
{ "title": "Testing Object-oriented"},
{ "title": "Application Debugging"}
]
}
Status Code:
200, 201, 400, 404
/?au=binder

REST Dataflow Model – Normal Paths
alpha
Defined
Used
Gone
PUT/201
GET/200
PUT|POST/200
DELETE/200
DELETE/200PUT|POST/200
GET/200

REST Dataflow Model – Method Errors
alpha
Defined
Used
Gone
DELETE|GET/404
DELETE|GET|PUT|POST/404

REST Dataflow Model
alpha
Defined
Used
Gone
Test Pattern: Non-Modal Class

Input variation, all sequences
• Nominal values
• Boundary values
• Operator mutants
• Fuzzing, each/all
• Domain model
• Pairwise selection
• Sequence
randomization
Sounds like a lot of work!

Model-based Testing
• Model-based testing tool
• Microsoft Research, 2001
• Test 500 MSFT APIs, 2007-12
• Robust and stable
• Visual Studio “power tool”
• C# code, not cartoons
• Generates standalone
executable test suite

Demo
• Synthetic Client
• Model Program
• Coordination File
• Test Cases
SUT HostTest Host
Test Suite
HTTP
Server
Synthetic
Client
Pass/Fail
Synthetic
Client
Interface
Spex
Rules
Spex
Cord
Test Modeling Test Execution
Service
Under
Test
Explore/
Generate

Synthetic Client
• The test model’s view of the SUT
• Static class wrapper for HTTP client
• Public methods correspond to SUT’s
HTTP methods and resources
• Manage server-side setup/cleanup
• Message serialize/deserialize
• Becomes part of the executable test
code assembly
• Example is a stub!

Model Program
• [Rule]
• Determines when an action
is called
• Selects argument values for
the action call
• Computes expected results
• Updates its model state as
needed
• Simulates environment
and/or system under test

Cord File
• Defines all model
actions
• action = Synthetic Client
public method
• machine
• Any action sequence
• Similar to regex
• May use other machines
• Model any use case,
scenario, slice, etc.
• Many options

What is Exploration?
• Find all action sequences and data
bindings that model program Rules
and a machine allow
• Search loop
• Select a rule for a machine action
• If enabling condition true:
• Update model program state
• Return expected results
• Stop when all selected inputs used or
size limit exceeded

Machine Exploration
• Shows all possible
action sequences for a
machine
• No data bindings
• Note similarity to
normal path dataflow

Model Program Exploration
• Rules + machine
• Rules add data
bindings, expected
results
• Many ways to
choose data values

Test Cases from an Exploration
• Spex chooses exploration steps
that end in accepting state
• Covers all states and steps at
least once

Generate Test Code
• Standalone code – does
not require model
• Run from VS Test
Explorer or command
line

SUT HostTest Host
Test Suite
HTTP
Server
Synthetic
Client
Pass/Fail
Synthetic
Client
Interface
Spex Rules
Spex Cord
Test Modeling Test Execution
Service
Under
Test
Explore/
Generate

Test Strategy
• Each resource path
• Interleave all DUG
variants
• Accepting sequence
• Wrong sequence
• Pairwise combination
• Parameters (path and value)
• Mutants, nominal, edge
• Security
• Interleave Fuzz cases
• Abuse case model
• All other HTTP methods
• Performance
• Virtual users/test drivers
• Randomize combos

THE TESTING TWOFER

The Testing BackBlob
Total
Number
of Test
Cases
Sprint 1
Available
Test Time
Manual Test
Cases not
executed
Automated Test
Cases not
maintained
Total Developed
Test Cases
Sprint 2 Sprint 3 Sprint 4

The Attack of the Testing BackBlob
Coming soon … to a scrum near you

Test Asset Size
Model
Test Code
Adapters
Model-based Testing Behavior Driven Development

Test Asset Maintenance Load
Model
Test Code
Adapters
Model-based Testing Behavior Driven Development

The Testing Twofer
Rock Solid APIs
• Documentation Scrutiny
• Fact-based Evaluation
• Multi-dimensional testing
• Dataflow coverage
• Everything wrong at least
once
• Fuzzing
• Repeat at scale
Icing the BackBlob
• Develop/maintain model
• Regenerate test suites

Q & A
rvbinder@sysverif.com
#MoreModelsLessTests
http://sysverif.com

ETC.
Say what you do, do what you say

Robert V. Binder
Robert Binder is a high-assurance
entrepreneur.
He has developed hundreds of application
systems and advanced automated testing
solutions. As test process architect for
Microsoft’s Open Protocol Initiative, he lead the
application of model-based testing to all of
Microsoft’s server-side APIs. He is the author of
the definitive Testing Object-Oriented Systems:
Models, Patterns, and Tools and two other
books. He holds a US patent for model-based
testing of mobile systems.
• MS, EECS, University of Illinois at Chicago
• MBA, University of Chicago
• BA, University of Chicago

System Verification Associates
Enabling High Assurance
• Chicago- based consulting boutique
• Clients are typically software development
organizations for whom system failure is not an option.
• We assist clients in achieving high reliability and
effectiveness in their IT processes and systems.
• Founded in 2009 and led by Robert V. Binder
• http://sysverif.com
• Advanced API Verification Datasheet
• Supported Microsoft’s Open Protocols project with a
team of experts; Robert Binder served process architect,
leading the technical work of over 300 staff located in
Redmond, China, India, and Argentina.
• Assessed and improved software process at several
FDA-regulated product companies, balancing quality
management system compliance and Agile practices.
• Developed model-based testing solutions for high-
frequency trading and aerospace applications.
• Helped software service and product companies
articulate unique high-value messaging for innovative
services.
• Conducted and published the Model-based Testing User
Survey of 2012 and 2014 (forthcoming.)

Does My API Suck?
 Your documentation is incomplete, wrong,
misleading, or just plain incomprehensible.
 Users complain that coding simple use cases
is just too much hassle.
 Users often rely on workarounds—they FTP
files instead of using your API’s getFile.
 Your API is unbalanced or incomplete—you
can turn something on, but not off.
 Your API’s service crashes or responds with
garbage when messages are out of order or
contain invalid data.
 Version mismatches have unpredictable
results.
 No one is really sure what will happen with
edge cases and they don’t want to know.
 Your API allows your service to be hacked
with common attack vectors.
 Your service supports several protocols (REST,
SOAP,…) or formats (JSON, XML,…), but
behavior and data isn’t consistent
 Your API doesn’t provide useful feedback—
good and bad input all get the same
response.
 Your service is so awesome that it draws
traffic spikes, but then your server chokes
and dies.
Buggy APIs are eating the world

How to Release Rock-solid RESTful APIs and Ice the Testing BackBlob

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to How to Release Rock-solid RESTful APIs and Ice the Testing BackBlob

Similar to How to Release Rock-solid RESTful APIs and Ice the Testing BackBlob (20)

More from Bob Binder

More from Bob Binder (18)

Recently uploaded

Recently uploaded (20)

How to Release Rock-solid RESTful APIs and Ice the Testing BackBlob