This document discusses Automatic Test Packet Generation (ATPG), which automatically generates test packets to check the health and functionality of a network. It aims to detect common network failures like hardware failures, software bugs, reachability issues, and performance degradation. The ATPG system works by performing reachability analysis to determine all possible packet flows, generating a minimum set of test packets to exercise the entire rule set, and using the results to localize any faults by eliminating passing rules and identifying broken ones. Implementing ATPG could provide more efficient network testing than current tools like ping and traceroute, helping reduce downtime and costs associated with network outages.

1. Automatic Test Packet
Generation
Tushar Jadhav

2. Contents
Introduction
Problem Statement
Concept
Conclusion
References

3. Introduction

4. Outages in world

5. Network Troubleshooting
Downtime:
25%take over an hour to resolve.
(estimated $60K-110K/hour ).
Current tools:
Ping, Traceroute, SNMP.
70%asked for better tools, automatic
tests

6. Automatic Test Packet Generation
(ATPG) generates packets :
- To test the liveness of the underlying
topology .
- To test the compatibility between
data plane state and configuration
specifications.
- To test performance assertions such
as packet latency.

7. Problem Definition
Most common causes of network
failure
1) Hardware failures
2) Software bugs
Others failure’s include reachability
failures and throughput/latency
degradation.
Our goal is to automatically detect
these types of failures.

8. Goals
Automatically generate test packets to
test the network state, faults before
being
noticed by application.
Augment human wisdom and intuition.
Reduce the downtime.
Save money.

9. Concept
ATPG System :

10. Roadmap
Reachability Analysis
Test Packet Generation & Selection
Fault Localization
Implementation & Evaluation

11. Reachability Analysis
Header Space Analysis
All-pairs reachability : Compute all
classes of packets that can flow
between every pair of ports.

12. All-pairs reachability

13. Test Packet Generation &
Selection
All-pairs reachability

14. Test Packet Selection
Packets in all-pairs reachability table
are more than necessary.
Goal :
Select a minimum subset of packets
whose histories cover the whole rule
set.

15. Min Set Cover

16. Test Packet Selection
Min Set Cover
-Exercise all rules
-Sent out
Periodically
- “Redundant”
- Will be used in
Fault Localization

17. Fault Localization

18. Fault Localization
In ATPG: we can choose packets!
Step 1:
Use results from regular test packets
– F (potentially broken rules) = Union from all failing packets
– P (known good rules) = Union from all passing packets
– Suspect Set = F –P

19. Step 2 :
Use reserved test packets
- Pick packets that test only one rule in the
suspect set, and send them out for testing
- Passed: eliminate
- Failed: label it as “broken”
Step 3:
Brute force
- Continue with test packets that test two or
more rules in the suspect set, until the set is
small enough

20. Implementation
Putting Them all together :

21. Cisco/Juniper Parsers
-Translate router configuration files and
forwarding tables (FIB) into Header space
representation.
Test Packet Generation/Selection
- Hassel: A python header space library
- Min-Set-Cover
- Python’s multiprocess module to parallelize
SDN can simplify the design

22. ATPG for Performance
Testing
ATPG can also be used for detecting
and localizing performance problems.
Intuition: generalize results of a test
from success/failure to performance
(e.g. latency)
Users can inject performance errors

23. Limitations
Dynamic/Non-deterministic boxes
- e.g. NAT
“Invisible” rules
- e.g. backup rules
Transient network states
Ambiguous states (work in progress)
- e.g. ECMP

24. Conclusion
Test’s the forwarding state by
generating minimal link, queue, rule
cover automatically.
Efficient liveness testing,reachability
testing.
Better tool than ping & traceroute.
Brings more testing and coverage of
networks

25. References
P. Kazemian, G. Varghese, and N. McKeown,
“Header space analysis : Static checking for
networks,” in Proc. NSDI, 2012, pp. 9–9.
Y. Bejerano and R. Rastogi, “Robust
monitoring of link delays and faults in IP
networks,” IEEE/ACM Trans. Netw., vol. 14,
no. 5, pp.1092–1103, Oct. 2006
A. Dhamdhere, R. Teixeira, C. Dovrolis, and
C. Diot, “Netdiagnoser:Troubleshooting
network unreachabilities using end-to-end
probes and routing data,” in Proc. ACM
CoNEXT, 2007, pp. 18:1–18:12.