Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Review: DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
Similar to Review: DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems (20)
Recently uploaded
Recently uploaded (20)
Review: DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
- 1. DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue , Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang will be published @ ASE 2018 Jinhan Kim 2018.8.3
- 3. How to Test Deep Learning System Systematically?
- 4. Not Like Traditional Software
- 5. Neuron Coverage of DeepXplore
- 6. Neuron
- 7. Neuron Coverage • # of neurons activated / # total neurons -11 Car Face hedge vedge ... Nose Eyes Wheel ... ... ... f: ReLU 3 1 2 1 2 3 f(1) 1 Neuron coverage: 4/7=57% when activation threshold=0.75
- 9. Neuron-Level Coverage • k-multisection Neuron Coverage Neuron Inputs 0.1 0.13 0.17 0.2 0.43 0.15 Outputs K-Section bucketing low high
- 10. Neuron-Level Coverage • Neuron Boundary Coverage K-Section bucketing low high Lower Corner Upper Corner Neuron
- 11. Neuron-Level Coverage • Strong Neuron Activation Coverage K-Section bucketing low high Lower Corner Upper Corner Neuron
- 12. Layer-Level Coverage • Top-k Neuron Coverage 6 7 1 2 3 4 5 0.5 0.2 0.6 Pick Top-2 “Neuron 3 and 5 are covered”
- 13. Layer-Level Coverage • Top-k Neuron Patterns 6 7 1 2 3 4 5 Pick Top-1 for each layer 1 4 7 Pattern
- 14. Study
- 16. Configuration
- 17. Result: k-multisection Neuron Coverage
- 18. Result: Strong Neuron Activation Coverage
- 19. Result: Neuron Boundary Coverage
- 20. Result: Top-k Neuron Coverage
- 21. Result: Top-k Neuron Patterns
- 22. Remarks • Adversarial test-sets boost coverage but.. • Note that increasing the test coverage does not necessarily imply that new defects could be detected in traditional software testing.
- 23. Comparison with DeepXplore’s NC
- 24. Result: DeepXplore’s Neuron Coverage (DNC)
- 25. Why?
- 26. Investigation on DNC • DNC uses the same threshold as the activation evaluation for all the neurons. • DNC normalizes the dynamic range of neuron outputs according to max and min output of neurons on the corresponding layer for each input image under analysis.