LEGaTO paper presented at ACM Middleware 2020 by Robert Krahn, Donald Dragoti, Franz Gregor, Do Le Quoc, Valerio Schiavoni, Pascal Felber, Clenimar Souza, Andrey Brito and Christof Fetzer

1. TEEMon: A continuous performance monitoring
framework for TEEs

2. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 2
Motivation
Datacenter

3. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 3
Motivation
Datacenter

4. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 4
Motivation
Datacenter
• Encryption for data in transit and at rest

5. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 5
Motivation
Datacenter
TEE TEE
• Encryption for data in transit and at rest
• Trusted execution environments (TEE) for application
security

6. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 6
Motivation
Datacenter
TEE TEE
• Encryption for data in transit and at rest
• Trusted execution environments (TEE) for application
security
• Intel SGX, AMD SEV, ARM TrustZone

7. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 7
Motivation
Datacenter
TEE TEE
• Encryption for data in transit and at rest
• Trusted execution environments (TEE) for application
security
• Intel SGX, AMD SEV, ARM TrustZone
• TEE security incurs overhead
• Monitoring for deployed apps needed
• Interest in TEE-related metrics

8. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 8
Existing Tools and Related Work
Trusted
Execution
Environments
(TEE)
Low
Overhead,
Easy to Use
Distributed
Applications
(cloud)
• Various debugging / profiling tools exist (perf, etc.)
• Limited support for trusted execution environments (TEEs)
• Existing tools include: SGX-Perf, VTune, TEE-Perf
• Intended for debugging, not suited for online monitoring
• Framework specific, code instrumentation, large overhead.
• Cloud centered monitoring lacks support for TEE

9. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 9
Existing Tools and Related Work
Trusted
Execution
Environments
(TEE)
Low
Overhead,
Easy to Use
Distributed
Applications
(cloud)
TEEMon
• Various debugging / profiling tools exist (perf, etc.)
• Limited support for trusted execution environments (TEEs)
• Existing tools include: SGX-Perf, VTune, TEE-Perf
• Intended for debugging, not suited for online monitoring
• Framework specific, code instrumentation, large overhead.
• Cloud centered monitoring lacks support for TEE

10. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 10
Design Goals
• Easy to use
• Lightweight, modular, and extensible
• Continuous monitoring of TEE-related metrics
• Without code instrumentation / debugger
• Cloud-deployable (docker / kubernetes)

11. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 11
Design / Implementation

12. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 12
Design / Implementation
TEE metrics Kernel metrics

13. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 13
Design / Implementation
TEE metrics Kernel metrics

14. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 14
Design / Implementation
TEE metrics Kernel metrics

15. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 15
Design / Implementation
TEE metrics Kernel metrics
Prometheus

16. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 16
Design / Implementation
TEE metrics Kernel metrics REST HTTP
Prometheus

17. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 17
Design / Implementation
TEE metrics Kernel metrics REST HTTP
Prometheus

18. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 18
Design / Implementation
TEE metrics Kernel metrics REST HTTP
Prometheus

19. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 19
User Interface (partial)
• Enclave page cache statistics
• Memory pages loaded back into an enclave
• During runtime of Redis (SGX)

20. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 20
User Interface (partial)
• System-calls resulting in enclave exits
• Per application (regex-filter)
• During runtime of Redis (SGX)

21. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 21
User Interface (partial)
• Context switches for monitored application.
• Switches into and from application.
• Filtered for „Redis“.

22. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 22
User Interface (partial)
• Page faults
• During runtime of Redis (SGX)
• Kernel and user space

23. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 23
Evaluation – Overhead of TEEMon
• Monitoring related overhead.
• 5% - 13% overall overhead.
• Frequency of metrics can be adjusted.

24. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 24
Evaluation – Throughput of Redis w/ and w/o Intel-SGX
• TEE-related overhead.
• Throughput with SGX at 25% or less.
• Overhead not only hardware related.

25. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 25
Evaluation – Context Switches of Redis w/ and w/o SGX
• Redis benchmark with SGX-frameworks
• Increasing number of connections
• Small and larger database size
• TEEMon monitoring of context switches during rutime
• Cause for context switches reduces performance

26. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 26
Evaluation – Cross Version Evaluation w.r.t. Performance
• Redis benchmark
• TEEMon monitoring of system calls
• System calls in SGX are expensive
• Excessive unrelated system calls found
• Bug in SGX-Framework (SCONE)

27. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 27
Evaluation – Cross Version Evaluation w.r.t. Performance
• Redis benchmark
• TEEMon monitoring of system calls
• System calls in SGX are expensive
• Excessive unrelated system calls found
• Bug in SGX-Framework (SCONE)

28. TEEMon: A continuous performance monitoring framework for TEEs — Middleware 2020 Slide 28
TEEMon: A continuous performance monitoring framework for TEEs
• Lightweight continuous monitoring.
• Extensible and distributed design.
• TEE-related metrics at runtime.
• Assist in finding code related bottlenecks.
• Demo-video at https://sconedocs.github.io/teemon/