Modern Linux Performance Tools for Application Troubleshooting. Mostly demos and focused on application/process troubleshooting, not systemwide summaries.

1. gluent.com 1
Modern Linux Performance Tools
Tanel Poder
a long time computer performance geek

2. gluent.com 2
Intro: About me
• Tanel Põder
• Oracle Database Performance geek (18+ years)
• Exadata Performance geek
• Linux Performance geek
• Hadoop Performance geek
• CEO & co-founder:
Expert Oracle Exadata
book
(2nd edition is out now!)
Instant
promotion

3. gluent.com 3
Gluent
Oracle
Teradata
NoSQL
Big Data
Sources
MSSQL
App
X
App
Y
App
Z
Gluent as a data
virtualization layer
Open Data
Formats!
Session:
Extending Oracle
Data Warehouses
with Hadoop
Wednesday 10am
Executive Room 6
(small room!)

4. gluent.com 4
Topics
• Not much about basic stuff like vmstat/iostat
• Also nothing about SystemTap (or Dtrace)
• Practical stuff that you can (relatively) safely run in production
• Measuring / tracing processes
• Measuring IO
• Measuring Memory
• Measuring CPU

5. gluent.com 5
Measuring Process Activity - 1
• Strace
• man strace
• strace -p <PID>
• https://fritshoogland.files.wordpress.com/2013/03/profiling_of_oracl
e_using_function_calls.pdf
• SysDig
• sysdig fd.name contains tnsnames
• sysdig -p "%evt.arg.name" proc.name=cat and
evt.type=open
• csysdig
• http://www.sysdig.org/wiki/sysdig-examples/

6. gluent.com 6
Measuring Process Activity - 2
• /proc filesystem
• Proc filesystem in Linux is like V$/X$ views in Oracle
• man proc
• https://www.kernel.org/doc/Documentation/filesystems/proc.txt
• http://blog.tanelpoder.com/2013/02/21/peeking-into-linux-kernel-
land-using-proc-filesystem-for-quickndirty-troubleshooting/
• psnapper (early prototype!)
• ./psnapper <seconds> <pid>
• ./psnapper 5 1234
• http://blog.tanelpoder.com/files/scripts/tools/unix/psnapper

7. gluent.com 7
Measuring IO
• iostat in RHEL 7
• /proc/<PID>/io
• iotop (or htop, atop)
• blktrace / blkparse (if we have time)
• blktrace -d /dev/sda -o - | blkparse -i -

8. gluent.com 8
Measuring Memory
• pmap -x <PID>
• http://tech.e2sn.com/oracle/performance/unix-performance-
tools/process-memory-matrix (Solaris example but roughly the same)
• cat /proc/PID/smaps
• grep MMUPageSize: /proc/`pgrep dbw0`/smaps |
sort | uniq –c
• https://www.kernel.org/doc/Documentation/filesystems/proc.txt
• Smem
• sudo smem –u
• https://www.selenic.com/smem/

9. gluent.com 9
Measuring CPU
• perf top
• https://fritshoogland.wordpress.com/2012/07/10/the-linux-perf-
utility-with-el-6/
• perf record –g && perf report
• http://www.brendangregg.com/perf.html
• http://externaltable.blogspot.jp/2014/05/flame-graphs-for-
oracle.html
• perf stat
• http://blog.tanelpoder.com/2015/08/09/ram-is-the-new-disk-and-
how-to-measure-its-performance-part-1/

10. gluent.com 10
Linux performance references
• Brendan Gregg:
• http://www.brendangregg.com/linuxperf.html
• Luca Canali’s articles:
• https://db-blog.web.cern.ch/blog/luca-canali/2015-12-extended-
stack-profiling-ideas-tools-and-comments
• Frits Hoogland:
• https://fritshoogland.wordpress.com/

11. gluent.com 11
Thanks!
http://blog.tanelpoder.com
tanel@tanelpoder.com
@tanelpoder
We are hiring developers & data engineers!!!
http://gluent.com