Greg Smith
Cover how to use simple low-level tools such as memtest86, dd, bonnie++, and sysbench to benchmark the hardware of a server intended for database use. A heavy dose of vendor management suggestions will be included as well, for the inevitable time when your shiny new server fails to deliver the performance it should.
2. About this presentation
The master source for these slides is
http://www.2ndquadrant.com/talks
Slides are released under the Creative Commons Attribution
3.0 United States License
http://creativecommons.org/licenses/by/3.0/us
Greg Smith Database Hardware Benchmarking
3. Why should you always benchmark your hardware?
Many useful tests will only run when the server isn’t being
used yet
Software stacks are complicated
Spending money on upgrades only helps if you upgrade the
right thing usefully
Vendors lie
Greg Smith Database Hardware Benchmarking
4. Systematic Benchmarking
Memory
CPU
Disk
Database server
Application
Greg Smith Database Hardware Benchmarking
5. Memory Tests
DOS: memtest86+ (also on many Linux CDs, like the Ubuntu
installer)
Windows: SiSoftware Sandra
UNIX: sysbench, stream
Greg Smith Database Hardware Benchmarking
6. sysbench memory write and CPU tests
sysbench --test=memory --memory-oper=write
--memory-block-size=64MB --memory-total-size=1024MB
run
sysbench --test=cpu run
Greg Smith Database Hardware Benchmarking
7. stream memory read test
wget http://www.cs.virginia.edu/stream/FTP/Code/stream.c
gcc -O stream.c -o stream
./stream
Greg Smith Database Hardware Benchmarking
10. Oracle Calling Center OLTP Benchmark
http://it.anandtech.com/IT/showdoc.aspx?i=3769&p=4
Greg Smith Database Hardware Benchmarking
11. Sources for slow memory results
Single channel RAM/slot mistakes
Incorrect SPD/timing/voltage
Bad RAM/CPU multiplier combination
Poor quality RAM
Greg Smith Database Hardware Benchmarking
12. PostgreSQL and the CPU
PostgreSQL uses only a single CPU per query
Queries executing against cached data will bottleneck on CPU
COPY is CPU intensive
Greg Smith Database Hardware Benchmarking
13. CPU Tests
Windows: SiSoftware Sandra
UNIX: sysbench CPU test
Custom test with timing and generate series
pgbench select-only on small database
Vary pgbench client count to test single or multiple CPUs
Greg Smith Database Hardware Benchmarking
15. Sources for slow CPU results
Slow memory
Power management throttling
Linux: /proc/cpuinfo shows 1000MHz suggests you need to
adjust the CPUFreq Governor to “performance”
Greg Smith Database Hardware Benchmarking
16. Disk Tests
Sequential write: INSERT, COPY FROM (when not CPU
limited)
Sequential read: SELECT * FROM and similar table
sequential scans
Seeks: SELECT using index, UPDATE
Commit fsync rate: INSERT, UPDATE
Greg Smith Database Hardware Benchmarking
17. dd test
Compute 2X the size of your RAM in 8KB blocks
blocks = 250,000 * gigabytes of RAM
time sh -c "dd if=/dev/zero of=bigfile bs=8k count=X &&
sync"
time dd if=bigfile of=/dev/null bs=8k
Watch vmstat and/or iostat during disk tests
vmstat’s bi and bo will match current read/write rate
Note the CPU percentage required to reach the peak rate
Greg Smith Database Hardware Benchmarking
18. bonnie++
./bonnie++
bon csv2html
Ignore the per-character and create results, look at the block
output/input ones
Random Seeks:
The test runs SeekProcCount processes (default 3) in parallel,
doing a total of 8000 random seek reads to locations in the
file. In 10% of cases, the block read is changed and written
back.
Greg Smith Database Hardware Benchmarking
19. bonnie++ ZCAV
./zcav -f/dev/sda > t500
Must get a recent version of bonnie++ for ZCAV to scale
properly for TB drives (1.03e works)
ZCAV on experimental branch (1.96) gave useless results for
me
Download somewhat broken gnuplot script sample and typical
results from:
http://www.coker.com.au/bonnie++/zcav/results.html
Greg Smith Database Hardware Benchmarking
20. Improved bonnie++ ZCAV gnuplot script
unset autoscale x
set autoscale xmax
unset autoscale y
set autoscale ymax
set xlabel "Position GB"
set ylabel "MB/s"
set key right bottom
set terminal png
set output "zcav.png"
plot "raid0" title "7200RPM RAID 0 3 Spindles",
"single" title "7200RPM Single Drive"
Greg Smith Database Hardware Benchmarking
25. Sample sysbench random read results
Read 78.125Mb Written 0b
Total transferred 78.125Mb (1.0059Mb/sec)
128.75 Requests/sec executed
That’s 128.75 seeks/second over 10GB, resulting in a net
throughput of 128.75 * 8KB/s = 1.01MB/s
Consider both the size of the disk used and the number of
clients doing seeks
Greg Smith Database Hardware Benchmarking
26. More customizable seek tests
bonnie++ experimental (currently at 1.96)
iozone
fio
Windows: HD Tune does everything but commit rate
Greg Smith Database Hardware Benchmarking
27. Sources for slow disk results
Poor mapping to underlying hardware
Buggy driver
Insufficient bandwidth to storage
Bottlenecking at CPU/memory limits
Bad performing filesystem or filesystem misaligned with stripe
sizes
Writes faster than reads? Probably low read-ahead settings
somewhere.
Vibration: don’t shout at your JBODs! They don’t like it!
http://it.toolbox.com/blogs/database-soup/the-problem-with-iscsi-30602
http://blog.endpoint.com/2008/09/filesystem-io-what-we-presented.html
http://www.youtube.com/watch?v=tDacjrSCeq4
Greg Smith Database Hardware Benchmarking
28. fsync tests
sysbench --test=fileio --file-fsync-freq=1 --file-num=1
--file-total-size=16384 --file-test-mode=rndwr run
| grep "Requests/sec"
pgbench insert-only test
PostgreSQL contrib/test fsync might work, but isn’t really
reliable
Greg Smith Database Hardware Benchmarking
29. Sample laptop disk specification
ST9320423AS Momentus 7200.4 320GB
7200 RPM
16MB Cache
Average seek: 11ms read/13ms write
Average rotational latency: 4.17ms
Greg Smith Database Hardware Benchmarking
31. IOPS Calculators and Info
http://www.wmarow.com/strcalc/
http://www.dbasupport.com/oracle/ora10g/disk_IO_02.shtml
http://storageadvisors.adaptec.com/2007/03/20/sata-iops-measurement/
Greg Smith Database Hardware Benchmarking
32. Sample disk results
Disk Seq Seq bonnie++ Read-only Commit Drive
Count Read Write seeks seeks Rate Model
1 71 58 232 @ 4GB 194 @ 4GB 105/s 7200.4
1 59 54 177 @ 16GB 56 @ 100GB 10212/s WD160
3 125 119 371 @ 16GB 60 @ 100GB 10855/s RAID0
Commit rate for 7200.4 laptop drive is 1048/s with unsafe
volatile write cache
Non-laptop drives include a 256MB battery-backed write
cache, Linux SW RAID
Greg Smith Database Hardware Benchmarking
33. Custom PostgreSQL tests
Quick CPU test (about 1140ms on my laptop):
timing
select sum(generate series) from
generate series(1,1000000);
Quick insert/plan test:
timing
CREATE TABLE test (id INTEGER PRIMARY KEY);
INSERT INTO test VALUES (generate series(1,100000));
EXPLAIN ANALYZE SELECT COUNT(*) FROM test;
Greg Smith Database Hardware Benchmarking
34. Benchmarking functions
http://justatheory.com/computers/databases/postgresql/benchmarking_upc_validation.html
http://justatheory.com/computers/databases/postgresql/benchmarking_functions.html
pg stat user functions handles this specific job in 8.4
The general technique is applicable for all sorts of custom
benchmarks
Greg Smith Database Hardware Benchmarking
35. pgbench select only test
set naccounts 100000 * :scale
setrandom aid 1 :naccounts
SELECT abalance FROM accounts WHERE aid = :aid;
Greg Smith Database Hardware Benchmarking
36. pgbench custom test: insert only, to measure commit rate
set nbranches :scale
set ntellers 10 * :scale
set naccounts 100000 * :scale
setrandom aid 1 :naccounts
setrandom aid 1 :naccounts
setrandom bid 1 :nbranches
setrandom tid 1 :ntellers
setrandom delta -5000 5000
BEGIN
INSERT INTO history (tid, bid, aid, delta, mtime)
VALUES (:tid, :bid, :aid, :delta, CURRENT TIMESTAMP);
END;
Greg Smith Database Hardware Benchmarking
37. pgbench standard test
BEGIN;
UPDATE accounts SET abalance = abalance + :delta
WHERE aid = :aid;
SELECT abalance FROM accounts WHERE aid = :aid;
UPDATE tellers SET tbalance = tbalance + :delta
WHERE tid = :tid;
UPDATE branches SET bbalance = bbalance + :delta
WHERE bid = :bid;
INSERT INTO history (tid, bid, aid, delta, mtime)
VALUES (:tid, :bid, :aid, :delta, CURRENT TIMESTAMP);
END;
Greg Smith Database Hardware Benchmarking
38. pgbench-tools
Available from git.postgresql.org and github
Set of shell scripts to automate running many pgbench tests
Results are saved into a database for analysis
Saves TPS, latency, background writer statistics
Most of the cool features require PostgreSQL 8.3
Inspired by the dbt2 benchmark framework
Greg Smith Database Hardware Benchmarking
39. Server configuration for pgbench results
Quad-Core Intel Q6600
8GB DDR2-800 RAM
Areca ARC-1210 SATA II PCI-e x8 RAID controller, 256MB
write cache
DB: 3x640GB Western Digital SATA disks, short-stroked,
Linux software RAID-0
WAL: 160GB Western Digital SATA disk
CentOS 5.4, Linux Kernel 2.6.18-164.11.1.el5xen x86 64
OS on separate disk
Untuned ext3 filesystems
Greg Smith Database Hardware Benchmarking
45. What should you do?
Trust no one
Don’t start on application benchmarks until you’ve proven
basic performance
Don’t spend too long on basic performance if you can switch
to application benchmarks
Vendors alternate among lying, misunderstanding what you
want, and trying to make you feel dumb
Use simple, standard tools whenever possible to minimize
vendor disputes
Be prepared to translate to your vendor’s language and
subvert their agenda
Never spend real money on hardware unless you can return it
if it sucks
Greg Smith Database Hardware Benchmarking
46. Credits
Special thanks to:
Mark Wong and the other contributors to dbt-2
Truviso for providing some of the test hardware included here
Greg Smith Database Hardware Benchmarking
47. For more information...
Some slides are taken from the upcoming book:
“High Performance PostgreSQL” by Greg Smith
Planned for release this summer by Packt Publishing
Performance tuning of PostgreSQL 8.1 through 9.0, from
hardware to scaling via replication
Greg Smith Database Hardware Benchmarking
48. Questions?
Remember that your coffee break awaits you
Greg Smith Database Hardware Benchmarking