Bart Oles - Severalnines AB Database performance affects organizational performance, and we tend to look for quick fixes when under stress. But how can we better understand our database workload and factors that may cause harm to it? What are the limitations in MongoDB that could potentially impact cluster performance? In this talk, we will show you how to identify the factors that limit database performance. We will start with the free MongoDB Cloud monitoring tools. Then we will move on to log files and queries. To be able to achieve optimal use of hardware resources, we will take a look into kernel optimization and other crucial OS settings. Finally, we will look into how to examine performance of MongoDB replication.

1. September 2018
Performance Tuning Cheat
Sheet for MongoDB
Bartłomiej Oleś
Presenter
bart@severalnines.com

3. Copyright 2017 Severalnines AB
Supported Databases

4. Copyright 2017 Severalnines AB
Free to download
Initial 30 days Enterprise trial
Converts into free Community Edition
Enterprise / paid versions available

5. Copyright 2017 Severalnines AB
Automation & Management
Deployment (Free Community)
● Deploy a Cluster in Minutes
○ On-Prem
○ Cloud (AWS/Azure/Google) - paid
Monitoring (Free Community)
● Systems View with 1 sec Resolution
● DB / OS stats & Performance Advisors
● Configurable Dashboards
● Query Analyzer
● Real-time / historical
Management (Paid Features)
● Backup Management
● Upgrades & Patching
● Security & Compliance
● Operational Reports
● Automatic Recovery & Repair
● Performance Management
● Automatic Performance Advisors

6. Agenda
Copyright 2018 Severalnines AB
● Why performance cheat sheet?
● Free monitoring for performance
● Logging database operations
● Capturing queries - database profiler
● Checking operating system parameters
● Working with the Explain Plan
● Measuring replication lag performance
● Live demo
● Other

7. Copyright 2017 Severalnines AB
Copyright 2018 Severalnines AB
Why performance cheat sheet?

8. Performance complexity
● Services running on multiple hosts
○ Replication
○ Sharding
○ Clustering
● Multiple Data Centers
○ Cloud and/or On-prem
○ Disaster Recovery
● Load balancing and Single point of contact IP
○ For workload management, HA, query caching...
○ E.g., HAProxy, KeepAlived/VIP, ProxySQL, MaxScale

9. Why we need a database monitoring system
● Data is a key asset of the organisation
● Databases are important as they manage the source of truth
● Database is complex - IO, transaction engine, query optimizer,
caches, locks, versioning,...
● Very dependent on OS, IO subsystems, network
● Distribution across multiple instances makes it even more complex
● Good database monitoring helps make sense of all that

10. MongoDB
● Similar to most other databases
● Understand the utilization of the hardware
● Capacity planning
● Determine the type of an issue
● I/O related?
● CPU related?
● Network related?

11. Why we need a database monitoring system
● CPU utilization (should I add more nodes to the cluster?)
● Network utilization (am I running out of bandwidth?)
● Ping (how badly latency affects my MongoDB cluster?)
● Disk throughput and IOPS (am I within my hardware limits?)
● Disk space (do I have to plan for larger disks?)
● Memory utilization (do I suffer from a memory leak?)

12. Why we need a database monitoring system
● Storage engine specific
● MMAP
● WiredTiger
● MongoRocks
● Insight in how the engine performs
● Internal congestion

13. Why we need a database monitoring system
● CPU, IO or lock related
● Outcome: similar to Galera
● Lagging behind could cause a full sync

14. Performance monitoring vs metrics
● Similar to most other databases
● Throughput of the cluster
● Relate throughput to cluster performance
● Determine the type of an issue
● Request spikes?
● Write amplification related?
● Queueing?

15. Monitoring vs Trending
● Monitoring system (i.e. Nagios)
● Checks if services are healthy
● Sends pages
● Trending system (i.e. Cacti, Graphite)
● Collects metrics
● Generate graphs
● Availability
● Do more than just opening a
connection
● Measure true status of nodes and
cluster
● Test read/write
● Open essential databases and
collections
● Keep an eye on the replication lag
● Increase oplog size?
● Check the full topology

16. Monitoring vs Trending
● Trending
● Plot trends of key (performance) metrics
● Find problems before they arise
● Pre-emptive problem management
● Trending tools
● Granularity of sampling
● More datapoints = better
● Periodical (daily/weekly) healthchecks
● Insight into all aspects of the database operations
● Post mortem and proactive monitoring
● Capacity planning

17. Copyright 2017 Severalnines AB
Copyright 2018 Severalnines AB
Performance monitoring

18. MongoDB monitoring
● Enable Free Monitoring
db.enableFreeMonitoring()
● Disable Free Monitoring
db.disableFreeMonitoring()

19. Logging database operations
Operation Execution Times (READ, WRITES, COMMANDS)
Disk utilization (MAX UTIL % OF ANY DRIVE, AVERAGE UTIL % OF ALL DRIVES)
Memory (RESIDENT, VIRTUAL, MAPPED)
Network - Input / Output (BYTES IN, BYTES OUT)
Network - Num Requests (NUM REQUESTS)
Opcounters (INSERT, QUERY, UPDATE, DELETE, GETMORE, COMMAND)
Opcounters - Replication (INSERT, QUERY, UPDATE, DELETE, GETMORE,
COMMAND)
Query Targeting (SCANNED / RETURNED, SCANNED OBJECTS / RETURNED)
Queues (READERS, WRITERS, TOTAL)
System Cpu Usage (USER, NICE, KERNEL, IOWAIT, IRQ, SOFT IRQ, STEAL,
GUEST)

20. Why we need a database monitoring system
db.getFreeMonitoringStatus()
{ resource: { cluster : true }, actions: [ "setFreeMonitoring",
"checkFreeMonitoringStatus" ] }
db.serverStatus()

21. Why we need a database monitoring system
{
"state" : "enabled",
"message" : "To see your monitoring data, navigate to the unique URL
below. Anyone you share the URL with will also be able to view this page.
You can disable monitoring at any time by running
db.disableFreeMonitoring().",
"url" :
"https://cloud.mongodb.com/freemonitoring/cluster/XEARVO6RB2OTXEAHKHLKJ5V
6KV3FAM6B",
"userReminder" : "",
"ok" : 1
}

22. Database profiler
db.getProfilingLevel()
To capture all queries set:
db.setProfilingLevel(2)
profile = <0/1/2>
slowms = <value>

23. Logging database operations
db.getLogComponents()
Log messages include many components. This is to provide a functional categorization of the
messages. For each of the component, you can set different log verbosity. The current list of
components is:
ACCESS, COMMAND, CONTROL, FTD, GEO, INDEX, NETWORK, QUERY, REPL_HB, REPL,
ROLLBACK, REPL, SHARDING, STORAGE, RECOVERY, JOURNAL, STORAGE, WRITE.

24. Examples
To list the 10 most recent:
db.system.profile.find().limit(10).sort(
{ ts : -1 }
).pretty()
To list all:
db.system.profile.find( { op:
{ $ne : 'command' }
} ).pretty()
To list all:
db.system.profile.find(
{ ns : 'mydb.test' }
).pretty()

25. MongoDB logging
/var/log/mongodb/mongod.log
You can find MongoDB configuration file at /etc/mongod.conf.
Here is sample data:
2018-07-01T23:09:27.101+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to
node1:27017
2018-07-01T23:09:27.102+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect
to node1:27017 - HostUnreachable: Connection refused
2018-07-01T23:09:27.102+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all
pooled connections to node1:27017 due to failed operation on a connection
2018-07-01T23:09:27.102+0000 I REPL_HB [replexec-2] Error in heartbeat (requestId: 21589) to
node1:27017, response status: HostUnreachable: Connection refused
2018-07-01T23:09:27.102+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to
node1:27017

26. MongoDB logging
db.runCommand({ logRotate : 1 });
db.setLogLevel(2, "query")

27. MongoDB Oplog
● Similar to MySQL binary logs
● Oplog: a special collection
● Limited size
● Eviction of transactions (FIFO)
● Replication window
● Time between first and last transaction in the oplog

28. MongoDB Connections
● Similar to MySQL when handling connections
● Client drivers may support connection pooling
● Multiple non-blocking queries can use the same
connection
● Spawns new connections when low on threshold
● Increase of connections
● Locking issues
● Application request bursts

29. Checking operating system parameters -
memory limits

30. Checking operating system parameters -
network
net.core.somaxconn (increase the value)
net.ipv4.tcp_max_syn_backlog (increase the value)
net.ipv4.tcp_fin_timeout (reduce the value)
net.ipv4.tcp_keepalive_intvl (reduce the value)
net.ipv4.tcp_keepalive_time (reduce the value)

31. Network stack
net.core.somaxconn = 4096
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_max_syn_backlog = 4096

32. Checking operating system parameters -
memory limits
$ sysctl -a | egrep “vm.dirty.*_ratio”
vm.dirty_background_ratio = 10
vm.dirty_ratio = 20

33. Security
sudo setenforce Enforcing
sudo getenforce

34. Swappiness
vi /etc/sysctl.conf
vm.swappiness = 1

35. Transparent huge pages
cat /proc/sys/vm/nr_hugepages
0

36. Filesystem options
ext4 rw,seclabel,noatime,data=ordered 0 0
XFS (MongoDB 3.0+)
disable access-time updates

37. NTP Demon
#Red Hat
sudo yum install ntp
#Debian
sudo apt-get install ntp

38. Explain plan
db.inventory.find( {
status: "A",
$or: [ { qty: { $lt: 30 } }, { item: /^p/ } ]
} ).explain('executionStats');
or append it to the collection:
db.inventory.explain('executionStats').find( {
status: "A",
$or: [ { qty: { $lt: 30 } }, { item: /^p/ } ]
} );

39. Measuring replication lag performance
db.getReplicationInfo()
db.getReplicationInfo()
{
"logSizeMB" : 2157.1845703125,
"usedMB" : 0.05,
"timeDiff" : 4787,
"timeDiffHours" : 1.33,
"tFirst" : "Sun Jul 01 2018 21:40:32 GMT+0000 (UTC)",
"tLast" : "Sun Jul 01 2018 23:00:19 GMT+0000 (UTC)",
"now" : "Sun Jul 01 2018 23:00:26 GMT+0000 (UTC)"

40. Measuring replication lag performance
rs.printSlaveReplicationInfo()
rs.status()
● Replication Metrics
● Throughput of the replication
● Durability of the oplog
● Replication lag
● Comparable to Galera replication
● Quorum based
● At least one secondary needs to acknowledge

41. MongoDB locking
● Three levels of (generic) locking
Global, Database, Collection
mongo_replica_0:PRIMARY> db.serverStatus().locks {
"Global" : { "acquireCount" : { "r" :
NumberLong(6050583), "w" : NumberLong(2416551), "R" :
NumberLong(1), "W" : NumberLong(7) }, "acquireWaitCount"
: { "r" : NumberLong(1), "w" : NumberLong(1), "W" :
NumberLong(1) }, … }

42. MongoDB locking
(WiredTiger)
Document level locking
Tickets (threads)
Read/Write
mongo_replica_0:PRIMARY>
db.serverStatus().wiredTiger.concurrentTransactions {
"write" : { "out" : 0, "available" : 128, "totalTickets"
: 128 }, "read" : { "out" : 0, "available" : 128,
"totalTickets" : 128 } }

43. MongoDB cache
● MongoDB uses three tiers of cache
○ Filesystem
○ Active memory
○ Storage engine (WiredTiger / MongoRocks)
● Page faults
● Evictions

44. Copyright 2017 Severalnines AB
Copyright 2018 Severalnines AB
How to automate?

45. Copyright 2017 Severalnines AB
Copyright 2018 Severalnines AB
Live demo

46. Links & Resources
● Download / install ClusterControl
● ClusterControl Community Edition Page
● Contact us: info@severalnines.com