How to handle Redis Troubleshooting

1. Troubleshooting Redis
@charsyam
KAKAO

2. About me
•Senior Software Engineer in KAKAO
•Redis/Twemproxy Contributor
•Redis-doc project merger.
•Apache Tajo Commiter

3. Kakaostory

4. Kakaostory
DAU: 8M
MAU: 15M

5. Kakaostory
420M API CALL COUNT

6. Kakaostory Service Stack
•For Storage
•MariaDB(Master/Slave for HA)
•Hbase
•Cassandra
•For Cache
•Redis
•Arcus
• (Memcached variant, opensource, supporting collections)

7. Redis
5.2TB, 274 Servers
(Arcus: 3.3TB, 137 Servers)

8. Why Redis?
•As lookaside Cache for service data
•Example)
•User Profile Information
•Feeds
•Activities
•Friends
•Notifications

9. Agenda
•Single Threaded
•Memory Fragmentation
•Redis Troubleshooting cases
•Redis Monitoring
•Redis HA

10. Single Threaded

11. Redis Event Loop
Client #1
Client #2
……
Client #N
Redis Event Loop
I/O Multiplexing
Process
Command
command #1
command #2

12. Only One Command
at Once

13. Long-time Spending
operations

14. KEYS
FlushAll/FlushDB
LUA Script
MULTI/EXEC
Delete Collections

15. Why slow?

16. O(n)

17. KEYS – Iterating all Keys
di = dictGetSafeIterator(c->db->dict);
allkeys = (pattern[0] == '*' && pattern[1] == '0');
while((de = dictNext(di)) != NULL) {
……
stringmatchlen(pattern,plen,key,sdslen(key),0)
}

18. FlushAll – Deleting all items
for (i = 0; i < ht->size && ht->used > 0; i++) {
dictEntry *he, *nextHe;
if ((he = ht->table[i]) == NULL) continue;
while(he) {
nextHe = he->next;
dictFreeKey(d, he);
dictFreeVal(d, he);
zfree(he);
ht->used--;
he = nextHe;
}
}

19. How slow?

20. Command Item Count Time
flushall 1,000,000 1000ms(1 second)
FlushAll

21. Delete collections
Item Count Time
list 1,000,000 1000ms(1 second)
set 1,000,000 1000ms(1 second)
Sorted set 1,000,000 1000ms(1 second)
hash 1,000,000 1000ms(1 second)
You can use Xscan commands from 2.8.x

22. Using Multiple Instances
in a Physical Server
(can use more cpus)

23. Fork for
Creating RDB,
AOF Rewrite

24. Maximum 2x Memory
Disk IO
CPU Load/Usage

25. CPU 4 core, 32G Memory
Mem: 24G
Mem: 8G
Mem: 8G
Mem: 8G
more Reliable

26. Set CPU Affinity
using taskset

27. Divide NIC Interrupt CPU
and Redis Process CPU

28. Memory Fragmentation

29. Memory Fragmentation #1
Used_memory RSS

30. Memory Fragmentation #2
Used_memory RSS
Starting to use Arcus at this case

31. Redis Troubleshooting Cases

32. Problem #1
KEYS

33. Performance Spike

34. INFO all
# Commandstats
cmdstat_psetex:calls=2326667,usec=9322929,usec_per_call=4.01
……
cmdstat_pexpire:calls=3695333,usec=10068580,usec_per_call=2.72
cmdstat_keys:calls=249,usec=1000314022,usec_per_call=4017325.50
cmdstat_ping:calls=27005,usec=30027,usec_per_call=1.11
……

35. Slowlog get 10

36. rename KEYS Command

37. Using Scan

38. Redis Dict Structure

39. Scan #1

40. Scan #2

41. Scan #3

42. Problem #2
All Write Commands Fail

43. “MISCONF Redis is configured to save RDB
snapshots, but is currently not able to persist on
disk. Commands that may modify the data set are
disabled. Please check Redis logs for details about
the error.”

44. Reason
if (((server.stop_writes_on_bgsave_err &&
server.saveparamslen > 0 &&
server.lastbgsave_status == C_ERR) ||
server.aof_last_write_status == C_ERR) &&
server.masterhost == NULL &&
(c->cmd->flags & CMD_WRITE ||
c->cmd->proc == pingCommand))
{
…
}

45. config set stop-writes-on-bgsave-error no

46. Problem #3
Using Default Option

47. Redis as Cache

48. SAVE 900 1
SAVE 300 10
SAVE 60 10000

49. Heavy Disk IO
High Cpu Load
with creating RDB

50. Config set SAVE “”

51. Problem #4
Using Swap Memory

52. Redis using 28G
on single 32G machine

53. Migrate or Restart

54. Monitor Redis Server
and keep within bounds

55. Problem #5
Simultaneous AOF Rewrite

56. A 256GB Single Machine
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB

57. Simultaneous AOF Rewrite
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
Redis
26GB
AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite
AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite

58. Stop all AOF Rewrites

59. Turn off Automatic
AOF Rewrite

60. Config set auto-aof-rewrite-percentage 0

61. Manually Run AOF Rewrite

62. Problem #6
Replication is Broken with
Network Line Failure

63. All redis replication
are broken
by Network line failure

64. What Happens
if network
is recovered

65. Replication
Master Slave
replicationCron
Health check Periodically

66. All slaves automatically
try to reconnect to
master.

67. Slave of no one

68. Problem #7
Replication Failure

69. Permission

70. Memory Allocation Fail
sysctl vm.overcommit_memory=1

71. Replication Failure
with OutputBufferSize

72. Hard Limit
Soft Limit

73. config set client-output-buffer-limit
"slave 1024mb 1024mb 60"

74. Problem #8
Hash Table Expansion

75. Redis Dict – Hash Table Expansion #1

76. Redis Dict – Hash Table Expansion #2

77. Redis Dict – Hash Table Expansion #3

78. Grows by twice

79. Maxmemory
and
freeMemoryIfNeeded

80. 1 Billion items

81. 1,000,000,000 * 4 = 4G

82. Maxmemory = 16G
Used_memory = 12G

83. Hash Table Expansion
is needed.

84. 4G * 2 = 8G.
You need 20G(12G + 8G)

85. 20G > 16G(maxmemory)

86. Need a feature that can
Set Initial size of Hash
Table
(Not Supported)
https://github.com/antirez/redis/pull/2812

87. Redis Monitoring

88. Monitoring is
important as much as
Management

89. Redis Monitoring Metrics
Factor System or Redis Info
CPU Usage, Load System
Network Inbound/outbound System
Client connections
Maxclient setting
Info
Key size
Processed commands
Redis
Memory Usage, RSS(very
Important)
Redis
Disk Usage, IO System
Expired Keys, Evicted Keys Redis

90. Redis HA

91. Using DNS for Failover

92. Private Internal
DNS Server
with TTL 0

93. DNS HA Flow
Detect A
Redis
Failure
Change
B can write
Change DNS
A with B
Send A
Client Kill
New clients
Will connect
to B
B Config
rewrite

94. JVM
add –Dsun.net.inetaddr.ttl=0

95. twemproxy
using 0.4.1

96. Using
Coordinator

97. Zookeeper

98. Zookeeper with Redis Information

99. Zookeeper with Redis
Application Servers
ZooKeeper
Redis
Shard-1
Redis
Shard-2
Redis
Shard-3
Redis Cluster
Monitor
Get Redis Shard Information
Health Check
Update Shard
Info
Event: Node Add or Remove, Master change

100. Summary
•Redis is Single Threaded
•Creating RDB or AOF Rewrite is expensive
•Don’t use KEYS command.
•Don’t use default redis configuration.
•Monitoring is very importatnt.

101. Thanks