1. REDIS Management
& Cell Architecture
charsyam@naver.com

2. Redis.io

3. Redis/Twemproxy
Contributor
카카오 홈 개발자

4. Agenda
REDIS Management
REDIS HA
CELL Architecture

5. Agenda
REDIS Management
REDIS HA
CELL Architecture

6. Redis
Single Threaded
In Memory
Collections
Persistent
Replication

7. Redis is
Single Threaded!

8. Single Thread
Means!

9. Don’t execute
long task.

10. Example:
Keys command
Flushall/flushdb command

11. O(n)

12. Keys *
di = dictGetSafeIterator(c->db->dict);
allkeys = (pattern[0] == '*' && pattern[1] == '0');
while((de = dictNext(di)) != NULL) {
……
stringmatchlen(pattern,plen,key,sdslen(key),0)
}

13. Memcache’s flush
is faster than
Redis’s flush.

14. Memcache’s flush
is faster than
?
Redis’s flush

15. FlushAll-Redis
for (i = 0; i < ht->size && ht->used > 0; i++) {
dictEntry *he, *nextHe;
if ((he = ht->table[i]) == NULL) continue;
while(he) {
nextHe = he->next;
dictFreeKey(d, he);
dictFreeVal(d, he);
zfree(he);
ht->used--;
he = nextHe;
}
}

16. FlushAll-Memcache
if (exptime > 0)
settings.oldest_live = realtime(exptime) - 1;
else /* exptime == 0 */
settings.oldest_live = current_time - 1;

17. FlushAll-Memcache
if (settings.oldest_live != 0 &&
settings.oldest_live <= current_time &&
it->time <= settings.oldest_live) {
do_item_unlink(it, hv);
do_item_remove(it);
it = NULL;
}

18. FlushAll
Cache
Memcache
Redis
Item Count
1,000,000
1,000,000
Time
1~2ms
1000ms(1 second)

19. Redis
Single Threaded
In Memory
Collections
Persistent
Replication

20. In Memory
Fast but can’t store big data.
32bit: Maximum 3G Data in Memory
64bit: No Max memory.
using swap memory in OS.

21. Redis
Single Threaded
In Memory
Collections
Persistent
Replication

22. Collections
Memcached supports just K-V
List
Set
Sorted Set
Hash

23. Don’t insert too
many items into
one collections.

24. Delete collections
list
set
Sorted set
hash
Item Count
1,000,000
Time
1000ms(1 second)

25. Redis
Single Threaded
In Memory
Collections
Persistent
Replication

26. Redis can make
its memory
snapshot.

27. RDB is not
Relation DBMS.

28. RDB is redis
snapshot name.

29. Fork()

30. RDB - BAD
• Bad Performance in Large memory.
• Twice memory problem.
• Denying write problem when storing
RDB fails.
• If you just want Cache. Turn off RDB

31. RDB – Large Memory
• Performance is relevant to Memory
Size.
117 GB
68.4 GB
17.1 GB
34.2 GB

32. RDB – Twice Memory
• fork() and COW(copy on write) Issue
–In Write Heavy System:

33. RDB – Twice Memory

34. RDB – Twice Memory

35. RDB – Twice Memory

36. Real Case Study
• Background
–Can’t write to Redis Server
–Sentinel doesn’t find Server’s failure.

37. Real Case Study
• Reason
–If redis fails to save RDB, Redis basically
denies write operations from client.
–“MISCONF Redis is configured to save RDB
snapshots, but is currently not able to
persist on disk. Commands that may modify
the data set are disabled. Please check Redis
logs for details about the error.”

38. Real Case Study
• Reason
if (server.stop_writes_on_bgsave_err &&
server.saveparamslen > 0
&& server.lastbgsave_status == REDIS_ERR &&
c->cmd->flags & REDIS_CMD_WRITE)
{
flagTransaction(c);
addReply(c, shared.bgsaveerr);
return REDIS_OK;
}

39. Real Case Study
• Solution #1
config set stop-writes-on-bgsave-error no
• Solution #2
Turn off RDB Setting

40. AOF

41. AOF is append
only file.

42. AOF memorizes
all requests.

43. AOF
*3rn$3rnsetrn$1rnA
rn$3rn123rn*3rn$3rn
setrn$1rnBrn$3rn123
rn*3rn$3rnsetrn$1r
nCrn$3rn123rn

44. AOF
*3
$3
Set
$1
A
……

45. AOF Rewrite
When size of AOF grows than redis
rewrite memory state to AOF.

46. Redis
Single Threaded
In Memory
Collections
Persistent
Replication

47. Redis support
master/slave
replication

48. Replication
• Support Chained Replication
Master
st
1
Slave
nd
2
Slave
1st slave is master of 2nd slave

49. Replication
Master
Slave
replicationCron
Health check

50. Replication
Master
Slave
replicationCron
Health check

51. Replication
Master
Slave
replicationCron
When master reruns, Resync with Master

52. Mistake: Replication
If master has no data.
Master
Slave
replicationCron
Slave will has no data after resyncing

53. Replication
Don’t forget
“slave of no one”

54. When Redis forks…
1. BGSAVE(RDB)
2. AOF Rewrite(AOF)
3. Replication(RDB)
even though, you turned it off

55. Agenda
REDIS Management
REDIS HA
CELL Architecture

56. Redis Sentinel

57. What
Redis Sentinel
do?

58. 1. Check Redis Liveness

59. 1. Check Redis Liveness
-Subjective Down
-Objective Down

60. 2. Choose good slave.

61. Good Slave
NOT SDOWN, ODOWN, DISCONNECTED
NOT DEMOTE
Ping reply > info_validity_time
Slave_priority != 0
Info reply > info_validity_time

62. Choose new master
Sort
Slave Priority
Runid

63. 3. Promote it

64. Promoting
Send Slaveof no one to new master
Send Slaveof [new master ip] [addr] to
Other redis
Notify new master to clietns
+switch-master
Set DEMOTE mark to old-Master

65. Is Sentinel Really Good?

66. Not Mature #1

67. Sentinel Conf
port 26379
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 2000
sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 900000

68. Quorum Count
sentinel monitor mymaster 127.0.0.1 6379 2
We should check this
count.

69. Sentinel Conf
sentinel down-after-milliseconds mymaster 2000
If this value is Too small,
It can cause some trouble.
-sdown/+sdown loop

70. Compare with Zookeeper Conf
tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888

71. Not Mature #2

72. If redis is down, when
redis is loading data(RDB,
AOF), sentinel can’t
register redis.

73. Even though, Sentinel is
not mature.
It is useful.

74. Redis Failover

76. Twemproxy

77. Client
Client
Client
Client
Client
Client
hashing
Twemproxy
Redis
Redis
Redis

78. Twemproxy Pros
One connection per server
Zero Copy
Multi Hashing Algorithm
Support Nickname

79. Twemproxy Cons
Not Support Full Command
of Redis(Pub/Sub or MULTI)
Not Support HA

80. Twemproxy
L4
Redis
Twemproxy
Redis
Twemproxy
Redis

81. Redis-Sentinel
TwemProxy Agent
http://www.codeproject.com/Articles/656965/R
edis-Sentinel-TwemProxy-Agent

82. Agent
Pub/Sub Shard
Redis Master
Restart
VRRP
Twemproxy
HA Proxy
Redis Slave
Redis Slave
Twemproxy
HA Proxy
Twemproxy
Shard
Sentinel
Redis Master
Redis Slave
Redis Slave

83. Sharding

84. Sharding
WEB/AS
Master
User A Data
User C Data
User D Data
Master
Master
User B Data
User X Data
User Z Data
User Y Data
User E Data
User F Data

85. Cell Architecture
User
ID: CharSyam
CellID: 1, Status: Normal
Cell Info
Server
Get/set
Cell 0
Cell 1
Cell 2

86. Cell Architecture
A Cell is Full-Set
Can serve Users

87. Cell Examples #1
WEB/AS
WEB/AS
Master
Slave
WEB/AS

88. Cell Examples #2
WEB/AS
WEB/AS
WEB/AS
WRITE only
Master
Slave
READ only
Slave
Slave

89. Failure of Cell Architecture
User
ID: CharSyam
CellID: 1, Status: Normal
Cell Info
Server
Can’t Service
Cell 0
Cell 1
Cell 2

90. Failure of Cell Architecture
User
ID: CharSyam
Can’t response
Cell Info
Server
Get/set
Cell 0
Cell 1
Cell 2

91. Cell Architecture
• Benefits
–Easy to extend
–Failure is limited to some Users in same cell.
–Can deploy specific feature to some cell users.
• Liabilities
–To need more servers.
• To build full-set

92. Q&A

93. Thank you