Supports session on NetPonto community to demonstrates Redis features such as data types, pipeline, transactions (MULTI) and scripting

1. Introducing Redis
Nuno Caneco
http://netponto.org64ª Reunião Presencial - 24/OUT/2016

2. Agenda
• Introducing Redis
• Data Structures
•Pipelining and "transactions"
•Publish / Subscriber
• Scripting

3. Introducing Redis

4. What is Redis?
Redis is an open source (BSD licensed), in-memory data
structure store, used as database, cache and message
broker.
It supports data structures such as strings, hashes, lists,
sets,sorted sets with range queries, bitmaps, hyperloglogs
and geospatial indexes with radius queries.
Redis has built-in replication, Lua scripting, LRU eviction,
transactions and different levels of on-disk persistence, and
provides high availability via Redis Sentinel and automatic
partitioning with Redis Cluster.
-- www.redis.io

5. What can it be used for?
• State Manager
• Cache Provider
– Application Data
– Sessions
– …
• Queuing
• Publisher / Subcriber storage point

6. It’s just a big hash map
Key Value
"pageSize" (String) 10
"onlineUsers" (Set) ["user:10", "user:24", "user:89"]
"user:10" (Hash) [{"name": "John Doe"}, {"login": "jdoe"}]
...
• Insert: O(1)
• Get by key: O(1)
• Keys lookups: O(n) n=keys

7. Redis Keys
• Redis Keys are Strings
– All Strings are binary safe
• Some recommendations:
– Avoid long keys (> 1024 bytes)
– Avoid very short keys (u100 vs user:100)
– Stick to a convention (e.g. user:100)

8. Redis Data Types

9. String
• Simplest basic type
• Can be interpreted as
numerics
– Allow add/subtract
operations
Documentation
• GET, SET
• SETNX
• SETEX
• MSET, MGET
• EXISTS
• DEL
For numerics:
• INCR, INCRBY
• DECR, DECRBY

10. String: Set and get
# set USD to EUR exchange rate
> SET exchangeRate:USD:EUR 0.9107
OK
# Get USD to EUR exchange rate
> GET exchangeRate:USD:EUR
"0.9107"
# Set multiple values at once
> MSET exchangeRate:USD:EUR 0.9107 exchangeRate:EUR:USD 1,0981
OK
> MGET exchangeRate:USD:EUR exchangeRate:EUR:USD
1) "0.9107"
2) "1,0981"

11. String: If not exists
> SETNX availableSeats 10
(integer) 1
> GET availableSeats
"10"
> INCRBY availableSeats -1
(integer) 9
> INCRBY availableSeats -1
(integer) 8
> GET availableSeats
"7"
> SETNX availableSeats 10 #Won't set the key
(integer) 0
> GET availableSeats
"7"

12. String: Use it as a counter
> INCR visitsCount
(integer) 1
> GET visitsCount
"1"
> INCRBY visitsCount 5
(integer) 6
# It can also decrement a counter
> INCRBY visitsCount -2
(integer) 4

13. List
•Collection of String elements
•Sorted according to the order of
insertion
A B C D
HEAD TAIL
List

14. List
• Commands are prefixed with
"L"
• Implemented as Linked List
– Op on Head or Tail: O(1)
– Op on Middle: O(n)
– Lookup: O(n)
• Enables implementation of
Queue
Documentation
> LSET
> LINDEX
> LLEN
> LINSERT
> LREM
> LTRIM
> LRANGE
> LPUSH/RPUSH
> LPOP/RPOP
> LRANGE
> RPOPLPUSH

15. List: Circular List
Use RPOPLPUSH with same source
and destination
> RPUSH list "a" "b" "c"
(integer) 3
> LRANGE list 0 -1
1) "a"
2) "b"
3) "c"
> RPOPLPUSH list list
"c"
> LRANGE list 0 -1
1) "c"
2) "a"
3) "b"
RPOPLPUSH returns the element that
is being exchanged between the lists

16. List: Reliable Queue
Queue
Processing
queue
Producer Consumer
LPUSH RPOPLPUSH
M5 M4 M3 M2 M1
LREM
Processing
message
NOTE: The consumer can call BRPOPLPUSH if it wishes to be blocked if the
queue is empty

17. Set
Collection of String:
• unique, non-repeating String
• unsorted
D
A
C
B
Set

18. Set
• Commands are prefixed
with "S"
Documentation
> SADD
> SCARD
> SISMEMBER
> SMEMBERS
> SRANDMEMBER
> SREM
> SSCAN
> SPOP
> SDIFF, SDIFFSTORE
> SMOVE
> SINTER, SINTERSTORE
> SUNION, SUNIONSTORE

19. Set: Example
> SADD prog-lang:alice C Java C# Ruby
(integer) 4
# Does Alice knows how to program in C?
> SISMEMBER prog-lang:alice C
(integer) 1 # Yes
# Does Alice knows how to program in Python?
> SISMEMBER prog-lang:alice Python
(integer) 0 # No
# So, which programming languages does Alice Know?
> SMEMBERS prog-lang:alice
1) "C#"
2) "C"
3) "Ruby"
4) "Java"

20. Set: Multiple Sets Example
> SADD prog-lang:bob C Python Ruby
(integer) 3
# Which languages both Bob and Alice know about?
> SINTER prog-lang:alice prog-lang:bob
1) "C"
2) "Ruby"
# Which languages could both Bob and Alice work on, if they teamed up?
> SUNION prog-lang:alice prog-lang:bob
1) "Java"
2) "Ruby"
3) "Python"
4) "C#"
5) "C"
# OK, so let's create a team that combines both their skills
> SUNIONSTORE prog-lang:team:1 prog-lang:alice prog-lang:bob
(integer) 5

21. Set: Random pick / pop
# Add a deck of cards
> SADD deck C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 CJ CQ CK D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 DJ DQ DK H1 H2 H3 H4 H5 H6 H7
H8 H9 H10 HJ HQ HK S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SJ SQ SK
(integer) 52
# Pick a random card, without taking it out the deck
> SRANDMEMBER deck
"H7"
# Create a game with the default deck
> SUNIONSTORE game:1:deck deck
(integer) 52
# Pick a card from the game deck
> SPOP game:1:deck
"D9" # A nine of Diamonds
# Now, the game deck has only 51 cards
> SCARD game:1:deck
(integer) 51

22. Sorted Set
Collection of Strings
• Unique, non-repeating
• Sorted by Score (float)
Set
E13
E26
E45
E39

23. Sorted Set
• Commands are prefixed
with "Z"
Documentation
> ZADD
> ZRANGE (BYLEX, BYSCORE)
> ZREM
> ZRANK
> ZREMRANGEBYLEX, (BYRANK, BYSCORE)
> ZCOUNT
> ZREVRANGE (BYLEX, BYSCORE)
> ZREVRANK
> ZSCAN
> ZSCORE
> ZINCRBY
> ZINTERSTORE
> ZUNIONSTORE

24. Sorted Set: Using Rank
Set
E13
E26
E45
E39
> ZSCORE set1 E1
"3"
> ZSCORE set1 E4
"5"
> ZRANK set1 E4
(integer) 1
> ZRANK set1 E1
(integer) 0
> ZREVRANK set1 E4
(integer) 2
> ZREVRANK set1 E1
(integer) 3
(*) Score starts at zero

25. Sorted Set: Example
> ZADD prog-lang-rank:alice 40 C 60 Java 80 C# 30 Ruby
# What is Alice's rank of C#
> ZREVRANK prog-lang-rank:alice C#
(integer) 0 # C# is what she knows best
# Top 2 languages for Alice
> ZREVRANGE prog-lang-rank:alice 0 1
1) "C#"
2) "Java"
# What are the programming languages for which Alice scores at least 50?
> ZREVRANGEBYSCORE prog-lang-rank:alice 100 50
1) "C#"
2) "Ruby"
3) "C"

26. Sorted Set: Much wow (!)
> ZADD prog-lang-rank:bob 30 C 50 Python 50 Ruby
# If we created a team with Alice and Bob, what would be the MAX score for each language?
> ZUNIONSTORE prog-lang-rank:team:1 2 prog-lang-rank:alice prog-lang-rank:bob AGGREGATE MAX
(integer) 5 # Their skills combined can work on projects using 5 different languages
> ZREVRANGE prog-lang-rank:team:1 0 -1 WITHSCORES
1) "C#"
2) "80"
3) "Java"
4) "60"
5) "Ruby"
6) "50"
7) "Python"
8) "50"
9) "C"
10) "40"

27. Hash
Collection of field-value pairs:
•Each field is unique
•Suitable to represent objects
Field Value
key1 value1
key2 value2

28. Hash
• Commands are prefixed
with "H"
• HINCRBY allows to
atomically increment or
decrement a value
Documentation
> HSET, HSETNX, HMSET
> HGET, HGETALL
> HKEYS
> HDEL
> HEXISTS
> HLEN
> HVALS
> HINCRBY, HINCREBYFLOAT

29. Hash: Using it to store objects
> HMSET lang:pt-PT locale "pt-PT" dateFormat "YYYY-MM-dd HH:mm:ss" currencyFormat "#.## EUR"
> HMSET lang:en-US locale "en-US" dateFormat "MM-YYYY-dd hh:mm:ss tt" currencyFormat "#.## USD"
> HGET lang:pt-PT dateFormat
"YYYY-MM-dd HH:mm:ss"
> HGETALL lang:pt-PT
1) "locale"
2) "pt-PT"
3) "dateFormat"
4) "YYYY-MM-dd HH:mm:ss"
5) "currencyFormat"
6) "#.## EUR"

30. Hash: Incrementing values
> HMSET comment:1001 text "I just love Redis" votes 0
> HINCRBY comment:1001 votes 1
> HGETALL comment:1001
1) "text"
2) "I just love Redis"
3) "votes"
4) "1"
HINCRBY performs atomic increments or decrements on an Hash Value

31. Bitmap
Set of bitwise instructions
implemented over (binary
safe) String
0 1 0 1 1 0 0 0 1
> SETBIT
> GETBIT
> BITOP (AND, OR, XOR and NOT)
> BITCOUNT
> BITPOS
> BITFIELD

32. Expiring entries
Entries can be configured to expire automatically
# Cache USD to EUR exchange rate for 1 hour (inline)
> SET exchangeRate:USD:EUR 0.9107 EX 3600
OK
# Cache an hash/object for 10 minutes
> HMSET user:100 name "John Doe" username jdoe
OK
> EXPIRE user:100 600
(integer) 1

33. Command batch and "transactions"

34. It DOES NOT ensure that all commands are run as a single
atomic set of commands
Pipelining
Most APIs allow to perform command pipeline.
Optimizes network usage, since all commands in a
batch or pipeline are sent all at once.

35. Pipelining
var batch = redisDb.CreateBatch();
// Commands will be kept pending on the client side
var counter1Value = batch.StringIncrementAsync("counter:1", 1);
var counter2Value = batch.StringIncrementAsync("counter:2", 10);
batch.Execute(); // Instructs the batch that all commands are ready
// When accessing the first item of the command, the entire batch is executed
Console.WriteLine("counter1 = {0}", counter1Value.Result);
Console.WriteLine("counter2 = {0}", counter2Value.Result);
"INCR" "counter:1"
"INCRBY" "counter:2" "10"

36. MULTI
Performs a set of commands as a
single atomic operation
MULTI
OK
INCR totalVisits
QUEUED
HINCRBY customer:10 totalvisits 1
QUEUED
EXEC # Both commands will be executed atomically at this point
1) (integer) 2 # The results of the commands are outputed from EXEC
2) (integer) 1

37. Publish/Subscriber

38. Publish/Subscribe
Allows implementation of
Publisher / Subscriber
pattern.
• Multiple publishers
• Multiple subscribers
Documentation
➢ PSUBSCRIBE
➢ PUBLISH
➢ PUBSUB
➢ PUNSUBSCRIBE
➢ SUBSCRIBE
➢ UNSUBSCRIBE

39. Publish/Subscribe
client1> SUBSCRIBE somefeed
Reading messages... (press Ctrl-C to
quit)
1) "subscribe"
2) "somefeed"
3) (integer) 1
1) "message"
2) "somefeed"
3) "Hi there!!"
client2> PUBLISH somefeed "Hi there!"
(integer) 1

40. Scripting

41. Scripting
Redis allows loading
and executing LUA
scripts
Scripts are executed
atomically.
> EVAL
> EVALSHA
> SCRIPT EXISTS
> SCRIPT FLUSH
> SCRIPT KILL
> SCRIPT LOAD

42. Scripting
# Execute an inline script
> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 first
second
1) "key1"
2) "key2"
3) "first"
4) "second"
# Load a script and execute it later
> set foo bar
OK
> eval "return redis.call('get','foo')" 0
"bar"
> evalsha 6b1bf486c81ceb7edf3c093f4c48582e38c0e791 0
"bar"

43. Performance Considerations

44. Commands performance
O(1) Not O(1)
General
Purpose
DEL
EXISTS
EXPIRE / PEXPIRE / PEXPIREAT
MOVE
PTTL
RENAME / RENAMENX
KEYS
SCAN
String GET
SET*
GETSET
INCRBY*
STRLEN
MGET
MSET*
GETRANGE
Sets SADD
SREM
SPOP
SCARD
SISMEMBER
SMOVE
SRANDMEMBER
SDIFF
SDIFFSTORE
SINTER
SINTERSTORE
SSCAN
SUNION
SUNIONSTORE

45. Commands performance
O(1) Not O(1)
Sorted Sets ZADD
ZREM
ZCARD
SISMEMBER
SMOVE
SRANDMEMBER
ZCOUNT
ZINCRBY
ZRANGE
ZRANK (O(log(n))
ZINTERSTORE
ZSCAN
ZUNIONSTORE
Hash Sets HGET
HMGET
HSET / HSETNX
HMSET
HINCRBY
HSTRLEN
HLEN
HGETALL (O(size of hash))
HKEYS (O(size of hash))
HKEYS
HSCAN

46. Tips and pitfalls
DO NOT:
• Call KEYS or SCAN
– You can keep track of keys on a SET
• Call MONITOR
DO:
• Use the NX variants instead of calling EXISTS
than SET to initialize keys

47. Questions?

48. References
Redis Website
– http://www.redis.io
Redis Commands Documentation
– http://redis.io/commands
Redis Documentation
– http://redis.io/documentation
Statistics LUA Script
– https://gist.github.com/thomasdarimont/60eefb3530f48de3d650

49. 64ª Reunião Presencial @ LISBOA
DateTime.Parse(”24-09-2016", new CultureInfo("pt-PT"));
http://netponto.org
hashtag #netponto

50. Patrocinadores “GOLD”
Twitter: @PTMicrosoft http://www.microsoft.com/portugal

51. Patrocinadores “Silver”

52. Patrocinadores “Bronze”

53. http://bit.ly/netponto-aval-63
* Para quem não puder preencher durante a reunião,
iremos enviar um email com o link à tarde

54. Próximas reuniões presenciais
24/09/2016 – Lisboa
22/10/2016 – Lisboa
19/11/2016 – Lisboa
17/12/2016 – Lisboa
Reserva estes dias na agenda! :)

55. Obrigado!
Nuno Caneco
nuno.caneco@gmail.com
http://twitter.com/ncaneco
https://pt.linkedin.com/in/nunocaneco