More Related Content
Similar to Scaling Crashlytics: Building Analytics on Redis 2.6
Similar to Scaling Crashlytics: Building Analytics on Redis 2.6 (20)
Scaling Crashlytics: Building Analytics on Redis 2.6
- 2. Redis Analytics
@JeffSeibert
CEO, Crashlytics
2 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 3. 3 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 4. 4 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 8. Strings
Lists
Hashes
Sets
Sorted Sets
8 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 9. Strings Activity Tracking
Lists
Hashes Event Tracking
Sets
Sorted Sets Leader boards
9 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 11. Active User Tracking
CREATE TABLE accounts (
id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
name varchar(255),
email varchar(255),
...
last_active_at datetime
);
11 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 12. Active User Tracking
CREATE TABLE events (
id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
type varchar(32),
account_id int(11),
happened_at datetime
);
12 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 14. Active User Tracking
accounts::active
0 0 0 0 1 0 0 1
SETBIT key offset value (>= 2.2) O(1)
> SETBIT “accounts::active” 4 1
> SETBIT “accounts::active” 7 1
14 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 15. Active User Tracking
accounts::active::2012-10
1 1 1 0 1 0 1 1
accounts::active::2012-10-22
0 0 1 0 1 0 0 1
accounts::active::2012-10-22-00
0 0 0 0 1 0 0 1
15 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 16. Active User Tracking
def record_active(obj, t=Time.now.utc)
key = "#{obj.class.name.downcase.pluralize}::active::"
key << t.year.to_s
key << "-" << '%02d' % t.month
REDIS.setbit key, obj.id, 1 # accounts::active::2012-10
key << "-" << '%02d' % t.day
REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22
key << "-" << '%02d' % t.hour
REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22-00
end
16 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 17. Active User Tracking
‣ We want to know…
• How many users were active today? This month?
BITCOUNT key (>= 2.6) O(N)
> BITCOUNT “accounts::active::2012-10-22”
(integer) 3
> BITCOUNT “accounts::active::2012-10”
(integer) 5
• Was user X active today? This month?
GETBIT key index (>= 2.2) O(1)
> GETBIT “accounts::active::2012-10-22” 6
(integer) 0
> GETBIT “accounts::active::2012-10” 6
(integer) 1
17 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 18. Active User Tracking
‣ Graphs and Heatmaps
• Monthly actives over time?
> BITCOUNT “accounts::active::2012-07”
> BITCOUNT “accounts::active::2012-08”
> BITCOUNT “accounts::active::2012-09”
> BITCOUNT “accounts::active::2012-10”
...
• Over time, when was user X active?
> GETBIT “accounts::active::2012-10-22” 6
> GETBIT “accounts::active::2012-10-21” 6
> GETBIT “accounts::active::2012-10-20” 6
> GETBIT “accounts::active::2012-10-19” 6
...
18 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 19. Active User Tracking
‣ Advanced Data-Mining: WAU
• Computing weekly active users:
BITOP op destkey srckey [srckeys...] (>= 2.6) O(N)
• > BITOP OR “accounts::active::2012-W42”
“accounts::active::2012-10-21”
“accounts::active::2012-10-20”
“accounts::active::2012-10-19”
“accounts::active::2012-10-18”
“accounts::active::2012-10-17”
“accounts::active::2012-10-16”
“accounts::active::2012-10-15”
> BITCOUNT “accounts::active::2012-W42”
19 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 20. Active User Tracking
‣ Advanced Data-Mining: Retention
• What % of users active last week are active this week?
BITOP op destkey srckey [srckeys...] (>= 2.6) O(N)
• > BITOP AND “accounts::active::2012-W41+W42”
“accounts::active::2012-W41”
“accounts::active::2012-W42”
> BITCOUNT “accounts::active::2012-W41”
> BITCOUNT “accounts::active::2012-W41+W42”
20 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 21. Active User Tracking
‣ Advanced Data-Mining: Churn
• Locate accounts that have been inactive for 3 months
BITOP op destkey srckey [srckeys...] (>= 2.6) O(N)
• > BITOP OR “accounts::active::2012-Q3”
“accounts::active::2012-09”
“accounts::active::2012-08”
“accounts::active::2012-07”
> BITOP NOT “accounts::churned::2012-Q3”
“accounts::active::2012-Q3”
> BITCOUNT “accounts::churned::2012-Q3”
21 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 22. Active User Tracking
def record_boolean(obj, topic=:active, t=Time.now.utc)
key = "#{obj.class.name.downcase.pluralize}::#{topic}::"
key << t.year.to_s
key << "-" << '%02d' % t.month
REDIS.setbit key, obj.id, 1 # accounts::active::2012-10
key << "-" << '%02d' % t.day
REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22
key << "-" << '%02d' % t.hour
REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22-00
end
22 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 24. Event Tracking
apps::crashes
0 0 0 0 ? 0 0 0
24 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 25. Event Tracking
apps::crashes {
0 => 34,
1 => 546457,
2 => 1
}
HINCRBY key field increment (>= 2.0) O(1)
> HINCRBY “apps::crashes” “0” 1
> HINCRBY “apps::crashes” “2” 1
25 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 26. Event Tracking
app::0::crash::by_day {
2012-10-22 => 34,
2012-10-21 => 46,
2012-10-20 => 29,
...
}
> HINCRBY “app::0::crash::by_day” “2012-10-22” 1
26 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 27. Event Tracking
def record_event(obj, topic=:crash, specificity=:day, t=Time.now.utc)
key = "#{obj.class.name.downcase}::#{obj.id}::#{topic}::by_#{specificity}"
# e.g. app::0::crash::by_day
field = t.year.to_s
field << "-" << '%02d' % t.month # 2012-10
REDIS.hincrby key, field, 1 if specificity == :month
field << "-" << '%02d' % t.day # 2012-10-22
REDIS.hincrby key, field, 1 if specificity == :day
field << "-" << '%02d' % t.hour # 2012-10-22-00
REDIS.hincrby key, field, 1 if specificity == :hour
end
27 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 28. Event Tracking
‣ We want to…
• Power a graph of crashes over the last week
HMGET key field1 [...] (>= 2.0) O(N)
> HMGET “app::0::crash::by_day” “2012-10-22”
“2012-10-21” “2012-10-20” “2012-10-19”
“2012-10-18” “2012-10-17” “2012-10-16”
1) ...
• “Zoom” the graph to see more detail
> HMGET “app::0::crash::by_hour” “2012-10-22-00”
“2012-10-22-01” “2012-10-22-02” “2012-10-22-03”
“2012-10-22-04” “2012-10-22-05” “2012-10-22-06” ...
1) ...
28 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 29. Grouped Event Tracking
“How often has app X crashed
on each type of iPad?”
29 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 30. Grouped Event Tracking
app::0::crash::iPad1,1 { device_models [
2012-10-22 => 34, “iPad1,1”,
2012-10-21 => 46, “iPad2,1”,
2012-10-20 => 29, ...
... ]
}
app::0::crash::iPad2,1 {
2012-10-22 => 12,
2012-10-21 => 17,
2012-10-20 => 11,
...
}
30 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 31. Grouped Event Tracking
app::0::crash::2012-10-22 {
ALL => 46,
iPad1,1 => 34,
iPad2,1 => 12,
...
}
HGETALL key (>= 2.0) O(N)
> HGETALL “app::0::crash::2012-10-22”
(multi-bulk)
31 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 32. Grouped Event Tracking
def record_grouped_event(obj, group, topic=:crash, t=Time.now.utc)
key = "#{obj.class.name.downcase}::#{obj.id}::#{topic}::"
key = t.year.to_s
key << "-" << '%02d' % t.month # app::0::crash::2012-10
REDIS.hincrby key, group, 1
REDIS.hincrby key, 'ALL', 1
field << "-" << '%02d' % t.day # app::0::crash::2012-10-22
REDIS.hincrby key, group, 1
REDIS.hincrby key, 'ALL', 1
field << "-" << '%02d' % t.hour # app::0::crash::2012-10-22-00
REDIS.hincrby key, group, 1
REDIS.hincrby key, 'ALL', 1
end
32 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 33. MongoDB
> Account.first.id
=> BSON::ObjectId('507db04798a3340ada000002')
33 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 34. Sequential ID Generation
sequential_ids::accounts {
10 5084bfbb98a33406f0000002,
9 5084bfa798a33406f0000001,
8 507db04798a3340ada000002,
...
}
ZADD key score member (>= 1.2) O(log(N))
> ZADD “sequential_ids::accounts” 10 507db04798a3340ada000002
(integer) 1
34 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 35. Sequential ID Generation
sequential_ids::accounts {
10 5084bfbb98a33406f0000002,
9 5084bfa798a33406f0000001,
8 507db04798a3340ada000002,
...
}
ZCARD key (>= 1.2) O(1)
> ZCARD “sequential_ids::accounts”
(integer) 9
ZADD key score member (>= 1.2) O(log(N))
> ZADD “sequential_ids::accounts” 10 5084bfbb98a33406f0000002
(integer) 1
35 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 36. Sequential ID Generation
sequential_ids::accounts {
10 5084bfbb98a33406f0000002,
9 5084bfa798a33406f0000001,
8 507db04798a3340ada000002,
...
}
ZSCORE key member (>= 1.2) O(1)
> ZSCORE “sequential_ids::accounts” 5084bfbb98a33406f0000002
(integer) 10
36 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 37. Sequential ID Generation
def sequential_id(obj)
key = "sequential_keys::#{obj.class.name.downcase.pluralize}"
id = obj.id.to_s
# Lua script to atomically determine the score of an id.
# If needed, adds it to the set with the next available score.
# In the general case, O(1). On add, O(log(N)). Requires Redis >= 2.6
monotonic_zadd = <<LUA
local sequential_id = redis.call('zscore', KEYS[1], ARGV[1])
if not sequential_id then
sequential_id = redis.call('zcard', KEYS[1])
redis.call('zadd', KEYS[1], sequential_id, ARGV[1])
end
return sequential_id
LUA
REDIS.eval(monotonic_zadd, [key], [id]).to_i
end
37 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 39. Redis Analytics Wish List
‣ MSETBIT, MGETBIT, MBITCOUNT, HMINCRBY
• Can already be addressed with scripting
‣ Native support for (insertion-)ordered sets
‣ Per-hash-key expiration policies
39 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
- 40. Q&A
@JeffSeibert
CEO, Crashlytics
40 CRASHLYTICS CONFIDENTIAL © 2012. All rights reserved
