SlideShare a Scribd company logo
1 of 21
Download to read offline
"Little Server of Awesome"


       2011 Dvir Volk
      Software Architect, DoAT
   dvir@doat.com http://doat.com
Redis: Key => DataStructure Server

● Memcache-ish in-memory key/value store
● But values are complex data types:
   ○ blobs
   ○ lists
   ○ sets
   ○ sorted sets
   ○ hash tables

● And it has persistence.
● Open source; very helpful and friendly community.
● Used in the real world: github, craigslist, bit.ly, engineyard...
● Used heavily in doAT as a front-end database, search, geo
  resolving
Key Features and Cool Stuff
● All data is in memory (almost)

● All data is eventually persistent

● Handles huge workloads easily

● Mostly O(1) behavior, Ideal for write-heavy workloads

● Support for atomic operations, transactions

● Has pub/sub functionality

● Single threaded, uses aync. IO
● Internal scripting with LUA in alpha
A little benchmark

This is on my laptop (core i7 @2.2Ghz)

 ● SET: 187265.92 requests per second
 ● GET: 185185.17 requests per second
 ● INCR: 190114.06 requests per second


 ● LPUSH: 190114.06 requests per second
 ● LPOP: 187090.73 requests per second
 ●
 ● SADD: 186567.16 requests per second
 ● SPOP: 185873.61 requests per second
Scaling it up

 ● Master-slave replication out of the box

 ● Slaves can be made masters on the fly

 ● Redis-cluster in alpha state.

 ● Sharding supported by some clients

 ● Single threaded - run num_cores/2 instances on the same
   machine
Persistence
● All data is synchronized to disk - eventually or immediately

● Pick your risk level Vs. performance

● Data is either dumped in a forked process, or written as a
  append-only change-log (AOF)

● Append-only mode supports transactional disk writes so you
  can lose no data (cost: 99% speed loss :) )

● You can save the state explicitly, background or blocking

● If the data is too big to fit in memory - redis can handle its
 own swap.
Show me the features!
The basics...
Get/Sets - nothing fancy. Keys are strings, anything goes - just quote spaces.
redis> SET foo "bar"
OK
redis> GET foo
"bar"

You can atomically increment numbers
redis> SET bar 337
OK
redis> INCRBY bar 1000
(integer) 1337

Getting multiple values at once
redis> MGET foo bar
1. "bar"
2. "1337"

Keys are lazily expired
redis> EXPIRE foo 1
(integer) 1
redis> GET foo
(nil)
Be careful with EXPIRE - re-setting a value without re-expiring it will remove the
expiration!
Atomic Operations
GETSET puts a different value inside a key, retriving the old one
redis> SET foo bar
OK
redis> GETSET foo baz
"bar"
redis> GET foo
"baz"

SETNX sets a value only if it does not exist
redis> SETNX foo bar
*OK*
redis> SETNX foo baz
*FAILS*
List operations
 ● Lists are your ordinary linked lists.
 ● You can push and pop at both sides, extract range, resize..
 ● Random access and ranges at O(N)
 ● BLPOP: Blocking POP - wait until a list has elements and pop
   them. Useful for realtime stuff.
Set operations
 ● Sets are... well, sets of unique values w/ push, pop, etc.
 ● Sets can be intersected/diffed /union'ed server side.
 ● Can be useful as keys when building complex schemata.
Sorted Sets
 ● Same as sets, but with score per element
 ● Ranked ranges, aggregation of scores on INTERSECT
 ● Can be used as ordered keys in complex schemata
 ● Think timestamps, inverted index, geohashing, ip ranges
Hashes
 ● Hash tables as values
 ● Think of an object store with atomic access to object
   members

 redis> HSET foo bar 1             redis> HINCRBY foo bar 1
 (integer) 1                       (integer) 2
 redis> HSET foo baz 2
 (integer) 1                       redis> HGET foo bar
 redis> HSET foo foo foo           "2"
 (integer) 1
                                   redis> HKEYS foo
 redis> HGETALL foo                1. "bar"
 {                                 2. "baz"
   "bar": "1",                     3. "foo"
   "baz": "2",
   "foo": "foo"
 }
PubSub - Publish/Subscribe
 ● Clients can subscribe to channels or patterns and receive
   notifications when messages are sent to channels.
 ● Subscribing is O(1), posting messages is O(n)
 ● Think chats, Comet applications: real-time analytics, twitter
Transactions
  ● MULTI, ...., EXEC: Easy because of the single thread.
  ● All commands are executed after EXEC, block and return
    values for the commands as a list.
  ● Example:
redis> MULTI
OK
redis> SET "foo" "bar"
QUEUED
redis> INCRBY "num" 1
QUEUED
redis> EXEC
1) OK
2) (integer) 1

  ● Transactions can be discarded with DISCARD.

  ● WATCH allows you to lock keys while you are queuing your
    transaction, and avoid race conditions.
Gotchas, Lessons Learned
● 64 bit instances consume much much more RAM.

● Master/Slave sync far from perfect.

● DO NOT USE THE KEYS COMMAND!!!

● vm.overcommit_memory = 1

● Use MONITOR to see what's going on
Recipe 1: Social Feed
 ● Each User's followers are list: user:$ID:followers => { 1, 2, 4 }

 ● Each User's feed is a sorted set with the time-stamp being the
   score: user:$ID:feed => { msg1234: 124950132, ... }

 ● Each message is either a blob or a HASH object with fields:
   msg1234 => {title, text, ... }

 ● When a message is added, we:
     ○ add the message
     ○ read the posting user's follower list
     ○ add the message id to each follower's feed
 ● When a user reads their feeds, we use ZREVRANGE to get the
   latest item ids, and HGETALL to fetch the message texts.

 ● For bonus points, connected users can receive push notifications
   with PubSub
Recipe 2: Real-Time Analytics
 ● We want to measure page views, per page, one minute resolution,
   in real time.
 ● Each page is represented by a sorted set, where the value is the
   minute, and the score is the number of hits.

 ● On each page hit, we do ZINCRBY <page> <current_minute> 1

 ● Page views for the last N minutes: ZREVRANGE <page> 0 N

 ● Group several pages together: ZUNIONSTORE ... AGGREGATE
   SUM

 ● Trimming for a specific window: ZREMRANGEBYRANK

 ● Again, PubSub can be used to push updates to clients
Recipe 3: Processing Queue
● Each task can be a blob describing it with json, using
  SET/GET/DEL.

● We keep a list of pending tasks, or multiple lists per task
  type/priority ("tasks:urgent")

● Worker procs can work from any machine over the network.

● Workers poll task queues using BRPOP tasks:urgent
● atomicity guaranteed!

● When a task is pulled it can be marked on a different list/set of
  "tasks:inprogress". When a task is done it is moved to a "done"
  or "failed" task list.

● A manager can monitor failed tasks and move them around.
Other use case ideas
● Geo Resolving with geohashing
● Implemented and opened by yours truly https://github.com/doat/geodis

● API Key and rate management
● Very fast key lookup, rate control counters using INCR

● Real time game data
● ZSETs for high scores, HASHES for online users, etc

● Database Shard Index
● map key => database id. Count size with SETS

● Comet - no polling ajax
● use BLPOP or pub/sub

● Machine learning data with automatic persistence.
● sorted sets are your vectors!
More resources

Redis' website:
http://redis.io

Excellent and more detailed presentation by Simon Willison (a
bit old):
http://simonwillison.net/static/2010/redis-tutorial/

Much more complex twitter clone:
http://code.google.com/p/redis/wiki/TwitterAlikeExample

Full command reference:
http://redis.io/commands

More Related Content

What's hot

Caching solutions with Redis
Caching solutions   with RedisCaching solutions   with Redis
Caching solutions with RedisGeorge Platon
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redisTanu Siwag
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcachedJurriaan Persyn
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to RedisDvir Volk
 
Paris Redis Meetup Introduction
Paris Redis Meetup IntroductionParis Redis Meetup Introduction
Paris Redis Meetup IntroductionGregory Boissinot
 
MySQL GTID 시작하기
MySQL GTID 시작하기MySQL GTID 시작하기
MySQL GTID 시작하기I Goo Lee
 
RedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ TwitterRedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ TwitterRedis Labs
 
Redis persistence in practice
Redis persistence in practiceRedis persistence in practice
Redis persistence in practiceEugene Fidelin
 
An Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdfAn Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdfStephen Lorello
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to RedisKnoldus Inc.
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesYoshinori Matsunobu
 
Tuning Autovacuum in Postgresql
Tuning Autovacuum in PostgresqlTuning Autovacuum in Postgresql
Tuning Autovacuum in PostgresqlMydbops
 
MyRocks introduction and production deployment
MyRocks introduction and production deploymentMyRocks introduction and production deployment
MyRocks introduction and production deploymentYoshinori Matsunobu
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDBMike Dirolf
 

What's hot (20)

Caching solutions with Redis
Caching solutions   with RedisCaching solutions   with Redis
Caching solutions with Redis
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
 
Redis and it's data types
Redis and it's data typesRedis and it's data types
Redis and it's data types
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Paris Redis Meetup Introduction
Paris Redis Meetup IntroductionParis Redis Meetup Introduction
Paris Redis Meetup Introduction
 
Redis Persistence
Redis  PersistenceRedis  Persistence
Redis Persistence
 
Redis database
Redis databaseRedis database
Redis database
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
 
MySQL GTID 시작하기
MySQL GTID 시작하기MySQL GTID 시작하기
MySQL GTID 시작하기
 
RedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ TwitterRedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ Twitter
 
Redis basics
Redis basicsRedis basics
Redis basics
 
Redis persistence in practice
Redis persistence in practiceRedis persistence in practice
Redis persistence in practice
 
An Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdfAn Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdf
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
 
Tuning Autovacuum in Postgresql
Tuning Autovacuum in PostgresqlTuning Autovacuum in Postgresql
Tuning Autovacuum in Postgresql
 
Redis 101
Redis 101Redis 101
Redis 101
 
MyRocks introduction and production deployment
MyRocks introduction and production deploymentMyRocks introduction and production deployment
MyRocks introduction and production deployment
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
 

Similar to Introduction to redis - version 2

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to RedisRizky Abdilah
 
Dragoncraft Architectural Overview
Dragoncraft Architectural OverviewDragoncraft Architectural Overview
Dragoncraft Architectural Overviewjessesanford
 
Logs @ OVHcloud
Logs @ OVHcloudLogs @ OVHcloud
Logs @ OVHcloudOVHcloud
 
Introduction to AWS Big Data
Introduction to AWS Big Data Introduction to AWS Big Data
Introduction to AWS Big Data Omid Vahdaty
 
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Hernan Costante
 
Redis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamRedis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamCodemotion
 
Redis for duplicate detection on real time stream
Redis for duplicate detection on real time streamRedis for duplicate detection on real time stream
Redis for duplicate detection on real time streamRoberto Franchini
 
Ledingkart Meetup #2: Scaling Search @Lendingkart
Ledingkart Meetup #2: Scaling Search @LendingkartLedingkart Meetup #2: Scaling Search @Lendingkart
Ledingkart Meetup #2: Scaling Search @LendingkartMukesh Singh
 
PostgreSQL: present and near future
PostgreSQL: present and near futurePostgreSQL: present and near future
PostgreSQL: present and near futureNaN-tic
 
Presentations from the Cloudera Impala meetup on Aug 20 2013
Presentations from the Cloudera Impala meetup on Aug 20 2013Presentations from the Cloudera Impala meetup on Aug 20 2013
Presentations from the Cloudera Impala meetup on Aug 20 2013Cloudera, Inc.
 
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQDocker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQJérôme Petazzoni
 
Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Jérôme Petazzoni
 
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...Hisham Mardam-Bey
 
Ceph Day New York 2014: Future of CephFS
Ceph Day New York 2014:  Future of CephFS Ceph Day New York 2014:  Future of CephFS
Ceph Day New York 2014: Future of CephFS Ceph Community
 
Amazon DynamoDB Lessen's Learned by Beginner
Amazon DynamoDB Lessen's Learned by BeginnerAmazon DynamoDB Lessen's Learned by Beginner
Amazon DynamoDB Lessen's Learned by BeginnerHirokazu Tokuno
 
Docker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12XDocker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12XJérôme Petazzoni
 
Amazon Redshift
Amazon RedshiftAmazon Redshift
Amazon RedshiftJeff Patti
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Jérôme Petazzoni
 

Similar to Introduction to redis - version 2 (20)

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Dragoncraft Architectural Overview
Dragoncraft Architectural OverviewDragoncraft Architectural Overview
Dragoncraft Architectural Overview
 
Logs @ OVHcloud
Logs @ OVHcloudLogs @ OVHcloud
Logs @ OVHcloud
 
Introduction to AWS Big Data
Introduction to AWS Big Data Introduction to AWS Big Data
Introduction to AWS Big Data
 
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
 
Redis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamRedis - for duplicate detection on real time stream
Redis - for duplicate detection on real time stream
 
Redis for duplicate detection on real time stream
Redis for duplicate detection on real time streamRedis for duplicate detection on real time stream
Redis for duplicate detection on real time stream
 
Ledingkart Meetup #2: Scaling Search @Lendingkart
Ledingkart Meetup #2: Scaling Search @LendingkartLedingkart Meetup #2: Scaling Search @Lendingkart
Ledingkart Meetup #2: Scaling Search @Lendingkart
 
PostgreSQL: present and near future
PostgreSQL: present and near futurePostgreSQL: present and near future
PostgreSQL: present and near future
 
Presentations from the Cloudera Impala meetup on Aug 20 2013
Presentations from the Cloudera Impala meetup on Aug 20 2013Presentations from the Cloudera Impala meetup on Aug 20 2013
Presentations from the Cloudera Impala meetup on Aug 20 2013
 
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQDocker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
 
Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9
 
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
 
Ceph Day New York 2014: Future of CephFS
Ceph Day New York 2014:  Future of CephFS Ceph Day New York 2014:  Future of CephFS
Ceph Day New York 2014: Future of CephFS
 
Amazon DynamoDB Lessen's Learned by Beginner
Amazon DynamoDB Lessen's Learned by BeginnerAmazon DynamoDB Lessen's Learned by Beginner
Amazon DynamoDB Lessen's Learned by Beginner
 
Redis
RedisRedis
Redis
 
Docker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12XDocker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12X
 
Amazon Redshift
Amazon RedshiftAmazon Redshift
Amazon Redshift
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
 

More from Dvir Volk

Searching Billions of Documents with Redis
Searching Billions of Documents with RedisSearching Billions of Documents with Redis
Searching Billions of Documents with RedisDvir Volk
 
Boosting Machine Learning with Redis Modules and Spark
Boosting Machine Learning with Redis Modules and SparkBoosting Machine Learning with Redis Modules and Spark
Boosting Machine Learning with Redis Modules and SparkDvir Volk
 
Redis modules 101
Redis modules 101Redis modules 101
Redis modules 101Dvir Volk
 
Tales Of The Black Knight - Keeping EverythingMe running
Tales Of The Black Knight - Keeping EverythingMe runningTales Of The Black Knight - Keeping EverythingMe running
Tales Of The Black Knight - Keeping EverythingMe runningDvir Volk
 
10 reasons to be excited about go
10 reasons to be excited about go10 reasons to be excited about go
10 reasons to be excited about goDvir Volk
 
Kicking ass with redis
Kicking ass with redisKicking ass with redis
Kicking ass with redisDvir Volk
 
Introduction to Thrift
Introduction to ThriftIntroduction to Thrift
Introduction to ThriftDvir Volk
 

More from Dvir Volk (8)

RediSearch
RediSearchRediSearch
RediSearch
 
Searching Billions of Documents with Redis
Searching Billions of Documents with RedisSearching Billions of Documents with Redis
Searching Billions of Documents with Redis
 
Boosting Machine Learning with Redis Modules and Spark
Boosting Machine Learning with Redis Modules and SparkBoosting Machine Learning with Redis Modules and Spark
Boosting Machine Learning with Redis Modules and Spark
 
Redis modules 101
Redis modules 101Redis modules 101
Redis modules 101
 
Tales Of The Black Knight - Keeping EverythingMe running
Tales Of The Black Knight - Keeping EverythingMe runningTales Of The Black Knight - Keeping EverythingMe running
Tales Of The Black Knight - Keeping EverythingMe running
 
10 reasons to be excited about go
10 reasons to be excited about go10 reasons to be excited about go
10 reasons to be excited about go
 
Kicking ass with redis
Kicking ass with redisKicking ass with redis
Kicking ass with redis
 
Introduction to Thrift
Introduction to ThriftIntroduction to Thrift
Introduction to Thrift
 

Recently uploaded

Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDropbox
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Zilliz
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsNanddeep Nachan
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 

Recently uploaded (20)

Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 

Introduction to redis - version 2

  • 1. "Little Server of Awesome" 2011 Dvir Volk Software Architect, DoAT dvir@doat.com http://doat.com
  • 2. Redis: Key => DataStructure Server ● Memcache-ish in-memory key/value store ● But values are complex data types: ○ blobs ○ lists ○ sets ○ sorted sets ○ hash tables ● And it has persistence. ● Open source; very helpful and friendly community. ● Used in the real world: github, craigslist, bit.ly, engineyard... ● Used heavily in doAT as a front-end database, search, geo resolving
  • 3. Key Features and Cool Stuff ● All data is in memory (almost) ● All data is eventually persistent ● Handles huge workloads easily ● Mostly O(1) behavior, Ideal for write-heavy workloads ● Support for atomic operations, transactions ● Has pub/sub functionality ● Single threaded, uses aync. IO ● Internal scripting with LUA in alpha
  • 4. A little benchmark This is on my laptop (core i7 @2.2Ghz) ● SET: 187265.92 requests per second ● GET: 185185.17 requests per second ● INCR: 190114.06 requests per second ● LPUSH: 190114.06 requests per second ● LPOP: 187090.73 requests per second ● ● SADD: 186567.16 requests per second ● SPOP: 185873.61 requests per second
  • 5. Scaling it up ● Master-slave replication out of the box ● Slaves can be made masters on the fly ● Redis-cluster in alpha state. ● Sharding supported by some clients ● Single threaded - run num_cores/2 instances on the same machine
  • 6. Persistence ● All data is synchronized to disk - eventually or immediately ● Pick your risk level Vs. performance ● Data is either dumped in a forked process, or written as a append-only change-log (AOF) ● Append-only mode supports transactional disk writes so you can lose no data (cost: 99% speed loss :) ) ● You can save the state explicitly, background or blocking ● If the data is too big to fit in memory - redis can handle its own swap.
  • 7. Show me the features!
  • 8. The basics... Get/Sets - nothing fancy. Keys are strings, anything goes - just quote spaces. redis> SET foo "bar" OK redis> GET foo "bar" You can atomically increment numbers redis> SET bar 337 OK redis> INCRBY bar 1000 (integer) 1337 Getting multiple values at once redis> MGET foo bar 1. "bar" 2. "1337" Keys are lazily expired redis> EXPIRE foo 1 (integer) 1 redis> GET foo (nil) Be careful with EXPIRE - re-setting a value without re-expiring it will remove the expiration!
  • 9. Atomic Operations GETSET puts a different value inside a key, retriving the old one redis> SET foo bar OK redis> GETSET foo baz "bar" redis> GET foo "baz" SETNX sets a value only if it does not exist redis> SETNX foo bar *OK* redis> SETNX foo baz *FAILS*
  • 10. List operations ● Lists are your ordinary linked lists. ● You can push and pop at both sides, extract range, resize.. ● Random access and ranges at O(N) ● BLPOP: Blocking POP - wait until a list has elements and pop them. Useful for realtime stuff.
  • 11. Set operations ● Sets are... well, sets of unique values w/ push, pop, etc. ● Sets can be intersected/diffed /union'ed server side. ● Can be useful as keys when building complex schemata.
  • 12. Sorted Sets ● Same as sets, but with score per element ● Ranked ranges, aggregation of scores on INTERSECT ● Can be used as ordered keys in complex schemata ● Think timestamps, inverted index, geohashing, ip ranges
  • 13. Hashes ● Hash tables as values ● Think of an object store with atomic access to object members redis> HSET foo bar 1 redis> HINCRBY foo bar 1 (integer) 1 (integer) 2 redis> HSET foo baz 2 (integer) 1 redis> HGET foo bar redis> HSET foo foo foo "2" (integer) 1 redis> HKEYS foo redis> HGETALL foo 1. "bar" { 2. "baz" "bar": "1", 3. "foo" "baz": "2", "foo": "foo" }
  • 14. PubSub - Publish/Subscribe ● Clients can subscribe to channels or patterns and receive notifications when messages are sent to channels. ● Subscribing is O(1), posting messages is O(n) ● Think chats, Comet applications: real-time analytics, twitter
  • 15. Transactions ● MULTI, ...., EXEC: Easy because of the single thread. ● All commands are executed after EXEC, block and return values for the commands as a list. ● Example: redis> MULTI OK redis> SET "foo" "bar" QUEUED redis> INCRBY "num" 1 QUEUED redis> EXEC 1) OK 2) (integer) 1 ● Transactions can be discarded with DISCARD. ● WATCH allows you to lock keys while you are queuing your transaction, and avoid race conditions.
  • 16. Gotchas, Lessons Learned ● 64 bit instances consume much much more RAM. ● Master/Slave sync far from perfect. ● DO NOT USE THE KEYS COMMAND!!! ● vm.overcommit_memory = 1 ● Use MONITOR to see what's going on
  • 17. Recipe 1: Social Feed ● Each User's followers are list: user:$ID:followers => { 1, 2, 4 } ● Each User's feed is a sorted set with the time-stamp being the score: user:$ID:feed => { msg1234: 124950132, ... } ● Each message is either a blob or a HASH object with fields: msg1234 => {title, text, ... } ● When a message is added, we: ○ add the message ○ read the posting user's follower list ○ add the message id to each follower's feed ● When a user reads their feeds, we use ZREVRANGE to get the latest item ids, and HGETALL to fetch the message texts. ● For bonus points, connected users can receive push notifications with PubSub
  • 18. Recipe 2: Real-Time Analytics ● We want to measure page views, per page, one minute resolution, in real time. ● Each page is represented by a sorted set, where the value is the minute, and the score is the number of hits. ● On each page hit, we do ZINCRBY <page> <current_minute> 1 ● Page views for the last N minutes: ZREVRANGE <page> 0 N ● Group several pages together: ZUNIONSTORE ... AGGREGATE SUM ● Trimming for a specific window: ZREMRANGEBYRANK ● Again, PubSub can be used to push updates to clients
  • 19. Recipe 3: Processing Queue ● Each task can be a blob describing it with json, using SET/GET/DEL. ● We keep a list of pending tasks, or multiple lists per task type/priority ("tasks:urgent") ● Worker procs can work from any machine over the network. ● Workers poll task queues using BRPOP tasks:urgent ● atomicity guaranteed! ● When a task is pulled it can be marked on a different list/set of "tasks:inprogress". When a task is done it is moved to a "done" or "failed" task list. ● A manager can monitor failed tasks and move them around.
  • 20. Other use case ideas ● Geo Resolving with geohashing ● Implemented and opened by yours truly https://github.com/doat/geodis ● API Key and rate management ● Very fast key lookup, rate control counters using INCR ● Real time game data ● ZSETs for high scores, HASHES for online users, etc ● Database Shard Index ● map key => database id. Count size with SETS ● Comet - no polling ajax ● use BLPOP or pub/sub ● Machine learning data with automatic persistence. ● sorted sets are your vectors!
  • 21. More resources Redis' website: http://redis.io Excellent and more detailed presentation by Simon Willison (a bit old): http://simonwillison.net/static/2010/redis-tutorial/ Much more complex twitter clone: http://code.google.com/p/redis/wiki/TwitterAlikeExample Full command reference: http://redis.io/commands