SlideShare a Scribd company logo
1 of 32
cache
   @TimYang
Agenda


• Feed
•      Cache
•
•   ,   status feed

•
Feed

•                 feed

    • Push(       )

    • Pull(   )

    •
Push
•     feed

    • Inbox:
    • Outbox:
•                   inbox( )

•               Inbox( )
Push
•
•


•
Pull


•                 outbox( )

•                       Inbox(   )

• Online computation
Pull


•
•
•
•


•
Cache
memory is the new disk,
and disk is the new tape.
for "real-time" web applications,
and systems that require massive scalability
     - Jim Gray
Cache
cache
          Weibo cache arch

  Inbox       hot cache




Outbox       Vector cache    Archive cache




 Social       Following        Followers     users
 Graph



Content       Hot cache          Total
Inbox
•       “         ”

• cache      id list

•                   (hot cache)

•                   get set

• redis list/set
Outbox

•
•     id list(e.g. 100)

•   id LRU
Social Graph cache
• Following ids
• Followers
 •
 •
 •            delete all
Content cache
•           cache

•
    •
•
    • Open API      xml, json...
Cache


•
•
Update Workflow


                  Update status




Content cache    Hot Inbox Vector   Outbox vector



Content cache
 replication
•   outbox hot vector

•   followers list

•   inbox
feed
                Home timeline Workflow


                     home_timeline


                      aggregator



                                         Content hot
Inbox cache         Outbox Vector
                                           cache


Inbox archive        Inbox archive      Content cache
feed

•   inbox cache

•
•         ,   following

•   id list         feed
cache

•
• hot keys
•
• mutex
•                     Content cache

• multi get n feed(n = items/ , e.g. 50)
• cache = n * (feed +                    ,
    e.g. 2k)
•                1,000   /
• 1,000    800Mbps

•1        8Gbps
•       1G       300~400Mbps

•
    •        local cache

    •
    •
hot keys
• content cache of
• create local cache
  1. get user_yaochen_local
  2. get user_yaochen
       1. set user_yaochen_local:value
  3.               delete all
cache
•                      key
      memcache

    •
    • LRU cache,
    • Memory storage
mutex
•       content item cache

    •           db

    •                        db

• key_mutex                   key

• mutex
•
    • S2
    • S2
    •
S2

• Web 2.0 Social
• What’s S2
 • Social Platform &
 • Social Applications
•   www.s2forum.org
•
• Sina App Engine
•
•
•
Q&A


• email: iso1600@gmail.com
•              @TimYang

More Related Content

What's hot

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
Dvir Volk
 
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
Using Spark Streaming and NiFi for the Next Generation of ETL in the EnterpriseUsing Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
DataWorks Summit
 

What's hot (20)

TiDBのトランザクション
TiDBのトランザクションTiDBのトランザクション
TiDBのトランザクション
 
Treasure Data Cloud Data Platform
Treasure Data Cloud Data PlatformTreasure Data Cloud Data Platform
Treasure Data Cloud Data Platform
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Pegasus KV Storage, Let the Users focus on their work (2018/07)
Pegasus KV Storage, Let the Users focus on their work (2018/07)Pegasus KV Storage, Let the Users focus on their work (2018/07)
Pegasus KV Storage, Let the Users focus on their work (2018/07)
 
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
 
AWS Webcast - Introduction to Amazon Kinesis
AWS Webcast - Introduction to Amazon KinesisAWS Webcast - Introduction to Amazon Kinesis
AWS Webcast - Introduction to Amazon Kinesis
 
Scale your Alfresco Solutions
Scale your Alfresco Solutions Scale your Alfresco Solutions
Scale your Alfresco Solutions
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
 
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, PresetStreaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
 
Kinesis Firehoseを使ってみた
Kinesis Firehoseを使ってみたKinesis Firehoseを使ってみた
Kinesis Firehoseを使ってみた
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 
Pegasus In Depth (2018/10)
Pegasus In Depth (2018/10)Pegasus In Depth (2018/10)
Pegasus In Depth (2018/10)
 
Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -
 
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
Using Spark Streaming and NiFi for the Next Generation of ETL in the EnterpriseUsing Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
 
MongoDB WiredTiger Internals
MongoDB WiredTiger InternalsMongoDB WiredTiger Internals
MongoDB WiredTiger Internals
 
Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...
Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...
Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...
 
Hive spark-s3acommitter-hbase-nfs
Hive spark-s3acommitter-hbase-nfsHive spark-s3acommitter-hbase-nfs
Hive spark-s3acommitter-hbase-nfs
 
PostgreSQL10徹底解説
PostgreSQL10徹底解説PostgreSQL10徹底解説
PostgreSQL10徹底解説
 
Elasticsearch勉強会#39 LT 20201217
Elasticsearch勉強会#39 LT 20201217Elasticsearch勉強会#39 LT 20201217
Elasticsearch勉強会#39 LT 20201217
 
大量時空間データの処理 ~ 現状の課題と今後OSSが解決すべきこと。(Open Source Conference 2021 Online/Osaka講演資料)
大量時空間データの処理 ~ 現状の課題と今後OSSが解決すべきこと。(Open Source Conference 2021 Online/Osaka講演資料)大量時空間データの処理 ~ 現状の課題と今後OSSが解決すべきこと。(Open Source Conference 2021 Online/Osaka講演資料)
大量時空間データの処理 ~ 現状の課題と今後OSSが解決すべきこと。(Open Source Conference 2021 Online/Osaka講演資料)
 

Viewers also liked

人人网技术经理张铁安 Feed系统结构浅析
人人网技术经理张铁安 Feed系统结构浅析人人网技术经理张铁安 Feed系统结构浅析
人人网技术经理张铁安 Feed系统结构浅析
isnull
 
设计模式分享
设计模式分享设计模式分享
设计模式分享
michaelsh
 
Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言
Feng Yu
 
Depoimento de Amaury Jr ne PF
Depoimento de Amaury Jr ne PFDepoimento de Amaury Jr ne PF
Depoimento de Amaury Jr ne PF
Luis Nassif
 
20140717 about openstreetmap
20140717 about openstreetmap20140717 about openstreetmap
20140717 about openstreetmap
Taichi Furuhashi
 
Lesson 7 1 Part 2 1
Lesson 7  1 Part 2 1Lesson 7  1 Part 2 1
Lesson 7 1 Part 2 1
mradcliffe
 
Fleury no inferno
Fleury no infernoFleury no inferno
Fleury no inferno
Luis Nassif
 
About OSM japan at the Association of Japanese Geographers
About OSM japan at the Association of Japanese GeographersAbout OSM japan at the Association of Japanese Geographers
About OSM japan at the Association of Japanese Geographers
Taichi Furuhashi
 
Michigan Ppt[1]
Michigan Ppt[1]Michigan Ppt[1]
Michigan Ppt[1]
mduncan
 
Hi France24 Iht Harris Poll Leaders Barometer April2009
Hi France24 Iht Harris Poll Leaders Barometer April2009Hi France24 Iht Harris Poll Leaders Barometer April2009
Hi France24 Iht Harris Poll Leaders Barometer April2009
Luis Nassif
 

Viewers also liked (20)

Build scalable microblog qcon beijing 2010
Build scalable microblog qcon beijing 2010Build scalable microblog qcon beijing 2010
Build scalable microblog qcon beijing 2010
 
新浪微博分布式缓存与队列-2013版
新浪微博分布式缓存与队列-2013版新浪微博分布式缓存与队列-2013版
新浪微博分布式缓存与队列-2013版
 
Feed服务架构-新浪微博新员工培训议题
Feed服务架构-新浪微博新员工培训议题Feed服务架构-新浪微博新员工培训议题
Feed服务架构-新浪微博新员工培训议题
 
高性能队列Fqueue的设计和使用实践
高性能队列Fqueue的设计和使用实践高性能队列Fqueue的设计和使用实践
高性能队列Fqueue的设计和使用实践
 
人人网技术经理张铁安 Feed系统结构浅析
人人网技术经理张铁安 Feed系统结构浅析人人网技术经理张铁安 Feed系统结构浅析
人人网技术经理张铁安 Feed系统结构浅析
 
设计模式分享
设计模式分享设计模式分享
设计模式分享
 
Java常见问题排查
Java常见问题排查Java常见问题排查
Java常见问题排查
 
Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言
 
13 tips for a highly engaging feed
13 tips for a highly engaging feed13 tips for a highly engaging feed
13 tips for a highly engaging feed
 
Depoimento de Amaury Jr ne PF
Depoimento de Amaury Jr ne PFDepoimento de Amaury Jr ne PF
Depoimento de Amaury Jr ne PF
 
20140717 about openstreetmap
20140717 about openstreetmap20140717 about openstreetmap
20140717 about openstreetmap
 
Lesson 7 1 Part 2 1
Lesson 7  1 Part 2 1Lesson 7  1 Part 2 1
Lesson 7 1 Part 2 1
 
Fleury no inferno
Fleury no infernoFleury no inferno
Fleury no inferno
 
About OSM japan at the Association of Japanese Geographers
About OSM japan at the Association of Japanese GeographersAbout OSM japan at the Association of Japanese Geographers
About OSM japan at the Association of Japanese Geographers
 
Miguli Basic
Miguli BasicMiguli Basic
Miguli Basic
 
Space hackathon5
Space hackathon5Space hackathon5
Space hackathon5
 
Collaborative 3D Modelling and Printing: What you See is not Directly What Yo...
Collaborative 3D Modelling and Printing: What you See is not Directly What Yo...Collaborative 3D Modelling and Printing: What you See is not Directly What Yo...
Collaborative 3D Modelling and Printing: What you See is not Directly What Yo...
 
Alfredorodriguez
AlfredorodriguezAlfredorodriguez
Alfredorodriguez
 
Michigan Ppt[1]
Michigan Ppt[1]Michigan Ppt[1]
Michigan Ppt[1]
 
Hi France24 Iht Harris Poll Leaders Barometer April2009
Hi France24 Iht Harris Poll Leaders Barometer April2009Hi France24 Iht Harris Poll Leaders Barometer April2009
Hi France24 Iht Harris Poll Leaders Barometer April2009
 

Similar to 微博cache设计谈

Timyang新浪微博设计谈
Timyang新浪微博设计谈Timyang新浪微博设计谈
Timyang新浪微博设计谈
Cevin Cheung
 
Tup2 新浪杨卫华
Tup2 新浪杨卫华Tup2 新浪杨卫华
Tup2 新浪杨卫华
yongboy
 
SDEC2011 NoSQL concepts and models
SDEC2011 NoSQL concepts and modelsSDEC2011 NoSQL concepts and models
SDEC2011 NoSQL concepts and models
Korea Sdec
 
BuilHigh Performance Weibo Platform-Qcon2011
BuilHigh Performance Weibo Platform-Qcon2011BuilHigh Performance Weibo Platform-Qcon2011
BuilHigh Performance Weibo Platform-Qcon2011
Yiwei Ma
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
DATAVERSITY
 
Introducing Amplify
Introducing AmplifyIntroducing Amplify
Introducing Amplify
appendTo
 

Similar to 微博cache设计谈 (20)

Timyang新浪微博设计谈
Timyang新浪微博设计谈Timyang新浪微博设计谈
Timyang新浪微博设计谈
 
Tup2 新浪杨卫华
Tup2 新浪杨卫华Tup2 新浪杨卫华
Tup2 新浪杨卫华
 
杨卫华:微博cache设计浅谈
杨卫华:微博cache设计浅谈杨卫华:微博cache设计浅谈
杨卫华:微博cache设计浅谈
 
Kiosk / PHP
Kiosk / PHP Kiosk / PHP
Kiosk / PHP
 
Cache on Delivery
Cache on DeliveryCache on Delivery
Cache on Delivery
 
SDEC2011 NoSQL concepts and models
SDEC2011 NoSQL concepts and modelsSDEC2011 NoSQL concepts and models
SDEC2011 NoSQL concepts and models
 
BuilHigh Performance Weibo Platform-Qcon2011
BuilHigh Performance Weibo Platform-Qcon2011BuilHigh Performance Weibo Platform-Qcon2011
BuilHigh Performance Weibo Platform-Qcon2011
 
04 standard class library c#
04 standard class library c#04 standard class library c#
04 standard class library c#
 
ログ収集プラットフォーム開発におけるElasticsearchの運用
ログ収集プラットフォーム開発におけるElasticsearchの運用ログ収集プラットフォーム開発におけるElasticsearchの運用
ログ収集プラットフォーム開発におけるElasticsearchの運用
 
Amazon Aurora로 안전하게 migration 하기
Amazon Aurora로 안전하게 migration 하기Amazon Aurora로 안전하게 migration 하기
Amazon Aurora로 안전하게 migration 하기
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
 
Introducing Amplify
Introducing AmplifyIntroducing Amplify
Introducing Amplify
 
Html5: Something wicked this way comes (Hack in Paris)
Html5: Something wicked this way comes (Hack in Paris)Html5: Something wicked this way comes (Hack in Paris)
Html5: Something wicked this way comes (Hack in Paris)
 
Nuxeo World Session: CMIS - What's Next?
Nuxeo World Session: CMIS - What's Next?Nuxeo World Session: CMIS - What's Next?
Nuxeo World Session: CMIS - What's Next?
 
OSDC 2013 | Introduction into Chef by Andy Hawkins
OSDC 2013 | Introduction into Chef by Andy HawkinsOSDC 2013 | Introduction into Chef by Andy Hawkins
OSDC 2013 | Introduction into Chef by Andy Hawkins
 
OSMC 2016 - Monitor your infrastructure with Elastic Beats by Monica Sarbu
OSMC 2016 - Monitor your infrastructure with Elastic Beats by Monica SarbuOSMC 2016 - Monitor your infrastructure with Elastic Beats by Monica Sarbu
OSMC 2016 - Monitor your infrastructure with Elastic Beats by Monica Sarbu
 
OSMC 2016 | Monitor your Infrastructure with Elastic Beats by Monica Sarbu
OSMC 2016 | Monitor your Infrastructure with Elastic Beats by Monica SarbuOSMC 2016 | Monitor your Infrastructure with Elastic Beats by Monica Sarbu
OSMC 2016 | Monitor your Infrastructure with Elastic Beats by Monica Sarbu
 
ActiveRecordのテーブル名の話
ActiveRecordのテーブル名の話ActiveRecordのテーブル名の話
ActiveRecordのテーブル名の話
 
初心者向けGAE/Java説明資料
初心者向けGAE/Java説明資料初心者向けGAE/Java説明資料
初心者向けGAE/Java説明資料
 
iPhone dev intro
iPhone dev introiPhone dev intro
iPhone dev intro
 

Recently uploaded

Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 

Recently uploaded (20)

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 

微博cache设计谈

Editor's Notes