SlideShare a Scribd company logo
1 of 20
Download to read offline
Developing Scalable User Search
for PlayStation 4
Ai	
  Sasho	
  
	
  
ai.sasho@am.sony.com	
  
Sr.	
  So/ware	
  Engineer	
  
Sony	
  Interac6ve	
  Entertainment	
  
	
  
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
About My Team
§  Developing	
  social	
  features	
  for	
  PS4	
  to	
  improve	
  social	
  gaming	
  experiences.	
  
	
  
§  Worked	
  on	
  User	
  Search	
  and	
  Players	
  You	
  May	
  know	
  recommenda6on	
  
features.	
  
§  Server	
  side:	
  Isaias,	
  Marlon,	
  Pavan,	
  Chris,	
  Janhavi,	
  Xifan,	
  Venkat	
  
§  Client	
  side:	
  Tomas,	
  Nythya,	
  Max,	
  Yukio,	
  Katsuya,	
  Eric,	
  Tong	
  
	
  
Sony	
  Interac8ve	
  Entertainment	
  
=	
  Sony	
  Network	
  Entertainment	
  Intn’l	
  
+	
  Sony	
  Computer	
  Entertainment	
  
=	
  Greatness	
  Awaits!	
  
	
  
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Outline
§  User Search Feature Overview
§  SolrCloud Setup
§  Personalized Search: Lucene + SolrCloud
§  Challenges
§  Solr4.8 to 5.4 Upgrade
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
User Search
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
User Search
§  Fast
•  query should return < 100 ms
§  Reliable / Fault Tolerant
§  Scalable
•  SolrCloud cluster need to handle:
o  Up to 1000 RPS query requests
o  Up to 250 RPS indexing requests
•  Appr. 300 millions documents
§  Ranking search results by friendship.
•  Up to n degrees of separation.
•  Friends, 2nd degree fiends (friends of friends), etc.
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
User Search: Requirements
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
SolrCloud: System Architecture
ZooKeeper	
  
SolrCloud	
  cluster	
  
Leader	
  
a	
  
Replica	
  
Leader	
  
a	
  
Replica	
  
Leader	
  
a	
  
Replica	
  
Leader	
  
a	
  
Replica	
  
ELB	
  
Applica6on	
  
Servers	
  
Database	
  
§  SolrCloud 5.4
§  Documents
•  User data (~ 1.5 kb per user)
•  ID, Online ID, Name (First, Middle, Last), Privacy, User Type, etc..
•  ~ 300 million documents
§  Shards
•  4 shards + many replicas.
•  # shards determined experimentally.
•  Most of the docs on each shard fit in the memory.
§  Cache
•  Query Result Cache, Document Cache, Filter Cache, etc ..
§  Commit
•  SoftAutoComit: 5 secs
•  AutoCommit: 15 mins (OpenSearcher=false)
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
SolrCloud: Configurations
§  Tokenizers
•  Whitespace Tokenizer
§  Filters
§  Ascii Folding Filter
o  Stored and queried with equivalent English alphabets.
o  Joan Miró -> Joan Miro
§  N-Gram Filter
o  abc -> a, b, c, ab, bc, abc
o  Takes up more space, but faster than wildcard (*) when
queried.
§  Lower Case Filter
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
SolrCloud: Configurations
§  People search users they know or they kind of know...
§  Search results should be ranked by the friendship
between the searcher and the searched (users).
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Personalized Search: Overview
User	
  A	
  	
  <-­‐	
  Friend	
  (1st	
  degree	
  of	
  separa6on)	
  
User	
  B	
  	
  <-­‐	
  Friend	
  (1st	
  degree	
  of	
  separa6on)	
  
User	
  C	
  	
  <-­‐	
  Friend	
  of	
  Friend	
  (2nd	
  degree	
  of	
  separa6on)	
  
...	
  
	
  
User	
  Y	
  <-­‐	
  Not	
  associated.	
  
User	
  Z	
  <-­‐	
  Not	
  associated.	
  
	
  
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Personalized Search: Ideas
q=ps4king&
bf=friends:(ID1 or ID2 or ID3 or …)^500&
bf=friends2nd:(ID4 or ID5 or ID6 or …)^50&
bf=friends3rd:(ID7or ID8 or ID9 or …)^5&
…
Possible	
  Solu8on	
  1	
  :	
  Query	
  SolrCloud	
  with	
  the	
  list	
  of	
  friend	
  IDs.	
  
	
  
Problems	
  
•  The	
  list	
  of	
  friends	
  can	
  be	
  very	
  long	
  (poten6ally	
  thousands).	
  
•  Increases	
  the	
  query	
  latency.	
  
Giving	
  a	
  higher	
  boost	
  for	
  users	
  who	
  are	
  closer	
  to	
  the	
  caller.	
  
Possible Solution 2: Index the friendship in SolrCloud.
Add “friends“ fields, if the caller is in one of the “friends”
fields, boost the document.
Problems:
o  Too many requests to Solr.
o  Maintaining friendship in Solr in addition to our database
might be overkill.
o  Requires a large disk space.
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Personalized Search: Ideas
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Personalized Search: Our Solution
+	
  
Personalized	
  Index	
  
	
  
Stores	
  people	
  close	
  to	
  the	
  caller	
  (friends,	
  
friends	
  of	
  friends,	
  up	
  to	
  n	
  degrees	
  of	
  
separa6on).	
  
	
  
§  Also	
  used	
  in	
  friend	
  recommenda6on	
  
system.	
  
§  Other	
  team	
  already	
  uses	
  Lucene	
  index	
  for	
  
user	
  owned	
  games.	
  
	
  
Global	
  Index	
  
	
  
Includes	
  all	
  the	
  users.	
  
	
  
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Personalized Search: Lucene + SolrCloud
Online	
  ID	
   First	
  
Name	
  
….	
   Degree	
  of	
  
Separa6on	
  
ps4Queen	
   Marge	
   …	
   1	
  
ps4King	
   Homer	
   ...	
   1	
  
ps4aweso
me	
  
Bart	
   …	
   2	
  
…	
   …	
   …	
   …	
  
Lucene	
  Index	
  (simplified)	
  
Applica6on	
  
Server	
  
Friendship	
  
Data	
  
§  Lucene	
  index	
  created	
  on-­‐
demand	
  for	
  the	
  caller	
  
	
  
§  Cached	
  temporarily	
  
+	
  
§  Hard to increase the performance using two
index systems. (Lucene + SolrCloud)
•  Tuned SolrCloud a lot (cache size, query optimization,
soft/auto commit settings, GC settings, etc.)
§  Not a problem anymore, but SolrCloud had been
unstable for a while.
•  Entire cluster would have gone down a couple of
times a month.
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Challenges
§  Increased the number of replicas
•  When leader goes in recovery, need to have
enough replicas to handle all the requests.
§  Reconfigured GC settings with CMS (concurrent mark
sweep).
§  Decreased the size of the document query cache.
o  Cache warm-up time was longer than the soft auto commit
duration -> was always warming the cache.
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
Challenges: Instability Solutions
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
SolrCloud Upgrade
§  Motivations
•  Originally Solr 4.8 was used, but due to the instability
issues, upgraded to Solr 5.4.
§  Challeges
•  Tried to data stream from a Solr 4.8 node to Solr 5.4 by
joining a node, but did not work.
•  Some data types have been deprecated.
o  IntegerType, LongType -> TrieInteger, TrieLong
o  schema.xml needed to be updated with the new data types.
o  Decided to full index the 300 million documents in Solr 5.4
cluster.
§  First, query out 300M docs and then full indexing.
§  Deep paging (specifying start index and limit) is too slow
•  Solr needs to cache documents up to the starting index.
§  The logical cursor cusorMark is solution to the deep paging problem.
The cursorMark returns the next cursor as part of the response.
§  cursorMark is not perfect. Sometimes the cursor stops before the end
of the documents. Could use filter query to query the certain range of
documents by ids.
©2016	
  Sony	
  Interac6ve	
  Entertainment	
  	
  
SolrCloud Upgrade: Full Indexing
...&rows=10&sort=id+asc&cursorMark=AoEjR0JQ	
  
Q	
  &	
  A	
  
Any	
  Ques6ons?	
  

More Related Content

What's hot

Introduction to Solr
Introduction to SolrIntroduction to Solr
Introduction to SolrErik Hatcher
 
Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...
Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...
Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...Lucidworks
 
Adding Search to the Hadoop Ecosystem
Adding Search to the Hadoop EcosystemAdding Search to the Hadoop Ecosystem
Adding Search to the Hadoop EcosystemCloudera, Inc.
 
Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...
Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...
Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...Lucidworks
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache SolrAndy Jackson
 
Rapid Prototyping with Solr
Rapid Prototyping with SolrRapid Prototyping with Solr
Rapid Prototyping with SolrErik Hatcher
 
Solr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance studySolr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance studyCharlie Hull
 
Solr Powered Lucene
Solr Powered LuceneSolr Powered Lucene
Solr Powered LuceneErik Hatcher
 
Introduction to Apache Lucene/Solr
Introduction to Apache Lucene/SolrIntroduction to Apache Lucene/Solr
Introduction to Apache Lucene/SolrRahul Jain
 
Enterprise Search Using Apache Solr
Enterprise Search Using Apache SolrEnterprise Search Using Apache Solr
Enterprise Search Using Apache Solrsagar chaturvedi
 
Benchmarking Solr Performance at Scale
Benchmarking Solr Performance at ScaleBenchmarking Solr Performance at Scale
Benchmarking Solr Performance at Scalethelabdude
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache SolrChristos Manios
 
Building and Running Solr-as-a-Service: Presented by Shai Erera, IBM
Building and Running Solr-as-a-Service: Presented by Shai Erera, IBMBuilding and Running Solr-as-a-Service: Presented by Shai Erera, IBM
Building and Running Solr-as-a-Service: Presented by Shai Erera, IBMLucidworks
 
The ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsThe ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsItamar
 
Lucene's Latest (for Libraries)
Lucene's Latest (for Libraries)Lucene's Latest (for Libraries)
Lucene's Latest (for Libraries)Erik Hatcher
 
Solr Black Belt Pre-conference
Solr Black Belt Pre-conferenceSolr Black Belt Pre-conference
Solr Black Belt Pre-conferenceErik Hatcher
 
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Apache Solr/Lucene Internals  by Anatoliy SokolenkoApache Solr/Lucene Internals  by Anatoliy Sokolenko
Apache Solr/Lucene Internals by Anatoliy SokolenkoProvectus
 
Integrating Spark and Solr-(Timothy Potter, Lucidworks)
Integrating Spark and Solr-(Timothy Potter, Lucidworks)Integrating Spark and Solr-(Timothy Potter, Lucidworks)
Integrating Spark and Solr-(Timothy Potter, Lucidworks)Spark Summit
 

What's hot (20)

Introduction to Solr
Introduction to SolrIntroduction to Solr
Introduction to Solr
 
Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...
Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...
Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...
 
Adding Search to the Hadoop Ecosystem
Adding Search to the Hadoop EcosystemAdding Search to the Hadoop Ecosystem
Adding Search to the Hadoop Ecosystem
 
Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...
Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...
Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache Solr
 
Rapid Prototyping with Solr
Rapid Prototyping with SolrRapid Prototyping with Solr
Rapid Prototyping with Solr
 
Solr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance studySolr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance study
 
Intro to Apache Solr
Intro to Apache SolrIntro to Apache Solr
Intro to Apache Solr
 
Solr Powered Lucene
Solr Powered LuceneSolr Powered Lucene
Solr Powered Lucene
 
Introduction to Apache Lucene/Solr
Introduction to Apache Lucene/SolrIntroduction to Apache Lucene/Solr
Introduction to Apache Lucene/Solr
 
Enterprise Search Using Apache Solr
Enterprise Search Using Apache SolrEnterprise Search Using Apache Solr
Enterprise Search Using Apache Solr
 
Benchmarking Solr Performance at Scale
Benchmarking Solr Performance at ScaleBenchmarking Solr Performance at Scale
Benchmarking Solr Performance at Scale
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache Solr
 
Building and Running Solr-as-a-Service: Presented by Shai Erera, IBM
Building and Running Solr-as-a-Service: Presented by Shai Erera, IBMBuilding and Running Solr-as-a-Service: Presented by Shai Erera, IBM
Building and Running Solr-as-a-Service: Presented by Shai Erera, IBM
 
The ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsThe ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch plugins
 
Lucene's Latest (for Libraries)
Lucene's Latest (for Libraries)Lucene's Latest (for Libraries)
Lucene's Latest (for Libraries)
 
Solr 4
Solr 4Solr 4
Solr 4
 
Solr Black Belt Pre-conference
Solr Black Belt Pre-conferenceSolr Black Belt Pre-conference
Solr Black Belt Pre-conference
 
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Apache Solr/Lucene Internals  by Anatoliy SokolenkoApache Solr/Lucene Internals  by Anatoliy Sokolenko
Apache Solr/Lucene Internals by Anatoliy Sokolenko
 
Integrating Spark and Solr-(Timothy Potter, Lucidworks)
Integrating Spark and Solr-(Timothy Potter, Lucidworks)Integrating Spark and Solr-(Timothy Potter, Lucidworks)
Integrating Spark and Solr-(Timothy Potter, Lucidworks)
 

Viewers also liked

Webinar: Fusion for Business Intelligence
Webinar: Fusion for Business IntelligenceWebinar: Fusion for Business Intelligence
Webinar: Fusion for Business IntelligenceLucidworks
 
Webinar: Search and Recommenders
Webinar: Search and RecommendersWebinar: Search and Recommenders
Webinar: Search and RecommendersLucidworks
 
Understanding the Solr security framework - Lucene Solr Revolution 2015
Understanding the Solr security framework - Lucene Solr Revolution 2015Understanding the Solr security framework - Lucene Solr Revolution 2015
Understanding the Solr security framework - Lucene Solr Revolution 2015Anshum Gupta
 
Apache Solr 5.0 and beyond
Apache Solr 5.0 and beyondApache Solr 5.0 and beyond
Apache Solr 5.0 and beyondAnshum Gupta
 
What's New in Apache Solr 4.10
What's New in Apache Solr 4.10What's New in Apache Solr 4.10
What's New in Apache Solr 4.10Anshum Gupta
 
Webinar: Fusion 2.3 Preview - Enhanced Features with Solr & Spark
Webinar: Fusion 2.3 Preview - Enhanced Features with Solr & SparkWebinar: Fusion 2.3 Preview - Enhanced Features with Solr & Spark
Webinar: Fusion 2.3 Preview - Enhanced Features with Solr & SparkLucidworks
 
Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...
Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...
Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...Lucidworks
 
What's new in Solr 5.0
What's new in Solr 5.0What's new in Solr 5.0
What's new in Solr 5.0Anshum Gupta
 
Solr JDBC: Presented by Kevin Risden, Avalon Consulting
Solr JDBC: Presented by Kevin Risden, Avalon ConsultingSolr JDBC: Presented by Kevin Risden, Avalon Consulting
Solr JDBC: Presented by Kevin Risden, Avalon ConsultingLucidworks
 
Scaling SolrCloud to a large number of Collections
Scaling SolrCloud to a large number of CollectionsScaling SolrCloud to a large number of Collections
Scaling SolrCloud to a large number of CollectionsAnshum Gupta
 
Ease of use in Apache Solr
Ease of use in Apache SolrEase of use in Apache Solr
Ease of use in Apache SolrAnshum Gupta
 
Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...
Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...
Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...Lucidworks
 
Solr security frameworks
Solr security frameworksSolr security frameworks
Solr security frameworksAnshum Gupta
 
SolrCloud Cluster management via APIs
SolrCloud Cluster management via APIsSolrCloud Cluster management via APIs
SolrCloud Cluster management via APIsAnshum Gupta
 
Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch, Wipro...
Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch,  Wipro...Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch,  Wipro...
Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch, Wipro...Lucidworks
 
Working with deeply nested documents in Apache Solr
Working with deeply nested documents in Apache SolrWorking with deeply nested documents in Apache Solr
Working with deeply nested documents in Apache SolrAnshum Gupta
 
Managing a SolrCloud cluster using APIs
Managing a SolrCloud cluster using APIsManaging a SolrCloud cluster using APIs
Managing a SolrCloud cluster using APIsAnshum Gupta
 
Coffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, Flax
Coffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, FlaxCoffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, Flax
Coffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, FlaxLucidworks
 
Slash n near real time indexing
Slash n   near real time indexingSlash n   near real time indexing
Slash n near real time indexingUmesh Prasad
 

Viewers also liked (20)

Webinar: Fusion for Business Intelligence
Webinar: Fusion for Business IntelligenceWebinar: Fusion for Business Intelligence
Webinar: Fusion for Business Intelligence
 
Webinar: Search and Recommenders
Webinar: Search and RecommendersWebinar: Search and Recommenders
Webinar: Search and Recommenders
 
Understanding the Solr security framework - Lucene Solr Revolution 2015
Understanding the Solr security framework - Lucene Solr Revolution 2015Understanding the Solr security framework - Lucene Solr Revolution 2015
Understanding the Solr security framework - Lucene Solr Revolution 2015
 
Apache Solr 5.0 and beyond
Apache Solr 5.0 and beyondApache Solr 5.0 and beyond
Apache Solr 5.0 and beyond
 
What's New in Apache Solr 4.10
What's New in Apache Solr 4.10What's New in Apache Solr 4.10
What's New in Apache Solr 4.10
 
Webinar: Fusion 2.3 Preview - Enhanced Features with Solr & Spark
Webinar: Fusion 2.3 Preview - Enhanced Features with Solr & SparkWebinar: Fusion 2.3 Preview - Enhanced Features with Solr & Spark
Webinar: Fusion 2.3 Preview - Enhanced Features with Solr & Spark
 
Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...
Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...
Building a Solr Continuous Delivery Pipeline with Jenkins: Presented by James...
 
What's new in Solr 5.0
What's new in Solr 5.0What's new in Solr 5.0
What's new in Solr 5.0
 
Solr JDBC: Presented by Kevin Risden, Avalon Consulting
Solr JDBC: Presented by Kevin Risden, Avalon ConsultingSolr JDBC: Presented by Kevin Risden, Avalon Consulting
Solr JDBC: Presented by Kevin Risden, Avalon Consulting
 
Scaling SolrCloud to a large number of Collections
Scaling SolrCloud to a large number of CollectionsScaling SolrCloud to a large number of Collections
Scaling SolrCloud to a large number of Collections
 
it's just search
it's just searchit's just search
it's just search
 
Ease of use in Apache Solr
Ease of use in Apache SolrEase of use in Apache Solr
Ease of use in Apache Solr
 
Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...
Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...
Cross Data Center Replication for the Enterprise: Presented by Adam Williams,...
 
Solr security frameworks
Solr security frameworksSolr security frameworks
Solr security frameworks
 
SolrCloud Cluster management via APIs
SolrCloud Cluster management via APIsSolrCloud Cluster management via APIs
SolrCloud Cluster management via APIs
 
Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch, Wipro...
Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch,  Wipro...Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch,  Wipro...
Using Apache Solr for Images as Big Data: Presented by Kerry Koitzsch, Wipro...
 
Working with deeply nested documents in Apache Solr
Working with deeply nested documents in Apache SolrWorking with deeply nested documents in Apache Solr
Working with deeply nested documents in Apache Solr
 
Managing a SolrCloud cluster using APIs
Managing a SolrCloud cluster using APIsManaging a SolrCloud cluster using APIs
Managing a SolrCloud cluster using APIs
 
Coffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, Flax
Coffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, FlaxCoffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, Flax
Coffee, Danish & Search: Presented by Alan Woodward & Charlie Hull, Flax
 
Slash n near real time indexing
Slash n   near real time indexingSlash n   near real time indexing
Slash n near real time indexing
 

Similar to Downtown SF Lucene/Solr Meetup: Developing Scalable User Search for PlayStation 4

AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...
AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...
AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...Amazon Web Services
 
Cassandra @ Sony: The good, the bad, and the ugly part 2
Cassandra @ Sony: The good, the bad, and the ugly part 2Cassandra @ Sony: The good, the bad, and the ugly part 2
Cassandra @ Sony: The good, the bad, and the ugly part 2DataStax Academy
 
MongoDB World 2019: Sharding: Stories From the Field
MongoDB World 2019: Sharding: Stories From the FieldMongoDB World 2019: Sharding: Stories From the Field
MongoDB World 2019: Sharding: Stories From the FieldMongoDB
 
DataStax Enterprise in the Field – 20160920
DataStax Enterprise in the Field – 20160920DataStax Enterprise in the Field – 20160920
DataStax Enterprise in the Field – 20160920Daniel Cohen
 
From 100s to 100s of Millions
From 100s to 100s of MillionsFrom 100s to 100s of Millions
From 100s to 100s of MillionsErik Onnen
 
MongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: ShardingMongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: ShardingMongoDB
 
Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Frontera распределенный робот для обхода веба в больших объемах / Александр С...Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Frontera распределенный робот для обхода веба в больших объемах / Александр С...Ontico
 
A Developer's View Into Spark's Memory Model with Wenchen Fan
A Developer's View Into Spark's Memory Model with Wenchen FanA Developer's View Into Spark's Memory Model with Wenchen Fan
A Developer's View Into Spark's Memory Model with Wenchen FanDatabricks
 
A Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen FanA Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen FanDatabricks
 
DSpace Under the Hood
DSpace Under the HoodDSpace Under the Hood
DSpace Under the HoodDuraSpace
 
Webinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You ThinkWebinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You ThinkStorage Switzerland
 
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod NarasimhaSpark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod NarasimhaSpark Summit
 
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod NarasimhaSpark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod NarasimhaSpark Summit
 
NAT64/DNS64 experiments, warnings and one useful tool
NAT64/DNS64 experiments, warnings and one useful toolNAT64/DNS64 experiments, warnings and one useful tool
NAT64/DNS64 experiments, warnings and one useful toolAPNIC
 
Coursera amazon cloudsearch presentation
Coursera amazon cloudsearch presentation Coursera amazon cloudsearch presentation
Coursera amazon cloudsearch presentation Michael Bohlig
 

Similar to Downtown SF Lucene/Solr Meetup: Developing Scalable User Search for PlayStation 4 (20)

AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...
AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...
AWS re:Invent 2016| GAM302 | Sony PlayStation: Breaking the Bandwidth Barrier...
 
Cassandra @ Sony: The good, the bad, and the ugly part 2
Cassandra @ Sony: The good, the bad, and the ugly part 2Cassandra @ Sony: The good, the bad, and the ugly part 2
Cassandra @ Sony: The good, the bad, and the ugly part 2
 
MongoDB World 2019: Sharding: Stories From the Field
MongoDB World 2019: Sharding: Stories From the FieldMongoDB World 2019: Sharding: Stories From the Field
MongoDB World 2019: Sharding: Stories From the Field
 
DataStax Enterprise in the Field – 20160920
DataStax Enterprise in the Field – 20160920DataStax Enterprise in the Field – 20160920
DataStax Enterprise in the Field – 20160920
 
From 100s to 100s of Millions
From 100s to 100s of MillionsFrom 100s to 100s of Millions
From 100s to 100s of Millions
 
MongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: ShardingMongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: Sharding
 
Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Frontera распределенный робот для обхода веба в больших объемах / Александр С...Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Frontera распределенный робот для обхода веба в больших объемах / Александр С...
 
A Developer's View Into Spark's Memory Model with Wenchen Fan
A Developer's View Into Spark's Memory Model with Wenchen FanA Developer's View Into Spark's Memory Model with Wenchen Fan
A Developer's View Into Spark's Memory Model with Wenchen Fan
 
A Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen FanA Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen Fan
 
WWDC 2017
WWDC 2017WWDC 2017
WWDC 2017
 
DSpace Under the Hood
DSpace Under the HoodDSpace Under the Hood
DSpace Under the Hood
 
Bh eu 05-kaminsky
Bh eu 05-kaminskyBh eu 05-kaminsky
Bh eu 05-kaminsky
 
Bh eu 05-kaminsky
Bh eu 05-kaminskyBh eu 05-kaminsky
Bh eu 05-kaminsky
 
Webinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You ThinkWebinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You Think
 
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod NarasimhaSpark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod Narasimha
 
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod NarasimhaSpark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod Narasimha
 
ION Durban - NAT64/DNS64 Experiments and the NAT64Check Tool
ION Durban - NAT64/DNS64 Experiments and the NAT64Check ToolION Durban - NAT64/DNS64 Experiments and the NAT64Check Tool
ION Durban - NAT64/DNS64 Experiments and the NAT64Check Tool
 
NAT64/DNS64 experiments, warnings and one useful tool
NAT64/DNS64 experiments, warnings and one useful toolNAT64/DNS64 experiments, warnings and one useful tool
NAT64/DNS64 experiments, warnings and one useful tool
 
Coursera amazon cloudsearch presentation
Coursera amazon cloudsearch presentation Coursera amazon cloudsearch presentation
Coursera amazon cloudsearch presentation
 
Data Science
Data ScienceData Science
Data Science
 

More from Lucidworks

Search is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategySearch is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategyLucidworks
 
Drive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in SalesforceDrive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in SalesforceLucidworks
 
How Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsHow Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsLucidworks
 
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks
 
Connected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesConnected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesLucidworks
 
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Lucidworks
 
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...Lucidworks
 
Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Lucidworks
 
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Lucidworks
 
AI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and RosetteAI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and RosetteLucidworks
 
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentThe Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentLucidworks
 
Webinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeWebinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeLucidworks
 
Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Lucidworks
 
Applying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchApplying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchLucidworks
 
Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1Lucidworks
 
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyWebinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyLucidworks
 
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Lucidworks
 
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceApply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceLucidworks
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchLucidworks
 
Why Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondWhy Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondLucidworks
 

More from Lucidworks (20)

Search is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategySearch is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce Strategy
 
Drive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in SalesforceDrive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in Salesforce
 
How Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsHow Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant Products
 
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
 
Connected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesConnected Experiences Are Personalized Experiences
Connected Experiences Are Personalized Experiences
 
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
 
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
 
Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020
 
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
 
AI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and RosetteAI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and Rosette
 
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentThe Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
 
Webinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeWebinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - Europe
 
Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19
 
Applying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchApplying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 Research
 
Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1
 
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyWebinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
 
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
 
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceApply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise Search
 
Why Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondWhy Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and Beyond
 

Recently uploaded

OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Projectwajrcs
 
Kubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxKubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxPrakarsh -
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9Jürgen Gutsch
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Neo4j
 
20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기Chiwon Song
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 

Recently uploaded (20)

OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
 
Kubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxKubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptx
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!
 
20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Program with GUTs
Program with GUTsProgram with GUTs
Program with GUTs
 

Downtown SF Lucene/Solr Meetup: Developing Scalable User Search for PlayStation 4

  • 1. Developing Scalable User Search for PlayStation 4 Ai  Sasho     ai.sasho@am.sony.com   Sr.  So/ware  Engineer   Sony  Interac6ve  Entertainment    
  • 2. ©2016  Sony  Interac6ve  Entertainment     About My Team §  Developing  social  features  for  PS4  to  improve  social  gaming  experiences.     §  Worked  on  User  Search  and  Players  You  May  know  recommenda6on   features.   §  Server  side:  Isaias,  Marlon,  Pavan,  Chris,  Janhavi,  Xifan,  Venkat   §  Client  side:  Tomas,  Nythya,  Max,  Yukio,  Katsuya,  Eric,  Tong     Sony  Interac8ve  Entertainment   =  Sony  Network  Entertainment  Intn’l   +  Sony  Computer  Entertainment   =  Greatness  Awaits!    
  • 3. ©2016  Sony  Interac6ve  Entertainment    
  • 4. ©2016  Sony  Interac6ve  Entertainment     Outline §  User Search Feature Overview §  SolrCloud Setup §  Personalized Search: Lucene + SolrCloud §  Challenges §  Solr4.8 to 5.4 Upgrade
  • 5. ©2016  Sony  Interac6ve  Entertainment     User Search
  • 6. ©2016  Sony  Interac6ve  Entertainment     User Search
  • 7. §  Fast •  query should return < 100 ms §  Reliable / Fault Tolerant §  Scalable •  SolrCloud cluster need to handle: o  Up to 1000 RPS query requests o  Up to 250 RPS indexing requests •  Appr. 300 millions documents §  Ranking search results by friendship. •  Up to n degrees of separation. •  Friends, 2nd degree fiends (friends of friends), etc. ©2016  Sony  Interac6ve  Entertainment     User Search: Requirements
  • 8. ©2016  Sony  Interac6ve  Entertainment     SolrCloud: System Architecture ZooKeeper   SolrCloud  cluster   Leader   a   Replica   Leader   a   Replica   Leader   a   Replica   Leader   a   Replica   ELB   Applica6on   Servers   Database  
  • 9. §  SolrCloud 5.4 §  Documents •  User data (~ 1.5 kb per user) •  ID, Online ID, Name (First, Middle, Last), Privacy, User Type, etc.. •  ~ 300 million documents §  Shards •  4 shards + many replicas. •  # shards determined experimentally. •  Most of the docs on each shard fit in the memory. §  Cache •  Query Result Cache, Document Cache, Filter Cache, etc .. §  Commit •  SoftAutoComit: 5 secs •  AutoCommit: 15 mins (OpenSearcher=false) ©2016  Sony  Interac6ve  Entertainment     SolrCloud: Configurations
  • 10. §  Tokenizers •  Whitespace Tokenizer §  Filters §  Ascii Folding Filter o  Stored and queried with equivalent English alphabets. o  Joan Miró -> Joan Miro §  N-Gram Filter o  abc -> a, b, c, ab, bc, abc o  Takes up more space, but faster than wildcard (*) when queried. §  Lower Case Filter ©2016  Sony  Interac6ve  Entertainment     SolrCloud: Configurations
  • 11. §  People search users they know or they kind of know... §  Search results should be ranked by the friendship between the searcher and the searched (users). ©2016  Sony  Interac6ve  Entertainment     Personalized Search: Overview User  A    <-­‐  Friend  (1st  degree  of  separa6on)   User  B    <-­‐  Friend  (1st  degree  of  separa6on)   User  C    <-­‐  Friend  of  Friend  (2nd  degree  of  separa6on)   ...     User  Y  <-­‐  Not  associated.   User  Z  <-­‐  Not  associated.    
  • 12. ©2016  Sony  Interac6ve  Entertainment     Personalized Search: Ideas q=ps4king& bf=friends:(ID1 or ID2 or ID3 or …)^500& bf=friends2nd:(ID4 or ID5 or ID6 or …)^50& bf=friends3rd:(ID7or ID8 or ID9 or …)^5& … Possible  Solu8on  1  :  Query  SolrCloud  with  the  list  of  friend  IDs.     Problems   •  The  list  of  friends  can  be  very  long  (poten6ally  thousands).   •  Increases  the  query  latency.   Giving  a  higher  boost  for  users  who  are  closer  to  the  caller.  
  • 13. Possible Solution 2: Index the friendship in SolrCloud. Add “friends“ fields, if the caller is in one of the “friends” fields, boost the document. Problems: o  Too many requests to Solr. o  Maintaining friendship in Solr in addition to our database might be overkill. o  Requires a large disk space. ©2016  Sony  Interac6ve  Entertainment     Personalized Search: Ideas
  • 14. ©2016  Sony  Interac6ve  Entertainment     Personalized Search: Our Solution +   Personalized  Index     Stores  people  close  to  the  caller  (friends,   friends  of  friends,  up  to  n  degrees  of   separa6on).     §  Also  used  in  friend  recommenda6on   system.   §  Other  team  already  uses  Lucene  index  for   user  owned  games.     Global  Index     Includes  all  the  users.    
  • 15. ©2016  Sony  Interac6ve  Entertainment     Personalized Search: Lucene + SolrCloud Online  ID   First   Name   ….   Degree  of   Separa6on   ps4Queen   Marge   …   1   ps4King   Homer   ...   1   ps4aweso me   Bart   …   2   …   …   …   …   Lucene  Index  (simplified)   Applica6on   Server   Friendship   Data   §  Lucene  index  created  on-­‐ demand  for  the  caller     §  Cached  temporarily   +  
  • 16. §  Hard to increase the performance using two index systems. (Lucene + SolrCloud) •  Tuned SolrCloud a lot (cache size, query optimization, soft/auto commit settings, GC settings, etc.) §  Not a problem anymore, but SolrCloud had been unstable for a while. •  Entire cluster would have gone down a couple of times a month. ©2016  Sony  Interac6ve  Entertainment     Challenges
  • 17. §  Increased the number of replicas •  When leader goes in recovery, need to have enough replicas to handle all the requests. §  Reconfigured GC settings with CMS (concurrent mark sweep). §  Decreased the size of the document query cache. o  Cache warm-up time was longer than the soft auto commit duration -> was always warming the cache. ©2016  Sony  Interac6ve  Entertainment     Challenges: Instability Solutions
  • 18. ©2016  Sony  Interac6ve  Entertainment     SolrCloud Upgrade §  Motivations •  Originally Solr 4.8 was used, but due to the instability issues, upgraded to Solr 5.4. §  Challeges •  Tried to data stream from a Solr 4.8 node to Solr 5.4 by joining a node, but did not work. •  Some data types have been deprecated. o  IntegerType, LongType -> TrieInteger, TrieLong o  schema.xml needed to be updated with the new data types. o  Decided to full index the 300 million documents in Solr 5.4 cluster.
  • 19. §  First, query out 300M docs and then full indexing. §  Deep paging (specifying start index and limit) is too slow •  Solr needs to cache documents up to the starting index. §  The logical cursor cusorMark is solution to the deep paging problem. The cursorMark returns the next cursor as part of the response. §  cursorMark is not perfect. Sometimes the cursor stops before the end of the documents. Could use filter query to query the certain range of documents by ids. ©2016  Sony  Interac6ve  Entertainment     SolrCloud Upgrade: Full Indexing ...&rows=10&sort=id+asc&cursorMark=AoEjR0JQ  
  • 20. Q  &  A   Any  Ques6ons?