How Apache Drives Music Recommendations At Spotify

•

12 likes•5,868 views

The slides go through the high-level process of generating personalized playlists for all Spotify's users, using Apache big data products extensively. Presentation given at Apache: Big Data Europe conference on September 29th, 2015 in Budapest.

Technology

HowApache
Drives Music
Recommendations
At Spotify
Josh Baer (jbx@spotify.com)
Note:The view expressed is my own and does not necessarily represent that of Spotify

WhoAm I?
• Technical Product Owner at
Spotify
• Working with batch and fast
processing infrastructure
@l_phant

What is Spotify?
• Music Streaming Service
• Launched in 2008
• Free and PremiumTiers
• Available in 58 Countries

How do we recommend a
personalized playlist of
new music to 75+ million
users?

10.123.133.333 - - [Mon, 3 June 2015 11:31:33 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1847 "https://my.analytics.app/
admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.123.133.222 - - [Mon, 3 June 2015 11:31:43 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1984 "https://my.analytics.app/
admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36”
10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.321.145.111 - - [Mon, 3 June 2015 11:33:03 GMT] "GET /api/loggedInUser HTTP/1.1" 304 - "https://my.analytics.app/dashboard/courses/
1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.112.322.111 - - [Mon, 3 June 2015 11:33:03 GMT] "POST /api/instrumentation/events/new HTTP/1.1" 200 2 "https://my.analytics.app/
dashboard/courses/1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81
Safari/537.36”
10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
It begins with a log

Apache Kafka at Spotify
•340 Kafka-related nodes
•30TB/day from logs

How do we store TBs of
new data every data?

Apache Hadoop at Spotify
• 1700 Nodes
• 60 PB of Data
• 70TB of Memory
• Over 1 Million jobs run in Q3, 2015

ProcessingGrowth
150%
250%
350%
450%
550%
Q4-2013 Q1-2014 Q2-2014 Q3-2014 Q4-2014 Q1-2015 Q2-2015 Q3-2015
Hadoop at Spotify

Processing Toolbox
• Apache Crunch
• Scalding
• Apache Hive
• Apache Spark
• Apache Storm
• Hadoop Streaming
• Apache Pig

Storage Formats
• ApacheAvro
• Apache Parquet

Collaborative Filtering
Justin Bieber Drake Avicii Major Lazer
Anna Listened Listened
Gustav Listened Listened Listened
Mary Listened Listened Listened Listened
Michael Listened ListenedSuggest

How do we serve new
playlists to all our users
every week?

Apache Cassandra at Spotify
• Number of Clusters: 113
• Number of Machines: 1155
• Largest Cluster: 60 Nodes

Thank YOU for your
contributions to
Apache products!

Spotify Luigi
•Workflow Manager
•Over 150 contributors
•Used by 10s, possibly 100s of companies

Maybe…Apache Luigi?
Sponsors/mentors/contributors wanted!

Think this stuff is interesting?
We have a great time building it!
spotify.com/jobs

Better Spotify ML Presentations
• Algorithmic Music Recommendations at Spotify (Chris Johnson)
• Interactive Recommender Systems with Netflix and Spotify (Chris Johnson)
• Music recommendations @ MLConf 2014 (Erik Bernhardsson)
• Machine learning @ Spotify (Andy Sloane)
• Recommending music on Spotifywith deep learning (Sander Dieleman)
• Scala Data Pipelines @ Spotify (Neville Li)
• Spotify's Music Recommendations LambdaArchitecture (Esh Kumar and Emily Samuels)

What's hot

Music recommendations @ MLConf 2014

Erik Bernhardsson

Music Personalization At Spotify

Vidhya Murali

Spotify: Data center & Backend buildout

David Poblador i Garcia

Are you still building data pipelines with Java and Python? Are you curious about the current buzz in the Big Data community surrounding Scala as a data processing environment? In this talk I'll discuss how Spotify migrated its music recommendations pipeline from Python to Scala. I'll dive into the language specific features that make Scala the ideal candidate for big data processing as well as highlight the rich set of tools and APIs that we take advantage of to process music recommendations for our 50 Million active users including Scalding, Breeze, Kafka, Spark, Parquet, Driven and Zeppelin.

Scala Data Pipelines for Music Recommendations

Chris Johnson

Music Personalization : Real time Platforms.

Esh Vckay

Spotify in the Cloud - An evolution of data infrastructure - Strata NYC

Josh Baer

How data drives spotify

Ali Sarrafi

Collaborative Filtering at Spotify

Erik Bernhardsson

Scala Data Pipelines @ Spotify

Neville Li

These are the slides of my talk at the 2019 Netflix Workshop on Personalization, Recommendation and Search (PRS). This talk is based on previous talks on research we are doing at Spotify, but here I focus on the work we do on personalizing Spotify Home, with respect to success, intent & diversity. The link to the workshop is https://prs2019.splashthat.com/. This is research from various people at Spotify, and has been published at RecSys 2018, CIKM 2018 and WWW (The Web Conference) 2019.

Personalizing the listening experience

Mounia Lalmas-Roelleke

Spotify: behind the scenes

Ricardo Vice Santos

Spotify uses a range of Machine Learning models to power its music recommendation features including the Discover page, Radio, and Related Artists. Due to the iterative nature of these models they are a natural fit to the Spark computation paradigm and suffer from the IO overhead incurred by Hadoop. In this talk, I review the ALS algorithm for Matrix Factorization with implicit feedback data and how we’ve scaled it up to handle 100s of Billions of data points using Scala, Breeze, and Spark.

Music Recommendations at Scale with Spark

Chris Johnson

Spotify architecture - Pressing play

Niklas Gustavsson

Building Data Pipelines for Music Recommendations at Spotify

Vidhya Murali

CF Models for Music Recommendations At Spotify

Vidhya Murali

These are the slides of a talk about some of our research at Spotify, as part of the celebration kickoff of Chalmers AI Research Centre in Gothenburg. I always like to make a story in my talk, and this time I wanted to reflect on the "push" (think recommender system) and "pull" (think search) paradigms. I am using this quote from Nicholas Belkin and Bruce Croft from their Communications of the ACM article published in 1992 to frame my story: "We conclude that information retrieval and information filtering are indeed two sides of the same coin. They work together to help people get the information needed to perform their tasks."

Recommending and Searching (Research @ Spotify)

Mounia Lalmas-Roelleke

Spotify uses a range of Machine Learning models to power its music recommendation features including the Discover page and Radio. Due to the iterative nature of training these models they suffer from IO overhead of Hadoop and are a natural fit to the Spark programming paradigm. In this talk I will present both the right way as well as the wrong way to implement collaborative filtering models with Spark. Additionally, I will deep dive into how Matrix Factorization is implemented in the MLlib library.

Collaborative Filtering with Spark

Chris Johnson

Recommending and searching @ Spotify

Mounia Lalmas-Roelleke

Storm at Spotify

Neville Li

In this presentation, I give an overview of the machine learning algorithms behind Spotify’s extraordinarily popular Discover Weekly playlist. I provide a brief introduction to what the playlist is, explain how music recommendation engines have evolved over time, then break down the three main algorithm types powering Spotify’s recommendations: (1) collaborative filtering, (2) Natural Language Processing (NLP), and (3) Raw audio analysis. Video of the presentation can be found here: https://www.youtube.com/watch?v=PUtYNjInopA

Spotify Discover Weekly: The machine learning behind your music recommendations

Sophia Ciocca

What's hot (20)

Music recommendations @ MLConf 2014

Music Personalization At Spotify

Spotify: Data center & Backend buildout

Scala Data Pipelines for Music Recommendations

Music Personalization : Real time Platforms.

Spotify in the Cloud - An evolution of data infrastructure - Strata NYC

How data drives spotify

Collaborative Filtering at Spotify

Scala Data Pipelines @ Spotify

Personalizing the listening experience

Spotify: behind the scenes

Music Recommendations at Scale with Spark

Spotify architecture - Pressing play

Building Data Pipelines for Music Recommendations at Spotify

CF Models for Music Recommendations At Spotify

Recommending and Searching (Research @ Spotify)

Collaborative Filtering with Spark

Recommending and searching @ Spotify

Storm at Spotify

Spotify Discover Weekly: The machine learning behind your music recommendations

Viewers also liked

Shortening the feedback loop

Josh Baer

Interactive recommender systems enable the user to steer the received recommendations in the desired direction through explicit interaction with the system. In the larger ecosystem of recommender systems used on a website, it is positioned between a lean-back recommendation experience and an active search for a specific piece of content. Besides this aspect, we will discuss several parts that are especially important for interactive recommender systems, including the following: design of the user interface and its tight integration with the algorithm in the back-end; computational efficiency of the recommender algorithm; as well as choosing the right balance between exploiting the feedback from the user as to provide relevant recommendations, and enabling the user to explore the catalog and steer the recommendations in the desired direction. In particular, we will explore the field of interactive video and music recommendations and their application at Netflix and Spotify. We outline some of the user-experiences built, and discuss the approaches followed to tackle the various aspects of interactive recommendations. We present our insights from user studies and A/B tests. The tutorial targets researchers and practitioners in the field of recommender systems, and will give the participants a unique opportunity to learn about the various aspects of interactive recommender systems in the video and music domain. The tutorial assumes familiarity with the common methods of recommender systems.

Interactive Recommender Systems with Netflix and Spotify

Chris Johnson

Big Data At Spotify

Adam Kawa

At Spotify we collect huge volumes of data for many purposes. Reporting to labels, powering our product features, and analyzing user growth are some of our most common ones. Additionally, we collect many operational metrics related to the responsiveness, utilization and capacity of our servers. To store and process this data, we use scalable and fault-tolerant multi-system infrastructure, and Apache Hadoop is a key part of it. Surprisingly or not, Apache Hadoop generates large amounts of data in the form of logs and metrics that describe its behaviour and performance. To process this data in a scalable and performant manner we use … also Hadoop! During this presentation, I will talk about how we analyze various logs generated by Apache Hadoop using custom scripts (written in Pig or Java/Python MapReduce) and available open-source tools to get data-driven answers to many questions related to the behaviour of our 690-node Hadoop cluster. At Spotify we frequently leverage these tools to learn how fast we are growing, when to buy new nodes, how to calculate the empirical retention policy for each dataset, optimize the scheduler, benchmark the cluster, find its biggest offenders (both people and datasets) and more.

Hadoop Operations Powered By ... Hadoop (Hadoop Summit 2014 Amsterdam)

Adam Kawa

Playlists at Spotify - Using Cassandra to store version controlled objects

Jimmy Mårdell

Spotify's Music Recommendations Lambda Architecture

Esh Vckay

Apache Hadoop In Theory And Practice

Adam Kawa

HCatalog

GetInData

Docker at Spotify - Dockercon14

dotCloud

Empowering Engineering Talent - an update from Spotify

Kevin Goldsmith

How spotify makes product

Ali Sarrafi

Activation: From thinking to tweaking it, how we do it at Spotify

TheFamily

Scaling Operations At Spotify

David Poblador i Garcia

Most of the people think that quality in software development is limited to manual testing on the latest stage before releasing a product. That might be true 20 years ago in the industrial era. But current world is much more dynamic than before. Time to market became the most crucial metric nowadays. Releasing code to production need to be done faster and faster. How to maintain quality on a sufficient level in this fast paced environment? How to find a time to work on quality improvements? Those are two main questions I want to answer during this talk. Do not expect a silver bullet or even receipt to success. But definitely expect a lot of information about continuous delivery/deployment/improvements with a case studies and lessons we learned at Spotify. Spotify Engineering Culture: https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/ https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/ Scaling Agile @ Spotify http://blog.crisp.se/2012/11/14/henrikkniberg/scaling-agile-at-spotify Scaled Agile @ Spotify http://vimeo.com/111131934

Quality Built In @ Spotify

Andrii Dzynia

Machine learning @ Spotify - Madison Big Data Meetup

Andy Sloane

This was an extended version of the talk that I gave at InfoShare 2016 in GDansk. This version of the talk was presented at ao.com and Think Money in Manchester UK in May 2016. This is a remix of several earlier talks and some new content to tie Spotify's autonomy and continuous improvement culture to it's data-driven product development approach to show the complete picture. As usual, I tend to talk to slides instead of putting a lot of the content into the slides themselves, so sorry if these don't have all the info.

How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...

Kevin Goldsmith

Yahoo: Experiences with MySQL GTID and Multi Threaded Replication

Yashada Jadhav

Présentation Spotify: Profils d'écoute et data storytelling @ Radio 2.0 2015 V rencontres Radio 2.0 - Mardi 13 octobre 2015 'La radio 2.0 et vous ? Raconte-moi une histoire' www.rr20.fr 50 intervenants / 10 keynotes / 4 tables rondes / 7 workshops / 4 études exclusives Grands Prix Radio 2.0 / La Radio des Rencontres - Organisateurs : Nicolas Moulard / Actuonda, Xavier Filliol / Editions de l'Octet, Sacem Université, INA - Partenaires Platinum : Mediametrie - Partenaires Grand Prix : Scam, Multivote - Partenaires Gold : Spotify, Deezer, Hyperworld, Musicovery, Targetspot, Triton Digital - Partenaires Média : CBNews, La Lettre Pro de la Radio, RadioPub, Media +, Edition Multimedia, Satellinet, French Web, La Correspondance de la Publicité - La Radio des Rencontres : Broadcast Associés, Radioline, Glowbl

Spotify: Profils d'écoute et data storytelling @ Radio 2.0 2015

ACTUONDA

Music Recommendations at Spotify

Emily Samuels

Impact et attractivité des concerts des radio pour les auditeurs Etude HyperWorld Salon de la Radio 2017 VI Rencontres Radio 2.0 ' Audio Digital Reboot' Lundi 30 janvier 2017 / Grande Halle de la Villette / Salon de la Radio-European Radio Show 45 intervenants : 7 Keynotes / 4 Tables Rondes / 4 Etudes exclusives / 7 Workshops www.rr20.fr Partenaires Gold : Deezer, Spotify, RCS, Blitzr, Sync, Coyote, ACPM, Sacem, Kantar Media Partenaires Communication : Mediamétrie, Scam, INA, Hyperworld Partenaires Media : La Lettre Pro, Radiopub, CBNews, Mind, Influencia, Frenchweb, Satellifax, Media+, Longeur D'ondes, RadioWorld Co-Organisateurs : Les Editions de l'Octet, Actuonda

Impact et attractivité des concerts des radio pour les auditeurs Etude HyperW...

ACTUONDA

Viewers also liked (20)

Shortening the feedback loop

Interactive Recommender Systems with Netflix and Spotify

Big Data At Spotify

Hadoop Operations Powered By ... Hadoop (Hadoop Summit 2014 Amsterdam)

Playlists at Spotify - Using Cassandra to store version controlled objects

Spotify's Music Recommendations Lambda Architecture

Apache Hadoop In Theory And Practice

HCatalog

Docker at Spotify - Dockercon14

Empowering Engineering Talent - an update from Spotify

How spotify makes product

Activation: From thinking to tweaking it, how we do it at Spotify

Scaling Operations At Spotify

Quality Built In @ Spotify

Machine learning @ Spotify - Madison Big Data Meetup

How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...

Yahoo: Experiences with MySQL GTID and Multi Threaded Replication

Spotify: Profils d'écoute et data storytelling @ Radio 2.0 2015

Music Recommendations at Spotify

Impact et attractivité des concerts des radio pour les auditeurs Etude HyperW...

Similar to How Apache Drives Music Recommendations At Spotify

Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...

Big Data Spain

fluent-plugin-beats at Elasticsearch meetup #14

N Masahiro

Guidelines for productive full stack data engineers

DataWorks Summit

Productive data engineer

Rafał Wojdyła

Music Hackday Boston - The Last.fm API

davidsingleton

How do volunteer open-source projects create and maintain so many compelling, competitive products? What is the Open Source Secret Sauce? Join open-source insider, Ted Husted, as he takes us deep inside the Apache Software Foundation, to show how the sausages are made. In this session, you will learn * Why open source matters; * How open source development works at the ASF; * What makes open source projects successful.

Open source-secret-sauce-rit-2010

Ted Husted

Siphon is a highly available and reliable distributed pub/sub system built using Apache Kafka. It is used to publish, discover and subscribe to near real-time data streams for operational and product intelligence. Siphon is used as a “Databus” by a variety of producers and subscribers in Microsoft, and is compliant with security and privacy requirements. It has a built-in Auditing and Quality control. This session will provide an overview of the use of Kafka at Microsoft, and then deep dive into Siphon. We will describe an important business scenario and talk about the technical details of the system in the context of that scenario. We will also cover the design and implementation of the service, the scale, and real world production experiences from operating the service in the Microsoft cloud environment.

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022

Timothy Spann

https://arcade.sqlbits.com/sessions/ Sql bits apache nifi 101 Introduction and best practices 11-March-2022 UK https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html https://github.com/tspannhw/EverythingApacheNiFi https://www.datainmotion.dev/2020/12/basic-understanding-of-cloudera-flow.html https://www.datainmotion.dev/2020/10/top-25-use-cases-of-cloudera-flow.html In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker or in CDP Public Cloud. I will cover: Terminology Flow Files Version Control Repositories Basic Record Processing Provenance Backpressure Prioritizers System Diagnostics Processors Process Groups Scheduling and Cron Bulletin Board Relationships Routing Tasks Networking Basic Cluster Architecture Listeners Controller Services Remote Ports Handling Errors Funnels Feedback LInk - https://sqlb.it/?7108 ROOM 04 Fri 12:00 - 12:50

Sql bits apache nifi 101 Introduction and best practices

Timothy Spann

The log shipping scene been between us for a long time: from syslog, rsyslog to nowadays Fluentd, Flume and Logstash. Logstash been pushing hard to introduce new features that make the experience better for everyone. At the end of the day, a healthy shipper means a happy sysadmin. The latest Logstash includes persistence to reduce the chance of data loss, monitoring to find how everything is going and configuration management to make your life a lot easier. But wait, there’s more! Offline support, improved shutdown semantics, etc … features that will make your logs shipped and you a rested sysadmin. In this talk we’ll see this features in action thought a real live sensor monitoring example. By the end of the session, you will be able to use the full power of Logstash in your own deployments.

OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes

NETWAYS

Apache Pulsar Community-Jennifer

StreamNative

Yuto Kawamura Building a company-wide data pipeline on Apache Kafka - engineering for 150 billion messages per day Summary: LINE is a messaging service with 200 million monthly active users. Our service architecture evolves daily with various collaborating components. I'll introduce overview of LINE's messaging service architecture, mainly focusing on our company-wide data pipeline infrastructure built upon Apache Kafka which accepts more than 150 billion messages every day, making it one of the largest in the world. In this talk I will introduce how we managed such scale keeping it highly reliable to be capable of being an infrastructure to build services.

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

LINE Corporation

Designing a social network offers some exciting challenges to engineers. The system needs to operate at scale, to provide a responsive user experience and to be able to inspect user activity in order to both generate new content and improve how the existing content is delivered. Event Driven Architectures are particularly suitable to handle these kind of challenges and highly scalable messaging systems such as Apache Kafka have been designed specifically to support the requirements of modern high volume applications. In this talk we are describing how the Crowdmix back-end has been designed as an Event Based system running on top of Kafka. We are going to present the overall system architecture and discuss in more detail some of the different sub components processing those events in different fashions, from streaming based processing to batch processing passing through a lambda-style batch and stream cooperation. We are going to conclude describing some lessons learned from our one-year journey in implementing and operating the system

Music streams

Stefano Galarraga

Create Your Own Chatbot with Hubot and CoffeeScript

Rob Scaduto

Trend Micro Big Data Platform and Apache Bigtop

Evans Ye

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Introduction Apache Kafka

Joe Stein

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

LINE Corporation

Last.fm API workshop - Stockholm

Matthew Ogle

What are the most common problems with testing environments? - You are not the only one who is using it. - Test failures are not repeatable. - Test data can be easily messed up due to tests overlap. Those problems are introducing flakiness in your tests, increase frustration level and decrease confidence in quality of a product you are building. Forcing your development team to have a testing queue increases delivery time dramatically. Creating zillions of environments does not sound as cheapest solution either. At Spotify we experimented with different approaches on how testing environments can be configured: from shared environment to mocks, stubs and hermetic servers. During my presentation I will share the lessons we learned, what worked, what not and what is the direction we are pursuing in order to stabilise our testing suites.

Hermetic environment for your functional tests

Andrii Dzynia

Similar to How Apache Drives Music Recommendations At Spotify (20)

Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...

fluent-plugin-beats at Elasticsearch meetup #14

Guidelines for productive full stack data engineers

Productive data engineer

Music Hackday Boston - The Last.fm API

Open source-secret-sauce-rit-2010

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022

Sql bits apache nifi 101 Introduction and best practices

OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes

Apache Pulsar Community-Jennifer

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

Music streams

Create Your Own Chatbot with Hubot and CoffeeScript

Trend Micro Big Data Platform and Apache Bigtop

Cloud lunch and learn real-time streaming in azure

Introduction Apache Kafka

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

Last.fm API workshop - Stockholm

Hermetic environment for your functional tests

Recently uploaded

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Developing An App To Navigate The Roads of Brazil

V3cube

Real Time Object Detection Using Open CV

Khem

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Recently uploaded (20)

Artificial Intelligence: Facts and Myths

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Advantages of Hiring UIUX Design Service Providers for Your Business

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Scaling API-first – The story of a global engineering organization

Axa Assurance Maroc - Insurer Innovation Award 2024

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Developing An App To Navigate The Roads of Brazil

Real Time Object Detection Using Open CV

How to Troubleshoot Apps for the Modern Connected Worker

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

GenCyber Cyber Security Day Presentation

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Powerful Google developer tools for immediate impact! (2023-24 C)

2024: Domino Containers - The Next Step. News from the Domino Container commu...

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Boost PC performance: How more available memory can improve productivity

presentation ICT roal in 21st century education

How Apache Drives Music Recommendations At Spotify

1. HowApache Drives Music Recommendations At Spotify Josh Baer (jbx@spotify.com) Note:The view expressed is my own and does not necessarily represent that of Spotify

2. WhoAm I? • Technical Product Owner at Spotify • Working with batch and fast processing infrastructure @l_phant

3. Music Discovery in the 90s

4. What is Spotify? • Music Streaming Service • Launched in 2008 • Free and PremiumTiers • Available in 58 Countries

5. 75+ MillionActive Users

6. 30+ Million Songs

7. 1+ Billion Plays/Day

8. Music Recommendations with Apache

10.

11. How do we recommend a personalized playlist of new music to 75+ million users?

12. 10.123.133.333 - - [Mon, 3 June 2015 11:31:33 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1847 "https://my.analytics.app/ admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.123.133.222 - - [Mon, 3 June 2015 11:31:43 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1984 "https://my.analytics.app/ admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36” 10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.321.145.111 - - [Mon, 3 June 2015 11:33:03 GMT] "GET /api/loggedInUser HTTP/1.1" 304 - "https://my.analytics.app/dashboard/courses/ 1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.112.322.111 - - [Mon, 3 June 2015 11:33:03 GMT] "POST /api/instrumentation/events/new HTTP/1.1" 200 2 "https://my.analytics.app/ dashboard/courses/1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36” 10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" It begins with a log

13.

14.

15. Apache Kafka at Spotify •340 Kafka-related nodes •30TB/day from logs

16.

17. How do we store TBs of new data every data?

18.

19. Apache Hadoop at Spotify • 1700 Nodes • 60 PB of Data • 70TB of Memory • Over 1 Million jobs run in Q3, 2015

20. ProcessingGrowth 150% 250% 350% 450% 550% Q4-2013 Q1-2014 Q2-2014 Q3-2014 Q4-2014 Q1-2015 Q2-2015 Q3-2015 Hadoop at Spotify

21. Processing Toolbox • Apache Crunch • Scalding • Apache Hive • Apache Spark • Apache Storm • Hadoop Streaming • Apache Pig

22. Storage Formats • ApacheAvro • Apache Parquet

23. How do we personalize the playlists?

24.

25. Collaborative Filtering Justin Bieber Drake Avicii Major Lazer Anna Listened Listened Gustav Listened Listened Listened Mary Listened Listened Listened Listened Michael Listened ListenedSuggest

26.

27. How do we serve new playlists to all our users every week?

28.

29. Apache Cassandra at Spotify • Number of Clusters: 113 • Number of Machines: 1155 • Largest Cluster: 60 Nodes

30. Driven By Data

31. Driven ByApache

32. Thank YOU for your contributions to Apache products!

33. One Last Thing…

34. Spotify Luigi •Workflow Manager •Over 150 contributors •Used by 10s, possibly 100s of companies

35. Maybe…Apache Luigi? Sponsors/mentors/contributors wanted!

36. Think this stuff is interesting? We have a great time building it! spotify.com/jobs

37. Better Spotify ML Presentations • Algorithmic Music Recommendations at Spotify (Chris Johnson) • Interactive Recommender Systems with Netflix and Spotify (Chris Johnson) • Music recommendations @ MLConf 2014 (Erik Bernhardsson) • Machine learning @ Spotify (Andy Sloane) • Recommending music on Spotifywith deep learning (Sander Dieleman) • Scala Data Pipelines @ Spotify (Neville Li) • Spotify's Music Recommendations LambdaArchitecture (Esh Kumar and Emily Samuels)

How Apache Drives Music Recommendations At Spotify

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to How Apache Drives Music Recommendations At Spotify

Similar to How Apache Drives Music Recommendations At Spotify (20)

Recently uploaded

Recently uploaded (20)

How Apache Drives Music Recommendations At Spotify