Modern Relationships — AI in Customer Experience w/ Dollar Shave Club
•Download as PPTX, PDF•
1 like•321 views
Zendesk and Dollar Shave Club present on using AI and ML in improving customer experiences
Recommended
Recommended
More Related Content
Similar to Modern Relationships — AI in Customer Experience w/ Dollar Shave Club
Similar to Modern Relationships — AI in Customer Experience w/ Dollar Shave Club (20)
Recently uploaded
Recently uploaded (20)
Modern Relationships — AI in Customer Experience w/ Dollar Shave Club
- 1. Machine Learning and Customer Experience MODERN RELATIONSHIPS
- 2. GM of Guide and Data Products, Zendesk @channelthetiger Manager - Analytics, Member Services, Dollar Shave Club @CrumpleyB Jason Maynard Brian Crumpley
- 4. MACHINE LEARNING FOR CUSTOMER SERVICE Predictive insights, recommendations, and automations to decrease the effort for customers Satisfaction Prediction Answer Bot
- 5. we <3 d8ta
- 7. STANDING ON THE SHOULDERS OF GIANTS Build great customer experiences with the best open source and cloud technology
- 8. ONE EMBEDDED MODEL PER LANGUAGE & DATA LOCALITY Solves the “cold start” problem and enabled small businesses to leverage AI
- 9. TWO PHASES OF LEARNING MEANS MORE TRAINING DATA Deep learning models are hungry, so find ways to stretch the training data you have RATED GOOD RATED BAD ANSWERS WITH AN ARTICLE
- 10. UX IS STILL KING The experience matters as much as the model (sometimes more).
- 11. CREATE EXPERIENCES THAT ADD TO YOUR TRAINING DATA Label more data wherever and whenever possible
- 21. TM and © 2017 Zendesk Inc. All rights reserved.
Editor's Notes
- Brian manages analytics at DSC who provide an amazing line of affordable men’s grooming products, available via subscription. If you haven’t seen their ad on YouTube, you should check it out. They are a great Zendesk customer and Brian will explain how they use some of our machine learning products.
- If you’re not familiar with Zendesk yet, Zendesk builds customer service software to help businesses deliver better customer service. We provide a family products to make it easier for customers to reach out for support across any channel or self-serve, or
- Our machine learning team works on taking the exhaust from all those customer interactions and turning them into products for our customers. Satisfaction Prediction is a feature for our Support product that uses historical satisfaction surveys to help customers predict current conversations that could result in a negative customer experience Answer Bot is our latest machine learning feature. It automates responses to customers when there is a relevant knowledge base article that can answer their question.
- Customer service is a great place to apply machine learning. Zendesk handles about 4.6M customer service interactions per day and many interactions by agents are repetitive answers to questions and we can automate a lot of the minutia of answering questions, so customers can get their answers faster and agents can focus on more nuanced customer conversations.
- To give you a sense of why this is important to customers. One of the biggest influencers of customer satisfaction is faster response time. The x-axis shows first reply time in minutes and the y-axis is customer satisfaction. Answer-bot solved tickets have the same or higher satisfaction rating - makes sense because 76% of people want to self serve. We’ve been working on building these products since 2014 and we’ve learned a few tricks
- I don’t know how big Google or AWS’s machine learning teams are, but it is safe to say that Zendesk’s data science squad is substantially smaller. That said, we aren’t in the business of building a machine learning platform. We're a product company focused on building great customer experiences and there are plenty of opportunities to use the latest machine learning cloud and open source technology. We use a of open source packages and even contribute back to serving platforms like TensorFlow. We also have a hybrid cloud footprint, which is great for leveraging the rapidly evolving cloud technologies for ML eg. GPUs on AWS to TPUs on Google. We also believe part of our differentiation is that we're leveraging the most advanced technology on our hybrid cloud footprint and the latest open source libraries for machine learning. Our infrastructure footprint across CoLos, AWS, and Google allows us to leverage the most cutting edge technologies and services for machine learning. For example, we are currently able to leverage AWS hosted Nvidia GPU processors needed for our most computationally intense neural net model training, but when Google launches their long awaited Tensor Processing Units (TPUs), purpose-built processing units for AI, we'll be able to evaluate and leverage that technology where it makes the most sense. If we were to rollout this infrastructure ourselves, we'd be locked in our own proprietary or private cloud technologies. We are also leveraging powerful open source libraries from TensorFlow like doc2vec, long short term memory (LSTM), and convolutional neural networks (CNNs) in our NN models. We feel our differentiation will be on applying these technologies to the 4.6M daily interactions between customers and business on Zendesk.
- Satisfaction Prediction and the first iteration of Answer Bot used vanilla ML models; Hashing, TFIDF, and logistic regression for classification. This worked well and was a great way to get started. That said, it had a drawback. It required us to train a model per customer and wait for a customer to get enough data in Zendesk to use the feature. For a company that has a lot of SMB customers and prize ourselves on having a seamless web-trial-buy experience this didn’t work. For Answer Bot, we bit the bullet and moved to an embedded space deep learning model. We train one global model by language and data locality. This global model not only solved the "cold start" problem, but it also meant that small businesses that don't have the data volumes could leverage the power of this AI R&D. This combined with our approach around easy-to-use UX will help our products receive broad adoption, further driving the virtuous cycle of creating more data and better recommendations.
- We receive about 6.6M customer satisfaction rating per month and we have 2.2M agent responses that contain links to knowledge base articles. We only wanted to use positive interactions where articles were used to train our model, but the intersection of that data is very small. The team developed a 2-phase training approach so a RNN first learns to recognize positive interactions, then learns to find the relevant answers within all articles with a link. With this approach we were able to extend our training set by nearly 60% to 12M ticket-article pairs.
- Does anyone know what one of the most impactful changes Google has made to click through rates on ads? Some segmentation change? Enhancements in ad targeting? These were important, but they found that when they changed the label on promoted ads from yellow to grey that a lot more people clicked on the promoted links… b/c they didn’t know it was an ad. We’ve talked a lot about what we did on the data and machine learning side to improve our resolution rates, but we’ve found some of our biggest gains from changes that had nothing to do with the model. We have a lot of customers that have their articles behind a login. In our first version, when we’d send a link to an article, a customer would click the link and have to authenticate before reading the article even though we already knew who that user was when they sent the request. We implemented a change to give the user a one time auth token at the login and it increased the deflection rates by 2 percentage points.
- Even though we have a baseline model to start, Answer Bot creates more data as customers use it. We capture feedback on suggestions from agents and customers and this feedback loop continues to improve Answer Bot the more customers that use it, creating a virtuous cycle of improvement. Feedback is king.
- So the second assumption was partially right. You will get better performance, but it will take work. Finally, to where we are today.