Analyzing Complex Behavior Graphs in Hadoop at Scale
•Download as PPTX, PDF•
12 likes•3,133 views
Employing traditional approaches to analyzing customer behavior graphs and event sequences requires data simplifications, generalizations, and segmentations that severely degrade prediction accuracy and can lead to the loss of valuable information. But there’s an alternative approach that retains the fidelity of customer profile data, the full sequence of events data, and enables powerful business intelligence and predictive analytics. In this webcast, join Apigee’s Joy Thomas, Chief Scientist, and Sanjeev Srivastav, VP Data Strategy, as they explore the superiority of new methods of behavior graph analysis over the simplifications required for traditional data storage and classical predictive algorithms. Join to learn: • How shortcomings of traditional approaches make it difficult to build and analyze complex behavior graphs • Why GRASP—graph and sequence processing technology on Hadoop presents a different and more effective approach to behavior graph analysis • How descriptive analytics using GRASP uncover hidden patterns in historical customer journey data • How Predictive analytics that use behavior graphs and Bayesian algorithms, along with machine learning, ensure model performance over time Download Video: http://youtu.be/CwxvlgW9aZY Download Podcast: https://soundcloud.com/apigee/analyzing-complex-behavior-graphs-in-hadoop-at-scale
Recommended
Recommended
More Related Content
More from Apigee | Google Cloud
More from Apigee | Google Cloud (20)
Recently uploaded
Recently uploaded (20)
Analyzing Complex Behavior Graphs in Hadoop at Scale
- 1. Analyzing Complex Behavior Graphs in Hadoop at Scale
- 2. Today’s speakers Sanjeev Srivastav @ssrivastav Joy Thomas @JoyAThomas1
- 3. Apigee social channels © 2014 Apigee – For Public display 3 YouTube http://youtube.com/apigee Slideshare http://slideshare.com/apigee
- 4. Agenda • Customer behavior graphs • GRASP: an event model for customer interactions • GRASP for – the customer journey – predictive modeling © 2014 Apigee – For Public display
- 6. Why do we need behavior graphs? © 2014 Apigee – For Public display
- 7. Customer view: a journey © 2014 Apigee – For Public display 7
- 8. Understand each customer’s journey © 2014 Apigee – For Public display 8 siloed view customer journey
- 9. Identify common interactions and influences © 2014 Apigee – For Public display 9 customer journey common interactions & influences
- 10. Customer behavior © 2014 Apigee – For Public display 10 Behavior graph • sequence of events: – actions experienced and taken Social graph • links between people & activities – at a particular point in time Behavior graph Social graph
- 11. Why do we need new technology for © 2014 Apigee – For Public display behavior graphs?
- 12. Challenges with current technologies © 2014 Apigee – For Public display 12 • SQL is not efficient for sequences of events and quick counting of customer journeys • Custom algorithms are expensive • Various graph systems are oriented toward social graphs and computations of neighborhoods, “friend of a friend,” etc.
- 13. Event model for customer interactions: GRASP
- 14. GRASP • Graph and sequence processing - time-sequenced graph analytics on Hadoop © 2014 Apigee C– oFnofri dPeunbtilaicl –diAsplll aRyights Reserved 14
- 15. Model for user behavior Users act on nodes in a temporal sequence of events © 2014 Apigee – For Public display 0 1 2 5 3 2 5 0 USER PROFILE UserID: U56 Gender: M Geo: San Francisco Interests: bikes, fashion USER PROFILE UserID: U57 Gender: F Interests: news, finance Age: 35-40 NODE PROFILE Type: Content PageID: P100 Category: product review SubCat: mountain bike NODE PROFILE Type: Creative ID: Creative95 Category: VideoAd Advertiser: BikePros EVENT Type: PageView UserID: U56 PageID: P100 TimeSpent: 180 sec. Scrolls: 3 EVENT Type: AdView UserID: U56 AdID: Creative95 PlayTime: 30 sec. Rewinds: 1
- 16. Aggregated behavior graph © 2014 Apigee – For Public display 0 1 Impressions: 5 TimeSpent: 30 Clicks: 1 2 3 5 0 Combine 1 2 5 Impressions: 1 TimeSpent: 20 Clicks: 1 3 2 5 0 0 Impressions: 4 TimeSpent: 10 Clicks: 0
- 17. Event streams Store visits Emails Phone calls Purchases F, 35, Married Combined event stream © 2014 Apigee – For Public display 17 Event stream Purchase health insurance Offer for Health Insurance M,25, Married GRASP merges event streams and normalizes time relative to responses
- 18. User dimension © 2014 Apigee – For Public display Data representation • Data model: events & dimensions • Data structure: aggregated behavior • Graphs for events, not tables • Data access: API GRASP: graphs vs. tables 18 Event facts are represented as graphs, not tables SQL is not effective for sequence queries 1 2 3 4 0 Node dimension Events Data storage & computation • Distributed data structure on Hadoop • Computation using map reduce
- 19. Dual use of GRASP: customer journey & predictive models
- 20. How has this been used for solving customer engagement problems? © 2014 Apigee – For Public display
- 21. Dual use of GRASP for analytics Graph and sequence processing (GRASP) descriptive analytics GRASP Adaptive applications Segments manager, GQM Business user GQM Data Profiles Text Events Feature extraction Events Predictive & Data scientist Developer Custom app Developer Predictive Descriptive Developer A X Y C D B © 2014 Apigee 2013 C– oFnofri dPeunbtilaicl –diAsplll aRyights Reserved 21
- 22. Visualize customer journeys across all interactions • Examples: telco, healthcare, retail © 2014 Apigee – For Public display
- 23. Build behavior prediction models • Machine learning using path sequences © 2014 Apigee 2013 C– oFnofri dPeunbtilaicl –diAsplll aRyights Reserved
- 24. Relevant problems to solve © 2014 Apigee – For Public display 24 • Multi-channel customer event data for understanding customer journeys and predicting behavior • Modeling evolving customer behavior using updates to behavior graphs
- 25. Questions? Sanjeev Srivastav @ssrivastav Joy Thomas @JoyAThomas1
- 26. Thank you © 2014 Apigee – For Public display
Editor's Notes
- At Apigee, Sanjeev leads the Data Strategy team Joy leads the Data Science team
- Multichannel interaction between customers and companies results in event streams that can be stored in the GRASP data structure, which is built on Hadoop HDFS Analysis of large event stream data sets provides detailed patterns of customer behavior Customer behavior, represented by sequences of event paths, is a very effective means of building predictive models
- Customers engage with companies through email, web sites, mobile apps, social networks, telephone and physical stores. Such engagement involves activities that can be mapped on a timeline, with consideration of the duration of engagement for each activity. External contextual data such as location and weather can also be recorded or used as a real-time input. From a customer’s point of view, as they interact with your business in the digital world, they go through a journey that involves a number of interactions sequenced in by time and spanning multiple channels. They might start their journey on one channel, then move to another, and so on. For example, Sally might receive an offer via email. She views product details from her laptop. She then tweets about it to check if any of her friends like it. Later she might be near your store, and it appears to be a nice, sunny day, so she pulls out her smartphone, checks directions and walks in. However, she doesn’t buy anything, but comes home and follows up with more questions on online chat or call center. And the journey continues. As customers get more accustomed to digital interactions from Amazon, facebook, google, and more, their expectations of how they are treated and served is also increasing. You’ve living through this. For example, We expect a journey and experiences that are easy, convenient, relevant, and contextual, and individualized to our specific needs. If we don’t find it, we tune out, we start looking at other brands, and so on. So for a business to be relevant to their customers, they have to understand the customer journey so they can optimize it to meet the needs of their customers.
- Insights generates the customer journey for each customer
- Insights uses GRASP to provide a comprehensive understanding of customer behavior across channels and gives you a comprehensive understanding of customer behavior across your entire customer base. With this understanding, so can discover common interactions & influences that lead to successful customer journeys. Now you can answer these questions: What journeys are most successful? What actions did customers take after receiving an offer? What actions did customers take before canceling service? Where are you failing to convert? What might be causing that? What can you change to improve conversion? How do you allocate sales credit or contribution across your channels?
- A behavior graph captures the sequence and duration of events, while a social graph is used for storing neighborhood relationships.
- Insights starts with all your customer interactions via web, mobile, store, call center, and other channels….and applies GRASP - Graph and Sequence Processing, a unique time sequenced graph analytics on Hadoop… Insights enables you to analyze the customer journey using a unique big data graph structure that is purpose built to help detect hidden patterns by finding strong signals in a sea of weak signals. Insights starts with fine grained event data - individual interactions that occur at specific points in time across different channels and devices – each product page viewed on web, each offer received on mobile, each call into the call center. Each interactions when viewed in isolation might not signal much. However, when you stitch them together using the unique Big Data structure that Insights offers – a temporal graph structure built directly on Hadoop that is built to analyze such time based sequence of interactions, you start to identify hidden patterns and relationships. In other words, Insights is able to looks at all the weak and strong signals to find strong signals that otherwise would lay dormant. Unlike social graphs which connect nodes based on relationships, GRASP connects events in time – it is ideally suited for analyzing customer interactions. This approach is also different from traditional predictive analytics which summarizes data into a tabular format – you lose precision by summarizing the data.
- Key points: Focus is on Flow & Behavior Flexible data model : Nodes, Users, Events are heterogeneous in the sense of not requiring a homogeneous attribute structure and permitting easy addition or dropping of attributes. For example, the attributes for an event may depend on its type and new types/attributes may be easily added. Customers data may need to be transformed: Customer may have modeled data differently and will need to be transformed to this model. For example, customer’s events may contain the structure of dynamically generated content (attributes that are not metrics) which will end up corresponding to nodes in the model. Sequential model of actions for a user : Concurrent actions are forced into a sequence. Scale - Implementation at scale that comes with analysis of consumer behavior
- Characteristics Represents flow & behavior of all users Automated construction from event logs Information preserving Aggregated representation Permits drill-down Useful for reasoning about traffic flows Count unique users at node/edge Aggregate metrics at nodes/edges Measure drop-offs on a path (funnel) Profile traffic at a node or edge Analyze flows for user segments
- Powerful functionality - multiple applications Patterns for Machine Learning Event sequences with time distances Profiles and Sequence combinations Social Scoring Exploit graph based relationships Segmentation & Profiling Flows/Metrics for segments of consumers Profiles of flows at nodes/edges Path/Tree/Funnel Analysis Flows by source or destination Flow analysis for paths Flows & Metrics Consumer/Session flows on nodes/edges Behavior metrics on nodes/edges
- Insights can help business users visualize the customer journey across all interactions Conduct ad-hoc analysis of customer journey View customer path across all interactions Eg: You can see customers who visited your website, then purchased a product, then called customer service, and then cancelled their order Visualize multiple paths – roll back or roll forward Visualize any path in the data Start with a specific event and view future events along the path E.g.: start with customers who visited website and determine the number who purchased that product Start with a specific event and view preceding events along that path E.g.: start with customers who canceled their contract, and determine the number who called customer service prior to canceling their contract Drill down into analyze additional details View additional details for each node
- Insights can help data scientists build sophisticated models faster using R Insights provides a R-based modeling environment for data scientists & statisticians Data Scientists can build models faster by using R, a popular tool among data scientists Data Scientists can Create, score, and test new models that leverage machine learning on GRASP The models they build can incorporate unstructured and structured data. The ability to include unstructured data allows them to models that result in higher precision They can experiment with different model configurations Background: Model Configuration = data + parameters Insights provides support for training and test environments They can measure model performance (lift curve)