Developing Data Products

•Download as PPTX, PDF•

25 likes•8,496 views

Examples, techniques, and lessons learned building data products over the last 3 years at LinkedIn. Pete Skomoroch is a Principal Data Scientist at LinkedIn where he leads a team focused on building data products leveraging LinkedIn's powerful identity and reputation data. The talk describes some techniques and best practices applied to develop products like LinkedIn Skills & Endorsements. This was the inaugural UberData Tech Talk, held in SF at Uber HQ.

Developing Data Products
Uber Tech Talk
Pete Skomoroch @peteskomoroch
December 5 2012

©2012 LinkedIn Corporation. All Rights Reserved.

Examples, Techniques, & Lessons Learned

Developing Data Products

Our Mission
Connect the world’s professionals to make them
more productive and successful.

Our Vision
Create economic opportunity for every
professional in the world.

Members First!

LinkedIn is the leading professional network site

187M+
1

LinkedIn Members

2

640M+ Worldwide
Professionals
2

3,300M+
Worldwide Workforce

©2012 LinkedIn Corporation. All Rights Reserved. 4

LinkedIn profiles represent our professional identity

1 2

187M Members 187M Member
Profiles

©2012 LinkedIn Corporation. All Rights Reserved. 5

We have a lot of data.

©2012 LinkedIn Corporation. All Rights Reserved.

We have a lot of data.
And (like everyone else), we store it in Hadoop.

©2012 LinkedIn Corporation. All Rights Reserved.

We have a lot of data.
And (like everyone else), we store it in Hadoop.
And people build awesome things with that data.

©2012 LinkedIn Corporation. All Rights Reserved.

Collaborative Filtering: LinkedIn Skill Pages

©2012 LinkedIn Corporation. All Rights Reserved.

Classification: giving structure to unstructured data

Extract

©2012 LinkedIn Corporation. All Rights Reserved.

Clustering & Disambiguation

©2012 LinkedIn Corporation. All Rights Reserved.

De-duplication and Normalization

©2012 LinkedIn Corporation. All Rights Reserved.

Network Algorithms: Relevance & Ranking

©2012 LinkedIn Corporation. All Rights Reserved. 15

Prediction: Personalized Skill Recommendations

©2012 LinkedIn Corporation. All Rights Reserved.

Skill Endorsements

©2012 LinkedIn Corporation. All Rights Reserved.

Social Proof and the Skill Endorsement Graph

©2012 LinkedIn Corporation. All Rights Reserved. 20

The Economic Graph: Skills, Jobs, People, Locations…

Location

©2012 LinkedIn Corporation. All Rights Reserved. 21

Lessons learned developing data
products

Collect the right data at the right time

Large amounts of data can reveal new patterns
Probability of Job Title

Months since graduation
©2012 LinkedIn Corporation. All Rights Reserved. 24

Be wary of “black-box” approaches

©2012 LinkedIn Corporation. All Rights Reserved. 25

Look at your data

©2012 LinkedIn Corporation. All Rights Reserved. 26

Aggregate statistics can be misleading

12

10

8

6

4

2

0
1 2 3 4 5 6 7 8 9 10

©2012 LinkedIn Corporation. All Rights Reserved. 27

Build a viewer app, “micro-listen”

©2012 LinkedIn Corporation. All Rights Reserved. 28

Algorithmic intuition: include data geeks in design

©2012 LinkedIn Corporation. All Rights Reserved. 29

OODA: Think like a jet fighter

©2012 LinkedIn Corporation. All Rights Reserved. 30

OODA: Observe, Orient, Decide, Act

©2012 LinkedIn Corporation. All Rights Reserved. 31

OODA: The speed you can move determines victory

©2012 LinkedIn Corporation. All Rights Reserved. 32

Red teaming: what can go wrong likely will

©2012 LinkedIn Corporation. All Rights Reserved. 33

Error data is super valuable, analyze it and adapt

©2012 LinkedIn Corporation. All Rights Reserved. 34

Conclusion: tips for developing data products

 Collect the right data at the right time
 Large amounts of data can reveal new patterns
 Be wary of “black box” approaches
 Look at your raw data
 Aggregate statistics can be misleading
 Build and use viewer apps
 Include data geeks in design process
 OODA: Think like a jet fighter
 Red-teaming: anticipate edge cases
 Find opportunity in your error data

©2012 LinkedIn Corporation. All Rights Reserved.

Questions?

More info: data.linkedin.com
@peteskomoroch

©2012 LinkedIn Corporation. All Rights Reserved. 36

What's hot

Microsoft + OpenAI: Recent Updates (Machine Learning 15minutes! Broadcast #74)

Naoki (Neo) SATO

When it comes to creating an enterprise AI strategy: if your company isn’t good at analytics, it’s not ready for AI. Succeeding in AI requires being good at data engineering AND analytics. Unfortunately, management teams often assume they can leapfrog best practices for basic data analytics by directly adopting advanced technologies such as ML/AI – setting themselves up for failure from the get-go. This presentation explains how to get basic data engineering and the right technology in place to create and maintain data pipelines so that you can solve problems with AI successfully.

Creating an Enterprise AI Strategy

AtScale

Responsible Generative AI

CMassociates

AI 2023.pdf

DavidCieslak4

More organizations are aspiring to become ‘data driven businesses’. But all too often this aim fails, as business goals and IT & data realities are misaligned, with IT lagging behind rapidly changing business needs. So how do you get the perfect fit where data strategy is driven by and underpins business strategy? This webinar will show you how by de-mystifying the building blocks of a global data strategy and highlighting a number of real world success stories. Topics include: •How to align data strategy with business motivation and drivers •Why business & data strategies often become misaligned & the impact •Defining the core building blocks of a successful data strategy •The role of business and IT •Success stories in implementing global data strategies

Becoming a Data-Driven Organization - Aligning Business & Data Strategy

DATAVERSITY

Analytics plays a critical role in supporting strategic business initiatives. Despite the apparent value of providing the data infrastructure for these initiatives, many executives question the economic feasibility of business intelligence and analytics. This requires information professionals to calculate and present the business value in terms business executives can understand. Unfortunately, most IT professionals lack the knowledge required to develop comprehensive cost-benefit analyses and return on investment (ROI) measurements. This session provides a framework to help IT professionals research, measure, and present the economic value of a proposed or existing analytics initiative. The session will provide practical advice about how to calculate ROI, the formulas in use, and how to collect necessary information.

Analytics ROI Best Practices

DATAVERSITY

Using Generative AI

Mark DeLoura

Despite the increased availability of ready-to-use generic tools, more and more enterprises are deciding to build in-house data platforms. This practice, common for some time in research labs and digital native companies, is now making its waves across large enterprises that traditionally used proprietary solutions and outsourced most of their IT. The availability of large volumes of data, coupled with more and more complex analytical use cases driven by innovations in data science have yielded these traditional and on premise architectures to become obsolete in favor of cloud architectures powered by open source technologies. The idea of building an in-house platform at a larger enterprise comes with many challenges of its own: Build an Architecture that combines the best elements of data lakes and data warehouses to accommodate all kinds from BI to ML use cases. The need to interoperate with all the company’s data and technology, including legacy systems. Cultural transformation, including a commitment to adopt agile processes and data driven approaches. This presentation describes a success story on building a Lakehouse in an enterprise such as LIDL, a successful chain of grocery stores operating in 32 countries worldwide. We will dive into the cloud-based architecture for batch and streaming workloads based on many different source systems of the enterprise and how we applied security on architecture and data. We will detail the creation of a curated Data Lake comprising several layers from a raw ingesting layer up to a layer that presents cleansed and enriched data to the business units as a kind of Data Marketplace. A lot of focus and effort went into building a semantic Data Lake as a sustainable and easy to use basis for the Lakehouse as opposed to just dumping source data into it. The first use case being applied to the Lakehouse is the Lidl Plus Loyalty Program. It is already deployed to production in 26 countries with more than 30 millions of customers’ data being analyzed on a daily basis. In parallel to productionizing the Lakehouse, a cultural and organizational change process was undertaken to get all involved units to buy into the new data driven approach.

Phar Data Platform: From the Lakehouse Paradigm to the Reality

Databricks

In this session, you'll get all the answers about how ChatGPT and other GPT-X models can be applied to your current or future project. First, we'll put in order all the terms – OpenAI, GPT-3, ChatGPT, Codex, Dall-E, etc., and explain why Microsoft and Azure are often mentioned in this context. Then, we'll go through the main capabilities of the Azure OpenAI and respective usecases that might inspire you to either optimize your product or build a completely new one.

Using the power of Generative AI at scale

Maxim Salnikov

A solid data architecture is critical to the success of any data initiative. But what is meant by “data architecture”? Throughout the industry, there are many different “flavors” of data architecture, each with its own unique value and use cases for describing key aspects of the data landscape. Join this webinar to demystify the various architecture styles and understand how they can add value to your organization.

Data Architecture, Solution Architecture, Platform Architecture — What’s the ...

DATAVERSITY

Because every organization produces and propagates data as part of their day-to-day operations, data trends are becoming more and more important in the mainstream business world’s consciousness. For many organizations in various industries, though, comprehension of this development begins and ends with buzzwords: “Big Data,” “NoSQL,” “Data Scientist,” and so on. Few realize that all solutions to their business problems, regardless of platform or relevant technology, rely to a critical extent on the data model supporting them. As such, data modeling is not an optional task for an organization’s data effort, but rather a vital activity that facilitates the solutions driving your business. Since quality engineering/architecture work products do not happen accidentally, the more your organization depends on automation, the more important the data models driving the engineering and architecture activities of your organization. This webinar illustrates data modeling as a key activity upon which so much technology and business investment depends. Specific learning objectives include: - Understanding what types of challenges require data modeling to be part of the solution - How automation requires standardization on derivable via data modeling techniques - Why only a working partnership between data and the business can produce useful outcomes

Data Modeling Fundamentals

DATAVERSITY

Creating a clearly articulated data strategy—a roadmap of technology-driven capability investments prioritized to deliver value—helps ensure from the get-go that you are focusing on the right things, so that your work with data has a business impact. In this presentation, the experts at Silicon Valley Data Science share their approach for crafting an actionable and flexible data strategy to maximize business value.

8 Steps to Creating a Data Strategy

Silicon Valley Data Science

The deep learning AI revolution has been sweeping the world for a decade now. Deep neural nets are routinely used for tasks like translation, fraud detection, and image classification. PwC estimates that they will create $15.7 trillion/year of value by 2030. But most current networks are "discriminative" in that they directly map inputs to predictions. This type of model requires lots of training examples, doesn't generalize well outside of its training set, creates inscrutable representations, is subject to adversarial examples, and makes knowledge transfer difficult. People, in contrast, can learn from just a few examples, generalize far beyond their experience, and can easily transfer and reuse knowledge. In recent years, new kinds of "generative" AI models have begun to exhibit these desirable human characteristics. They represent the causal generative processes by which the data is created and can be compositional, compact, and directly interpretable. Generative AI systems that assist people can model their needs and desires and interact with empathy. Their adaptability to changing circumstances will likely be required by rapidly changing AI-driven business and social systems. Generative AI will be the engine of future AI innovation.

The Future of AI is Generative not Discriminative 5/26/2021

Steve Omohundro

Generative AI is here, and it can revolutionize your business. With its powerful capabilities, this technology can help companies create more efficient processes, unlock new insights from data, and drive innovation. But how do you make the most of these opportunities? This guide will provide you with the information and resources needed to understand the ins and outs of Generative AI, so you can make informed decisions and capitalize on the potential. It covers important topics such as strategies for leveraging large language models, optimizing MLOps processes, and best practices for building with Generative AI.

Unlocking the Power of Generative AI An Executive's Guide.pdf

PremNaraindas1

Data Storytelling: The only way to unlock true insight from your data

Bright North

Exploring Opportunities in the Generative AI Value Chain.pdf

Dung Hoang

Data tends to pile up and can be rendered unusable or obsolete without careful maintenance processes. Reference and Master Data Management (MDM) has been a popular Data Management approach to effectively gain mastery over not just the data but the supporting architecture for processing it. This webinar presents MDM as a strategic approach to improving and formalizing practices around those data items that provide context for many organizational transactions: its master data. Too often, MDM has been implemented technology-first and achieved the same very poor track record (one-third succeeding on-time, within budget, and achieving planned functionality). MDM success depends on a coordinated approach typically involving Data Governance and Data Quality activities. Learning objectives: - Understand foundational reference and MDM concepts based on the Data Management Body of Knowledge (DMBOK) - Understand why these are an important component of your Data Architecture - Gain awareness of Reference and MDM Frameworks and building blocks - Know what MDM guiding principles consist of and best practices - Know how to utilize reference and MDM in support of business strategy

Essential Reference and Master Data Management

DATAVERSITY

What does it mean for an organization to be data-driven? How does an organization get there? Many organizations think that they are data-driven but the reality is that few genuinely are and that we could all do better. In this talk, I cover what it truly means to be data driven. The answer, it turns out, is not to do with the latest tools and technologies (although they can help) but having an appropriate data culture than spans the whole organization, where data is accessible broadly, embedded into operations and processes, and enables effective decision making. In this presentation, I dissect what an effective data-driven culture entails, covering facets such as data leadership, data literacy, and A/B testing, illustrating concepts with examples from different industries as well as personal experience.

Creating a Data-Driven Organization, Crunchconf, October 2015

Carl Anderson

Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.

Building a Data Strategy – Practical Steps for Aligning with Business Goals

DATAVERSITY

Data Architecture is foundational to an information-based operational environment. Without proper structure and efficiency in organization, data assets cannot be utilized to their full potential, which in turn harms bottom-line business value. When designed well and used effectively, however, a strong Data Architecture can be referenced to inform, clarify, understand, and resolve aspects of a variety of business problems commonly encountered in organizations. The goal of this webinar is not to instruct you in being an outright Data Architect, but rather to enable you to envision a number of uses for Data Architectures that will maximize your organization’s competitive advantage. With that being said, we will: Discuss Data Architecture’s guiding principles and best practices Demonstrate how to utilize Data Architecture to address a broad variety of organizational challenges and support your overall business strategy Illustrate how best to understand foundational Data Architecture concepts based on “The DAMA Guide to the Data Management Body of Knowledge” (DAMA DMBOK)

Data Architecture Strategies

DATAVERSITY

What's hot (20)

Microsoft + OpenAI: Recent Updates (Machine Learning 15minutes! Broadcast #74)

Creating an Enterprise AI Strategy

Responsible Generative AI

AI 2023.pdf

Becoming a Data-Driven Organization - Aligning Business & Data Strategy

Analytics ROI Best Practices

Using Generative AI

Phar Data Platform: From the Lakehouse Paradigm to the Reality

Using the power of Generative AI at scale

Data Architecture, Solution Architecture, Platform Architecture — What’s the ...

Data Modeling Fundamentals

8 Steps to Creating a Data Strategy

The Future of AI is Generative not Discriminative 5/26/2021

Unlocking the Power of Generative AI An Executive's Guide.pdf

Data Storytelling: The only way to unlock true insight from your data

Exploring Opportunities in the Generative AI Value Chain.pdf

Essential Reference and Master Data Management

Creating a Data-Driven Organization, Crunchconf, October 2015

Building a Data Strategy – Practical Steps for Aligning with Business Goals

Data Architecture Strategies

Similar to Developing Data Products

Examples, techniques, and lessons learned building data products over the last 4 years at LinkedIn. Pete Skomoroch is a Principal Data Scientist at LinkedIn where he leads a team focused on building data products leveraging LinkedIn's powerful identity and reputation data. The talk describes some techniques and best practices applied to develop products like LinkedIn Skills & Endorsements. This talk was presented at the SF Data Science Meetup on September 19th, 2013

SF Data Science: Developing Data Products

Peter Skomoroch

Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...

Vitaly Gordon

Computing Professional Identity for the Economic Graph

Vitaly Gordon

We are all visual thinkers. 75 percent of the sensory neurons in our brains process visual information. As prospects and customers continue to be bombarded with information in the form of pure text such as whitepapers and blogs, it can be difficult to differentiate your content from the competition. Visual content can help and Slideshare is the perfect place to publish. One of the top 150 sites on the web, Slideshare is not just a repository for your slide presentations. It’s a social channel where you can establish your brand as a thought leader and authority around topics and keywords. With 60 million visitors a month and 3 billion slide views a month (That’s 1140 Slides per second) if you are not publishing content here, you are simply missing opportunities. The presentation below is from a session I recently did at Social Fresh West in San Diego. I pulled together seven essential tactics that successful Slideshare content publishers are using to turn their content marketing efforts up to eleven. Also included are some of the top businesses and brands using SlideShare effectively as part of their overall integrated marketing approach. I chose the theme of Badass because it’s a style that’s understated yet instantly recognizable. Like a chopped Harley or the perfect pair of sunglasses, this style is simple, direct and functional. (Just like a good piece of content should be) Visual language aids in decision making, is more persuasive, and makes a better, longer overall impression than simple text. Quite frankly, visual often kicks texts but in terms of overall views and virality across the social sphere. Who’s ready to get started?

7 Badass Tactics for SlideShare Content Domination

7 Badass Tactics for Slideshare Content Domination

Jason Miller

Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

Hakka Labs

7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)

Social Fresh Conference

Big Data Ecosystem @ LinkedIn

Minh-Hoang Nguyen

Are Your LinkedIn or Linked Out?

virallysocial

Enterprise Search, more relevant now than ever

Mike Davis

Big data arch_analytics

Srinu Adira

The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...

CityAge

Building the perfect profile on LinkedIn

Alex Charraudeau

Solving Biz Problems with SugarExchange: Session 9: How to Run Contributor Ca...

SugarCRM

Partner - Talent Solutions - Staffing Agency

LinkedIn_EMEA_Events

From Audit to Revenue – Content Marketing Tips, Tricks & Ideas Discovered, Created and Stolen The one repeated demand, common to us all, is the cry for more content. More content will solve all ills, right? Sales needs this, customer service wants that, marketing needs more of something else. Balderdash – more web pages or more product data sheets are not the answer. Content marketing is not about volume, it is about having the right content, at the right time, in front of the right audience and connecting with that audience and driving a satisfactory result for you and them. In this presentation we’ll look at what is working for organizations like yours; the tips, tricks and ideas that have changed approaches to content development, tactics, execution and ultimately driven success. From the nitty-gritty of a content audit to what marketers consider effective today, we packed a lot into this American Marketing Association New Orleans lunch on November 20th 2014.

Remarkable Content for AMA New Orleans, Nov 2014

David Shephard

Partner - Talent Solutions - Corporate

LinkedIn_EMEA_Events

Tamm & kitt

Jeff Roy

Office 2012 LinkedIn slides

Luke Williams

Link Me In Guides is Asia's leading LinkedIn™ specialist firm with offices in Singapore, Australia & the Philippines. Link Me In Guides optimises your LinkedIn™ profile so that it reflects your professional strengths in a positive light and garners you the maximum amount of the right attention. With this advantage, you land on the front page of the world’s largest network for professionals! In short, we elevate you to the front page of the world’s largest network for professionals and place your profile ahead of 90% of your competition on LinkedIn™. Link Me In Guide’s team of copywriters have several decades of combined experience in journalism, marketing, human resources, teaching, and English literature.With over 16,000 hours of LinkedIn™ consulting experience, we have created 1,500 LinkedIn™ profiles for professionals from every major industry. Every account receives the following treatment: * Insightful profile analysis * Engaging profile summary * Prospect-heavy organisational selection * Improved profile photo * Punchy work experience summary * Personalised URL * Optimised account settings * Google-optimised skills selection * Starter Guide for LinkedIn™ * Standout profile headline * Skills-based educational summary * 30-day guarantee * Over 140+ unique edits For more information go to: www.linkmeinguides.com/ To learn more about how Link Me In Guides - go to www.linkmeinguides.com or email contactus@linkmeinguides.com. *LinkedIn®, the LinkedIn logo, the IN logo and InMail are registered trademarks of LinkedIn Corporation and its affiliates in the United States and/or other countries. Specialities: LinkedIn Profile Optimisation, LinkedIn Keyword Optimisation, LinkedIn Lead Generation, LinkedIn Networking, LinkedIn Research, LinkedIn Keyword Tagging, LinkedIn Groups, LinkedIn Invitations, LinkedIn SEO, Australia, Canada, Hong Kong, New Zealand, Singapore, United Kingdom, United States of America, Real Estate.

Business Networking on LinkedIn

Realty Profiler

Similar to Developing Data Products (20)

SF Data Science: Developing Data Products

Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...

Computing Professional Identity for the Economic Graph

7 Badass Tactics for SlideShare Content Domination

7 Badass Tactics for Slideshare Content Domination

Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)

Big Data Ecosystem @ LinkedIn

Are Your LinkedIn or Linked Out?

Enterprise Search, more relevant now than ever

Big data arch_analytics

The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...

Building the perfect profile on LinkedIn

Solving Biz Problems with SugarExchange: Session 9: How to Run Contributor Ca...

Partner - Talent Solutions - Staffing Agency

Remarkable Content for AMA New Orleans, Nov 2014

Partner - Talent Solutions - Corporate

Tamm & kitt

Office 2012 LinkedIn slides

Business Networking on LinkedIn

More from Peter Skomoroch

Bridging the AI Gap: Building Stakeholder Support

Peter Skomoroch

Machine learning drove massive growth at consumer internet companies over the last decade, and this was enabled by open software, datasets, and AI research. For many problems, machine learning will produce better, faster, and more repeatable decisions at scale. Unfortunately, building and maintaining these systems is still extremely difficult and expensive. As more machine learning software moves to production, many of our traditional tools and best practices in software development will change. Pete Skomoroch walks you through what you need to know as we shift from a world of deterministic programs to systems that give unpredictable results on ever-changing training data. To navigate this world powered by nondeterministic data-dependent programs, we’ll also need a new development stack to help us write, test, deploy, and monitor machine learning software. Presented at OSCON Portland July 18, 2019

Managing Machines: The New AI Dev Stack

Peter Skomoroch

Companies that understand how to apply AI will scale and win their respective markets over the next decade. That said, delivering on this promise and managing machine learning projects is much harder than most people anticpate. Many organizations hire teams of PhDs and data scientists, then fail to ship products that move business metrics. The root cause is often a lack of product strategy for AI, or the failure to adapt their product development processes to the needs of machine learning systems. This talk will cover some of the common ways machine learning fails in practice, the tactical responsibilities of AI product managers, and how to approach product strategy for AI. Peter Skomoroch, former Head of Data Products at Workday and LinkedIn, will describe how you can navigate these challenges to ship metric moving AI products that matter to your business. Peter will provide practical advice on: * The role of an AI Product Manager * How to evaluate and prioritize your AI projects * The ways AI product management differs from traditional product management * Bridging the worlds of design and machine learning * Making trade offs between data quality and other business metrics

Product Management for AI

Peter Skomoroch

Companies that understand how to apply machine intelligence will scale and win their respective markets over the next decade. That said, delivering on this promise is much harder than most executives realize. Without large amounts of labeled training data, solving most AI problems isn’t possible. The talent and leadership to bridge the worlds of product design, machine learning research, and user experience are scarce. Many organizations will tackle the wrong problems and fail to ship successful AI products that matter to their customers. Pete Skomoroch explains how to navigate these challenges and build a business where every product interaction benefits from your investment in machine intelligence. This talk was presented at the 2019 Strata Data Conference in London. Topics include: Who defines the data vision and roadmap in your organization? Who is accountable for building and expanding your competitive moat? Investing in foundational data infrastructure, training, logging, and tools Fostering executive support for exploration and innovation, including user-facing data product and algorithm development How to evaluate new machine intelligence projects and develop a portfolio that delivers How AI product management differs from traditional product management How to bridge the worlds of design and machine learning to get to product-market fit Defining a framework for trading off investments in data quality, machine learning relevance, and other business objectives

Executive Briefing: Why managing machines is harder than you think

Peter Skomoroch

Building Competitive Moats With Data

Peter Skomoroch

Talk from the first O'Reilly Strata, Feb 2011. Learn how to leverage data exhaust, the digital byproduct of our online activities, to solve problems and discover insights about the world around you. We will walk through a real world example which combines several datasets and statistical techniques to discover insights and make predictions about attendees at O'Reilly Strata. Includes a preview of some of the technology behind LinkedIn Skills, which I launched in a Keynote with DJ Patil the following day. Video: http://blip.tv/oreilly-promos/distilling-data-exhaust-4780870

O'Reilly Strata: Distilling Data Exhaust

Peter Skomoroch

Skills, Reputation, and Search

Peter Skomoroch

Endorsements are a one-click system to recognize someone for their skills and expertise on LinkedIn, the largest professional online social network. This is one of the latest “data features” in LinkedIn’s portfolio, and the endorsement ecosystem generates a large graph of reputation signals and viral user activity. In this talk, we’ll examine the practical aspects of building a data feature like Endorsements. We’ll talk about marrying product design and data, deep diving into several of the lessons we’ve learned along the way - all using skills & endorsements as an empirical case study. We’ll include technical detail on our approaches and how we combine crowdsourcing, machine learning, and large scale distributed systems to recommend topics to users.

LinkedIn Endorsements: Reputation, Virality, and Social Tagging

Peter Skomoroch

Practical Problem Solving with Data - Onlab Data Conference, Tokyo

Peter Skomoroch

Practical problem solving with data involves more than just visualization or applying the latest machine learning techniques. Intuition, domain knowledge, and reasonable approximations can mean the difference between a successful model and a catastrophic failure. We’ll dive into some best practices I’ve extracted from solving real world problems like computing trending topics, cleaning election data, and ranking experts on social networks. New analysts or engineers are often lost when textbook approaches fail on real world data. Drawing inspiration from problem solving techniques in mathematics and physics, we will walk through examples that illustrate how come up with creative solutions and solve problems with big data.

Street Fighting Data Science

Peter Skomoroch

Data Mashups -Data Science Summit

Peter Skomoroch

O'Reilly Where 2.0 2011 As a result of cheap storage and computing power, society is measuring and storing increasing amounts of information. It is now possible to efficiently crunch Petabytes of data with tools like Hadoop. In this O'Reilly Where 2.0 tutorial, Pete Skomoroch, Sr. Data Scientist at LinkedIn, gives an overview of spatial analytics and how you can use tools like Hadoop, Python, and Mechanical Turk to process location data and derive insights about cities and people. Topics: * Data Science & Geo Analytics * Useful Geo tools and Datasets * Hadoop, Pig, and Big Data * Cleaning Location Data with Mechanical Turk * Spatial Tweet Analytics with Hadoop & Python * Using Social Data to Understand Cities

Geo Analytics Tutorial - Where 2.0 2011

Peter Skomoroch

LinkedIn is the premiere professional social network with over 60 million users and a new user joining every second. One of LinkedIn's strategic advantages is their unique data. While most organizations consider data as a service function, LinkedIn considers data a cornerstone of their product portfolio. To rapidly develop these products LinkedIn leverages a number of technologies including open source, 3rd party solutions, and some we've had to invent along the way. This LinkedIn talk at the NYC Hadoop Meetup held 3/18 at ContextWeb focused on best practices for quickly uncovering patterns, visualizing trends, and generating actionable insights from large datasets.

Rapid Data Exploration With Hadoop

Peter Skomoroch

Prototyping Data Intensive Apps: TrendingTopics.org

Peter Skomoroch

Amazon EC2 may offer the possibility of high performance computing to programmers on a budget. Instead of building and maintaining a permanent Beowulf cluster, we can launch a cluster on-demand using Python and EC2. This talk will cover the basics involved in getting your own cluster running using Python, demonstrate how to run some large parallel computations using Python MPI wrappers, and show some initial results on cluster performance.

Elasticwulf Pycon Talk

Peter Skomoroch

More from Peter Skomoroch (15)

Bridging the AI Gap: Building Stakeholder Support

Managing Machines: The New AI Dev Stack

Product Management for AI

Executive Briefing: Why managing machines is harder than you think

Building Competitive Moats With Data

O'Reilly Strata: Distilling Data Exhaust

Skills, Reputation, and Search

LinkedIn Endorsements: Reputation, Virality, and Social Tagging

Practical Problem Solving with Data - Onlab Data Conference, Tokyo

Street Fighting Data Science

Data Mashups -Data Science Summit

Geo Analytics Tutorial - Where 2.0 2011

Rapid Data Exploration With Hadoop

Prototyping Data Intensive Apps: TrendingTopics.org

Elasticwulf Pycon Talk

Developing Data Products

2. Examples, Techniques, & Lessons Learned Developing Data Products

3. Our Mission Connect the world’s professionals to make them more productive and successful. Our Vision Create economic opportunity for every professional in the world. Members First!

9. What do we mean by data products?

10. Building products from data at LinkedIn A few examples:  People You May Know  Skills and Endorsements  Year in Review  Network Updates Digest  InMaps  Who’s viewed my profile  Collaborative Filtering  Groups You May Like  and more… ©2012 LinkedIn Corporation. All Rights Reserved.

17.

18.

22. Lessons learned developing data products

23. Collect the right data at the right time

35. Conclusion: tips for developing data products  Collect the right data at the right time  Large amounts of data can reveal new patterns  Be wary of “black box” approaches  Look at your raw data  Aggregate statistics can be misleading  Build and use viewer apps  Include data geeks in design process  OODA: Think like a jet fighter  Red-teaming: anticipate edge cases  Find opportunity in your error data ©2012 LinkedIn Corporation. All Rights Reserved.

Editor's Notes

Mission: For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in. Vision: But, we’re just getting started. By our measure,there are more than 640 million professionals in the world. And roughly 3.3 billion people in the global workforce. Ultimately, our vision is to create economic opportunity for every professional, which we believe is an especially crucial objective in light of current macroeconomic trends.Our most important core value is that members come first.

Developing Data Products

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Developing Data Products

Similar to Developing Data Products (20)

More from Peter Skomoroch

More from Peter Skomoroch (15)

Developing Data Products

Editor's Notes