Josh Wills, Director of Data Science, Cloudera at MLconf SEA - 5/01/15

•Download as PPTX, PDF•

1 like•1,253 views

Brainwashed: Building an IDE for Feature Engineering: Feature engineering- writing code to map raw input data into a set of signals that will be fed into a machine learning algorithm- is the dark art of data science. Although the process of crafting new features is tedious and failure-prone, the key to a successful model is a diverse set of high-quality features that are informed by domain experts. Recently, academic researchers have begun to focus on the problem of feature engineering, and have started to publish research that addresses the relative lack of tools that are designed to support the feature engineering process. In this talk, I will review some of my favorite papers and present some efforts to convert these ideas into tools that leverage the principles of reactive application design in order to make feature engineering (dare I say it) fun.

Technology

1© Cloudera, Inc. All rights reserved.
Or, A Less Pretentious Title
Josh Wills | Senior Director of Data Science
Meditations on on
Feature Engineering

2© Cloudera, Inc. All rights reserved.
About Me

3© Cloudera, Inc. All rights reserved.
Data Scientists At Work

4© Cloudera, Inc. All rights reserved.
Data Scientists at Home

5© Cloudera, Inc. All rights reserved.
I For One…

6© Cloudera, Inc. All rights reserved.
Deep Learning

7© Cloudera, Inc. All rights reserved.
Deeper Deep Learning

8© Cloudera, Inc. All rights reserved.
The Importance of Context

9© Cloudera, Inc. All rights reserved.
Help Me Help You

10© Cloudera, Inc. All rights reserved.
Brainwash!

11© Cloudera, Inc. All rights reserved.
The Operational/Analytical Impedance Mismatch

12© Cloudera, Inc. All rights reserved.
Supernova Schemas

13© Cloudera, Inc. All rights reserved.
Exhibit: http://github.com/jwills/exhibit

14© Cloudera, Inc. All rights reserved.
Demo Time!
jwills@cloudera.com

Similar to Josh Wills, Director of Data Science, Cloudera at MLconf SEA - 5/01/15

Analyzing Hadoop Data Using Sparklyr 

Cloudera, Inc.

NOVA Data Science Meetup 2-21-2018 Presentation Cloudera Data Science Workbench

NOVA DATASCIENCE

Building Data Science Teams: A Moneyball Approach

joshwills

Harnessing the Power of Apache Hadoop Series

Cloudera, Inc.

The Edge to AI Deep Dive Barcelona Meetup March 2019 A deep dive demo of using MiNiFi, NiFi, CDSW for real-time AI at the edge, in a local cluster, in the cloud and in a Data Science platform at scale with real-time streaming and data storage. Apache NiFi, MiNiFi, NiFi Registry, Cloudera Data Science Workbench (CDSW), Python, Pyspark, Spark SQL, Apache Calcite, Apache Parquet, Apache MXNet, GluonCV.

The Edge to AI Deep Dive Barcelona Meetup March 2019

Timothy Spann

The Art of Data Science

odsc

Data Science in the Enterprise

The Hive

Introducing the data science sandbox as a service 8.30.18

Cloudera, Inc.

Edge to AI: Analytics from Edge to Cloud with Efficient Movement of Machine ...

Timothy Spann

Federated Learning makes it possible to build machine learning systems without direct access to training data. The data remains in its original location, which helps to ensure privacy, reduces network communication costs, and taps edge device computing resources. The principles of data minimization established by the GDPR, and the growing prevalence of smart sensors make the advantages of federated learning more compelling. Federated learning is a great fit for smartphones, industrial and consumer IoT, healthcare and other privacy-sensitive use cases, and industrial sensor applications. We’ll present the Fast Forward Labs team’s research on this topic and the accompanying prototype application, “Turbofan Tycoon”: a simplified working example of federated learning applied to a predictive maintenance problem. In this demo scenario, customers of an industrial turbofan manufacturer are not willing to share the details of how their components failed with the manufacturer, but want the manufacturer to provide them with a strategy to maintain the part. Federated learning allows us to satisfy the customer's privacy concerns while providing them with a model that leads to fewer costly failures and less maintenance downtime. We’ll discuss the advantages and tradeoffs of taking the federated approach. We’ll assess the state of tooling for federated learning, circumstances in which you might want to consider applying it, and the challenges you’d face along the way. Speaker Chris Wallace Data Scientist Cloudera

Federated Learning

DataWorks Summit

Introducing Cloudera Data Science Workbench for HDP 2.12.19

Cloudera, Inc.

Part 2: A Visual Dive into Machine Learning and Deep Learning  

Cloudera, Inc.

You like to use R, and you need to use big data. dplyr, one of the most popular packages for R, makes it easy to query large data sets in scalable processing engines like Apache Spark and Apache Impala. But there can be pitfalls: dplyr works differently with different data sources—and those differences can bite you if you don’t know what you’re doing. Ian Cook is a data scientist, an R contributor, and a curriculum developer at Cloudera University. In this webinar, Ian will show you exactly what you need to know about sparklyr (from RStudio) and the package implyr (from Cloudera). He will show you how to write dplyr code that works across these different interfaces. And, he will solve mysteries: Do I need to know SQL to use dplyr? When is a “tbl” not a “tibble”? Why is 1 not always equal to 1? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems? 3 things to learn: Do I need to know SQL to use dplyr? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems?

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...

Cloudera, Inc.

Data Science and Machine Learning for the Enterprise

Cloudera, Inc.

Data Science in Enterprise

Josh Yeh

Part 1: Introducing the Cloudera Data Science Workbench

Cloudera, Inc.

The Vision & Challenge of Applied Machine Learning

Cloudera, Inc.

Cloudera Analytics and Machine Learning Platform - Optimized for Cloud

Stefan Lipp

One Hadoop, Multiple Clouds - NYC Big Data Meetup

Andrei Savu

One Hadoop, Multiple Clouds

Cloudera, Inc.

Similar to Josh Wills, Director of Data Science, Cloudera at MLconf SEA - 5/01/15 (20)

Analyzing Hadoop Data Using Sparklyr 

NOVA Data Science Meetup 2-21-2018 Presentation Cloudera Data Science Workbench

Building Data Science Teams: A Moneyball Approach

Harnessing the Power of Apache Hadoop Series

The Edge to AI Deep Dive Barcelona Meetup March 2019

The Art of Data Science

Data Science in the Enterprise

Introducing the data science sandbox as a service 8.30.18

Edge to AI: Analytics from Edge to Cloud with Efficient Movement of Machine ...

Federated Learning

Introducing Cloudera Data Science Workbench for HDP 2.12.19

Part 2: A Visual Dive into Machine Learning and Deep Learning  

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...

Data Science and Machine Learning for the Enterprise

Data Science in Enterprise

Part 1: Introducing the Cloudera Data Science Workbench

The Vision & Challenge of Applied Machine Learning

Cloudera Analytics and Machine Learning Platform - Optimized for Cloud

One Hadoop, Multiple Clouds - NYC Big Data Meetup

One Hadoop, Multiple Clouds

More from MLconf

Understanding Human Impact: Social and Equity Assessments for AI Technologies Social and Equity Impact Assessments have broad applications but can be a useful tool to explore and mitigate for Machine Learning fairness issues and can be applied to product specific questions as a way to generate insights and learnings about users, as well as impacts on society broadly as a result of the deployment of new and emerging technologies. In this presentation, my goal is to advocate for and highlight the need to consult community and external stakeholder engagement to develop a new knowledge base and understanding of the human and social consequences of algorithmic decision making and to introduce principles, methods and process for these types of impact assessments.

Jamila Smith-Loud - Understanding Human Impact: Social and Equity Assessments...

Josh Wills, Director of Data Science, Cloudera at MLconf SEA - 5/01/15

Recommended

Recommended

More Related Content

Similar to Josh Wills, Director of Data Science, Cloudera at MLconf SEA - 5/01/15

Similar to Josh Wills, Director of Data Science, Cloudera at MLconf SEA - 5/01/15 (20)

More from MLconf

More from MLconf (20)

Recently uploaded

Recently uploaded (20)

Josh Wills, Director of Data Science, Cloudera at MLconf SEA - 5/01/15

Editor's Notes