Presenter: Michael Conover, LinkedIn Interesting, impactful problems are rarely easy to define, model, or evaluate - in fact, the very newness of a problem often suggests a scarcity of data and structured thinking about viable approaches. In this talk, we'll work through some of the challenges faced by LinkedIn's machine learning research teams in building and shipping intelligent systems that make sense of the world's economy. From creating novel training data and productionizing models with complex structure to evangelizing and evaluating the results of unsupervised algorithms, this talk will examine real-world case studies describing some of the hardest and most interesting challenges faced by one of the world's largest technology companies.

1. Condense Fact from the Vapor of Nuance
Mike Conover, Ph.D.
Staff Data Scientist, LinkedIn

2. How do industries evolve in time?
Which companies are positioned for explosive growth?
Which jobs will employ the workforce in ten years?
Industrial
Structure
YY Ahn, et al.

3. Good Problems

4. Unambiguous
Concrete Response Variables
Ad Clicks
Real Estate Pricing
Flower Species

5. Qualitative
Multidimensional
Non-Euclidean
Expertise
Nebulous
TrustSerendipity Sentiment

6. High-Fidelity Proxy Variables
Crowdsourcing / In-House Evaluation
Tip of the Spear
Operationalize
“All models are wrong, but some are useful.”
George Box

7. Execution

8. Quality Control

9. Scale is the Premise

10. Spectrum

11. Qualitative
Evaluation
Heteroscedasticity
Stratified Sampling
Temporal Factors

12. Confirmation Bias

13. Fit to Print

14. Lightweight
Narrative Frame
Toy Examples
Propaganda
“Given enough eyeballs, all bugs are shallow.”
Eric S. Raymond

15. Infrastructure
Notebooks
Model Viewers
Soup to Nuts

16. Ship It!

17. Devil’s Bargain
Feature Transformations
Transcription Errors
Model Specification

18. Power Tools
Singularity
Something’s Wrong. Go.
Spark & MLLib + Notebooks

19. Condense Fact from the Vapor of Nuance
Mike Conover, Ph.D.
Staff Data Scientist, LinkedIn

20. Biomorphs
Iterating on Response Variables