Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dark data

1,458 views

Published on

A Short Presentation on Dark Data containing an introduction on tools and frameworks one needs to utilize it.

Published in: Data & Analytics
  • I also order from www.HelpWriting.net
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • If u need a hand in making your writing assignments - visit ⇒ www.HelpWriting.net ⇐ for more detailed information.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Dark data

  1. 1. Amir Sedighi February 2017 Dark Data Risks and Opportunities @amirsedighi
  2. 2. Speaker Amir Sedighi Software Engineer
 Data Solutions Architect Founder at recommender.ir twitter: @amirsedighi
  3. 3. By even the most conservative estimates, the amount of data in the world doubles every two years. Data Era
  4. 4. May Venn Diagram helps us! Big Data
  5. 5. May Venn Diagram helps us! Tabular/ Relational/ RDBMS Data Big Data
  6. 6. May Venn Diagram helps us! Dark Data Tabular/ Relational/ RDBMS Data Big Data
  7. 7. May Venn Diagram helps us! Dark Data Tabular/ Relational/ RDBMS Data (Structured/Unstructured) (Almost Unstructured) (Structured) Big Data
  8. 8. May Venn Diagram helps us! Dark Data Tabular/ Relational/ RDBMS Data (Structured/Unstructured) (Almost Unstructured) (Structured) Big Data Almost can’t be processed or analyzed
  9. 9. Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing). Dark Data Definition by Gartner
  10. 10. Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing). Similar to dark matter in physics, dark data often comprises most organizations’ universe of information assets. Dark Data Definition by Gartner
  11. 11. Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing). Similar to dark matter in physics, dark data often comprises most organizations’ universe of information assets. Thus, organizations often retain dark data for compliance purposes only. Storing and securing data typically incurs more expense (and sometimes greater risk) than value. Dark Data Definition by Gartner
  12. 12. Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing). Similar to dark matter in physics, dark data often comprises most organizations’ universe of information assets. Thus, organizations often retain dark data for compliance purposes only. Storing and securing data typically incurs more expense (and sometimes greater risk) than value. Dark Data Definition by Gartner
  13. 13. Dark Data - A more Sensible Definition
  14. 14. Dark Data - A more Sensible Definition Organizations Generate and Gather Data
  15. 15. Dark Data - A more Sensible Definition Organizations Generate and Gather Data A large portion of the collected data are never even analyzed!
  16. 16. Dark Data - A more Sensible Definition Organizations Generate and Gather Data A large portion of the collected data are never even analyzed! 90% of the data are never analyzed
  17. 17. Dark Data - A more Sensible Definition Organizations Generate and Gather Data A large portion of the collected data are never even analyzed! 90% of the data are never analysed. • Customer Information • Log Files • Previous Employee Information • Previous Webpages • Sensor Data • Email Correspondences • Account Information • Notes or Presentations • Old Versions of Relevant Documents
  18. 18. 80%..90% is Dark Data
  19. 19. Does Your Org have any Dark Data? I am just going to check if we have any dark data in the cellar…
  20. 20. Brining Dark Data into Light 1. Gathering 2. Storing/Processing 3. Analyzing and Bringing it into decisions
  21. 21. Brining Dark Data into Light
  22. 22. Brining Dark Data into Light
  23. 23. Brining Dark Data into Light
  24. 24. Brining Dark Data into Light
  25. 25. Brining Dark Data into Light
  26. 26. Brining Dark Data into Light
  27. 27. Question All companies know data is going to provide value. Why there is so much of dark data?
  28. 28. Why there is so much of dark data? • Lack of insight about data • Lack of ambitions to improve • Disconnect among departments • Lopsided priorities • Lack of technologies to Capture and Store • Lack of resources/infrastructures to make it available • Lack of CPU and technics to analyze the data
  29. 29. The issues you face with Dark Data • Legal and Regulatory Issues • Loss of Reputation • Intelligence Risk • Operation Costs • Opportunity Costs
  30. 30. Some essential questions • What can we gather? • What may we extract from it? • How we may prune it? • How long should we keep it? • What are the storage options? • What are the processing options? • How much is the value of each block of data (Approximately) • Running limited boundary scenarios
  31. 31. Software Tools & Frameworks on DD
  32. 32. Software Tools & Frameworks on DD
  33. 33. Software Tools & Frameworks on DD Log Management
  34. 34. Software Tools & Frameworks on DD Indexing and Search
  35. 35. Software Tools & Frameworks on DD Data Streaming
  36. 36. Software Tools & Frameworks on DD
  37. 37. Software Tools & Frameworks on DD
  38. 38. Software Tools & Frameworks on DD Machine Learning and Graph Processing • Mahout • MLLib • FlinkMK • Theano • Torch • TensorFlow • GraphX • Gelly
  39. 39. A common Pipeline Machine Learning Steam Processing Query Already Processed Data Real World RT Events
  40. 40. A common Pipeline Machine Learning Steam Processing Query Already Processed Data Real World RT Events New Pipeline
  41. 41. Questions? Keep in touch: twitter: @amirsedighi
  42. 42. 1. http://www.gartner.com/it-glossary/dark-data/ 2. http://www.itproportal.com/2016/03/07/5-benefits-of-putting-dark-data-to-work/ 3. http://www.kdnuggets.com/2015/11/importance-dark-data-big-data-world.html 4. https://www.youtube.com/watch?v=_fBMmQo-Z4E 5. http://confluent.io 6. https://www.ecmconnection.com/doc/the-various-shades-of-dark-data-0001 7. https://www.datanami.com/2015/11/30/spark-streaming-what-is-it-and-whos-using-it/ References

×