Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Splunk Spark Integration
Gang Tao
About Me
• Software Engineer with 15+ Years experience
• Now architect working on Data acquisition and
Cloud App
• Used to...
Splunk'Company'Overview'
3"
Company''
•  Global"HQs:""
!  San"Francisco"
!  London""
!  Hong"Kong"
•  1,800+"employees"glo...
Splunk'–'a'Data'Pla-orm'
Mainframe)
Data)
VMware)
Pla0orm)for)Machine)Data)
Exchange) PCI)Security)
Rela=onal)
Databases)
...
Demo
Splunk Technical Stack
Presenting
Processing
Store
Acquisition
Splunk Deployment Architecture
Indexer

store	
  data,	
  transform	
  row	
  data	
  into	
  
events	
  and	
  searches	
...
Splunk VS Open Source
Splunk VS Open Source
SQL of Machine Data - SPL
SPL	
  –	
  Splunk	
  Processing	
  Language	
  
SQL	
  
*nix	
  Pipe	
  
Google	
  Search
Extensibility - Splunk App
h0p://apps.splunk.com/	
  	
  
Enterprise	
  Security	
  
ITSI	
  
DB	
  Connect	
  
Technology...
Why Integration?
• Splunk to Spark
• Data Ingestion
• Unstructure/Semi
Structure data Indexing
• Data processing with
Splu...
Solution A
Solution B
Solution C
Indexer
Virtual Indexer (Spark)
SPL
Enhanced
Search Command
Spark
Driver
(SPL Parser)
Spark
Worker
Spark
Worker...
Challenges
• Avoid big data movement
• keep good user experience
• Adapt to SPL concept
Upcoming SlideShare
Loading in …5
×

Splunk Spark Integration

4,233 views

Published on

An introduction of Splunk and the possible solutions to do an integration.

Published in: Data & Analytics
  • I have always found it hard to meet the requirements of being a student. Ever since my years of high school, I really have no idea what professors are looking for to give good grades. After some google searching, I found this service ⇒ www.HelpWriting.net ⇐ who helped me write my research paper. The final result was amazing, and I highly recommend ⇒ www.HelpWriting.net ⇐ to anyone in the same mindset as me.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I can definitely recommend a website that really helped me with my essay. I found out it was due the day before I had to submit it. Went into full-on panic mode. Worst experience of my senior year by far. It’s called ⇒ HelpWriting.net ⇐. The quality of the writing is passable but the completion rate is super quick. You get to pick your own writer to do your stuff and that’s also a big bonus.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ♥♥♥ http://bit.ly/2F90ZZC ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ❤❤❤ http://bit.ly/2F90ZZC ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Splunk Spark Integration

  1. 1. Splunk Spark Integration Gang Tao
  2. 2. About Me • Software Engineer with 15+ Years experience • Now architect working on Data acquisition and Cloud App • Used to be working on BI, ERP and other Enterprise application development • Like data science and open source
  3. 3. Splunk'Company'Overview' 3" Company'' •  Global"HQs:"" !  San"Francisco" !  London"" !  Hong"Kong" •  1,800+"employees"globally" •  Annual"Revenue:" $450.9M"(YoY"+49%)" •  NASDAQ:"SPLK" Products' •  Free"trial"to"massive"scale" •  Splunk"products:"" !  Splunk"Enterprise" !  Splunk"Cloud" !  Hunk" !  Splunk"Light" !  Splunk"MINT" !  Premium"SoluWons" Customers'' •  10,000+"customers" •  Across"100"countries" •  Small"to"large" organizaWons" •  More"than"80"of"the" Fortune"100" •  Largest"license:"" !  400+"Terabytes/day"
  4. 4. Splunk'–'a'Data'Pla-orm' Mainframe) Data) VMware) Pla0orm)for)Machine)Data) Exchange) PCI)Security) Rela=onal) Databases) Mobile)Forwarders) Syslog)/)) TCP)/)Other) Sensors)&) Control)Systems) Wire)) Data) Mobile)Intel) Splunk'Premium'Apps' Rich'Ecosystem'of'Apps' MINT' ) Splunk - a Machine Data Platform
  5. 5. Demo
  6. 6. Splunk Technical Stack Presenting Processing Store Acquisition
  7. 7. Splunk Deployment Architecture Indexer
 store  data,  transform  row  data  into   events  and  searches  the  indexed   data  in  response  to  search   requests.   Search  Head
 directs  search  requests  to  a  set  of   indexers,  merges  the  results  and   presents  them  to  the  user   Forwarder
 get  data  into  indexers  
  8. 8. Splunk VS Open Source
  9. 9. Splunk VS Open Source
  10. 10. SQL of Machine Data - SPL SPL  –  Splunk  Processing  Language   SQL   *nix  Pipe   Google  Search
  11. 11. Extensibility - Splunk App h0p://apps.splunk.com/     Enterprise  Security   ITSI   DB  Connect   Technology  Add-­‐ons
  12. 12. Why Integration? • Splunk to Spark • Data Ingestion • Unstructure/Semi Structure data Indexing • Data processing with Splunk search • Data Presenting • Spark to Splunk • Powerful computing capability • Machine Learning • Open Source community
  13. 13. Solution A
  14. 14. Solution B
  15. 15. Solution C Indexer Virtual Indexer (Spark) SPL Enhanced Search Command Spark Driver (SPL Parser) Spark Worker Spark Worker Spark Worker
  16. 16. Challenges • Avoid big data movement • keep good user experience • Adapt to SPL concept

×