Splunk's api how we built it
•
4 likes•1,439 views
These are the slides for my September API Craft SF talk on how we built / design Splunk's API. You can find the video here: https://www.youtube.com/watch?v=vHXcDKK4eGY. My talk starts at about 60 mins. The other two are on Uber and Sift Science and you should watch them as well!
Recommended
Recommended
More Related Content
Similar to Splunk's api how we built it
Similar to Splunk's api how we built it (20)
More from Glenn Block
More from Glenn Block (18)
Recently uploaded
Recently uploaded (20)
Splunk's api how we built it
- 1. Copyright © 2014 Splunk, Inc. Splunk’s API How we built it!
- 2. Agenda Why we built Splunk’s API How we built Splunk’s API
- 3. What is Splunk A product for handling REALLY large and varied sets of evented data
- 4. What is Splunk From megabytes to hundreds of terabytes daily
- 5. What is Splunk It is highly scalable and distributed
- 6. What is Splunk Useful in many domains: IT/Ops/DevOps, security, healthcare, financial,IOT/Devices
- 7. 7 Splunk storage Other Big Data stores Developer Pla6orm Data collecUon and indexing Report and analyze Custom dashboards Monitor and alert Ad hoc search
- 8. Why – Product need Need to build a Splunk UI that surfaces all of Splunk’s capabiliUes
- 9. Why – Customer need Need to allow customers to integrate their applicaUons and scripts with Splunk
- 10. Why – Cost Less code to maintain, a single source of truth
- 11. Why – Reach HTTP is ubiquitous, every plaZorm has a client
- 12. The arch 12
- 13. Server Tech Stack 13 C/C++ Python – Cherry PI Python – Django Javascript Node
- 14. Client Tech Stack 14 Javascript Node Python Ruby PHP C#
- 15. The API 15
- 16. Log directly to Splunk via HTTP Run historical and real-‐Ume searches What can you do with Splunk’s API? 16 Search Manage Add/Delete Users ReporUng/Alerts Manage Inputs ConfiguraUon Index Login to a Splunk instance and get a session token Auth
- 17. The API design 17 Service Categories Endpoints Endpoints
- 18. The API design -‐ Categories 18
- 19. The API design – Endpoints 19
- 20. The API design – Endpoints 20
- 21. Responses -‐ Feeds and Hypermedia <entry xmlns="hjp://www.w3.org/2005/Atom" xmlns:s="hjp://dev.splunk.com/ns/rest" xmlns:opensearch="hjp://a9.com/-‐/spec/opensearch/1.1/"> <Utle>search index</Utle> <id>hjps://localhost:8089/services/search/jobs/mysearch_02151949</id> <updated>2011-‐07-‐07T20:49:58.000-‐07:00</updated> <link href="/services/search/jobs/mysearch_02151949" rel="alternate"/> <published>2011-‐07-‐07T20:49:57.000-‐07:00</published> <link href="/services/search/jobs/mysearch_02151949/search.log" rel="search.log"/> <link href="/services/search/jobs/mysearch_02151949/events" rel="events"/> <link href="/services/search/jobs/mysearch_02151949/results" rel="results"/> <link href="/services/search/jobs/mysearch_02151949/results_preview" rel="results_preview"/> <link href="/services/search/jobs/mysearch_02151949/Umeline" rel="Umeline"/> <link href="/services/search/jobs/mysearch_02151949/summary" rel="summary"/> <link href="/services/search/jobs/mysearch_02151949/control" rel="control"/> </entry> 21
- 22. Auth 22 HTTP Basic Token based LDAP/AD Cookie based
- 23. Auth – HTTP Basic 23 curl -‐k -‐u admin:changeme hjps://localhost:8089/services/auth/login -‐ d username="admin" -‐d password="changeme"
- 24. Auth – Splunk Token 24 curl -‐k -‐H "AuthorizaUon: Splunk SfH2D^zvPyLu^mO61C9kWtB7TOuQs0i9oSzh4lD7ho7Gvw26I61VYRjXkgj LQlJDJ0hER^q^A6v0BHYiKNba^CMbOmC63frGCrDqr2Zt" hjps:// localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get
- 25. Search – Oneshot – Get me results! 25 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot"
- 26. Search – Oneshot – Get me results in json 26 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json"
- 27. Search – Oneshot – Get me json columns 27 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
- 28. Search – Oneshot – Get me json rows 28 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"
- 29. Search – Blocking – Wait Ull done! 29 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 5” -‐d exec_mode=”blocking” output_mode="json” | python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
- 30. Search – List search jobs 30 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d output_mode="json" -‐-‐get | python -‐mjson.tool
- 31. Search – Normal– Run in the background 31 curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐ d search="search sourcetype=sysmon | head 10000" -‐d exec_mode="normal" output_mode="json"| python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool
- 32. Search -‐ Export 32 curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/ search/search/jobs/export -‐d search="search index%3D_internal | head 100000" -‐d output_mode="raw"
- 33. Search – Export REALTIME 33 curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/ search/search/jobs/export -‐d search="search index%3D_internal" -‐d output_mode="raw" earliest_Ume="rt-‐1m" latest_Ume="rt"
- 34. Copyright © 2014 Splunk, Inc. Splunk’s API How we built it! dev.splunk.com splunk.com/jobs