Islands of Node in a sea of big data
•Download as PPTX, PDF•
2 likes•845 views
In this presentation, Glenn Block discusses how Splunk uses node.js in its products. Surprising, huh? Node shows up in various islands of Splunk’s architecture from the Splunk Server, to middleware components, and finally in the SDKs. The focus here is on real world usage, the specific places where we chose to use node and why, as well as the team’s experiences deploying it into production. What’s Splunk? It’s a product designed for data ingest and query for massive realtime data work loads. It is used by some of the largest organizations in the world.
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Viewers also liked
Viewers also liked (20)
Similar to Islands of Node in a sea of big data
Similar to Islands of Node in a sea of big data (20)
More from Glenn Block
More from Glenn Block (11)
Islands of Node in a sea of big data
- 1. Islands of node in a sea of Big Data Glenn Block Splunk Dev Platform CC Image Carlota and Isabel Islands by Storm Crypt Flic
- 2. About me
- 4. 4
- 6. What is Splunk? Any Machine Data HA Indexes and Storage Search and Investigation Proactive Monitoring Operational Visibility Real-time Business Insights Commodity Servers Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom ApplicationsMessaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID
- 7. 7
- 8. node in the box
- 9. Let's generate some PDFs 9
- 10. How we do it 10 node JSON JSON pdfchart jsdom
- 11. Why node? 11 • We needed an easy way to convert a web page into PDF Node has modules for rendering web pages on the server (JSDOM) Node has modules for PDF generation (PDFKIT)
- 12. 12
- 13. 13
- 14. 14
- 15. 15
- 16. Why node? 16 We needed a mobile-friendly façade for our non-mobile friendly API The server is heavily IO bound / async, which node is perfect for. It is easy and lightweight to spin up API servers and proxies in node. Great support for Auth as well.
- 17. 17 node for integration and extensiblity
- 18. 18
- 19. Log directly to Splunk via TCP, UDP, HTTP Integrate search results with other applications using custom visualizations Create and run searches from other applications Splunk SDK 19 VisualizeSearch Manage Add/Delete Users Manage Inputs Index
- 20. node inputs 20 Public APIs DevicesOS level Internal Systems Generating test data Input
- 21. Ingesting data with Socket.IO 21
- 22. Why node? 22 Inputs by their nature are heavily IO bound and async Inputs are streaming results back to Splunk and node has great support in the box for streams. It has a great ecosystem of modules It’s JavaScript!
- 24. Where to go for More Info • Tutorials, Code Samples, Getting Started, Downloads – http://dev.splunk.com/javascript – https://github.com/glennblock/splunk-socket.io • Support – https://www.splunk.com/page/submit_issue • GitHub – https://github.com/splunk/splunk-sdk-javascript • Twitter – https://twitter.com/splunkdev or follow me: https://twitter.com/gblock • Blog – http://blogs.splunk.com/dev/ 24
- 25. Thank You!
Editor's Notes
- Splunk is the leading platform for machine data analytics with over 5,200 organizations using Splunk (as of 7/1/13) – from tens of GB to many tens of TBs of data PER DAY. Splunk software is optimized for real-time, low latency and interactivity. Splunk software reliably collects and indexes all the streaming data from IT systems and technology devices in real-time - tens of thousands of sources in unpredictable formats and types. The value from Splunking machine data is described as Operational Intelligence. This enables organizations to: 1. Find and fix problems dramatically faster 2. Automatically monitor to identify issues, problems and attacks 3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions 4. Gain real-time insight from operational data to make better-informed business decisions
- The Splunk SDKs empower developers to deliver greater operational agility throughout the enterprise by making it easy to integrate data from Splunk with other applications. Splunk provides a fully-documented and supported REST API with nearly 200 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs, built on that API, make it easy for developers to integrate data from Splunk with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions. Splunk offers SDKs for Python, Java, JavaScript, PHP, Ruby and C#. Developers can use the Splunk SDKs to: Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards Build mobile applications with real-time KPI dashboards and alerts powered by Splunk Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP Build customer-facing dashboards in your applications powered by user-specific data in Splunk Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk Programmatically extract data from Splunk for long-term data warehousing Interact with data stored in Hadoop using HUNK