How Autodesk Leverages Splunk as an Assurance Platform on AWS
•
4 likes•2,093 views
In this presentation I talked about how a couple divisions at Autodesk run Splunk on AWS and leverage Splunk as a platform to provide operational and security visibility and confidence for our AWS adoption
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to How Autodesk Leverages Splunk as an Assurance Platform on AWS
Similar to How Autodesk Leverages Splunk as an Assurance Platform on AWS (20)
Recently uploaded
Recently uploaded (20)
How Autodesk Leverages Splunk as an Assurance Platform on AWS
- 1. November 13, 2014 | Las Vegas, NV Alan Williams, Principal Engineer at Autodesk Consumer Group Praveen Rangnath, Director of Cloud Product Marketing at Splunk
- 2. © 2014 Autodesk Engineer @ Autodesk Infrastructure Background AWS for ~4 years Splunk for ~1 year Motorcyclist Soft spot for pit bulls Who Am I?
- 3. © 2014 Autodesk Leader in 3D design, engineering and entertainment software Introduced AutoCAD in 1982 Empowering the Maker movement Help our customers imagine, design and create a better world Who is Autodesk? http://www.autodesk.com/products/personal-design-and-creativity
- 4. © 2014 Autodesk iOS Apps on App Store
- 5. © 2014 Autodesk Android Apps on Play Store
- 6. © 2014 Autodesk students.autodesk.com Get free access to the software used to make the games, movies, buildings, and products that inspire you.
- 7. © 2014 Autodesk Why AWS?
- 8. © 2014 Autodesk Take inventory of existing hardware Use the AWS Calculator http://calculator.s3.amazonaws.com/index.html Cost/compute analysis Let’s begin with cost analysis…
- 9. © 2014 Autodesk Cost Analysis – Account for Everything Support & Maintenance Power & Cooling Rack space Storage (FC + SATA) Servers Load Balancers
- 10. © 2014 Autodesk What we noticed… Total cost of server hardware vs Total cost of AWS instances = 35% lower for AWS Total cost of all on- premise infra vs Total cost of all AWS infra = 50% lower for AWS
- 11. © 2014 Autodesk We can’t compete on price Economies of scale We can’t compete on speed Time to provision Time to innovate - deliver new features Outcome
- 12. © 2014 Autodesk Production workloads Customer facing Business critical Big data analytics Dev / Test environments Net new systems Many AWS Use Cases
- 13. © 2014 Autodesk Why Splunk?
- 14. © 2014 Autodesk Leverages existing investments Standard log aggregation platform Familiar technology Logging = Splunk Single view across all accounts Splunk App for AWS Why Splunk?
- 15. © 2014 Autodesk Operations Insights & Troubleshooting Analyzing data from thousands of endpoints globally Centralized visibility across all AWS accounts Splunk App for AWS Security Incidents Response Event auditing Splunk App for Enterprise Security Business Intelligence Product metrics dashboards – executive visibility using Splunk Product analytics Provide Assurance for AWS adoption
- 16. © 2014 Autodesk CloudTrail Dashboard
- 17. © 2014 Autodesk Business Intelligence
- 18. © 2014 Autodesk Incident Response
- 19. © 2014 Autodesk Has this compromised host made any API calls?
- 20. © 2014 Autodesk Where were these IAM keys used?
- 21. © 2014 Autodesk Where are sign-ins originating?
- 22. © 2014 Autodesk Running Splunk On AWS
- 23. © 2014 Autodesk Tenets to running Splunk on AWS Automation AWS CloudFormation template - http://goo.gl/Hn309p Ansible Playbook - http://goo.gl/fulJPc Scalability Easy and quick to add/remove nodes Auto-scale everything Splunk Search Head Pooling* Performance Search heads (CPU bound) – C3 instances Indexers (IO bound) – C3+EBS, I2, HS1 instances Maximize IOPS with RAID O
- 25. © 2014 Autodesk CloudTrail + Splunk SNS Topic SQS Queue CloudTrail S3 Bucket SNS TopicCloudTrail 1 1 2 2 3 3 44 5 Account A Account B Core Services Account Simple to configure Decoupled components Scalable to many accounts Central logging view across all accounts
- 26. © 2014 Autodesk AWS + Splunk = Happy Marriage Scalable to 100s of accounts Platform Use Cases for Operations, Security and Business Summary
- 27. © 2014 Autodesk CloudFormation Splunk Cluster Template https://github.com/alanwill/cfn-splunk Ansible Splunk Playbook https://github.com/alanwill/ansible-splunk Try it yourself… Pull Requests encouraged
- 28. Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document. © 2014 Autodesk. All rights reserved. @alanwill alanwill
- 29. 29 Splunk Company Overview Company (NASDAQ: SPLK) Founded 2004, first software release in 2006 HQ: San Francisco / Regional HQ: London, Hong Kong Over 1,200 employees, based in 12 countries Annual revenue: $302.6M (YoY +52%) Business Model / Products Free download / online sandbox to massive scale On-premises, in the cloud and SaaS 8,000+ Customers Customers in 100 countries Over 2/3 of the Fortune 100 Largest license: 100 Terabytes per day Fast Company 2013: Named Splunk #4 Most Innovative Company in the World and #1 Big Data Innovator Leader: Gartner SIEM Magic Quadrant, 2014
- 30. What Is Machine Data? Volume | Velocity | Variety | Variability GPS, RFID, Hypervisor, Web Servers, Email, Messaging, Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, Servers, Storage, Security Devices, AWS CloudTrail and AWS Config Machine data is the fastest growing, most complex, most valuable area of big data 30
- 31. 31 What Does Machine Data Look Like? Sources Twitter Care IVR Middleware Error Order Processing
- 32. 32 Machine Data Contains Critical Insights Customer ID Order ID Customer’s Tweet Time Waiting On Hold Twitter ID Product ID Company’s Twitter ID Sources Twitter Care IVR Middleware Error Order Processing Customer IDOrder ID Customer ID
- 33. 33 Machine Data Contains Critical Insights Order ID Customer’s Tweet Time Waiting On Hold Product ID Company’s Twitter ID Sources Twitter Care IVR Middleware Error Order Processing Order ID Customer ID Twitter ID Customer ID Customer ID
- 34. IT Operations Security and Compliance Digital Intelligence App Dev and App Mgmt. Developer Platform (REST API, SDKs) Business Analytics Industrial Data and Internet of Things Small Data. Big Data. Huge Data. Use Cases for Machine Data Analytics 34 Core Use Cases Emerging Use Cases
- 35. Industry-Leading Platform For Machine Data Machine Data: Any Location, Type, Volume Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom ApplicationsMessaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID On- Premises Private Cloud Public Cloud Platform Support (Apps / API / SDKs) Enterprise Scalability Universal Indexing Answer Any Question Developer Platform Report and analyze Custom dashboards Monitor and alert Ad hoc search 35
- 36. Comprehensive AWS Solutions Software-as-a-Service Self-managed software App for AWS Integrations Amazon Machine Images Hunk – EMR Integration CloudTrai l CloudWatch S3ConfigBilling ELB 36
- 38. Splunk Cloud – Full-Featured SaaS Platform Full Power of Splunk Enterprise Access to 600+ Apps Hybrid Deployment Architecture 38
- 39. Hybrid Search Search Head(s) Indexer(s) Search Head(s) Indexer(s) On Premises Private Cloud Public Cloud On Premises Private Cloud Public Cloud Single Pane of Glass Visibility 39
- 40. SplunkCloud–Enterprise-ReadyService Industry-leading scalability & flexibility Architected for uptime & performance 100% Uptime SLA Robust enterprise security 5GB/day - 5TB/day plans No data comingling; Security attestations 40
- 41. High availability across indexers & search heads Multiple AWS availability zones Dedicated cloud environments Splunk Cloud fully monitored using Splunk Enterprise Builtfor100%Uptime 41
- 42. SplunkCloud–Easy ACCELERATED TIME TO VALUE & FASTER ROI Starts at $675 per month / 33% price reduction Splunk Online Sandbox Immediate deployment 42
- 43. Forward data Search Monitor Get value fast What You Do Hardware setup Storage Scaling Monitoring What We Do 43
- 44. Copyright © 2014 Splunk, Inc. ThankYou!
- 45. ENT212
Editor's Notes
- Splunk now has more than 1,200 employees worldwide, with headquarters in San Francisco and 14 offices around the world. Since first shipping its software in 2006, Splunk now has over 7,900 customers in 100 countries. These organizations are using Splunk software to improve service levels, reduce operations costs, mitigate security risks, enable compliance, enhance DevOps collaboration and create new product and service offerings. Please always refer to latest company data found here: http://www.splunk.com/company.
- Data is growing and embodies new characteristics not found in traditional structured data: Volume, Velocity, Variety, Variability/Veracity. Machine data is one of the fastest, growing, most complex and most valuable segments of big data. All the webservers, applications, network devices – all of the technology infrastructure running an enterprise or organization – generates massive streams of data, in an array of unpredictable formats that are difficult to process and analyze by traditional methods or in a timely manner. Why is this “machine data” valuable? Because it contains a trace - a categorical record - of user behavior, cyber-security risks, application behavior, service levels, fraudulent activity and customer experience.
- Unlike traditional structured data or multi-dimensional data– for example data stored in a traditional relational database for batch reporting – machine data is non-standard, highly diverse, dynamic and high volume. You will notice that machine data events are also typically time-stamped – it is time-series data. Take the example of purchasing a product on your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data generated by the different systems supporting these different interactions. Each of the underlying systems can generate millions of machine data events daily. Here we see small excerpts from just some of them.
- When we look more closely at the data we see that it contains valuable information – customer id, order id, time waiting on hold, twitter id … what was tweeted. What’s important is first of all the ability to actually see across all these disparate data sources, but then to correlate related events across disparate sources, to deliver meaningful insight.
- If you can correlate and visualize related events across these disparate sources, you can build a picture of activity, behavior and experience. And what if you can do all of this in real-time? You can respond more quickly to events that matter. You can extrapolate this example to a wide range of use cases – security and fraud, transaction monitoring and analysis, web analytics, IT operations and so on.
- This is why Splunk is a platform technology every CIO should know and care about. Splunk deployments are up and running quickly, they return value in days or weeks (not months or years) and our product supports an broad set of use cases supporting almost every vertical industry. Add to this a restful API and seven SDKs allow your enterprise architects and developers to integrate Splunk with all of the existing technology infrastructure investments you have made. We help your turn machine data into insights for IT and the business.
- Splunk is the leading platform for machine data analytics with over 7,900 organizations using Splunk – for data volumes ranging from tens of GBs to tens of TBs to over 100 TBs of data PER DAY. Splunk software reliably collects and indexes all the streaming data from IT systems, technology devices and the Internet of Things in real time - tens of thousands of sources in unpredictable formats and types. Splunk software is optimized for real time, low latency and interactivity. Organizations use Splunk software and their data the following ways: 1. Find and fix problems dramatically faster 2. Automatically monitor to identify issues, problems and attacks 3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions 4. Gain real-time insight from operational data to make better-informed business decisions This is described as Operational Intelligence: visibility, insights and intelligence from operational data. Splunk Cloud is currently only available in the United States and Canada.
- First, we built a full-featured service. You told us you want to do in the cloud everything you can do on-premises, with no compromises. All the features, all the functionality, all the use cases. Next, we built an enterprise-ready service. You told us Splunk is essential to how you run your IT and business, and that there is no tolerance for anything but an enterprise ready service. Last, we built a service that’s easy. You told us this is the cloud, so you expect easy. Easy across every dimension of the customer experience. I’m now going to go in to more detail on each of these three areas. I’ll start w/full featured.
- Last aspect of easy is how you, as existing customers, can take advantage of Splunk Cloud. You may think Splunk Cloud is SaaS and Splunk Enterprise is ent. Software so they are separate deployments, as this image might indicate. But what we built is infinitely better. Only vendor in industry w/hybrid search, ability to maintain single pane of glass visibility. Search, correlate, report, visualize across both Splunk Ent and Splunk cld. Ultimately means to you, really easy to leverage Splunk Cld for certain data sets, Splunk Ent. For other data sets, and maintain centralized visibility.
- First, Spl. Cld is architected across multiple AWS availability zones. In rare event one is down, we seamlessly fail over to another. Second, we incorporate HA across indexer and search heads. Again, in the event any go down, we seamlessly fail over. Third, we provision dedicated cloud environments for each customer, so we are not vulnerable to system-wide outages. Last, and very importantly, we monitor in real time using our very own Splunk software. As you know, core use case… leading cloud services such as salesforce, concur, successfactors use Splunk software to ensure uptime of their cloud services. So do we. One might say we eat our own dog food, but I say no, we drink our own champagne.
- First, Splunk Cld is architected across multiple AWS availability zones. In rare event one is down, we seamlessly fail over to another. Second, we incorporate HA across indexer and search heads. Again, in the event any go down, we seamlessly fail over. Third, we provision dedicated cloud environments for each customer, so we are not vulnerable to system-wide outages. Last, and very importantly, we monitor in real time using our very own Splunk software. As you know, core use case… leading cloud services such as salesforce, concur, successfactors use Splunk software to ensure uptime of their cloud services. So do we. One might say we eat our own dog food, but I say no, we drink our own champagne.
- First, Spl. Cld is architected across multiple AWS availability zones. In rare event one is down, we seamlessly fail over to another. Second, we incorporate HA across indexer and search heads. Again, in the event any go down, we seamlessly fail over. Third, we provision dedicated cloud environments for each customer, so we are not vulnerable to system-wide outages. Last, and very importantly, we monitor in real time using our very own Splunk software. As you know, core use case… leading cloud services such as salesforce, concur, successfactors use Splunk software to ensure uptime of their cloud services. So do we. One might say we eat our own dog food, but I say no, we drink our own champagne.
- The first aspect of easy means you can get started quickly. W/ Splunk Cld, you will be production live faster than you thought possible. We handle all the operational work of running Splunk. All you do is forward data, start searching, and see the value. Simple and easy.