Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scaling Security Workflows in Government Agencies

For most federal agencies dealing with increased security threats, limiting machine-data collection is not an option. But faced with finite IT budgets, few agencies can continue to absorb the high costs of scaling high-end network attached storage (NAS) or moving to and expanding a block-based storage footprint. During this webcast, you’ll learn about more cost-effective solutions to support large-scale machine-data ingestion and fast data access for security analytics.

You’ll learn about:
- The common challenges organizations face when scaling security workflows
- Why a high-performance cache works to solve these issues
- How to integrate cloud into processing and storage for additional scalability and efficiencies

  • Login to see the comments

  • Be the first to like this

Scaling Security Workflows in Government Agencies

  1. 1. Scaling Security Workflows in Government Agencies September 28, 2017 | 11:00 AM ET WEBINAR
  2. 2. Housekeeping 2 • Recording • Today’s Slides • Attachments • Questions • Feedback
  3. 3. Our Presenters 3 Keith Ober Systems Engineer Avere Systems Bernie Behn Principal Product Engineer Avere Systems
  4. 4. What’s the problem? Data, Data Everywhere
  5. 5. Security Analysis Workflow
  6. 6. Security Analysis Workflow • Acquire and Aggregate Inputs • Normalize Data • Archive Raw / Archive Normalized Data • Analyze for Patterns • Alert or Remediate
  7. 7. Types of Inputs • Network Equipment: routers, switches, firewalls, VPN appliances, etc. • IT: physical servers, VM infrastructure, virtual machines, directory services, end user desktops/laptops • Application Layer: log files, access logs for applications, web servers • Miscellaneous: sensor data
  8. 8. Typical Ingest Workflow Ingest Node(s) Normalize/Filter Storage Analyze data Report/Alert Logs, streamed to Ingest Nodes
  9. 9. Typical Ingest Workflow Ingest Node(s) Normalize/Filter Storage I/O at scale can slow down storage, backing up entire workflow
  10. 10. Typical Ingest Workflow Ingest Node(s) Normalize/Filter Storage Analyze data Report/Alert If analysis is not co-located, latency can impede analysis
  11. 11. Typical Analysis Workflow Ingest Node(s) Normalize/Filter Storage Analyze data Report/Alert If analysis is not co-located, latency can impede analysis
  12. 12. The meta-problem: Log File Ingest and Processing 12 All router1 log files from the beginning of time...well, from when we started gathering them... All log files from firewall 1 All log files from server 1Time and Volume NET: A lot of historical data over time, applying pressure both in terms of storage and processing
  13. 13. Log Ingest Writ Large TheTrue Scale of Enterprise Ingest!
  14. 14. Five Big Challenges Let’s Break It Down.
  15. 15. 5 challenges when engineering security workflows 15 1. Ingest Latency and Throughput 2. Vendor Lock-In 3. Life Cycle Management 4. Data Availability and Redundancy 5. Cloud Integration
  16. 16. Challenge 1: Ingest Latency and Throughput Ingest Node Normalize/Filter Storage I/O at scale can slow down storage, backing up entire workflow
  17. 17. Ingest Latency Scales Too Storage The scale forces multiple storage sites, and on some products requires a replication mechanism, introducing more cost, overhead and latency. Ingest Node(s) Normalize/Filter Storage Storage Storage Storage Volume of inputs will drive the number of ingest nodes required. IOT devices are increasing the amount of log data being generated and ingested.
  18. 18. Challenge 2: Vendor Lock-In Storage Ingest Node(s) Normalize/Filter Storage Storage Storage Storage As solution scales: 1. Additional demand for storage increases costs, and lowers performance. 2. Deficiencies in current storage solution amplifies as deployment grows, longer upgrade outages. 3. Vendors often limit interoperability with other products when it comes to replication and tiering.
  19. 19. Vendor Lock-In Ability to transfer data to new solution is difficult: • Business Continuity -- How do you stop ingesting and processing logs? • Interoperability -- How do you ensure that your new/proposed storage solution will work well in a high-performance environment : Ingest performance : Read performance : Scale
  20. 20. Challenge 3: Data Lifecycle Management Storage Ingest Node(s) Normalize/Filter Storage Storage StorageMore Expensive Storage Value of the data 1. Lowering the cost to store means you can store more and derive greater value. 2. New analytic methods/tools bring a fresh round of analysis and burst workloads. 3. How can we begin to build AI based workloads? Cheaper Storage Warm Data Cold Data
  21. 21. Lifecycle Management Storage Performance • Ingest & analysis requires higher performance storage = More expensive storage • Over time simply too much data to store in performance tier • Deletion of older data possible?
  22. 22. Challenge 4: Data Availability and Redundancy Ingest Node Normalize/Filter Storage Ingest Node Normalize/Filter Ingest Node Normalize/Filter Storage Storage Reporting and Processing Node Reporting and Processing Node Reporting and Processing Node
  23. 23. Data Availability and Redundancy • Performance at scale requires distributing the reporting/analysis • Geographical location of ingest may also be distributed • Critical data: can’t lose it so avoid single point of failure • Large data sets with streaming data are extremely difficult to backup with traditional methods and are cost prohibitive
  24. 24. Challenge 5: Cloud Integration: Storage Storage Ingest Node(s) Normalize/Filter Storage Storage Storage Storage 1. How to start archiving data to Cloud Storage, in order to lower cost? 2. How can businesses leverage cloud-based AI workloads against the same data they have today?
  25. 25. Cloud Integration: Storage Cloud Storage Pros • Cloud Storage can be very inexpensive • Reduces the need to own and maintain additional IT assets • Public clouds have built-in redundancy Cloud Storage Cons • Cloud storage is eventually consistent...a query immediately after a write may not succeed • Lowest-cost storage is object, and requires S3-compliant application access
  26. 26. And, so, what do we do? Cache It If You Can
  27. 27. How can Avere address those challenges? 27 Challenges Avere Solution Ingest Latency and Throughput Avere write caching Avere FXT NVRAM 10GB+ Bandwidth Vendor Lock-In Global Namespace Flash Move & Mirror Life Cycle Management Data Availability and Redundancy HA, Clustering Flash Move & Mirror Cloud Integration Avere vFXT compute-based appliance Avere FlashCloud for AWS S3 and GCS
  28. 28. Speed Ingest via write-behind caching • Gather writes (ack’ing clients immediately) and flushing in parallel • Hardware: NVRAM for write protection and caching • Clustered caching solution distributes writes across multiple nodes Accelerate read performance with distributed, read-ahead caching • Read-ahead a request (read a bit more than what was requested) • Cache requests for other readers (typical in analytic workloads) • Writes cached as written, speeding analysis workloads The Power of High-Performance Caching
  29. 29. Caching Basic Architecture Ingest Node Normalize/Filter Storage ● Ingest nodes write to NFS mount points distributed across a caching layer Ingest Node Normalize/Filter Ingest Node Normalize/Filter ● Writes are flushed to storage over time, smoothing the ingest ● Writes are protected within the cluster via HA mirror Reporting / Analysis Node(s) ● Reporting / Analysis nodes access data via the cluster. ● Reads are cached, eventually aged ● Written data in the near term is cached and available Avere FXT Cluster
  30. 30. Data Placement Ingest Node Normalize/Filter Storage Ingest Node Normalize/Filter Ingest Node Normalize/Filter Avere can Mirror data to cloud storage for longer- term archiving Data is accessible through the cluster, as though it were on the primary storage Reporting Analysis Node(s)
  31. 31. Does this Really Work? From the Field Use Case
  32. 32. Avere Security Workflow FXT Cluster DMZ Network Central Control Container based applications to Normalize data DATA Syslog/NetFlow/… DATA Streaming data to Avere with no direct access Core Filers Splunk Configure Splunk to eat data from separate vServer (isolating traffic) Splunk Data Consumers Web access to visualize data ingested and analyzed by Splunk Mirror/Migrate/Cloud Core Filers
  33. 33. 33 SC17 November 12-17, 2017 Denver, Colorado AIRI 2017 October 1-4, 2017 Washington DC AWS re:Invent Nov. 27- Dec. 1, 2017 Las Vegas, Nevada
  34. 34. Contact Us! 34 Keith Ober Systems Engineer Avere Systems kober@averesystems.com Bernie Behn Principal Product Engineer Avere Systems bbehn@averesystems.com AvereSystems.com 888.88 AVERE askavere@averesystems.com Twitter: @AvereSystems

×