Big data, big challenge－ splunk 幫你解決 big data 議題帶來的挑戰
•
8 likes•1,672 views
講者：SYSTEX 數據加值應用發展部產品經理 | 陶靖霖 議題簡介：認清現實吧! Big Data 是個熱門詞彙、熱門議題，但是問題的核心仍然圍繞在資料處理的流程、架構與技術，要踏入 Big Data 的領域，使用者會遭遇哪些挑戰? Splunk 被譽為「全球最佳的 Big Data Company」，究竟在資料處理的流程中擁有什麼獨特的技術優勢，能夠幫助使用者克服這些挑戰?又有哪些成功幫助使用者從資料中萃取出價值的應用案例?歡迎來認識 Splunk 以及全球 Big Data 成功案例。
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Big data, big challenge－ splunk 幫你解決 big data 議題帶來的挑戰
Similar to Big data, big challenge－ splunk 幫你解決 big data 議題帶來的挑戰 (20)
Big data, big challenge－ splunk 幫你解決 big data 議題帶來的挑戰
- 1. 1 Big Data, Big Challenge－ Splunk 幫你解決 Big Data 議題 帶來的挑戰 精誠資訊 Splunk 產品經理 陶靖霖
- 4. 4 Big Data 其實仍是「資料處理」議題 生 流 存 算 用 看
- 5. 5 資料處理的難題是共通的 生 流 存 算 用 看 我需要的資料從 哪裡來？ 要怎麼收集資料？ 資料收集能力？ 資料格式的限制？ 資料如何ETL？ 資料如何正規化？ 該如何呈現運算分析 結果？ 該用什麼工具呈現？ 需要客製開發嗎？ 資料可否壓縮保存？ 資料可否備份？ 保存年限與生命週 期管理？ 如何尋找與計算？ 如何撰寫MR程式？ 需要多久才能學會？ 如何輸出運算結果？ 如何與其他系統整 合？
- 6. 6 Hadoop Ecosystems 已解決了部分難題 生 流 存 算 用 看
- 8. 8 全球最佳 Big Data 創新應用公司
- 9. 9 獲獎無數 Big Data Innovator 2013 SIEM Magic Quadrant LEADER 2012 Security Market Growth #1 Worldwide 2012 IT Operations Market Growth #3 Worldwide Best SIEM North America Best Enterprise Security Solution EMEA #1 Most Innovative#4
- 12. 12 來看看 Splunk 如何應對 處理 Big Data 的難題 生 流 存 算 用 看
- 13. 13 • 利用既有的資料產生機制 – 只是常常在應用時發現缺東漏西 • 自行產生需要的資料 – 這件事情，還是得靠自己來 – 可以定製需要的資料與格式 • 若自行產生資料，哪些資料格式比較適合？ – 收集工具容易處理的資料格式優先使用 生 流 存 算 用 看
- 14. 14 Key-Value Pair (KVP) 很理想 • 欄位容易識別，易於 ETL • 半結構化，不會受限於固定的資料結構
- 15. 15 是個好選擇 • 欄位容易識別，易於 ETL • 巢狀結構易於表現資料維度
- 18. 18 資料產生的 Best Practices • 用 Text 不要用 Binary – 省去 parsing • 別用 XML – JSON 可讀性更高 • 避免一筆多行資料 • 使用可讀的 timestamp • 不要使用 time offsets • Timestamp 放最前面 • 留下 UUID 或 Session ID – 資料關連性 • 要考慮未來 – 資料價值、業務發展 – 可計算、可彙總的 • 分類分級
- 20. 20 用單一平臺收集各式機器資料 不用事先定義資料欄位，不用客製化連接器，不用資料庫，不需要事先過濾 Web logs Log4J, JMS, JMX .NET events Code and scripts Configurations syslog SNMP netflow Configurations Audit/query logs Tables Schemas Hypervisor Guest OS, Apps Cloud Configurations syslog File system ps, iostat, top Registry Event logs File system sysinternals Logfiles Configs Messages Traps Alerts Metrics Scripts TicketsChanges UNIX平台 Linux/Unix 視窗平台 Windows 網路設備 Networking 資料庫 Databases 應用系統 Applications 虛擬化 & 雲端 Virtualization Click-stream data Shopping cart data Online transaction data 客戶使用資料 資料中心以外 其他資料 Manufacturing, logistics… CDRs & IPDRs Power consumption RFID data GPS data 凡走過，必留下足跡 => Splunk 收集 所有的足跡，進行 問題追蹤、使用分析
- 21. 21 • 資料的傳遞與收集需考量： – 收集工具支援的資料收集管道 – 收集工具支援的資料類型 – 傳遞與收集的安全性、穩定性與效能 • 資料的 ETL (Extract, Transform, Load) – 若使用前述資料格式或已結構化格式通常簡單 – 預處理的需求 生 流 存 算 用 看
- 22. 22 Splunk 能收集多種資料結構 結構化資料 非結構化資料 關聯式 資料庫 半結構化資料 Log Email XM L 聲音 圖片 影像 Extract Metadata
- 23. 23 Splunk 資料收集方式 彈性的資料收集方式，可使用代理程式或免用代理程式 perf shell code Mounted File Systems hostnamemount syslog TCP/UDP WMI Event Logs Performance Active Directory syslog compatible hosts and network devices Unix, Linux and Windows hosts (NFS, CIFS, SMB…etc.) Windows hosts DB Connection, SNMP, REST API, STDOUT…etc. Local File Monitoring log files config files dumps and trace files Windows Inputs Event Logs performance counters registry monitoring Active Directory monitoring virtual host Windows hosts Scripted Inputs shell scripts custom parsers batch loading 免安裝代理程式 Splunk Forwarder
- 25. 25 • 資料的 ETL (Extract, Transform, Load) – 若使用前述資料格式或已結構化格式比較簡單 – ETL 對預處理的需求 – 資料處理流程最花時間的大概就在這邊 • 資料的儲存與生命週期管理 – 視這些資料有多重要而定 – 巨量資料的備份與備援議題 生 流 存 算 用 看
- 26. 26 Splunk 能在資料收集後再欄位定義 • 可以依需求動態定義所需要的欄位 • 提供多種工具協助定義欄位 • 能自動識別多種資料格式： – JSON, CSV, W3C…etc.
- 28. 28 便利的欄位定義工具
- 29. 29 也有 Hadoop 協助 ETL 的案例
- 30. 30 Splunk 叢集確保資料高可用性 • 向 Hadoop 學習 • 資料高可用性 – 可自由設定 Replication Factor 數 • Multi-site Clustering – 跨資料中心備援 • 可自由橫向擴充 – 可達 PB 等級
- 32. 32 • 學習曲線是重要考量 – 對企業來說，Time to Value 仍然很重要 – 人才搜尋或養成 • 時間與人力，都是成本 生 流 存 算 用 看
- 33. 33 Search your IT infrastructure J2EE exception Last 60 minutes fail* password sshd Last 30 minutes Last 60 minutes Last 3 hours Last 24 hours Last 7 days All time Last 24 hours 以關鍵字方式快速探索巨量資料
- 34. 34 搜尋速度令人滿意
- 35. 35 搜尋功能也滿足需求 • 關鍵字搜尋 • 欄位搜尋 • 正規表示式(Regular Expression)搜尋 • 資料關連 • 子查詢與 Join • 資料參照(Lookup) • 資料加工處理 • 資料運算與統計 • 資料標記(Tag)與事件類型 (Event Type)管理 • 資料模型(Data Modeling)與樞 紐分析(Pivot) • 預測分析 (Predictive Analysis)
- 36. 36 Splunk 的 Map Reduce 作法 36 以分散式搜尋從各 Indexer 取出資料，並將結果交由 Search Head 彙整 分散式搜尋 Search Head Map Reduce Indexer Indexer Indexer Indexer
- 37. 37 也可以跟 Hadoop 合作 匯入 瀏覽 匯出 利用 Splunk Hadoop Connect 與 Hadoop 交換資料 Splunk Hadoop Connect HA Indexes and Storage Commodity Servers Hadoop (MapReduce & HDFS) Report and analyze Custom dashboards Monitor and alert Ad hoc search
- 38. 38 或乾脆讓 Hadoop 來計算 可完全與 Hadoop 整 合，運用 Hadoop 完成搜尋與計算 能夠立即為 Hadoop 的導入帶來效益 可直接使用已存在 Hadoop 內的資料 搜尋與探索 視覺化 儀表板 分享分析 NoSQL and Other Data Stores Hadoop Client Libraries Streaming Resource Libraries 保留 Splunk 全功能， 操作方式與 Splunk 完全相同
- 40. 40 • 資料的應用場景才是關鍵 – 新的技術需要創新思維 – 顧問很重要：專業領域與Big Data技術 • 運算結果的利用與整合 – 呈現？ – 偵測與發出訊息 – 與第三方系統介接 生 流 存 算 用 看
- 41. 41 機器資料可以拿來做什麼？ 避險 節流 開源 日誌管理 資安稽核 法規遵循 個資法 ISO27001 / ISO20000 資安監控 資安事件分析 網站/App分析 使用者行為分析 Customer Insight 精準行銷 (CEM)用戶經驗管理 商業情報分析 Business Insight IT 與系統狀態監控 效能與資源監控 應用系統管理/監控 故障查找排除 異常問題調查 資源規劃
- 42. 42 Splunk 的應用橫跨 IT 與 Business 層面 • Splunk的應用領域 – 日常維運管理 – 資源與效能管理 – 資訊安全管理 – 市場分析與商業智慧 – … Splunk 帶來「一份日誌，多重應用」的複合價值 Mush up Web Apps 網站管理 團隊 VPs of Infrastruc ture
- 43. 43 適用於各種角色與應用的報表與儀表板 網站維運分析 應用系統管理 商情分析 資安與法規遵循 LOB Owners/ Executives Customer Support System Administrator IT 維運管理 Operations Teams Security Analysts IT Executives Development Teams Auditors Website/Business Analysts 提供給 IT 與 Business 層面透徹的資料可視度與洞察力 稽核員 IT負責人 行銷與商 務分析人 員 業務負責人
- 44. 44 Splunk Apps 讓你想怎麼用就怎麼用 超過 500 種 Apps 提供各種應用方式 REST API XenApp XenDesktop Server, Storage, Network Server Virtualization Operating Systems Infrastructure Applications Mobile Applications Cloud Services Other Monitoring Ticketing/Help Desk Custom Biz Applications SDKs Web Framework
- 45. 45 Splunk 提供客製化開發的彈性 REST API Web Framework 開發人員可以利用熟悉的程式語言、SDK 介接其他系統或開發應用 Web Framework JAVA JavaScript Python Data Models Search Extensibility Modular Inputs SDKsSimple XML JavaScript Django Ruby C# PHP
- 47. 47 透過 DB Connect 連結 RDBMS 以及 Hadoop 世界
- 48. 48 Splunk 分別與 Hortonworks 與 Cloudera 合作
- 49. 49 Splunk 與 Tableau 合作
- 50. 50 • Hadoop 缺乏原生工具，需要整合外部工具 • 手刻 vs. 工具 的考量 – 熟悉程度、整合彈性、呈現方式 – 學習曲線、用途、時間成本… • Data Visualization 是另外一門專門學科/藝術 – 訊息含量、美感、解讀能力… 生 流 存 算 用 看
- 53. 53 或是整合外部工具
- 54. 54 Splunk 是完整的巨量資料解決方案 生 流 存 算 用 看 完整成熟的解決方案，涵蓋巨量資料處理各階段會遭遇的挑戰 成熟穩定的資料 收集機制 完善的資料格式 處理能力 妥善的資料生命 週期管理 強大的資料搜尋 運算功能 貼心的 Hadoop 整合應用 開放彈性的資料 整合開發平台 優異便利的 資料視覺化工具
- 55. 55 結論 Splunk & Hunk 是 Hadoop 的好朋友 Splunk 是優異的機器資料應用平台 從資料來源開始規劃，事半功倍 Big Data 議題不只是分散式或是計算的問題
- 56. 56 Q & A
Editor's Notes
- Splunk is a data engine for your machine data. It gives you real-time visibility and intelligence into what’s happening across your IT infrastructure – whether it’s physical, virtual or in the cloud. Everybody now recognizes the value of this data, the problem up to now has been getting to it. At Splunk we applied the search engine paradigm to being able to rapidly harness any and all machine data wherever it originates. The “no predefined schema” design, means you can point Splunk at any of your data, regardless of format, source or location. There is no need to build custom parsers or connectors, there’s no traditional RDBMS, there’s no need to filter and forward. Here we see just a sample of the kinds of data Splunk can ‘eat’. Reminder – what’s the ‘big deal’ about machine data? It holds a categorical record of the following: User transactions Customer behavior Machine behavior Security threats Fraudulent activity You can imagine that a single user transaction can span many systems and sources of this data, or a single service relies on many underlying systems. Splunk gives you one place to search, report on, analyze and visualize all this data.
- Getting data into Splunk is designed to be as flexible and easy as possible. Because the indexing engine is so flexible and doesn’t generally require configuration for most IT data, all that remains is how to collect and ship the data to your Splunk. There are many options. First, you can collect data over the network, without an agent. The most common network input is syslog; Splunk is a fully compliant and customizable syslog listner over both TCP and UDP. Further, because Splunk is just software, any remote file share you can mount or symlink to via the operating system is available for indexing as well. To facilitate remote Windows data collection, Splunk has a its own WMI query tool that can remotely collect Windows Event logs and performance counters from your Windows systems. Finally, Splunk has a AD monitoring tool that can connect to AD and get your user meta data to enhance your searching context and monitor AD for replication, policy or user security changes. When Splunk is running locally as an indexer or lightweight forwarder, you have additional options and greater control. Splunk can directly monitor hundreds or thousands of local files, index them and detect changes. Additionally, many customers use our out-of-the-box scripts and tools to generate data – common examples include performance polling scripts on *nix hosts, API calls to collect hypervisor statistics and for detailed monitoring of custom apps running in debug modes. Also, Splunk has Windows-specific collection tools, including native Event Log access, registry monitoring drivers, performance monitoring and AD monitoring that can run locally with a minimal footprint.
- A single indexers it can index 100-200 gigabytes per day depending the data sources and load from searching. If you have terabytes a day you can linearly scale a single, logical Splunk deployment by adding index servers, using Splunk’s built in forwarder load balancing to distribute the data and using distributed search to provide a single view across all of these servers. Unlike some log management products you get full consolidated reporting and alerting not simply merged query results. When in doubt, the first rule of scaling is ‘add another commodity indexer.’ Splunk indexers are designed to enable nearly limitless fan-out with linear scalability by leveraging techniques like MapReduce to fan-out work in a highly efficient manner.
- A single indexers it can index 100-200 gigabytes per day depending the data sources and load from searching. If you have terabytes a day you can linearly scale a single, logical Splunk deployment by adding index servers, using Splunk’s built in forwarder load balancing to distribute the data and using distributed search to provide a single view across all of these servers. Unlike some log management products you get full consolidated reporting and alerting not simply merged query results. When in doubt, the first rule of scaling is ‘add another commodity indexer.’ Splunk indexers are designed to enable nearly limitless fan-out with linear scalability by leveraging techniques like MapReduce to fan-out work in a highly efficient manner.
- To address some of the challenges, we released Splunk Hadoop Connect in October last year. This enables bi-directional integration - users can browse and move data into Splunk and act on it. And since launch we’ve seen nearly 1,000 downloads! (as of June 2013).
- Hunk offers Full-featured Analytics in an Integrated Platform Explore, analyze and visualize data, create dashboards and share reports from one integrated platform. Hunk enables everyone in your organization to unlock the business value of data locked in Hadoop Hunk integrates the processes of data exploration, analysis and visualization into a single, fluid user experience designed to drive rapid insights from your big data in Hadoop. Enable powerful analytics for everyone with Splunk’s Data Models and the Pivot interface, first released in Splunk Enterprise 6. And Hunk works with what you have today Hunk works on Apache Hadoop and most major distributions, including those from Cloudera, Hortonworks, IBM, MapR and Pivotal, with support for both first-generation MapReduce and YARN (Yet Another Resource Negotiator, the technical acronym for 2nd generation MapReduce). Preview results and interactively search across one or more Hadoop clusters, including from different distribution vendors. Use the ODBC driver for saved searches with report acceleration to feed data from Hunk to third-party data visualization tools or business intelligence software. Streaming Resource Libraries enables developers to stream data from NoSQL and other data stores, such as Apache Accumulo, Apache Cassandra, Couchbase, MongoDB and Neo4j, for exploration, analysis and visualization in Hunk.
- Here are just some of the new Splunk Apps that have been delivered over the past year. Their goal is to make it easier to use Splunk for specific technologies and use cases – prepackaging inputs, field extractions, searches and visualizations. Highlight a few apps. These apps along with 100’s of others have been developed not only by Splunk but by partners, customers and members of the Splunk community.
- BUILD SPLUNK APPS The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to BUILD interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof). EXTEND AND INTEGRATE SPLUNK Splunk Enterprise is a robust, fully-integrated platform that enables developers to INTEGRATE data and functionality from Splunk software into applications across the organization using Software Development Kits (SDKs) for Java, JavaScript, C#, Python, PHP and Ruby. These SDKs make it easier to code to the open REST API that sits on top of the Splunk Engine. With almost 200 endpoints, the REST API lets developers do programmatically what any end user can do in the UI and more. The Splunk SDKs include documentation, code samples, resources and tools to make it faster and more efficient to program against the Splunk REST API using constructs and syntax familiar to developers experienced with Java, Python, JavaScript, PHP, Ruby and C#. Developers can easily manage HTTP access, authentication and namespaces in just a few lines of code. Developers can use the Splunk SDKs to: - Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications - Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards - Build mobile applications with real-time KPI dashboards and alerts powered by Splunk - Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP - Build customer-facing dashboards in your applications powered by user-specific data in Splunk - Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk - Programmatically extract data from Splunk for long-term data warehousing Developers can EXTEND the power of Splunk software with programmatic control over search commands, data sources and data enrichment. Splunk Enterprise offers search extensibility through: - Custom Search Commands - developers can add a custom search script (in Python) to Splunk to create own search commands. To build a search that runs recursively, developers need to make calls directly to the REST API - Scripted Lookups: developers can programmatically script lookups via Python. - Scripted Alerts: can trigger a shell script or batch file (we provide guidance for Python and PERL). - Search Macros: make chunks of a search reuseable in multiple places, including saved and ad hoc searches. Splunk also provides developers with other mechanisms to extend the power of the platform. - Data Models: allow developers to abstract away the search language syntax, making Splunk queries (and thus, functionality) more manageable and portable/shareable. - Modular Inputs: allow developers to extend Splunk to programmatically manage custom data input functionality via REST.