Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Telco - Yousun Jeong

Big Telco - Yousun Jeong

  • Login to see the comments

Big Telco - Yousun Jeong

  1. 1. ‹#› Big Telco 
 Real-Time Network Analytics Yousun Jeong
  2. 2. Who am I • Senior Software Engineer of SK Telecom, South Korea’s largest wireless communications provider • Work on commercial products (~ ’15)
 - She worked with Hadoop DW
 - She worked with IaaS(OpenStack)
 - She worked with PaaS(CloudFoundry)
 • Mail to : jerryjung@sk.com 2
  3. 3. 3 Table of Contents 1. Big Data in SK Telecom 2. Benefit of Spark 3. Spark Real Workload 
 Real-Time Network Analytics 4. Ongoing R&D
  4. 4. Big Data in SKT in a Nutshell ✓ Data Size - Currently collecting 250 TB/day ! ✓ Big Data Management Infrastructure - Hadoop cluster (1400+ nodes); migrated from 
 MPP RDBMS ✓ Use cases
 - Real-Time Analytics of Base Stations
 - Network Enterprise DW ! ✓ Ongoing R&D
 - SKT Hadoop DW Appliance with H/W acceleration 4
  5. 5. Operating over 1400 nodes (30 PB+) of Hadoop cluster SKT Hadoop Infrastructure • Optimized configuration • Fault tolerant and effective resource management system 5 Data Collector Data Collect " & pre-processing Main Cluster Analysis R&D Cluster ~250 TB/day (500+ node) Service! Logic Repository (400+ Node) (100+ node) Service Cluster (400+ node) Marketing NW 
 Analytics VoC SKT Hadoop Infra Data Feeding Data Feeding Commercialize Develop.
  6. 6. Batch LayerInterface Layer Flume Kafka" HDFS 
 (Data Mart) oozie (workflow) Hive (ETL) Spark (ETL) Analytics Layer 1 2 Spark SQL Spark MlLib Spark GraphX Spark R YARN (Unified Resource Manager) Real-Time Layer NoSQL Elastic
 Search HDFS Data Service Layer BI Legacy App 3 Analytics Layer Batch Processing Layer - Hadoop EDW Real-Time Processing Layer – Real Time Analysis 3 1 2 【 Components 】 Spark Streaming" ! H/W Accelerator (SSD, FGPA) Cluster Manger Ambari SKT Big Data Reference Architecture Designed to handle both real-time & batch data processing and high level analysis using Spark as a core technology 6
  7. 7. Benefit of Spark Spark help us to have the gains in processing speed and implement various big data applications easily and speedily ▪ Support for Event Stream Processing ▪ Fast Data Queries in Real Time ▪ Improved Programmer Productivity ▪ Fast Batch Processing of Large Data Set Why SKT use spark … 7
  8. 8. Use cases: Summary Network Enterprise DW APOLLO • End-to-end network quality assurance and
 fault analysis in a timely manner • Real-time analysis of radio access network to improve operation efficiency Network analytics 8
  9. 9. 9 DC
 Parser Kafka" Broker Kafka" Producer
 Kafka" Topic Spark Streaming Kafka Direct" Stream" 1 minute widow 10 s HDFS ES 10 s Real-Time Dashboard Spark SQL BI
 Analysis JDBC" ODBC 1 2 4 5 Data Collector" (Flume) 3 Spark
 MLlib 6 Timely Processing" Quick Response Requirements Parallelism • Executors • Partitions • Using Akka Use case 1: Requirements & Challenges
  10. 10. “Hadoop S/W and Commodity H/W Based Cost-effective IT Infrastructure System” 【 SKT DW Infrastructure】 “High-price, High-performance Proprietary IT Infrastructure System” 【 Legacy IT Infrastructure 】 ※ MPP Massively Parallel Processing, SAN Storage Area Network, NAS Network Attached Storage, RDBMS Relational DB Management System Structured/Un-structured Data Scale-out Structure (Petabyte, Exabyte)Data Structured Data Scale-up Structure (Terabyte) Commodity H/W (x86 Server)H/W High Performance H/W (MPP, Fabric Switch, etc.) Hadoop Architecture SQL on Hadoop S/W Proprietary S/W
 (RDBMS, etc.) Transaction/Batch Processing" (SQL) Hadoop File System Hadoop DW can handle telco big data with scalability & cost efficiency Use case 2: Hadoop based Enterprise DW 10 ※ MPP Massively Parallel Processing
  11. 11. 11 Use case 2: Network Enterprise DW NMS#1 DBMS … NMS#1 DBMS NMS#N-1 DBMS [ Current ]
 Siloed Data & IT Management Access NW Core NW Transport Expected advantages • Unification of 130+ legacy DMBSs, each of which was managing separate network monitoring system, enabling thorough analysis over the entire network • Quick and accurate identification of root causes of network failure Data scientists need unified platform to collect data from all network equipment for management and analysis purpose NMS
 #1 … NMS
 #2 NMS
 #N-1 Legacy NMS
 #N Hadoop DW DW Legacy NEWN MS#1 … NEW
 NMS# N BI &
 Analytic … [ Goal (4Q, 2015) ]" Network Enterprise DW
  12. 12. Network EDW is a Hadoop-based data warehouse built on Spark for various network statistics or raw data User Benefits • End-to-End quality assurance,
 Fault analysis • Reduces analysis lead time
 (days → minutes) • Saves TCO (1/5 less than legacy DW) ! Hadoop DW • Spark-SQL functions and query optimizer • Bulk-loading and timely processing of large data • SSD caching applied for 
 performance enhancement Acess Core Transport EMS EMS T-Pani EMS Hadoop DW DW Data Data Mart SQL on Hadoop
 (Spark SQL) IP EMS AnalyticsSQL ETL ETL O! D! S MQE*
 (Meta Query
 Engine) H/W Accelerator ! SSD Caching H/W Accelerator
 SSD Caching BI * MQE (Meta Query Engine) : Heterogeneous database integration query, including the Hadoop. Use case 2: Network Enterprise DW 12
  13. 13. 13 https://github.com/bitnine-oss/octopus Use case 2: Meta Query Engine Features" 1. Subset of ANSI-SQL" 2. Queries on multiple databases 
 including Spark-SQL, Oracle." 3. SQL-based authorization" 4. User authentication" 5. Unified schema view
  14. 14. Use case 2: Requirements & Challenges Timely Processing -ETL" Integrated BI Tools" Quick Response Requirements 14 MDS #1 MQE #1 HA Proxy Thrift Server 
 #1 Thrift Server 
 #2 Spark SQL HDFS YARN WEB MDS BI MQE Meta Store Octopus NW EDW # 96 ETL Spark 3 2 1 4
  15. 15. Use case 2: YARN(Dynamic Resource Allocation) 15 spark.dynamicAllocation.enabled true! spark.shuffle.service.enabled true! spark.dynamicAllocation.minExecutors 50! spark.dynamicAllocation.maxExecutors 150! spark.dynamicAllocation.initialExecutors 50! spark.dynamicAllocation.cacheExecutorIdleTimeout 600! spark.dynamicAllocation.executorIdleTimeout! 5! spark.dynamicAllocation.schedulerBacklogTimeout! ! 5! spark.dynamicAllocation.sustainedSchedulerBacklogTimeout! 5 <property>! <name>yarn.nodemanager.aux-services</name>! <value>mapreduce_shuffle,spark_shuffle</value>! </property>! <property>! <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>! <value>org.apache.spark.network.yarn.YarnShuffleService</value>! </property> Configuration
  16. 16. Use case 2: BI Integration 16 spark.sql.thriftServer.incrementalCollect true! spark.driver.maxResultSize 10g Configuration
  17. 17. Use case 2: Patches 17 SPARK-7792! - HiveContext registerTempTable not thread safe! SPARK-7936! - Add configuration for initial size and limit of hash for aggregation! SPARK-8153! - Add configuration for disabling partial aggregation in runtime! SPARK-8285! - CombineSum should be calculated as unlimited decimal first! SPARK-8312! - Populate statistics info of hive tables if it's needed to be! SPARK-8333! - Spark failed to delete temp directory created by HiveContext! SPARK-8334 ! - Binary logical plan should provide more realistic statistics! SPARK-8357! - Memory leakage on unsafe aggregation path with empty input! SPARK-8420! - Inconsistent behavior with Dataframe Timestamp between 1.3.1 and 1.4.0! SPARK-8552! - Using incorrect database in multiple sessions! SPARK-8707! - RDD#toDebugString fails if any cached RDD has invalid partitions! SPARK-8826! - Fix ClassCastException in GeneratedAggregate! SPARK-9685! - Unspported dataType: char(X) in Hive! SPARK-10151! - Support invocation of hive macro! SPARK-10152! - Support Init script for hive-thriftserver! SPARK-10679! - javax.jdo.JDOFatalUserException in executor! SPARK-10684! - StructType.interpretedOrdering need not to be serialised! SPARK-10216 - Avoid creating empty files during overwrite into Hive table with group by query Open Issues
  18. 18. Use case 2: Performance 18 TPC-H
  19. 19. Use case 2: Performance 19 Job Server
  20. 20. Hadoop DW Appliance (ongoing) 【 SKT Hadoop DW Appliance 】 Management & Automation Core Software Solution Hardware Acceleration 3 1 2 ▪ Develop Interactive Spark SQL ▪ Develop Meta Query Engine ▪ Develop Flash Storage-based I/O Acceleration ▪ Develop FPGA-based CPU Acceleration ▪ Develop Data & System Security ▪ Workload Optimization & Automation Industry Oriented Solution4 ▪ Fault Detection & Classification in Manufacturing ▪ Mobile Network Data Analytic Solution ▪ Unstructured Data Collection/Processing Solution Develop a Hadoop DW appliance combining optimized S/W layer and H/W acceleration 20 H/W Acceleration Layer Data Processing Layer * Meta Query Engine DW Management Layer Industry" Oriented Solution ! ! ! ! ! ! ! Monitoring DB Migration Security OptimizationPackaging SQL Engine/Storage " ! ! ! * SPARK HIVE Legacy RDBMS FDC Telco others Hadoop Storage DB Storage * Flash based I/O Accelerator * FPGA Accelerator 2 1 3 4
  21. 21. 21 Thank You!

×