TeraStream - Data Integration/Migration/ETL/Batch Tool

© 2014 DataStreams Corp. All Rights Reserved.
DataStreams Corp.
"Always find the better value of your data"
www.datastreams.co.kr

1.Company Profile
2.Data Integration as Data Infrastructure
3.TeraStream™ Overview
Performance & Cost Savings of TeraStream™
Features & Benefits
Customers & Application
Contents

1.Company Profile

Name
Data Streams Corp.
CEO
Mr. Young-sang Lee
Business
Area
Data Integration Solutions Development and Sales
Data Quality Solutions Development and Sales
Data Warehouse / BI / FDS / Forensic / Audit Consulting and Construction
Big Data Analytic Consulting and Platform Construction
Data Governance Platform Development/Consulting/Sales
Data Migration Consulting and Construction
Large Volume Data Batch Processing Improvement Consulting and Construction
Data Standardization and Quality Management Consulting and System Construction
Data Architecture Consulting
Office
Address
HQ Chungho-nais B/D 6F, 28 Saimdang-ro, Seocho-gu, Seoul, Korea
R&D U-Spacemall #2 B-601, 670 Daewangpangyo-ro, Bundang-gu, Seongnam, Korea
China Office Room 1216, 12th Floor, Intersection of Hopson Kirin Society Building 2,
Wang Jing Fu Tong West Street, Wangjing, Chaoyang District, Bejing
Contact
Tel) +82-2-3473-9077 / Fax) +82-2-3473-9084
Investor
JAFCO ASIA
Capital
USD 2M
Sales Amount
USD 19M (2013)
Established
Sep 19, 2001
Employees
121
Company Profile
3

2012
•Established R&D Center in Pangyo Techno Valley
•Released Social Cube for SNS Data Analytics
•Participated in Original Technology Development Project for Next Generation Memory Based Big Data Analytics and Management
2009
•TeraStream™, Selected as Standard Data Integration Tool by The Korea Federation of Banks
•Selected as Contractor for Building Resource Management Data Standardization and Meta-data Management System by Ministry of National Defense
•Released DeltaStream™, QualityStream™, and ImpactStream™
2007
•Awarded for Excellent Venture Company by Deputy Prime Minister
•MetaStream™, Awarded for Digital Business Innovation by SMBA
•Released TeraStream™ Version 2.0
•JAFCO, Japan invested 4 million USD
2005
•Mr. Young-sang Lee, CEO, Was Awarded a Grand Prize for Korea Digital Competitiveness
•TeraStream™ Won New Technology Certification from Ministry of Knowledge Economy
•Released MetaStream™ Version 1.0
•Acquired KDB Solution Co., Ltd., Korea’s First Meta-data Management Solution Company
•TeraStream™ Version 1.4, Acquired GS(Good Software) Certification
2003
•KEB selected TeraStream™ as Standard Batch/ETL Solution for Next Generation Banking System
•First Worldwide Sales Contract of FACT™
•Presented FACT™ to Oracle Open World 2003 in San Francisco, USA
•Released MetaStream™ Version 2.7
•CEO, Mr. Young-sang Lee, Was Elected as a Chairman of KOSEA(Korea Software Enterprise Association)
•Released TeraStream™ Version 3.2
2010
•TeraStream™ Version 2.2, Acquired GS Certification
•Changed Company Name to DataStreams Corp.
•Contracted with Intellectual Property Office for Enterprise Data Quality Management
2008
•Awarded for Top Private Company for Population and Housing Census by Deputy Prime Minister
•TeraStream™, Selected as Standard ETL Tool by Ministry of Government Administration and Home Affairs
2006
•Selected as Technically Innovated Company of 2004 by SMBA
•Selected as Technically Innovated Company of 2004 by Small & Medium Business Administration(SMBA)
•TeraStream™, Selected for a Next Generation Banking Data Migration Tool by Shinhan Bank
2004
•Registered TeraStream™ as a Trademark
•Released TeraStream™ Designer Version 1.1
•First TeraStream™ V.1.1 Contract with National Statistics Office
2002
2011
•Awarded Prime Minister Citation for SW Achievement
•Selected as an ATC(Advanced Technology Center) by Ministry of Knowledge Economy
•Established China Office in Beijing
2013
•TeraStream™ for Hadoop, Selected as Base Solution for Building Government-Wide Big Data Infrastructure
•Acquired Patent for Readable Data Encryption and Decryption
•CEO, Mr. Young-sang Lee, Was Awarded Digital Management Innovation Prize
•Joined Int’l SOFT China in Beijing
• MetaStream™ Version 3.0, Acquired GS Certification
• Launched DQ Appliance
2001
•Released TeraStream™ Beta Version
•Innovative Data Solutions Corp., Was Established
Company History
4
2014
•TeraStream™, Was Awarded 2014 Korea Software Award by Ministrer of Science, ICT and Future
•DataStreams Is Listed on KONEX(Korea New Exchange)

Organizational Structure
Present Condition of Engineering Employees
96
Consultants
Data Governance
36
Data Integration & Migration
Big Data Management
DW & BI
SNS Analytics
Engineers
Meta Data & Data Quality Management Solution
60
Data Integration & Migration Solution
Big Data Management Platform
Number
Of
Employees
Total
Consultants/Developers
Management/Sales
121
96
25
Engineering Lev.
Total
Consultants
Engineers
Total
96
36
60
Professional
27
18
9
Qualified
26
11
15
Intermediate
19
3
16
Beginning
24
4
20
•Government Offices
•Banking
•Manufacturing Business
•Logistics/Services
•Planning Products
•Presales & Consulting
•DW/BI
•Big Data
•SNS
•QA(Quality Assurance)
•Marketing
•Overseas Sales
•Overseas Corp.
•HR/General Affairs
•Financial Admin.
•Knowledge Mgmt.
•Sales Support
•PI
•DI Technical Support
•DQ Technical Support
•DI
•RTI
•DQ
•UI
CEO
Auditor
Counselor
Sales Div.
PPC Div.
Business Consulting Div.
Global Business Div.
Management Support Div.
Technical Service Div.
R&D Center
5

Business Area
Data Governance
• Data Governance Architecture
• Data Quality Management
• Meta Data Management
• Master Data Management
• Data Quality Appliance
Data Integration
• High Performance ETL ∙ Batch
• Data Integration
• Deferred(Near Real Time)
• CDC, Real Time Data Transition
• High Speed Data Extraction
• High Speed Data Sort
• Data Integration with Hadoop
• Data Integration with Grid
• Test Data Management
Big Data
• Big Data Platform with Hadoop
• Big Data Anaysis & Visualization
• Structured & Unstructured Data Analysis
• SNS Data Analysis ∙ Monitoring
Consulting
• ISP & Big Data Consulting
• Fraud Detection System(FDS) Consulting
• DW ∙ CRM ∙ BI Consulting
• Data Integration & Migration Consulting
• Data Standardization ∙ Quality Management ∙ Architecture Consulting
• Master Data Management Consulting
• Data Lineage Management Consulting
DW/BI
• Building DW ∙ CRM ∙ BI
• QPI Methodology
• Fraud Detection System(FDS)
• Information System Planning(ISP)
• Alternative Trading System Consulting
• Transaction Cost Data Analysis Framework
• Transaction Cost Data Analysis Framework & Consulting(TCA)
• Financial Analysis Services
DataStreams Is a Company Which Has Expertise in Data Processing and Analysis to Provide Total Data Management Services in Data Integration and Quality Management.
Data Lineage Management
• Data Lineage Analysis Platform
• Visualization for Data Lineage
• Relative Tool, Program & Script Language Analysis
• Table Column Search & Monitoring
6

Market Recognition & Share
7
60%
25%
10%
5%
80%
15%
5%
Korean Market Share
for ETL Solution
Korean Market Share
for Data Migration
(Banking Industry)
DataStreams Corp.
IBM
Informatica
Others
* The market share for ETL solutions is self-researched in 2013.
55%
30%
15%
Korean Market Share
for Metadata Management
W Company (Korean)
G Company (Korean)
ETL
Data
Migration
Metadata
Management
No. 1 Total Data Management Technologies in Korea
Vendor Report of Magic Quadrant for Data Integration and Data Quality Tools
(2013)
Reference URL : http://www.citia.co.uk/
Mentioned DataStreams’ capabilities of offering wide range of data integration
products through ETL, CDC and near-real time technologies.

Private
Banking /
Finance Companies
Public Finance
Companies
Government &
Public /
Educational
Institutions
Enterprises
Major Domestic Customers
8

DataStreams Is Exporting & Expanding…
Columbia
Banco Colpatria
Bogota City Government
Credibanco
China
Kookmin Bank
Hana Bank
USA
Merklenet, Inc.
CSC Consulting
Bisys
Comcast
Merkle Data Tech
USA
Airweb
Sungard
American Airlines
Highmark, Inc.
Mexico
Sodexhopass
Procesar
Peru
Banco Ripley
Chile
Banco Estado de Chile
Australia
National Wealth Management
(MLC/NAB)
Spain
Procecard
Tecnocom
Telefonica Soluciones
ITnow!
Germany
Accenture GmbH
India
Reliance Industry
Indonesia
Excelcom
Aviva
Telkomsel
Hana Bank
Global Customers
9
Japan
with Reliable Business Partners
U.S.A.
BellaDati(US)
Vietnam
HIPT
FPT IS
Lac Viet
QTSC
EU
BellaDati(CZE)
Gibkie(RUS)
IMBI(Europe, N. Africa)
China
China Mobile
Fuchen Telecom
& Banking / Insurance Companies

Business Challenges
Data Integration Architecture
2. Data Integration as Data Infrastructure

Data Integration Landscape: Business Challenges
 Inaccurate data leads to bad or no decisions
 More than 30% of IT budgets typically spent on Data integration
 Inconsistent enterprise and application architecture for integration
Disparate data
Inaccurate data
Incomplete data
Untimely data
Fragmented Integration Approach
Factors
 Bad decisions
 Lost revenue
 Lost productivity
 Lost market opportunity
 Bad Citizen relationships
Results
Multiple versions of the “Truth”
Wasted time and resources aggregating information
Difficult to use Data
Delayed Decision making
Uninformed management
Impacts
11

Data Oriented Business Intelligence Architecture
Administration of data integration and quality should be based on solid data
infrastructure which requires data transformation and quality management.
DM
Legacy
Legacy Channel
ERP CRM
DBMS
EDW
ODS
SAM file
DW
ETL/DQ/RTI
ETL/DQ
ETL/DQ
Data Architecture Application Architecture
OLAP
…
E
D
W
P
O
R
T
A
L
Casual
User
Power
User
Data
Architect
IT
Developer
Metadata
manage-ment
system
Aggregation
Cu tsatbolme er
Contract
Revenue
…
…
Fundamental
Rule set
Column
Dependency
Description of
column
…
Involution
Rule Set vs Column
Talbe vs Column
F(FundamentFa(lF) undamental)
A(Association)
I(Involution)
D(Dependency) D(Dependency)
Central Metadata Repository
Relational
Integrity
User Defined
Integrity
DQI
aggreg
ation
Rule
Set
Inspec
-tion
Inspec
tion
Result
Inspecti
on
report
Domain
Integrity
Metadata Oriented Service Architecture
QualityStreamTM MetaStreamTM ImpactStreamTM
------
------
------
------
------
------
------
------
------
Program source
Pl/SQL
Stored Procedure
...
Token
Special String
SQL script
Analysis
PGM source
Impact Analysis
Engine
TeraStreamTM / DeltaStreamTM/TeraNRT/ QualityStreamTM
12

Performance & Cost Savings of TeraStream™
Features & Benefits
Customers & Application
3. TeraStream™ Overview

TeraStream™ Overview (1/2)
Variety of data types and formats transport from source to target as needed.
Covers enterprise-wise data flow from operational to subject data mart.
Also applied to high volume batch processing and near real-time data integration.
Data extraction from a various commercial DBMS in high speed
High performance sort engine resolves time bottleneck due to Sort//Join/Aggregation
Automatic generation of scripts can be used for loading to various DBMSs
Transform / Cleansing
Load
Files
New Systems
Files
Conversion
Sort/Join/Aggregation
Reformat
Databases
Databases
Extraction
14

TeraStream overview (2/2) – product configuration
TeraStream™ includes a sort engine and a high volume data extraction engine(FACT™), and meta data is stored and managed in DBMS.
• easy to use GUI for developers.
 User Interface
• High performance (FACT/CoSORT)
• External command(shell/SortCL)
• Query processing
• Data conversion (Korean/Japanese)
• Function processing
 Data Processing
 Metadata Management
 Operations & Administration
User Interface
Operations & Administration
Data Processing Engine
TeraStream Designer
Metadata Management Engine
TeraStream DB
(Repository)
Log
Manager
Project
Scheduler
FFD
Manager
Process
Manager
Data
Access
Manager
Message
Broker
FACTTM
CoSORTTM
Converter
USQL
External
command
User SCL
• Job and system log management
• Job scheduling
• File Format Description for metadata
• Real-time job monitoring
• Authentication Management
•Data format, job & system information in TSDB(Repository)
Monitor
15

TeraStreamTM for Data Integration
 TeraStream™ is a high-performance ETL solution with convenient GUI
which is proved for its reliability in variety of enterprises for a decade.
Experiences Cost
Plenty of customers
Various industry
Decades of
experiences
TeraStream™
 Easy to use
 Easy to operate
 Easy to maintain
 Sort engine
(CoSORT™)
High-speed
extraction (FACT™)
Reuse of data (EBH)
 Low resource use
 Low development
cost
 quick development
Performance Conveniences
16

FILE → DB
DB → DB
FILE → FILE
FILE → DB
Excellent Performance and Resource Usage
TeraStreamTM out-performed 3-times in speed against its competitor with 30% of CPU resource.(Data Migration in Shinhan Bank, Korea)
D product
TeraStream™
 Elapse time : 20 minutes
 Wasted System Resource : 800
( 40% Avg. CPU usage X 20 mins )
 Elapse time : 59 minutes
 Wasted System Resource : 3000
(50% Avg. CPU usage X 60 mins)
 thread MAX for sort =3
 File manipulation : 35% CPU usage
Load : 80% of peak CPU usage
 Parallel = 4
 File manipulation : 58% of CPU usage.
 Load: 58% of peak CPU usage
Conclusion
Conclusion
17

Excellent Performance in NRT Implementation
Transportation of up to 1 million records per minute by reading flat files through
EAI and splitting them per tables eliminating the duplicated business days to
Sybase IQ.
3 X
0
10
20
30
40
50
60
70
100 1,000 5,000 10,000 20,000
D product
minutes
Thousand records
[Shinhan bank DW Benchmark in August, 2006)]
18

Time Table for NRT Implementation
Unit
(records in thousand)
TeraStream™
D product
mapping/processing/loading
mapping/processing/loading
start
end
time
start
end
time
100
18:02:39
18:02:55
0:16
15:08:16
15:10:33
00:53
1000
18:05:25
18:06:23
0:58
15:11:13
15:20:34
03:32
5000
18:07:20
18:12:02
4:42
15:25:14
15:43:44
15:28
10,000
18:13:54
18:24:20
10:26
15:47:57
16:23:45
31:09
20,000
18:29:10
18:49:55
20:45
16:31:40
17:36:10
58:41
10,000
(concurrent execution)
11:35:48
11:50:35
14:47
11:35:48
12:17:10
41:22
19

Excellent Performance in Batch Jobs
TeraStream™’s excellent performance can be applied to not only ETL but also daily batch jobs.
[Batch Job of POST Insurance Service Company, 2007]
High Performance
Effective use of resources
Convenience
No. of Records
Oracle (SQL)
TeraStream
400,000
1m 32s
28s
1,000,000
5m 01s
41s
2,500,000
12m 21s
59s
No. of Recs
Oracle
Time
250,000~500,000
Tth
20

ETL Performance Improvement
 Using EBH, TeraStreamTM can cut down data path from Legacy to MART saving ETL time and resource usage.
 Massive volume of files extracted from Legacy Systems are stored in EBH for further reuse in next step.
 ETL time is reduced by avg. 56%. (In L-Telecom from D-3 to D-1)
EDW Server
IBM p690
NCR 10Node
Teradata
D-1
Oracle 8i
ETL Server
ODS
Customer/Call/
Billing
Connection
PPS/BSS
Mining Input Variable
MOLAP Analysis
Mining Analysis
Campaign Analysis
Sybase IQ/ASE
OLAP
MART Server
CSM/AR
Billing
Oracle 8.0.6
CCS/MPS/ERP
CTI /PPS/NMS
SRDF
Legacy
ETL
EBH
Infomatica
EBH (ETL and Batch Hub) stores temporary and result files which is shared for further table generation in EDW and MART.
21

Job
Task
Cycle
System
Before
After
Improvementrate
Billing
Sales
Month
EDW
12:50
5:00
61%
OLAP Mart
18:35
8:20
55%
Calls
Charges
day
EDW
5:50
3:00
49%
OLAP Mart
8:00
4:00
50%
ACCUM
week
EDW
4:20
1:55
56%
OLAP Mart
7:20
3:00
60%
receiving CDR (NMS)
day
EDW
1:00
0:30
50%
OLAP Mart
2:20
0:55
61%
Sending CDR (NMS)
Day
EDW
1:40
1:05
35%
ERP batch
Month
EDW
11:20
3:15
71%
receiving CDR (NMS)
Month
EDW
5:00
2:15
55%
OLAP mart
11:40
2:20
80%
sending CDR (NMS)
Month
EDW
8:20
4:50
42%
ERP provided BATCH
Month
EDW
16:20
5:15
68%
Customer Service
After service
month
EDW
5:30
5:05
9%
Improvement Details
22

Cost Saving Factors
 TeraStream has many cost saving factors.
Ease of Maintenance
Operational Efficiency
Quick Development
 Creating standard operation and new development procedures
 Efficient operation of test system
 Ability to fast make test file
Ease of maintenance
Quality improvement of developing programs - Reliability - Maintainability - Efficiency - Functionality
 Productivity improvement of developers and administrators
 Developing correct and efficient logic programs
 Quality improvement
23

Cost Saving : Development Stage
The higher complexity, the bigger cost saving in development .
(Courtesy of Hanhwa Insurance Co. and SKC&C
in 2007)
Job
complexity
No. of recs
Input
Size
(Gb)
TeraStream™
In- house
coding
Speed- up
1:1 mapping
90
22
30min
2hour
75%
1:N mapping
900
21
2hour
6hour
66%
N:1 mapping
1700
15
2hour
10hour
80%
N:N mapping, complex
1300
8
2hour
20hour
90%
Avg. 70% of development speed-up
90% speed-up for more complex jobs
Overhead from modification, test and preliminary data checking.
Development
(4Month)
Test
(4Month)
Stabilization
(1Month)
24M/M
48M/M
54M/M
TeraStream™
In-house coding
(Estimated)
40M/M
80M/M
90M/M
40%
Reduction
24

Features & Benefits
TeraStream™ guarantees to meet your need for enterprise data integration as well as excellent batch job hub.
Sort Engine
Using CoSORT™, the first sort package since 1985, TeraStream™ can accelerate sort-related data manipulation (dedup, average, min, max, join, summary and etc.)
FAst extraCT
FACT™ performs high speed bulk extraction from various commercial DBMS.
Automatic Metadata Generation
TeraStream™ provides direct reading of DBMS data dictionary to create its own metadata information.
High Speed Lookup
It provides in-memory lookup function which is high speed mapping conversion using lookup tables.
Variety of conversion function calls
It provides more than 100 user friendly mapping functions.
Developers can easily add their own functions.
Pre/Post Processing
TeraStream™ provides inter-record and inter-table conversion through pre/post mapping.
Major Features
Description
25

Features & Benefits
TeraStream™ has been evolved to meet various parallel processing needs and to give convenience through highly efficient GUIs.
Inter-node Operation
Remote call is possible to initiate the projects of other nodes between TeraStream™s.
Distributed Computing using idle nodes is possible by easy transfer of data.
Near Real-Time ETL
Data transportation every minute is possible including complex data mapping
Efficient GUI
Using GUI, no skills on programming language are necessary.
Unified monitor and control in single screen or specialied monitoring is possible through web browser.
Scheduling of jobs is made in unified GUI but even for distributed servers.
Multi Language Support
UTF-8 is supported.
Major Features
Description
26

Improved GUI
Supports for data integration activities(develop, execute, monitor, validation) in integrated GUI environment
Intuitive task flow
Project monitor
Editor window
GUI for developers
Intuitive task flow
checking standard output/error/file information/
number of files processed
Execution log
real time job monitoring
 Project Monitor
scheduling by time/ period/ business calendar
 Scheduler
Mapping creation
 Editor window
Scheduler
Task block execution log
Metadata property
Impact analysis
Change history manager
 Metadata Repository
27

DBMS Connectivity
Powerful connection between different DBMS types.
Both DB-to-DB and File-to-DB data transportation are supported.
•N:N mapping
•Conversion while transportation
•Click to choose record processing types : (Insert/delete/update/insert- update/delete-insert)
•DBMS types : Oracle, DB2, Sybase, Informix, Teradata, Greenplum, MSSQL, MySQL, (Altibase, Tibero)
Transformation Logic
Source Table
Target Table
28

High speed data extraction with FACT™
High speed data extraction of commercial database with SQL is supported.
Automatic extraction query is generated.
Select * from table
•High speed extraction engine(FACT™) with optimized database API.
•DBMS Supported :
- Oracle - Informix - DB2 / UDB - Sybase IQ /ASE - Teradata - Greenplum - MSSQL /MySQL - Altibase
•File split and filtering while extraction
•Time, time stamp, and user data format specification
29

Data Conversion
By mapping source to target, conversion of formats, types, character sets, dates, bytes/bits, encryption
•Convenient data conversion using mapping window of “converter task block”
•Data character set conversion including EBCDIC to ASCII
•Data conversion from NDB(Unisys 9-bit) or HDB(IBM) data type to RDB
•300 built-in functions
•DATE, Time Stamp Conversion between different date formats
•CLOB/BLOB supported
•Users can add more functions as needed
List of provided functions
CALLED_NO function editor
=addday(cdate(“",”",” (N)")
addday(cdate("2005/05/12 12:08:24", "YYYY/HH/DD HH:MI:SS"),2)
Converter task block
30

Data Transport
TeraStream uses various transportation method according to file structure, transportation distance, security, amount of record and etc.
•File to DB data load for bulk data
•“Load task block” generates load scripts automatically.
•Remote transportation using FTP
•Encryption while transporting
•Near Real-time and Bulk transportation is possible
Load Scripts
31

Korea Exchange Bank
Expected Result
 TeraStream™ is in charge of data flow from core-banking to data mart.
 Batch job is also executed during ETL process in the same manner.
Issues
Plans
System Configuration
Extract
24 hours
DB SPLIT
Profit Management
RISK
KPI
EUC
current ODW
ORACLE 9i
EDW
Sybase ASIQ 12.5
New ODW
ORACLE 9i
Bidirectional ETL
ETL
ETL
ETL
Batch
Batch
Batch
FTP transfer after extraction
Accounting System
Information System DW
DM & Sub System
Open in Feb. 2005
Batch
• A next-generation conversion of M/F and IMS HDB
• Interactive ETL process
• High volume data handling within batch process time
• Converting main frame data into Unix data (1TB → 1.5TB) including Korean character conversion
• ETL task from accounting system server to new ODW server(extracting appx. 200 GB of daily changed data within 1 hour and 30 minutes by using FACT™ of TeraStream™)
• Data hub is used in ETL process from new core banking system to subjective marts. (EUC, KPI, RISK, IFRS, Basel2)
• ETL and Batch process in unified way.
• Batch process is designed to be done in both ODW and EDW.
 Unified Data Processing in ETL & Batch
32

System configuration
Issues
Plans
Kookmin Bank
IBM M/F
HDB, DB2
Server RDB
Sybase ASIQ 12.7
IMS HDB
- Seg. split
- conversion & Array split
- logic applied
- conversion
- logic applied
- Logic applied
EDW
A-SOR
DM
ETL
ETL
ETL
Informover
TS(FACT)
Informover
Source system
File process flow
DB QUERY
Expected Result
 Various DBMS(IMS HDB, HOST DB2, Oracle, DB2 UDB) integration by using TeraStream™
 Meeting batch target time of 2 hours and 30 minutes for 4TB of EBCDIC data.
• M/F and IMS HDB conversion
• Processing changed data in absence of time-series column
• Processing large size data within batch process time(10TB/day based on source data)
• How to process high volume files in parallel
• Converting main frame data into data in Unix environment (10TB → 25TB) within 18 hours.
• Various data conversion and processing including Korean character conversion
• ETL task from accounting system server to new ODW server(extracting appx. 200 GB of daily changed data within 1 hour and 30 minutes by using FACT module of TeraStream™)
• ETL and Batch process in unified way.
• Batch job in core banking system within 6 hours.
 EDW and integrated DM installation
33

Samsung Electronics
• Rea-time data transportation between Germany and
China.
• Bi-directional synchronization between TeraStreams
of Germany and China.
• 20 min. MAX loading time for transported data is
implemented using TeraStream NRT.
•Web Monitoring is developed
• Registration in one country should have the same
service at other country.
• duplicated record should be avoided due to cross
transportation.
• 20 minutes Near Real-time
• Perfect Recovery scheme should be presented
Plans
Issues System Configuration
Smart Phone System
in Germany
DBs in
Service
 Efficiency is maintained despite cross transportation
Bi-directional NRT integration allows the same service regardless of system type
and country (Time from extraction to loading is 20 minutes.)
 Bi-directional remote data transportation using TeraStream
NRT Extract
프로그램 성공, 실패 등 실행 결과
Web Monitoring
Sam To DB
UPSERT
NRT Extract
SAM To DB
UPSERT
 Global Database Integration using NRT ETL
DBs in
Service
Smart Phone System
in China
Expected
Result
34

Department of Domestic Administration
 Building an E-government and improving the accessibility of government Information
•The required data is interactively transferred between DW and DM by EAI and configured with ETL.
•Target : the central office and 16 cities and provinces
•Work status by provinces
1) Administration -> Seoul City
2) Environment : 2 cities and 2 provinces
3) Health Service : 1 city
4) Economy and trade : 2 cities and 7 provinces
•50 GB of data processing per day
•How to save the processed data : transfer data to a file system and use a backup solution
•The central office data is provided in file through EAI.
• Oracle data from central office, other cities and provinces is directly extracted by FACT and stored in DW/DM(Sybase).
• To complete substantial E-Government service
• To provide integrated high-quality information
• To build a fast and accurate decision support system
Issues
Plan
Integrated Data Mart
Data Warehouses in 16 cities and provinces
ODS
DW
ODS
DW
DM
DM
Central
governmental
system
Other
provincial s
ystems
ETL
EAI
 Provincial Administrative DW
Expected Result
35

 Batch processing time for daily closing and unified collection reporting is reduced.
 ETL flows unified in TeraStream.
 ETL Batch Hub reduces data flow path.
• Complex extraction process & difficult job tracking
• Absence of ODS makes data referencing hard.
• Long DW time cause delayed service start in the morning.
•Heavy loads in closing process
Issues
• Unified processing of ODS, DW & DM
• Simple extraction scheme and easy tracking of errors.
•ETL time and load reduction using EBH
Plans
SQL
script
stage
DW
DM
Procedure
Source
ODS
DM
DW
ETL HUB
D-1
D-1
Source
Before Change
After Change
 Data Warehouse ETL
Expected Result
Korea Worker’s Compensation & Welfare Service
36

Health and Welfare Department’s e-Voucher
E-Voucher DW Performance Improvement
Statistics reporting time is dramatically reduced from 1~6 days to a few second or minutes.
 Statistics reporting process made simple and easy to get report.
 Consistent data delievry increase data reliability.
• daily transportation to ODS
• build ODS, DW and DM for better table model
• e-Voucher System (DB2 -> DW Server)
• Platform - OS : AIX 5.3(ASIS,TOBE ) - CPU : Power5, 2.1GHz, 6core , IBM P-serise
- MEM : 12 GB
- H/W : 1TB
• Simple logic made MA easy
•Low data integrity
•Lack of expeditious response
•Fraud detection was hard.
•Low reliability of statistic data caused dispute between data users and generators
Plans
Issues
E-Voucher Statistical DW
Operational
Source DB
(oracle)
- ODS data conversion
- update/insert at ODS
- 1:1 mapping
- Daily batch
- Load to ODS
IBM P-serise
Voucher Servide
Mis-settlement
Pregnancy & Birth
History
Target DB
(oracle)
FACT
ODS
DM
DW
ETL
- ODS/ DW data manuplation
- update/insert to data mart
ETL
ETL
Expected Result
37

Korea National Open University (Synthetic Statistic DW )
Expected Result
 Building a complete view and improving the accessibility of Statistical DW .
 Reducing the institutional service time from 15 days to 4 hours.
 Expeditious response of university’s administration statistics report become possible.
•Extract an operational data for the institutional service and university administration service, transfer the data into ODS, DW and DM, and then load Oracle DBMS by ETL for producing typical and atypical statistic reports
•Institutional Statistic Service Data (Initial/Alteration) :
20GB/1GB, Alteration data load time: 4 hours
•University Administration Statistic Service Data (Initial/Alteration) : 5GB/100MB, Alteration data load time: 1 hour
• Reducing time and automation for the existing manual based university statistic services
• To provide integrated high-quality academic, administration, and management information
• To build a fast and accurate academic administration decision support system
Issues
Plan
Univ, Admin.
Graduate
Institutional Services
Electronic
settlement
Tutor
Administ
-ration
Lifelong
Education
Institution
Graduation
Grade
Registration
Admission
Synthetic Statistic DW Server
TeraStream
ODS
DW
DM
Data Extract/Load
Data
Extract/Load
Data Extract/Transfer/Load
ETL Control
Data Extract/Load
Data Extract/Load
Unit Process
Processing Time
Institutional Statistics Services
Within 4 hours
University Statistic Services
Within 1 hour
Huge time reduction from of institutional statistic sservice reports 14 days to 4 hours
38

TeraStream - Data Integration/Migration/ETL/Batch Tool

TeraStream - Data Integration/Migration/ETL/Batch Tool

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to TeraStream - Data Integration/Migration/ETL/Batch Tool

Similar to TeraStream - Data Integration/Migration/ETL/Batch Tool (20)

Recently uploaded

Recently uploaded (20)

TeraStream - Data Integration/Migration/ETL/Batch Tool