This ppt gives information about: 1. Delivery Process 2. System Processes 3.Architecture

1. 1

2. May 6, 2016
Data Mining: Concepts and
Techniques
2
Introduction toIntroduction to
DataData
WarehousingWarehousing

3. Chapter 3: Data Warehousing and OLAPChapter 3: Data Warehousing and OLAP
Technology: An OverviewTechnology: An Overview
• What is a data warehouse?
• Data warehouse architecture
• From data warehousing to data mining

4. What is Data Warehouse?What is Data Warehouse?
• Defined in many different ways, but not rigorously.
o A decision support database that is maintained separately from
the organization’s operational database
o Support information processing by providing a solid platform of
consolidated, historical data for analysis.
• “A data warehouse is a subject-oriented, integrated, time-variant,
and nonvolatile collection of data in support of management’s
decision-making process.”—W. H. Inmon
o Data warehousing:The process of constructing and using
data warehouses

5. Data Warehouse—Subject-OrientedData Warehouse—Subject-Oriented
• Organized around major subjects, such as customer, product,
sales
• Focusing on the modeling and analysis of data for decision
makers, not on daily operations or transaction processing
• Provide a simple and concise view around particular subject issues
by excluding data that are not useful in the decision support
process

6. Data Warehouse—IntegratedData Warehouse—Integrated
• Constructed by integrating multiple, heterogeneous data sources
o relational databases, flat files, on-line transaction records
• Data cleaning and data integration techniques are applied.
o Ensure consistency in naming conventions, encoding structures,
attribute measures, etc. among different data sources
• E.g., Hotel price: currency, tax, breakfast covered, etc.
o When data is moved to the warehouse, it is converted.

7. Data Warehouse—Time VariantData Warehouse—Time Variant
• The time horizon for the data warehouse is significantly longer
than that of operational systems
o Operational database: current value data
o Data warehouse data: provide information from a historical
perspective (e.g., past 5-10 years)
• Every key structure in the data warehouse
o Contains an element of time, explicitly or implicitly
o But the key of operational data may or may not contain
“time element”

8. Data Warehouse—NonvolatileData Warehouse—Nonvolatile
• A physically separate store of data transformed from the
operational environment
• Operational update of data does not occur in the data
warehouse environment
o Does not require transaction processing, recovery, and
concurrency control mechanisms
o Requires only two operations in data accessing:
• initial loading of data and access of data

9. Data Warehouse vs. Heterogeneous DBMSData Warehouse vs. Heterogeneous DBMS
• Traditional heterogeneous DB integration: A query driven approach
o Build wrappers/mediators on top of heterogeneous databases
o When a query is posed to a client site, a meta-dictionary is used to
translate the query into queries appropriate for individual
heterogeneous sites involved, and the results are integrated into a
global answer set
o Complex information filtering, compete for resources
• Data warehouse: update-driven, high performance
o Information from heterogeneous sources is integrated in advance and
stored in warehouses for direct query and analysis

10. Data Warehouse vs. Operational DBMSData Warehouse vs. Operational DBMS
• OLTP (on-line transaction processing)
o Major task of traditional relational DBMS
o Day-to-day operations: purchasing, inventory, banking,
manufacturing, payroll, registration, accounting, etc.
• OLAP (on-line analytical processing)
o Major task of data warehouse system
o Data analysis and decision making
• Distinct features (OLTP vs. OLAP):
o User and system orientation: customer vs. market
o Data contents: current, detailed vs. historical, consolidated
o Database design: ER + application vs. star + subject
o View: current, local vs. evolutionary, integrated

11. OLTP vs. OLAPOLTP vs. OLAP
OLTP OLAP
users clerk, IT professional knowledge worker
function day to day operations decision support
DB design application-oriented subject-oriented
data current, up-to-date
detailed, flat relational
isolated
historical,
summarized, multidimensional
integrated, consolidated
usage repetitive ad-hoc
access read/write
index/hash on prim. key
lots of scans
unit of work short, simple transaction complex query
# records accessed tens millions
#users thousands hundreds
DB size 100MB-GB 100GB-TB
metric transaction throughput query throughput, response

12. Why Separate Data Warehouse?Why Separate Data Warehouse?
• High performance for both systems
o DBMS— tuned for OLTP: access methods, indexing, concurrency
control, recovery
o Warehouse—tuned for OLAP: complex OLAP queries,
multidimensional view, consolidation
• Different functions and different data:
o missing data: Decision support requires historical data which
operational DBs do not typically maintain
o data consolidation: DS requires consolidation (aggregation,
summarization) of data from heterogeneous sources
o data quality: different sources typically use inconsistent data
representations, codes and formats which have to be reconciled

13. Chapter 3: Data Warehousing and OLAP Technology: An OverviewChapter 3: Data Warehousing and OLAP Technology: An Overview
• What is a data warehouse?
• Data warehouse architecture
• From data warehousing to data mining

14. Design of Data Warehouse: A Business Analysis FrameworkDesign of Data Warehouse: A Business Analysis Framework
• Four views regarding the design of a data warehouse
o Top-down view
• allows selection of the relevant information necessary for the
data warehouse
o Data source view
• exposes the information being captured, stored, and
managed by operational systems
o Data warehouse view
• consists of fact tables and dimension tables
o Business query view
• sees the perspectives of data in the warehouse from the view
of end-user

15. Data Warehouse Design ProcessData Warehouse Design Process
• Top-down, bottom-up approaches or a combination of both
o Top-down: Starts with overall design and planning (mature)
o Bottom-up: Starts with experiments and prototypes (rapid)
• From software engineering point of view
o Waterfall: structured and systematic analysis at each step before
proceeding to the next
o Spiral: rapid generation of increasingly functional systems, short turn
around time, quick turn around
• Typical data warehouse design process
o Choose a business process to model, e.g., orders, invoices, etc.
o Choose the grain (atomic level of data) of the business process
o Choose the dimensions that will apply to each fact table record

16. Data Warehouse: A Multi-Tiered ArchitectureData Warehouse: A Multi-Tiered Architecture
OLAP Engine
Metadata
Data Sources Front-End ToolsData Storage
Data
Warehouse
Extract
Transform
Load
Refresh
Analysis
Query
Reports
Data mining
Monitor
&
Integrator
Serve
Data Marts
Operational
DBs
Other
sources
OLAP Server

17. Three Data Warehouse ModelsThree Data Warehouse Models
• Enterprise warehouse
o collects all of the information about subjects spanning the entire
organization
• Data Mart
o a subset of corporate-wide data that is of value to a specific groups
of users. Its scope is confined to specific, selected groups, such as
marketing data mart
• Independent vs. dependent (directly from warehouse) data mart
• Virtual warehouse
o A set of views over operational databases
o Only some of the possible summary views may be materialized

18. Data Warehouse Development: A Recommended ApproachData Warehouse Development: A Recommended Approach
Multi-Tier Data
Warehouse
Enterprise
Data
Warehouse
Define a high-level corporate data model
Data
Mart
Data
Mart
Distributed
Data Marts
Model refinementModel refinement

19. Data Warehouse Back-End Tools and UtilitiesData Warehouse Back-End Tools and Utilities
• Data extraction
o get data from multiple, heterogeneous, and external sources
• Data cleaning
o detect errors in the data and rectify them when possible
• Data transformation
o convert data from legacy or host format to warehouse
format
• Load
o sort, summarize, consolidate, compute views, check integrity,
and build indicies and partitions
• Refresh
o propagate the updates from the data sources to the
warehouse

20. Metadata RepositoryMetadata Repository
• Meta data is the data defining warehouse objects. It stores:
• Description of the structure of the data warehouse
o schema, view, dimensions, hierarchies, derived data defn, data mart
locations and contents
• Operational meta-data
o data lineage (history of migrated data and transformation path), currency of
data (active, archived, or purged), monitoring information (warehouse
usage statistics, error reports, audit trails)
• The algorithms used for summarization
• The mapping from operational environment to the data warehouse
• Data related to system performance
o warehouse schema, view and derived data definitions
• Business data
o business terms and definitions, ownership of data, charging policies

21. OLAP Server ArchitecturesOLAP Server Architectures
• Relational OLAP (ROLAP)
o Use relational or extended-relational DBMS to store and manage
warehouse data and OLAP middle ware
o Include optimization of DBMS backend, implementation of aggregation
navigation logic, and additional tools and services
o Greater scalability
• Multidimensional OLAP (MOLAP)
o Sparse array-based multidimensional storage engine
o Fast indexing to pre-computed summarized data
• Hybrid OLAP (HOLAP) (e.g., Microsoft SQLServer)
o Flexibility, e.g., low level: relational, high-level: array
• Specialized SQL servers (e.g., Redbricks)
o Specialized support for SQL queries over star/snowflake schemas

22. Chapter 3: Data Warehousing and OLAP Technology: An OverviewChapter 3: Data Warehousing and OLAP Technology: An Overview
• What is a data warehouse?
• Data warehouse architecture
• From data warehousing to data mining

23. Data Warehouse UsageData Warehouse Usage
• Three kinds of data warehouse applications
o Information processing
• supports querying, basic statistical analysis, and reporting using
crosstabs, tables, charts and graphs
o Analytical processing
• multidimensional analysis of data warehouse data
• supports basic OLAP operations, slice-dice, drilling, pivoting
o Data mining
• knowledge discovery from hidden patterns
• supports associations, constructing analytical models,
performing classification and prediction, and presenting the
mining results using visualization tools

24. From On-Line Analytical Processing (OLAP) to On Line Analytical Mining (OLAM)From On-Line Analytical Processing (OLAP) to On Line Analytical Mining (OLAM)
• Why online analytical mining?
o High quality of data in data warehouses
• DW contains integrated, consistent, cleaned data
o Available information processing structure surrounding data
warehouses
• ODBC, OLEDB, Web accessing, service facilities, reporting and
OLAP tools
o OLAP-based exploratory data analysis
• Mining with drilling, dicing, pivoting, etc.
o On-line selection of data mining functions
• Integration and swapping of multiple mining functions,
algorithms, and tasks

25. An OLAM System ArchitectureAn OLAM System Architecture
User GUI API
Data
Warehouse
Meta
Data
MDDB
OLAM
Engine
OLAP
Engine
Data Cube API
Database API
Data cleaning
Data integration
Layer3
OLAP/OLAM
Layer2
MDDB
Layer1
Data
Repository
Layer4
User Interface
Filtering&Integration Filtering
Databases
Mining query Mining result

26. Chapter 3: Data Warehousing and OLAP Technology: An OverviewChapter 3: Data Warehousing and OLAP Technology: An Overview
• What is a data warehouse?
• A multi-dimensional data model
• Data warehouse architecture
• Data warehouse implementation
• From data warehousing to data mining
• Summary

27. Summary: Data Warehouse and OLAP TechnologySummary: Data Warehouse and OLAP Technology
• Why data warehousing?
• Data warehouse architecture
• From OLAP to OLAM (on-line analytical mining)

28. ThankThank You !!!You !!!
For More Information click below link:
Follow Us on:
http://vibranttechnologies.co.in/datawarehousing-classes-in-mumbai.html