More Related Content Similar to SAP World Tour 2010: Impact of Column-Oriented Main-Memory Databases on Enterprise Applications Similar to SAP World Tour 2010: Impact of Column-Oriented Main-Memory Databases on Enterprise Applications (20) More from Matthieu Schapranow More from Matthieu Schapranow (20) SAP World Tour 2010: Impact of Column-Oriented Main-Memory Databases on Enterprise Applications1. Impact of Column-OrientedMain-Memory Databases on Enterprise Applications Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld Hasso Plattner Institute March 02, 2010 2. © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 2 Disclaimer This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent. 3. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 3 4. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 4 5. Key Facts about the Hasso Plattner Institute Founded as a public private partnershipin 1998 in Potsdam near Berlin, Germany Institute belongs to theUniversity of Potsdam Ranked 1st in “CHE” 340 B.Sc. and M.Sc. students 10 professors, 91 PhD students Course of study: IT Systems Engineering © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 5 6. Prof. Dr. h.c. Hasso Plattner / Dr. Alexander Zeier Research focuses on the technical aspects of enterprise software anddesign of complex applications Memory-Based Data Management for Enterprise Applications Human-Centered Software Design and Engineering Maintenance and Evolution of Service-Oriented Enterprise Software Integration of RFID Technology in Enterprise Platforms Architecture-based Performance Simulation Research co-operations with Stanford, MIT, etc. Industry co-operations with SAP, Siemens, Audi, etc. Research GroupEnterprise Platform & Integration Concepts Partner of Stanford Center for Design Research Partner of MIT in Supply Chain Innovation © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 6 7. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 7 8. Two separate worlds: OLTP and OLAP? © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 8 9. Two separate worlds: OLTP and OLAP? © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 9 10. Dominant Hardware Trends Multi-Core Technology Moore’s Law: “…number of transistors … doubling approximately CPU frequency hit limitin 2002, but Moore’s law holds today In-Memory Technology Increased size: up to 2TB of main-memory on one main board in 2010 Constantly dropping costs © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 10 11. 3 Aspects for a Hybrid Solution Columnar Storage New database layout accessing only needed portions of data Improve access for subsets of attributes In-Memory Fastest possible data access Spatial proximity Compression Reduce amount of data to fit in main memory Use cache and bus capacities more efficient © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 11 12. Row Store Column Store Storages: Row vs. Column © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 12 13. Columnar Storage: Architecture Claim: Columnar storage is suited for update-intensive applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 13 14. In-Memory: Aggregate Processing Time © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 14 The value of an attribute changes by calculation 15. Compression: Types © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 15 16. Dictionaries Compression:Advantages ofColumnar Storages © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 16 17. Scalability: Multiple CPU Cores Set processing is most frequent access type in EAs(scan is dominant pattern) Sequential column-wise scans show best bandwidth utilization between CPU cores and main memory Independence of tuples per column allows: easy partitioning, and parallel processing (see Hennessy [1]) Faster memory scans by improved memory bandwidth in next generation CPUs Neither materialized views nor aggregateseverything is calculated on-the-fly © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 17 [1] John L. Hennessy, David A. Patterson: Computer Architecture: A Quantitative Approach 34. Agenda The Hasso Plattner Institute Technical Foundation of Columnar In-Memory Databases Impact on Enterprise Applications © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 21 35. Architecture of ExistingFinancials Systems © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 22 36. Architecture of Simplified Financials Systems © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 23 Only base tables, algorithms, and some indices 37. Analyzing Real Customer Data 1M records in BSEG ~ 1GB disk storage © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 24 38. Results:Distinct Values per Attribute Results on analyzing Financials Distinct values in accounting document headers (99 attributes) CPG Logistics Banking High Tech Discrete Manufacturing © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 25 39. Results:Accounting Document Updates Percentage of rows updated © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 26 40. Dunning © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 27 41. Available to Promise © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 28 42. Demand Planning © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 29 43. Insert Only Tuple visibility indicated by timestamps (POSTGRES-style time-travel [2]) Additional storage requirements can be neglected due to low update frequency Timestamp columns are not compressed to avoid additional merge costs Snapshot isolation Application-level locks Insert Only © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 30 44. Memory Consumption Experiments show a general factor 10 in compression (using dictionary compression and bit vector encoding) Additional storage savings by removing materialized aggregates, save ~2× Keep only the active partition of the data in memory (based on fiscal year), save ~5× Next generation blade servers will allow up to 512 GB RAM. Arrays of 100 blades already available 50 TB main memory would allow to cover the majority of SAP Business Suite customers © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 31 45. Impact on Application Development Formalized logic must be moved close to the engine Calculations must take place close to the data Reduction of application code OLTP queries must use minimal projections (SELECT * is not allowed) No caching necessary anymore © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 32 46. Conclusion Technology improvements allow re-thinking of how we build enterprise apps: A combined OLTP and OLAP system can share the same in-memory column store data base Our experiments with real applications and data prove it Open research challenges: Disaster recovery, extension for unstructured data, life cycle based data management © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 33 47. Further Information è SAP Public Web: EPIC@HPI: https://epic.hpi.uni-potsdam.de Hasso Plattner Institute: http://www.hpi-web.de © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 34 48. Thank you! Contact us! Hasso Plattner Institute EA²L / Enterprise Platform & Integration Concepts Matthieu-P. Schapranow August-Bebel-Str. 88 D-14482 Potsdam, Germany Matthieu-P. Schapranow matthieu.schapranow@hpi.uni-potsdam.de Responsible: Deputy Prof. of Prof. Hasso PlattnerDr. Alexander Zeierzeier@hpi.uni-potsdam.de © SAP 2008 / SAP TechEd 08 / <Session ID> Page 35 49. Feedback Please complete your session evaluation. Be courteous — deposit your trash, and do not take the handouts for the following session. Thank You ! © HPI & SAP 2010 / SAP World Tour 10 / Impact of Column-Oriented Main-Memory Databases on Enterprise Applications, Dr. Alexander Zeier, Matthieu-P. Schapranow, Christian Tinnefeld / Page 36 Editor's Notes ccdcdMoore’s Law: “…number of transistors … doubling approximately every two years”CPU frequency hit limitin 2002, but Mooreslaw holds todayHow? Multi-Core and Parallelization Select required attributes only X: number of aggregatesY: log. time required for aggregate calculation ordered/few: tarif ratesUnordered/few: sexOrdered/Distinct: temperature values Partitioning! Remove data redundancy Partioning! Insert-Only Stress on:Materialized aggregatesMaterialized viewsIndicesRedudant data in cubes, change history, … Analysis of accounting tablesBkpf= accounting document headersBseg = accounting document line items VieleSELECTs: bspw. Dunning ablauf (mituntersehr complex) row-oriented, relational programming pattern select via attributes (column-wise) cp. to OLAP needs a rewrite!!! What about rescheduling for high-prio customer now: manual rescheduling necessary dank main-memory jedes mal neuberechnenmöglich rescheduling on-demand ATP combining with pricing, e.g. customer demands for a certain price per product you can name shipping date (cupper, metals, oil, etc.) Aggregates narrow your flexibility interactive planning