Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Quality & Data Governance

1,022 views

Published on

Fundamentals of data quality and data governance

Published in: Data & Analytics
  • Login to see the comments

Data Quality & Data Governance

  1. 1. Data Quality and Data Governance Tuba Yaman Him
  2. 2. Why is Data Quality Important? • Wrong Reports = Wrong Decisions
  3. 3. Why is Data Quality Important? • Wrong Reports = Wrong Decisions • Bad Reputation
  4. 4. Why is Data Quality Important? • Wrong Reports = Wrong Decisions • Bad Reputation • Wasted Money According to a recent study in the UK, US and France, 16% to 18% of departmental budgets are eaten up because of poor data quality. The research also indicates that 90% of surveyed companies admit that inaccurate data – such as duplicate accounts, lost contacts and missed sales opportunities – contributes to budget waste. On top of this, a 2009 Gartner study revealed that the average organization surveyed loses $8.2 million annually because of poor data quality and that most of this is due to lost productivity.
  5. 5. Modern Data Environment Enterprise Data Warehouse ERP Systems (SAP/Oracle etc) CRM (Salesforce, Dynamics etc) Manufacturing Systems Financial Systems Web Applications Documents Marketing Data Mart Sales Data Mart Financial Data Mart
  6. 6. Modern Data Environment Enterprise Data Warehouse ERP Systems (SAP/Oracle etc) CRM (Salesforce, Dynamics etc) Manufacturing Systems Financial Systems Web Applications Documents Marketing Data Mart Sales Data Mart Financial Data Mart
  7. 7. Dimensions Of Data Quality IntegrityAccuracy Currency Uniqueness Validity Completeness
  8. 8. Dimensions Of Data Quality • Do data objects accurately represent the “real-world” values? • Is data correct? • Example: Wrong sales amount, wrong contact information of a customer etc. Accuracy
  9. 9. Dimensions Of Data Quality • Is there are any data missing important relationship linkages? • Example: A product ownership without a valid owner/customer record. Integrity
  10. 10. Dimensions Of Data Quality • Is any neccessary part of data is missing? • Example:A customer record which has an address without city, although city is mandatory. Completeness
  11. 11. Dimensions Of Data Quality • Is data up-to-date? • Do we provide real-time data to our clients? • Example: Customers with old address information. A bank which can not provide the real-time amount of funds of its customers. Currency
  12. 12. Dimensions Of Data Quality • Are there multiple, unnecessary representations of the same data objects within your data? • Example: 3 different records which indicate the same customer. Misspelling can be the reason. CurrencyUniqueness
  13. 13. Dimensions Of Data Quality • Do data values comply with the specified formats and rules? • Example: A customer record whose DOB is dd/mm/1735. A customer record with invalid postal code for UK like WC3T. CurrencyValidity
  14. 14. Methods and Tools For Data Quality Objective How to Validation Regular Expressions Data Merging For Duplicate Data SSIS Fuzzy Lookup, Fuzzy Grouping Packages Integrity Proper ETL and ELT Process Completeness Mandatory Fields Rules, ETL/ELT Verification For Important Information Activation E-mails, Verification SMS Prevent Typographical Error Autocomplete Tools Minimizing Human Errors Employee Training
  15. 15. SSIS Fuzzy Matching • Tuba Yaman Him • Tuba.yamanhim@yopmail.com • Deniz Apt. • Ataşehir • İstanbul • Tuba Him • Tuba.yamanhi@yopmail.com • Deniz Apt. • Ataşehir • istanbul • Tuğba Yaman Him • Tuba.yamanhim@yopmail.com • Deniz Apt. • Ataşehir • İstanbul • Tuba Him • Tuba.yamanhim@yopmail.com • Deniz Apt. • Ataşehir • istanbul
  16. 16. Data Governance Data governance is a set of policies, rules and standarts in order to increase and maintain enterprise data quality. It is about putting people in charge of fixing and preventing issues with data so that the enterprise can become more efficient. Data governance also describes an evolutionary process for a company, altering the company’s way of thinking and setting up the processes to handle information so that it may be utilized by the entire organization. It’s about using technology when necessary in many forms to help aid the process. When companies desire, or are required, to gain control of their data, they empower their people, set up processes and get help from technology to do it
  17. 17. Data Governance
  18. 18. Data Governance –Job Ads USA1.885 India290 UK253 Canada113 Germany83 Singapore25 Switzerland24 Turkey 0
  19. 19. Data Governance Team Missions
  20. 20. Data Quality Scorecard Objective Action Plan KPI Target Jul.2016 Aug.2016 Sep.2016 Decrease Duplicates A Merging flow will be implemented Number of duplicate records in CDB 0 11.276 3.500 200 Increase the Correctness of email info Verification process will be implemented Number of invalid email addresses in Customer DB <500 25.500 4.700 4.700 Decrease wrong relationship of product and customer ETL enhancement is planned. Number of incorrect relations between products and customers in DB 0 2.700 2.700 2.900

×