DataEd Online: Data Architecture and Data Modeling Differences — Achieving a Common Understanding

© Copyright 2021 by Peter Aiken Slide # 1
paiken@plusanythingawesome.com+1.804.382.5957 Peter Aiken, PhD
Data Architecture and
Data Modeling Differences:
Data mapping from two perspectives–achieving a common understanding
Peter Aiken, Ph.D.
• I've been doing this a long time
• My work is recognized as useful
• Associate Professor of IS (vcu.edu)
• Institute for Defense Analyses (ida.org)
• DAMA International (dama.org)
• MIT CDO Society (iscdo.org)
• Anything Awesome (plusanythingawesome.com)
• 11 books and dozens of articles
• Experienced w/ 500+ data
management practices worldwide
• Multi-year immersions
– US DoD (DISA/Army/Marines/DLA)
– Nokia
– Deutsche Bank
– Wells Fargo
– Walmart …
https://plusanythingawesome.com

Event Pricing
• 20% off
directly from the publisher on
select titles
• My Book Store @
http://plusanythingawwsome.com
• Enter the code
"anythingawesome" at the
Technics bookstore checkout
where it says to
"Apply Coupon"
anythingawesome
4
© Copyright 2021 by Peter Aiken Slide #
• Data Maps ➜ Models
– Why do we need them?
– How are they used?
– Challenges to increased use
(social, political, economic)
• Architecture/Engineering
– Two sides of same data coin
– Must operate on standard,
shared data of known quality
• From the Top
– Means: Forward engineering
– Goal: Composition/Building
• From the Bottom
– Means: Reverse engineering
– Goal: Understanding
• Working Together
– Functions required for effective
data management
– Need for simplicity
• Take Aways/Q&A
Achieving a common understanding
Program

Data ...
• As a subject is
– Complex and detailed
– Taught inconsistently, and
– Poorly understood
• Data maps (models) are necessary but
insufficient prerequisites to data architectures
– Fully leveraging data assets
• Maps are incomplete without purpose statements
– More powerful than definitions
– Remedy
• Add purpose statements
• Validate resulting model
• Maps are required to share information about data
• Data architectures are composed of data models
– Data modeling is an engineering activity required to product data maps that are
necessary but insufficient prerequisites to leveraging data assets
Metadata
Management
6
Data
Management
Body
of
Knowledge
(DM
BoK
V2)
Practice
Areas
from The DAMA Guide to the Data Management Body of Knowledge 2E © 2017 by DAMA International
• Analysis
• Database Design
• Implementation
• Additional data
development

https://www.amazon.com/Infonomics-Monetize-Information-Competitive-Advantage/dp/1138090387
Data's Unique Properties
• Does not obey all of the laws of physics
– Not really visible (visualization expertise)
• Non rivalrous
– the cost of providing an additional copy is zero
• Non depleting
– Does not require replenishment
• Regenerative
• Nearly unlimited
• Low inventory and transportation/transmission costs
• More difficult to control and own
• Eco friendly
• Impossible to clean-up if you spill it
Data modeling
• The process of discovering, analyzing, and
scoping data requirements
– Understand what the data things are?
– What do they do?
– How do they interact?
• Representing/communicating requirements in
a precise form called a data model
– Maps of critical business assets
– Compose and contain metadata essential to data
consumers
– Function as a kind of sheet music language
– Metadata is essential to other business functions
(definitions for governance, lineage for analytics, etc.)
• The process is iterative and may include a
conceptual, logical, and physical model

process of discovering, analyzing, and scoping data requirements
• List organizational places
• These are called Attributes
– Attributes are characteristics of "things"
• List organizational places that need to be
persons
places
things
created
read
updated
deleted
archived
• An organization might decide to
characterize the parts of a THING as:
– Attributes: ID, description, status,
sex.to.be.assigned, reserve.reason
• Decisions to manage information
about each specific attribute has
direct consequences
– A decision to use the above data
attributes permits the organization to
determine if it has female THINGs are available to be reserved
• Characteristics can be shared
– All THINGs may have a status
– Many THINGs can be assigned to females
• Characteristics may be required to be unique
– ID permits identification every THING as distinct for every other THING
– Description is likely to be unique for each THING
THING
Thing.Id #
Thing.Description
Thing.Status
Thing.Sex.To.Be.Assigned
Thing.Reserve.Reason
Attributes arranged into an
entity named "thing" – the
attribute Thing.Id is the means
used to identify a unique
occurrence of thing

Data modeling
consumers
representing and communicating these in a precise form called a data model
Thing.Id #
Thing.Description
Thing.Status
Thing 1
Thing.Id #
…
Thing 2
Each THING 2 must be accompanied by a THING 1

representing/communicating requirements in a precise form called a data model
• Defines mandatory/optional
relationships between using
minimum/maximum
occurrences from one entity
to another
Data modeling
consumers

!

!

!

!

15
Organizational Needs
become instantiated
and integrated into a
Data Models
Informa(on)System)
Requirements
authorizes and
articulates
satisfy
specific
organizational
needs
The process is iterative
Data models and data architectures are
developed in response to needs
Data modeling
consumers

ANSI-SPARC 3-Layer Schema
1. CONCEPTUAL - Allows independent
customized user views:
– Each should be able to access the same
data, but have a different customized view
of the data.
2. LOGICAL - This hides the physical
storage details from users:
– Users should not have to deal with physical
database storage details. They should be
allowed to work with the data itself, without
concern for how it is physically stored.
3. PHYSICAL - The database
administrator should be able to
change the database storage
structures without affecting the
users’ views:
– Changes to the structure of an
organization's data will be required. The
internal structure of the database should
be unaffected by changes to the physical
aspects of the storage.
For example, a changeover to a new DBMS
technology. The database administrator
should be able to change the conceptual or
global structure of the database without
affecting the users.
Data modeling
consumers

Each data arrangement is a data structure
Some data structure characteristics:
• Grammar for data objects
– Grammar is the principles
or rules of an art, science,
or technique "a grammar
of the theater"
• Constraints for data objects
• Sequential order
• Uniqueness
• Order
– Hierarchical, relational,
network, lake, other
• Balance
• Optimality
http://www.nist.gov/dads/HTML/datastructur.html
"An organization of information, usually in memory, for better algorithm efficiency, such
as queue, stack, linked list, heap, dictionary, and tree, or conceptual unity, such as the
name and address of a person. It may include redundant information, such as length of
the list or number of nodes in a subtree."
Hierarchy
• A hierarchy is an
arrangement of items
(objects, names, values,
categories, etc.) in which
the items are represented
as being "above",
"below", or "at the same
level as" one another.
• Hierarchy is an important
concept in a wide variety
of fields, such as
philosophy, mathematics,
computer science,
organizational theory,
systems theory, and the
social sciences
(especially political
philosophy).
Data Architecture
Data Maps
/Models
Mess
Data Maps
/Models
Data Maps
/Models
Data Maps
/Models
Data Maps
/Models
Model View

Social
Differences between Programs and Projects
• Programs are Ongoing, Projects End
– Managing a program involves long term strategic planning and
continuous process improvement is not required of a project
• Programs are Tied to the Financial Calendar
– Program managers are often responsible for delivering
results tied to the organization's financial calendar
• Program Management is Governance Intensive
– Programs are governed by a senior board that provides direction,
oversight, and control while projects tend to be less governance-intensive
• Programs Have Greater Scope of Financial Management
– Projects typically have a straight-forward budget and project financial management
is focused on spending to budget while program planning, management and control
is significantly more complex
• Program Change Management is an Executive Leadership Capability
– Projects employ a formal change management process while at the program level,
change management requires executive leadership skills and program change is
driven more by an organization's strategy and is subject to market conditions and
changing business goals
Adapted from http://top.idownloadnew.com/program_vs_project/ and http://management.simplicable.com/management/new/program-management-vs-project-management
Your data program must
last at least as long as
your HR program!
22

What do we teach knowledge workers about data?
What percentage of the deal with it daily?
Political

What do we teach IT professionals about data?
• 1 course
– How to build a
new database
• What
impressions do IT
professionals get
from this
education?
– Data is a technical
skill that is needed
when developing
new databases
If the only tool you
know is a hammer
you tend to see
every problem as a
nail (slightly reworded
from Abraham Maslow)

Bad data decisions
Technical deci-
sion makers are not
data knowledgable
Business decision
makers are not
data knowledgable
Poor organizational outcomes
Poor treatment of
organizational data
assets
Poor
quality
data
Bad Data Decisions Spiral
Economic

Tacoma Narrows Bridge/Gallopin' Gertie
• World's 3rd longest suspension span
• Slender, elegant and graceful
• Opened on July 1st 1940, collapsed in a windstorm on 7 Nov 1940
• "The most dramatic failure in
bridge engineering history"
• Changed forever how engineers
design suspension bridges leading
to safer spans today.
Tacoma Narrows Bridge/Gallopin' Gertie

Similarly data failures cost organizations
minimally 20-40% of their IT budget
Data incoherence is a hidden expense
• How does maltreated data cost money?
• Consider the opposite question:
– Were your systems explicitly designed to
be integrated or otherwise work together?
– If not then what is the likelihood that they
will work well together?
• Organizations spend 20-40% of their IT
budget evolving their data - including:
– Data migration
• Changing the location from one place to another
– Data conversion
• Changing data into another form, state, or product
– Data improving
• "Inspecting and manipulating, or re-keying data to prepare it for
subsequent use" - Source: John Zachman
PETER AIKEN WITH JUANITA BILLINGS
FOREWORD BY JOHN BOTTEGA
MONETIZING
DATA MANAGEMENT
Unlocking the Value in Your Organization’s
Most Important Asset.

Doing a poor job with data
• Takes longer
• Costs more
• Delivers less
• Presents greater risk (with thanks to Tom DeMarco)
34
• From the Top
• From the Bottom
data management
• Take Aways/Q&A
Program

4 Minute Architecture Lesson from Steve Jobs, Introducing iCloud
Architecture is about ...
• Things
– (components)
• The functions of the things
– (individually)
• How the things interact
– (as a system,
– towards a goal)
• Business
• Process
• Systems
• Security
• Technical
• Data / Information

Typically Managed Architectures
• Business Architecture
– Goals, strategies, roles, organizational structure, location(s)
• Process Architecture
– Arrangement of inputs -> transformations = value -> outputs
– Typical elements: Functions, activities, workflow, events, cycles, products, procedures
• Systems Architecture
– Applications, software components, interfaces, projects
• Security Architecture
– Arrangement of security controls relation to IT Architecture
• Technical Architecture/Tarchitecture
– Relation of software capabilities/technology stack
– Structure of the technology infrastructure of an enterprise, solution or system
– Typical elements: Networks, hardware, software platforms, standards/protocols
• Data / Information Architecture
– Arrangement of data assets supporting organizational strategy
– Typical elements: specifications expressed as entities, relationships, attributes,
definitions, values, vocabularies
1 in 10 organizations manage 1
or more of the formally
Data Architectures: here, whether you like it or not
deviantart.com
• All organizations
have data
architectures
– Some are better
understood and
documented (and
therefore more
useful to the
organization) than
others
Business
Process
Systems
Security
Technical
Data/Information

Data model focus is typically domain specific
Program A
Program C
Program B
Focus of a software engineering effort
Underutilized
data modeling
effort
Database architecture focus Can vary
Application
domain 1
Program A
Program C
Program B
Focus of a software engineering effort
Underutilized
data modeling
effort
Better utilized
data modeling
effort
ERPs and COTS are marketed as
being similarly integrated!
Program F
Program E
Program G
Program H
Program I
Application
domain 2
Application
domain 3
Program D

Application
domain 1
Program A
Program C
Program B
D
a
t
a
D
a
t
a
D
a
t
a
D
a
t
a
D
a
t
a
D
a
t
a
D
a
t
a
Program F
Program E
Program D
Program G
Program H
Program I
Application
domain 2
Application
domain 3
D
a
t
a
D
a
t
a
D
a
t
a
Data Architecture Focus has Greater Potential Value
• Broader focus than
either software
architecture or
database
architecture
• Analysis scope is
on the system
wide use of data
• Problems caused
by data exchange
or interface
problems
• Architectural goals
more strategic
than operational
Metadata
Management
42
Data
Management
Body
of
Knowledge
(DM
BoK
V2)
Practice
Areas
from The DAMA Guide to the Data Management Body of Knowledge 2E © 2017 by DAMA International
• Enterprise data
modeling
• Value chain analysis
• Related data
architecture

How are components expressed as architectures?
• Details are
organized into
larger components
• Larger components
are organized into
models
• Models are
organized into
architectures
(composed of
architectural
components)
A B
C D
A B
C D
A
D
C
B
Intricate
Dependencies
Purposefulness
How are data structures expressed as architectures?
• Attributes are organized into entities/objects
– Entitles/objects are "things" whose
information is managed in support of strategy
– Example(s)
• Entities/objects are organized into models
– Combinations of attributes and entities are
structured to represent information requirements
– Poorly structured data, constrains organizational
information delivery capabilities
– Example(s)
• Models are organized into architectures
– When building new systems, architectures are used to plan development
– More often, data managers do not know what existing architectures are and -
therefore - cannot make use of them in support of strategy implementation
• Why no examples?
Intricate
Dependencies
Purposefulness
THING
Thing.Id #
Thing.Description
Thing.Status

Data architectures are composed of data models
Data
Data
Data
Information
Fact Meaning
Request
A model defining 3 important concepts composing a data architecture
[Built on definitions from Dan Appleton 1983]
Intelligence
Strategic Use
Data
Data
Data Data
46
“You can have data without information, but
you cannot have information without data”
— Daniel Keys Moran, Science Fiction Writer
1. Each FACT combines with one or more MEANINGS.
2. Each specific FACT and MEANING combination is referred to as a DATUM.
3. An INFORMATION is one or more DATA that are returned in response to a specific REQUEST
4. INFORMATION REUSE is enabled when one FACT is combined with more than one MEANING.
5. INTELLIGENCE is INFORMATION associated with its STRATEGIC USES.
6. DATA/INFORMATION must formally arranged into an ARCHITECTURE.
Wisdom & knowledge are
often used synonymously
Useful Data

Data Architectures Determine
Interoperability
• Required to enable
self-correction/generation
capabilities
• Permits governance of data as an
asset
• Prerequisite to meaningful data
exchanges
• Lowers costs of organization-wide
and extra-organizational data
sharing
• Permits managed evolution - rapidly
responding to changing needs, new
partners, time criticality's
• Required for (full) role-based
security implementation
• Decreases the cost of maintaining
data inventories
Data Architectures
• Capture the business meaning of the
data required to run the organization
• Living document – constantly
evolving to meet upcoming and
discovered business requirements
• A potential entry point for
architecture engagements
• Validated data architectural
components can be used to
populate a business glossary
• Major collection of metadata
Levels of Abstraction, Completeness and Utility
• Models more downward facing - detail
• Architecture is higher level of abstraction - integration
• In the past architecture attempted to gain complete (perfect)
understanding
– Not timely
– Not feasible
• Focus instead on
architectural components
– Governed by a framework
– More immediate utility
• http://www.architecturalcomponentsinc.com

Data structures organized into an Architecture
• How do data structures support strategy?
• Consider the opposite question?
– Were your systems explicitly designed to be
integrated or otherwise work together?
– If not then what is the likelihood that they will
work well together?
– In all likelihood your organization is spending
between 20-40% of its IT budget compensating
for poor data structure integration
– They cannot be helpful as long as their
structure is unknown
• Two answers/two separate strategies
– Achieving efficiency and
effectiveness goals
– Providing organizational dexterity
for rapid implementation
Engineering
Architecture
50
Engineering/Architecting
Relationship
• Architecting is used to
create and build systems too
complex to be treated by
engineering analysis alone
– Require technical details as the
exception
• Engineers develop the
technical designs for
implementation
– Engineering/Crafts-persons
deliver work product
components supervised by:
• Manufacturer
• Building Contractor

You cannot architect after implementation!
USS Midway
& Pancakes
What is this?
• It is tall
• It has a clutch
• It was built in 1942
• It is cemented to the floor
• It is still in regular use!

Definition of Bed
• An organization might decide to
characterize the parts of a BED as:
– Attributes: ID, description, status,
sex.to.be.assigned, reserve.reason
• Decisions to manage information
about each specific attribute has
direct consequences
– A decision to use the above data
attributes permits the organization to
determine if it has female beds are available to be reserved
• Characteristics can be shared
– All beds may have a status
– Many beds can be assigned to females
• Characteristics may be required to be unique
– ID permits identification every bed as distinct for every other bed
– Description is unlikely to be the same for each bed
BED
Bed.Id #
Bed.Description
Bed.Status
Bed.Sex.To.Be.Assigned
Bed.Reserve.Reason
Attributes arranged into an
entity named "bed" – the
attribute Bed.Id is the means
used to identify a unique
occurrence of bed

Q: What is the proper relationship for these entities?
Bed Room
Bed Room
Data Maps at the Entity Level ➜ Stored Facts
Bed Room
a BED is related to a ROOM
More precision:
many BEDS are related to many ROOMS
Bed Room
Better information:
many BEDS may be contained in each ROOM and each room may contain many beds
What if beds can
be moved?

Eventually One or Many (optional)
Eventually One (optional)
Zero, or Many (optional)
One or Many (mandatory)
Exactly One (mandatory)
Possible Entity Relationship Cardinality Options
Families of Modeling Notation Variants
Information Engineering

What is a Relationship?
• Natural associations between two or more entities
Ordinality & Cardinality
• Defines mandatory/optional relationships using minimum/
maximum occurrences from one entity to another
A BED is placed
in one and only
one ROOM A ROOM
contains zero
or more BEDS
A BED is occupied by zero or more
PATIENTS
A PATIENT
occupies at least
one or more BEDS
ROOM
BED
PATIENT

Standard definition reporting does not provide conceptual context
BED
Something you sleep in
Purpose statement incorporates motivations
Entity: BED
Data Asset Type: Principal Data Entity
Purpose: This is a substructure within the Room
substructure of the Facility Location. It contains
information about beds within rooms.
Source: Maintenance Manual for File and Table
Data (Software Version 3.0, Release 3.1)
Attributes: Bed.Description
Bed.Status
Bed.Reserve.Reason
Associations: >0-+ Room
Status: Validated
Draft
A purpose statement describing
– Why the organization is maintaining information about this business concept;
– Sources of information about it;
– A partial list of the attributes or characteristics of the entity; and
– Associations with other data items(read as "One room contains zero or many beds.")

63
• From the Top
• From the Bottom
data management
• Take Aways/Q&A
Program
BUILD?
WHAT? HOW?
As Is Requirements
Assets WHAT?
As Is Design Assets
HOW?
As Is Implementation
Assets AS BUILT
Forward Engineering
New
Building new stuff - in this case, new databases

Systems Development Life Cycle (SDLC)
WHAT?
HOW?
BUILD?
66
• From the Top
• From the Bottom
data management
• Take Aways/Q&A
Program

Data Representation is the Essence of Programming
• Mythical Man Month ➜ 9 parallel effort x 1 month each ≠ baby
• Fred Brooks Jr.'s observation
– Data representation is the essence of programming
– "Show me your flowchart and
conceal your tables, and
I shall continue to be mystified.
– Show me your tables, and
I won't usually need your flowchart;
it'll be obvious."
As Is Requirements
Assets WHAT?
As Is Design Assets
HOW?
Assets AS BUILT
Existing
Reverse Engineering
A structured technique aimed at recovering rigorous knowledge
of the existing system to leverage enhancement efforts
[Chikofsky & Cross 1990]

69
• From the Top
• From the Bottom
data management
• Take Aways/Q&A
Program
As Is Requirements
Assets WHAT?
As Is Design Assets
HOW?
Assets AS BUILT
Existing
New
Reengineering
Reverse Engineering
Forward engineering
Reimplement
To Be
Implementation
Assets
To Be
Design
Assets
To Be Requirements
Assets
• First, reverse engineering the existing system
to understand its strengths/weaknesses
• Next, use this information to inform the design
of the new system

Data Modeling Process
1. Identify entities
2. Identify key for each
entity
3. Draw rough draft of
entity relationship
data model
4. Identify data
attributes
5. Map data attributes
to entities
Model evolution is good, at first ...
1. Identify entities
2. Identify key for each
entity
3. Draw rough draft of
entity relationship
data model
4. Identify data
attributes
5. Map data attributes
to entities

Relative use of time allocated to tasks during modeling
Preliminary Modeling Wrapup
Activity activities Cycles activities
Evidence Analysis
collection &
analysis Collection
Project
coordination
requirements Declining coordination requirements
Target
system
analysis Increasing amounts of target system analysis
Modeling Validation
cycle
focus Refinement
Source:http://dmreview.com/article_sub.cfm?articleID=1000941 used with permission
Meta Data Models

75
• From the Top
• From the Bottom
data management
• Take Aways/Q&A
Program
Take Aways
• Attributes are organized into entities/objects
– Entitles/objects are "things" whose
information is managed in support of strategy
– Example(s)
• Entities/objects are organized into models
– Combinations of attributes and entities are
structured to represent information requirements
– Poorly structured data, constrains organizational
information delivery capabilities
– Example(s)
• Models are organized into architectures
– When building new systems, architectures are used to plan development
– More often, data managers do not know what existing architectures are and -
therefore - cannot make use of them in support of strategy implementation
– Why no examples?
Entity: BED
Attributes: Bed.Description
Bed.Status
Bed.Reserve.Reason
Intricate
Dependencies
Purposefulness

Learning Objectives
• Both data architecture and data models are
made more useful by each other
– Power derived from their interdependence
– Data models are a primary means to achieve shared
understanding of data challenge specifics
– They are literally the pages that intersect data assets
and the organizational response
• Data architecture is the sum of the data
models
– Coverage is rarely complete
• Data models
– Documentation
– The currency of data coordination
– Used to verify integration
– Mandated input to any data systems evolution
• What is taught
– Forward engineering with a goal of building
• What is also needed
– Reverse engineering with a goal of understanding
Upcoming Events (Webinars begin @ 19:00 UTC/2:00 PM NYC)
Brought to you by:
Why Data Modeling
Is Fundamental
8 June 2021
Business Value through
Reference & Master Data Strategies
13 July 2021
Getting (Re)Started with Data Stewardship
10 August 2021

paiken@plusanythingawesome.com +1.804.382.5957
Questions?
Thank You!
Book a call with Peter to discuss anything - https://plusanythingawesome.com/OfficeHours.html
+ =

DataEd Online: Data Architecture and Data Modeling Differences — Achieving a Common Understanding

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to DataEd Online: Data Architecture and Data Modeling Differences — Achieving a Common Understanding

Similar to DataEd Online: Data Architecture and Data Modeling Differences — Achieving a Common Understanding (20)

More from DATAVERSITY

More from DATAVERSITY (20)

Recently uploaded

Recently uploaded (20)

DataEd Online: Data Architecture and Data Modeling Differences — Achieving a Common Understanding