The document discusses considerations for building an enterprise data lake. It notes that traditional data warehousing approaches do not scale well for new data sources like sensors and streaming data. It advocates adopting a data lake approach with separate systems for data acquisition, management, and access instead of a monolithic architecture. A data lake requires a distributed architecture and platform services to support various data flows, formats, and processing needs. The data architecture should not enforce models or limitations upfront but rather allow for evolution and change over time.

1. Building the
Enterprise Data Lake
Considerations before you
jump in
December, 2015
Mark Madsen
www.ThirdNature.net
@markmadsen1

2. What This Session Isn’t
SQL..
.
SQL!
SQL?
SQL

3. The craft model of information delivery does not scale

4. © Third Nature, Inc.
So we shifted to data publishing
Industrialized data delivery for self-service access.

5. Events and sensors are a relatively new data source
Sensor data doesn’t fit well with current methods of modeling,
collection and storage, or with the technology to process and analyze it.

6. There’s lots of other new data involved

7. © Third Nature, Inc.
You can store this data in an RDBMS, but…

8. These sorts of things slow user requests down
Conclusion: any methodology built on the premise that you
must know and model all the data first is untenable

9. © Third Nature, Inc.
Analytics embiggens data volume problems
Many of the processing problems are O(n2) or worse, so
moderate data can be a problem for scale-up platforms

10. © Third Nature, Inc.
Old market says: There’s nothing wrong with what
you have, just keep buying new products from us

11. The emerging big data market has an answer…

12. © Third Nature, Inc.
The data lake

13. © Third Nature, Inc.
Views of the lake
Is the business vs supports the business?
Application vs infrastructure?

14. © Third Nature, Inc.
The naïve idea of a data lake leads to predictable results

15. © Third Nature, Inc.
You can’t install Hadoop and hope it solves all the problems
Big data no 2

16. Slide 16
The answer isn’t just technology, it’s architecture

17. Schema
In the DW world both data and processing are bounded
No consideration for feedback loops and change
Processing only
happens here
Carefully
controlled
access
here
Nobodyherecreates
newinformation
Sources few and
well understood
Complex DI
is controlled
by IT
Schemas are few
and designed
Tools are authorized,
few in number and
kind
One way flow
This is a monolithic, layered architecture

18. © Third Nature, Inc.
In the big data world flow is unbounded and continuous
Feedback
loops allowed
End-of-analysis
dataset may be
start of a BI dataset
Continuous data
integration and delivery
Files are back as both
input and storage
Minimal
barrier of /
control on
collection
Areas of
provisioned
data
Any shape in,
rectangles out
This needs a distributed service architecture

19. © Third Nature, Inc.
Deconstructing data environments
There are three
things happening in
a data warehouse:
▪ Data acquisition
▪ Data management
▪ Data delivery
Isolate them from one
another, allow read-
write use, and you are
on the path.
Data
Warehouse

20. Data lake subsystems / components
The acquisition component allows any data to be collected at any latency. The
management component allows some data to be standardized and integrated. The
access component provides access at any latency and via any means an application
chooses. Processing can be done to any data at any time from any area.
Data Acquisition
Collect & Store
Incremental
Batch
One-time copy
Real time
Data Lake Platform Services
Data Management
Process & Integrate
Data Access
Deliver & Use
Data storage
In reality, you are building three systems, not one. Avoid the monolith.

21. © Third Nature, Inc.
Data lake functions depend on platform services
Base Platform Services
Data Movement MetadataData Persistence
Workflow
Management
Processing Engines Dataflow Services
Data Curation
Data Access
Services
Data Acquisition
Collect & Store
Data Management
Process & Integrate
Data Access
Deliver & Use
Platformservicesneeded

22. DATA ARCHITECTURE
We’re so focused on the light switch that we’re not
talking about the light

23. © Third Nature, Inc.
Decouple the Data Architecture
The core of the data lake isn’t a database or HDFS,
it’s the data architecture that the tools implement.
We need a data architecture that is not limiting:
▪ Deals with change easily and at scale
▪ Does not enforce requirements and models up front
▪ Does not limit the format or structure of data
▪ Assumes the range of data latencies in and out, from
streaming to one-time bulk

24. © Third Nature, Inc.
Food supply chain: an analogy for data
Multiple contexts of use, differing quality levels
You need to keep the original because just like baking,
you can’t unmake dough once it’s mixed.

25. © Third Nature, Inc.
Data architecture is required by the services, and vice versa
Raw data in an immutable
storage area
Standardized or
enhanced data
Common or
usage-
specific data
Transient data
DataAcquisition
Collect&Store
Platform Services
DataAccess
Deliver&Use
Data Management
Process & Integrate

26. © Third Nature, Inc.
The data areas map (mostly) to functional areas of the lake
Collection can’t be limited by database scale and latency.
Immutability, persistence and concurrency are required.
Incremental
Collect
Batch
One-time copy
Real time
Manage & Integrate Process, Deliver, Use

27. © Third Nature, Inc.
Stages, not layers
Some tools require specific repositories or models.
Others can reach in to get what they need. Do not
enforce a single access point or model.

28. © Third Nature, Inc.
The geography has been redefined
The box IT created:
• not any data, rigidly typed data
• not any form, tabular rows and
columns of typed data
• not any latency, persist what the
DB can keep up with
• not any process, only queries
The digital world was diminished
to only what’s inside the box until
we forgot the box was there.

29. © Third Nature, Inc.
Layered data architecture
The DW assumed a single flat
model of data, DB in the center.
The data lake enables new ways
to organize data:
▪ Raw – straight from the source
▪ Enhanced –cleaned, standardized
▪ Integrated – modeled,
augmented, ~semi-persistent
▪ Derived – analytic output,
pattern based sets, ephemeral
Implies a new technology architecture
and data modeling approaches.

30. © Third Nature, Inc.
The data lake enables evolutionary design for data
Evolutionary design is required because data needs change. You
need a system not for stability – we have that in the DW - but for
evolution and change, the data lake.
Data Acquisition
Collect & Store
Incremental
Batch
One-time copy
Real time
Data Lake Platform Services
Data Management
Process & Integrate
Data Access
Deliver & Use
Data storage
You can’t build this all at once. You need to grow it over time.

31. © Third Nature, Inc.
Away from “one throat to choke”, back to best of breed
Tight coupling leads to efficient
reuse and standardization, and
to slow changes.
In a rapidly evolving market
componentized architectures,
modularity and loose coupling
are favorable over monolithic
stacks, single-vendor
architectures and tight
coupling.
Architecture, not blueprints:
there is no single answer. It
depends on your goals and
starting position.

32. Questions?“When a new technology rolls over you, you're either part of
the steamroller or part of the road.” – Stewart Brand

33. © Third Nature, Inc.
CC Image Attributions
Thanks to the people who supplied the creative commons licensed images used in this presentation:
donuts_4_views.jpg - http://www.flickr.com/photos/le_hibou/76718773/
glass_buildings.jpg - http://www.flickr.com/photos/erikvanhannen/547701721

34. © Third Nature, Inc.
About the Presenter
Mark Madsen is president of Third Nature, a
consulting and advisory firm focused on
analytics, business intelligence and data
management. Mark is an award-winning
author, architect and CTO. Over the past ten
years Mark received awards for his work
from the American Productivity & Quality
Center, TDWI, and the Smithsonian Institute.
He is an international speaker, a contributor
to Forbes, member of the O’Reilly Strata
program committee. For more information
or to contact Mark, follow @markmadsen on
Twitter or visit http://ThirdNature.net

35. About Third Nature
Third Nature is a consulting and advisory firm focused on new and emerging technology
and practices in information strategy, analytics, business intelligence and data
management. If your question is related to data, analytics, information strategy and
technology infrastructure then you‘re at the right place.
Our goal is to help organizations solve problems using data. We offer education, consulting
and research services to support business and IT organizations as well as technology
vendors.
We fill the gap between what the industry analyst firms cover and what IT needs. We
specialize in strategy and architecture, so we look at emerging technologies and markets,
evaluating how technologies are applied to solve problems rather than evaluating product
features.