This document discusses considerations for building an enterprise data lake. It begins by introducing the presenters and stating that the session will not focus on SQL. It then discusses how the traditional "crab" model of data delivery does not scale and how organizations have shifted to industrialized data publishing. The rest of the document discusses important aspects of data lake architecture, including how different types of data like sensor data require new approaches. It emphasizes that the data lake requires a distributed service architecture rather than a monolithic structure. It also stresses that the data lake consists of three core subsystems for acquisition, management, and access, and that these depend on underlying platform services.

1. Building The
Enterprise Data Lake
Important Considerations Before
You Jump In
December 8, 2015

2. Building The Enterprise Data Lake
Today’s Presenters
Mark
Madsen
Industry
Analyst
Third Nature
@markmadsen
Craig
Stewart
Sr. Dir.
Product
Management
SnapLogic
@01Badger
Erin
Curtis
Sr. Dir.
Product
Marketing
SnapLogic
@erncrts

3. Building
the
Enterprise
Data
Lake
Considera6ons
before
you
jump
in
December,
2015
Mark
Madsen
www.ThirdNature.net
@markmadsen1

4. What
This
Session
Isn’t
SQL..
.
SQL!
SQL?
SQL

5. The
craB
model
of
informa6on
delivery
does
not
scale

6. ©
Third
Nature,
Inc.
So
we
shiBed
to
data
publishing
Industrialized
data
delivery
for
self-­‐service
access.

7. Events
and
sensors
are
a
rela6vely
new
data
source
Sensor
data
doesn’t
fit
well
with
current
methods
of
modeling,
collecEon
and
storage,
or
with
the
technology
to
process
and
analyze
it.

8. There’s
lots
of
other
new
data
involved

9. ©
Third
Nature,
Inc.
You
can
store
this
data
in
an
RDBMS,
but…

10. These
sorts
of
things
slow
user
requests
down
Conclusion:
any
methodology
built
on
the
premise
that
you
must
know
and
model
all
the
data
first
is
untenable

11. ©
Third
Nature,
Inc.
Analy6cs
embiggens
data
volume
problems
Many
of
the
processing
problems
are
O(n2)
or
worse,
so
moderate
data
can
be
a
problem
for
scale-­‐up
plaOorms

12. ©
Third
Nature,
Inc.
Old
market
says:
There’s
nothing
wrong
with
what
you
have,
just
keep
buying
new
products
from
us

13. The
emerging
big
data
market
has
an
answer…

14. ©
Third
Nature,
Inc.
The
data
lake

15. ©
Third
Nature,
Inc.
Views
of
the
lake
Is
the
business
vs
supports
the
business?
ApplicaEon
vs
infrastructure?

16. ©
Third
Nature,
Inc.
The
naïve
idea
of
a
data
lake
leads
to
predictable
results

17. ©
Third
Nature,
Inc.
You
can’t
install
Hadoop
and
hope
it
solves
all
the
problems
Big
data
no
2

18. Slide 18
The
answer
isn’t
just
technology,
it’s
architecture

19. Schema
In
the
DW
world
both
data
and
processing
are
bounded
No consideration for feedback loops and change
Processing only
happens here
Carefully
controlled
access
here
Nobodyherecreates
newinformation
Sources few and
well understood
Complex DI
is controlled
by IT
Schemas are few
and designed
Tools are authorized,
few in number and
kind
One way flow
This
is
a
monolithic,
layered
architecture

20. ©
Third
Nature,
Inc.
In
the
big
data
world
flow
is
unbounded
and
con6nuous
Feedback
loops allowed
End-of-analysis
dataset may be
start of a BI dataset
Continuous data
integration and delivery
Files are back as both
input and storage
Minimal
barrier of /
control on
collection
Areas of
provisioned
data
Any shape in,
rectangles out
This
needs
a
distributed
service
architecture

21. ©
Third
Nature,
Inc.
Deconstruc6ng
data
environments
There
are
three
things
happening
in
a
data
warehouse:
▪ Data
acquisiEon
▪ Data
management
▪ Data
delivery
Isolate
them
from
one
another,
allow
read-­‐
write
use,
and
you
are
on
the
path.
Data
Warehouse

22. Data
lake
subsystems
/
components
The
acquisi6on
component
allows
any
data
to
be
collected
at
any
latency.
The
management
component
allows
some
data
to
be
standardized
and
integrated.
The
access
component
provides
access
at
any
latency
and
via
any
means
an
applica6on
chooses.
Processing
can
be
done
to
any
data
at
any
6me
from
any
area.
Data
AcquisiEon
Collect
&
Store
Incremental
Batch
One-­‐Eme
copy
Real
Eme
Data
Lake
PlaOorm
Services
Data
Management
Process
&
Integrate
Data
Access
Deliver
&
Use
Data
storage
In
reality,
you
are
building
three
systems,
not
one.
Avoid
the
monolith.

23. ©
Third
Nature,
Inc.
Data
lake
func6ons
depend
on
plaUorm
services
Base Platform Services
Data Movement MetadataData Persistence
Workflow
Management
Processing Engines Dataflow Services
Data Curation
Data Access
Services
Data
AcquisiEon
Collect
&
Store
Data
Management
Process
&
Integrate
Data
Access
Deliver
&
Use
PlaOorm
services
needed

24. DATA
ARCHITECTURE
We’re
so
focused
on
the
light
switch
that
we’re
not
talking
about
the
light

25. ©
Third
Nature,
Inc.
Decouple
the
Data
Architecture
The
core
of
the
data
lake
isn’t
a
database
or
HDFS,
it’s
the
data
architecture
that
the
tools
implement.
We
need
a
data
architecture
that
is
not
limiEng:
▪ Deals
with
change
easily
and
at
scale
▪ Does
not
enforce
requirements
and
models
up
front
▪ Does
not
limit
the
format
or
structure
of
data
▪ Assumes
the
range
of
data
latencies
in
and
out,
from
streaming
to
one-­‐Eme
bulk

26. ©
Third
Nature,
Inc.
Food
supply
chain:
an
analogy
for
data
MulEple
contexts
of
use,
differing
quality
levels
You
need
to
keep
the
original
because
just
like
baking,
you
can’t
unmake
dough
once
it’s
mixed.

27. ©
Third
Nature,
Inc.
Data
architecture
is
required
by
the
services,
and
vice
versa
Raw data in an immutable
storage area
Standardized or
enhanced data
Common or
usage-
specific data
Transient data
Data
AcquisiEon
Collect
&
Store
PlaOorm
Services
Data
Access
Deliver
&
Use
Data
Management
Process
&
Integrate

28. ©
Third
Nature,
Inc.
The
data
areas
map
(mostly)
to
func6onal
areas
of
the
lake
CollecEon
can’t
be
limited
by
database
scale
and
latency.
Immutability,
persistence
and
concurrency
are
required.
Incremental
Collect
Batch
One-­‐Eme
copy
Real
Eme
Manage
&
Integrate
Process,
Deliver,
Use

29. ©
Third
Nature,
Inc.
Stages,
not
layers
Some
tools
require
specific
repositories
or
models.
Others
can
reach
in
to
get
what
they
need.
Do
not
enforce
a
single
access
point
or
model.

30. ©
Third
Nature,
Inc.
The
geography
has
been
redefined
The
box
IT
created:
• not
any
data,
rigidly
typed
data
• not
any
form,
tabular
rows
and
columns
of
typed
data
• not
any
latency,
persist
what
the
DB
can
keep
up
with
• not
any
process,
only
queries
The
digital
world
was
diminished
to
only
what’s
inside
the
box
un6l
we
forgot
the
box
was
there.

31. ©
Third
Nature,
Inc.
Layered
data
architecture
The
DW
assumed
a
single
flat
model
of
data,
DB
in
the
center.
The
data
lake
enables
new
ways
to
organize
data:
▪ Raw
–
straight
from
the
source
▪ Enhanced
–cleaned,
standardized
▪ Integrated
–
modeled,
augmented,
~semi-­‐persistent
▪ Derived
–
analyEc
output,
pacern
based
sets,
ephemeral
Implies
a
new
technology
architecture
and
data
modeling
approaches.

32. ©
Third
Nature,
Inc.
The
data
lake
enables
evolu6onary
design
for
data
EvoluEonary
design
is
required
because
data
needs
change.
You
need
a
system
not
for
stability
–
we
have
that
in
the
DW
-­‐
but
for
evoluEon
and
change,
the
data
lake.
Data
AcquisiEon
Collect
&
Store
Incremental
Batch
One-­‐Eme
copy
Real
Eme
Data
Lake
PlaOorm
Services
Data
Management
Process
&
Integrate
Data
Access
Deliver
&
Use
Data
storage
You
can’t
build
this
all
at
once.
You
need
to
grow
it
over
6me.

33. ©
Third
Nature,
Inc.
Away
from
“one
throat
to
choke”,
back
to
best
of
breed
Tight
coupling
leads
to
efficient
reuse
and
standardizaEon,
and
to
slow
changes.
In
a
rapidly
evolving
market
componenEzed
architectures,
modularity
and
loose
coupling
are
favorable
over
monolithic
stacks,
single-­‐vendor
architectures
and
Eght
coupling.
Architecture,
not
blueprints:
there
is
no
single
answer.
It
depends
on
your
goals
and
starEng
posiEon.

34. Ques6ons?
“When
a
new
technology
rolls
over
you,
you're
either
part
of
the
steamroller
or
part
of
the
road.”
–
Stewart
Brand

35. ©
Third
Nature,
Inc.
CC
Image
Abribu6ons
Thanks
to
the
people
who
supplied
the
creaEve
commons
licensed
images
used
in
this
presentaEon:
donuts_4_views.jpg
-­‐
hcp://www.flickr.com/photos/le_hibou/76718773/
glass_buildings.jpg
-­‐
hcp://www.flickr.com/photos/erikvanhannen/547701721

36. ©
Third
Nature,
Inc.
About
the
Presenter
Mark
Madsen
is
president
of
Third
Nature,
a
consulEng
and
advisory
firm
focused
on
analyEcs,
business
intelligence
and
data
management.
Mark
is
an
award-­‐winning
author,
architect
and
CTO.
Over
the
past
ten
years
Mark
received
awards
for
his
work
from
the
American
ProducEvity
&
Quality
Center,
TDWI,
and
the
Smithsonian
InsEtute.
He
is
an
internaEonal
speaker,
a
contributor
to
Forbes,
member
of
the
O’Reilly
Strata
program
commicee.
For
more
informaEon
or
to
contact
Mark,
follow
@markmadsen
on
Twicer
or
visit
hcp://ThirdNature.net

37. About
Third
Nature
Third
Nature
is
a
consulEng
and
advisory
firm
focused
on
new
and
emerging
technology
and
pracEces
in
informaEon
strategy,
analyEcs,
business
intelligence
and
data
management.
If
your
quesEon
is
related
to
data,
analyEcs,
informaEon
strategy
and
technology
infrastructure
then
you‘re
at
the
right
place.
Our
goal
is
to
help
organizaEons
solve
problems
using
data.
We
offer
educaEon,
consulEng
and
research
services
to
support
business
and
IT
organizaEons
as
well
as
technology
vendors.
We
fill
the
gap
between
what
the
industry
analyst
firms
cover
and
what
IT
needs.
We
specialize
in
strategy
and
architecture,
so
we
look
at
emerging
technologies
and
markets,
evaluaEng
how
technologies
are
applied
to
solve
problems
rather
than
evaluaEng
product
features.

38. About SnapLogic

39. Anything
apps | APIs | things | data
Anytime
batch | streaming | real-time
Anywhere
on premises | in the cloud
SnapLogic helps enterprises
connect data and
applications faster

40. Modern Architecture: Hybrid and Elastic
Streams: No data is
stored/cached
Secure: 100%
standards-based
Elastic: Scales out &
handles data and app
integration use cases
Metadata
Data
Databases
On Prem
Apps
Big Data
Cloud Apps
and DataCloud-Based Designer, Manager,
Dashboard
Cloudplex
Groundplex
Hadooplex
Sparkplex
Firewall

41. z
Data
Acquisition
On Prem Apps
and Data
Data
Access
z
Data
Management
Data Lake
Add information
and improve data
Spark
Python
Scala
Java
R
Pig
Collect and
integrate data
from multiple
sources
HDFS
AWS S3
MS Azure Blob
• ERP
• CRM
• RDBMS
Cloud Apps
and Data
• CRM
• HCM
• Social
IoT Data
• Sensors
• Wearables
• Devices
Lakeshore
Data Mart
• MS Azure
• AWS
Redshift
• …
BI / Analytics
• Tableau
• MS
PowerBI /
Azure
• AWS
QuickSight
Organize and
prepare data for
visualization
HDFS
AWS S3
MS Azure Blob
Hive
Batch
Streaming
Schedule and manage:
Oozie, Ambari
Kafka, Sqoop,
Flume
Real-time
Ingest Prepare Deliver
Impala, HiveSQL,
SparkSQL

42. z
Data
Acquisition
On Prem Apps
and Data
Data
Access
z
Data
Management
The Modern Data Lake
Powered by SnapLogic
• ERP
• CRM
• RDBMS
Cloud Apps
and Data
• CRM
• HCM
• Social
IoT Data
• Sensors
• Wearables
• Devices
Lakeshore
Data Mart
• MS Azure
• AWS
Redshift
• …
BI / Analytics
• Tableau
• MS
PowerBI /
Azure
• AWS
QuickSight
Batch
Streaming
Schedule and manage: SnapLogicSnapLogic Pipelines
Real-time
Ingest Prepare Deliver
SnapLogic Pipelines
Sort,
Aggregate,
Join, Merge,
Transform
SnapLogic
abstracts and
operationalizes
with
SnapReduce or
Spark pipelines
Collect and
integrate data
from multiple
sources
SnapLogic
pipelines with
standard mode
execution
Organize and
prepare data for
visualization
SnapLogic
pipelines with
standard mode
execution

43. Thank You
Watch SnapLogic in action:"
video/snaplogic.com
Contact us:
info@snaplogic.com
Follow us on Twitter:
@SnapLogic