Discover & identify ideal storage solution for our needs by examining the history of data storage & the modern database systems including Key Value, Relational, Graph and Document databases. This presentation was given at RootsTech 2013 in March

1. Modern Database
Systems

2. @spf13
AKA
Steve Francia
Chief Evangelist @
responsible for drivers,
integrations, web & docs

3. What’s the Point?
๏ Goal: Discover & identify ideal
storage solution for our needs
๏ History is important
๏ Many options today
๏ Document databases are good
for Genealogy

4. History of the
World

5. Over 5500 years ago
2 People

6. 1804
1 Billion People

7. 1927
2 Billion People

8. World Population Growth

9. World Population Growth
(last ~200 years in Billions)
8
6
4
7
6
5
4 2
3
2
1
1804 1927 0
1960 1974 1987 1999 2012

10. Really Big Data
In the last 50 years...
over 4 % of the world people
were born...
in less than 1 % of the time

11. History of
Databases

12. 1970
๏ Oracle
creates the relational
database
๏ Everyone happily uses it for
the next 43 years

13. What really
happened

14. Let’s start at
the beginning

15. It’s a story about...
Storing & Retrieving
Information

20. Even today we still use
the same mediums for
data storage

23. With the advent of
the computer things
really took off

24. 1960 : DBMS Emerges
๏ Ordered set of fixed length fields
๏ Low level pointer operations (flat
files)
๏ Most popular was IMS (created at
IBM)
๏ Shockingly still in use today at IBM &
American Airlines

25. Lots of Problems
๏ Complex and inflexible
๏ User had to know physical structure of the
DB in order to query for information
๏ Adding a field to the DB required rewriting
the underlying access/modification scheme
๏ Records isolated (no relations)
๏ Emphasis on records to be processed, not
overall structure

26. 1970 : Relational DB
๏ Edgar Frank “Ted” Codd
๏ Relational Database
theory
๏ Codd’s 13 rules
(aka 12 rules)

27. 3 HUGE Advantages
๏ Data independence from hardware
and storage implementation
๏ Ability to process more than one
record at a time with a single
operation
๏ Establishing a relationship
between records

28. IBM vs Codd
๏ IBM bet on IMS
๏ Codd bets on relational DB
๏ Eventually
2 relational
prototypes emerge

29. Ingres
๏ Built at UC Berkley
๏ Uses QUEL
๏ Inspires Sybase & MSSQL

30. System R
๏ Built at IBM
๏ Leads to SEQUEL... later SQL
๏ Evolved into SQL/DS which
evolved into DB2
๏ Project concludes that relational
model is viable

31. Oracle
๏ Larry Ellison watches IBM
๏ Starts Relational Software Inc.
๏ Oracle 1st commercial RDBMS
released in 1979
๏ Beats IBM by 2 years to market

32. Entity Relationship
๏ Proposed by Peter
Chen in 1976
๏ Focuses on data use
and not logical table
structure

33. 1980s
๏ RDBMS dominates
๏ Some fields (medicine,
physics, multimedia) need
more than RDBMS offers
๏ Object Databases emerge

34. Object Databases
๏ Inspired by Entity Relationship
๏ More flexible than relational permits
๏ Tightly coupled with OO
programming language (c++, later
Java)
๏ Full object: data & methods stored

35. 1990s
๏ Internet emerges
๏ Data demand spikes
๏ Databases used for
archiving historical data

36. Early 2000s
๏ Internet booms
๏ RDBMS fails to scale
๏ Indesperation we take a
step backwards

37. MemcacheD
๏1 dimensional
๏ No persistence
๏ No ACI or D
๏ but...

38. ... FAST

39. 2005 ish
๏ Relational + MemcacheD
broken (and we didn’t know it)
๏ Scale redefined with high
volume & social
๏ Infrastructure reinvented with
cloud computing & SSDs

40. Alternatives Emerge
๏ Dynamo / Key Value
๏ Document
๏ Graph

41. Modern Data
Storage

42. A lot going on
Easiest to define databases in
broad terms
• What is a record?
(data model)
• CAP : CA, AP, CP ?
(infrastructure model)

43. Data Storage Structure
1D 2D nD
Key Key Value Key Value(s)
Key Value Key Value(s)
Value Key Value Key
Key Value Key Value
Key Value(s)
Key
Key Value
Key Value(s)

44. Database structure
1D 2D nD
Key Value
Relational Document
Dynamo
Graph

45. CAP Theorem
Availability
Partitioning Consistency

46. CAP Theorem
xx
Node Node
App

47. CAP Theorem
Availability
Dynamo
RDBMS
t
Key Value
ten
Int
o
sis
ler
NoSQLs
on
ant
Inc
Unavailable
Partition Consistency
Tolerant MongoDB
BigTable

48. Key Value
๏ ๏ Often
1 Dimensional
storage (tupal) MultiMaster...
๏
meaning
Query key only availability over
๏ Bucket index consistency
(range) on keys ๏ Partitioning easy
๏ Records cannot be thanks to single
updated, only value
replaced
Cassandra, Redis, MemcacheD, Riak, DynamoDB

49. Relational
๏ Single master
๏ 2 Dimensional
storage (map) meaning
consistency >
๏ Query any availability
field ๏ Partitioning hard
๏ due to
BTree Indexes transactions &
joins
Oracle, MSSQL, MySQL, PostgreSQL, DB2

50. Document
๏ ๏ Single master
n Dimensional
storage (hash meaning
w/ nesting) consistency >
availability
๏ Query any field
๏ Partitioning easy
at any level
thanks to richer
๏ BTree Indexes data model
MongoDB, CouchDB, RethinkDB

51. Graph
๏ 1 Dimensional storage... but grouped to appear
2D
๏ Differentiated by indexes
๏ Large indexes cover many relationships
๏ Query time depends on # records returned,
not distance to get them
๏ Doesn’t require traversing to determine
relationship
Neo4j, about 20 more... nobody talks much about

52. MongoDB for
Genealogy

53. Right Data
Model

54. Types of
genealogy data
๏
Events ๏
Photographs
(birth, death, etc)
๏
๏ Diaries & letters
Official records
๏
๏ Ship passenger list
Census
๏
๏ Occupation
Names
๏
๏ and more
Relationships

55. Challenges of
genealogy data
๏
Lots of possible data points... need flexible
schema
๏
Multiple versions of same data point
(3 different dates for death date, 4 variations on
name).
๏
Lots of data associated with physical records
๏
Multiple versions of same nodes
(intelligent nondestructive merge needed)
๏
Need to have meta data associated

56. Individual User
Events[] • Name
• AFN • type • Email Address
• Modification Date • date • Password
• contributor[] • Individual_id
• record[]
Name
• First[]
• Middle[] Location
• Last[] • city
• state
• county
Record
• contributor
• country • type
• coordinates[] • thumbnail
• content
• description
• tags[]

57. Individual
individual = {
_id : ObjectId("4f2978dfaa999d9db02618ce"),
AFN : '1XYK-KQJ',
name: {
first: ['john', 'johannes'],
middle: 'peter',
last: ['smith', 'sandvik']
}
}
db.individual.find(
{name.first : ‘john’, name.middle : ‘peter’})

58. Individual.Events
events : [
death : {
date : ISODate('1989-07-14'),
location : {
city: 'pensacola',
state: 'fl',
county: 'escambia',
country: 'usa'
coordinates : [30.26,87.12]},
contributor : ObjectId("4eeac...691")}]
db.individual.find(
{events.death.date : ISODate(‘1989-07-14’)})
db.individual.find(
{events.death.location : { $near:[30,90]}})

59. Event Versions
events : [
birth : [ {
date : ISODate('1928-04-06'),
location : {
city: 'brattleboro',
state: 'vt',
county: 'windham',
country: 'usa'
coordinates : [42.51,72.34]},
contributor : ObjectId("4ee...00000"),
records: ObjectId("4ed8a...7b000000")
},
{
date : ISODate('1928-04-16'),
location : {
city: 'brattleboro',
state: 'vt',
county: 'windham',
country: 'usa'
coordinates : [42.51,72.34]},
contributor : ObjectId("4ee...37bb"),
records: ObjectId("4eea...0000c8"),
}],
}

60. Query with Versioned Events
events : [
birth : [
{ date : ISODate('1928-04-06')},
{ date : ISODate('1928-04-16')}
],
]
db.individual.find(
{events.birth.date : ISODate(‘1928-04-16’)})

61. Records
record1 = {
_id : ObjectId("4ed8aea7d8562f7d7b")
contributor : ObjectId("4eeab...1537bb"),
type : 'birth certificate',
thumbnail : BinData(0,"/9j/4AAQSkZJ...."),
content : BinData(0,"j6b/Id11lWqs..."),
tags : ['NY', 'certified'],
description : "John's birth certificate"
}

62. Right Scale

63. MongoDB: Scale built in
๏ Intelligent replication
๏ Automatic partitioning of data
(user configurable)
๏ Horizontal Scale
๏ Targeted Queries
๏ Parallel Processing

64. Intelligent Replication
Node 1 Node 2
Secondary Secondary
Heartbeat
Re
on
p
i
cat
lic
ati
pli
on
Re
Node 3
Primary

65. Scalable Architecture
App Server App Server App Server
Mongos Mongos Mongos
Config
Node 1
Server
Secondary
Config
Node 1
Server
Secondary
Config
Node 1
Server
Secondary
Shard Shard Shard

66. x
High Availability in Shards
Shard Shard
Primary
Mongod
or
Secondary
Secondary

67. Targeted Requests
1
4
Mongos
2
3
Shard Shard Shard

68. Parallel processing
1
6
Mongos 5
2 2 2
4 4 4
Shard Shard Shard
3 3 3

69. Right Feature
Set

70. Broad Feature Set
๏ Rich query language
๏ Native support for over 12 languages
๏ GeoSpatial
๏ Text search
๏ Aggregation & MapReduce
๏ GridFS
(distributed & replicated file storage)
๏ Integration with Hadoop, Solr & more

71. Last Year I
presented
on Graph in
MongoDB
http://j.mp/XvJ3dl

72. FamilySearch
presented in
December
2012
http://j.mp/X03TXp

73. http://j.mp/X03TXp

74. http://j.mp/X03TXp

75. http://j.mp/X03TXp

76. http://spf13.com
http://github.com/spf13
@spf13
Questions?
download at mongodb.org