Gil Elbaz, CEO and founder of Factual, gave a talk at the 2011 Web 2.0 Conference in San Francisco. His talk was entitled: "Big Data Challenges: Getting Some."

1. Big Data Challenges: Getting Some
March 31, 2011
Gil Elbaz
@factual
@gilelbaz

2. Road to Information Singularity
Conf dential
i 2

3. Networks Underlying Information Flow
! Density:
number of
connecting paths
! Plasticity:
ease of forming
new paths
!
Speed & Flow:
!""#$%%&&&'()*++,-+.*/(01,-211(**3'4*5%()*++,-+6.*/6(01,-211%
rate of information
transfer
Conf dential
i 3

4. The Internet
!""#$%%&&&'7578*-'4*5%9,+,"7):;/*11/*7<1:=52/,47-:>2)24*550-,47",*-1:?-"2/-2"%<#%@ABACD@ECF
Conf dential
i 4

5. Search Engines
Conf dential
i 5

6. Social Networks: Facebook
!""#$%%A'(#'()*+1#*"'4*5%
600 million Facebook users
130 average friends
8 friend requests / month
15 messages / day / user
Conf dential
i 6

7. Trending of Unfriending
Conf dential
i 7

8. Conf dential
i 8

9. Unfriending
Conf dential
i 9

10. Another Network: The Brain
100 billion neurons
1000 ‘hardwired’ synapses
!""#$%%&2)4*52"*G57/"'4*5%A@CC%@C
Conf dential
i 10

11. Web 3.0: Data Web
Conf dential
i 11

12. Web Scale Data = More Pain
Findability
Access
Rights
Economics
Standards
Integration & Aggregation
Trust
Conf dential
i 12

13. Web 2.0 Model: Scale-Free Networks
&&&'.0"0/22H#)*/7",*-'-2" Conf dential
i 13

14. Book Data: Progress Being Made
Google Book Search API
Open Library Books API
ISBNdb
Amazon API
LibraryThing
GoodReads
WorldCat
Conf dential
i 14

15. Google Book Search API Amazon API
Open Library Books API LibraryThing
ISBNdb WorldCat GoodReads
I,-<7(,),"JKKKKKKKKKKKK=44211KKKKKKK
L,+!"1KKKKKKKKKKKKM4*-*5,41KKKKKKK
N"7-<7/<1KKKKKKKKKKKKKKK>/01"KKKKKK
Conf dential
i

16. Another Case Study: Local Data
!""#$%%1"2O24!2-2J'#*1"2/*01'4*5%
Conf dential
i 16

17. Another Case Study: Local Data
!"##$%$&$'$(#)*+()(,-&(##)%.'/!"#$%"$&"'$()*$*!)$%+*+0
!"#$$%& !"#$$%&
'()%*++, Examine Twitter sentiment '()%*++,
(avoid dirty coffee shops)
-++.$ -++.$
'+/&01/(&% Identify areas of highest '+/&01/(&%
bike thefts
2%3. 2%3.
4#33+" 4#33+"
Correlate check-ins with
5++63% property values 5++63%
7+8%9:/;)$#+; 7+8%9:/;)$#+;
Conf dential
i 17

18. HomeJunction
Conf dential
i 18

19. Factual is Example of New Information Network
"#$#%&'( )'$&*+*#(&( 345&*'6&'$ ,-./#'&01&-*'&2
,-."'-%$%+*+
Aggregate Mash Curate
Dedupe Canonicalize
Developers Publishers Search Engines
!"#$%"&'()"*+$,-.-/(0(1("*+$%231#-&"$4..*
Conf dential
i 19

20. Factual’s Open Data Model
Free, access via APIs, SDKs, and downloads BUT…
we ask you to contribute back into ecosystem.
Benef ts
i
! Drive down costs
! Rapid iteration
! Differentiate on user
experience
! Only need small %
participation from world
(e.g. Wikipedia)
Conf dential
i 20

21. Equivalence Measurements
=?
Subway Sandwiches Subway
52 E Court St 52 West Court St
Cincinnati, OH 45202
(513)-241-6699 (800)-653-2323
Conf dential
i 21

22. Large-Scale Aggregation Technologies
Conf dential
i 22

23. Large-Scale Aggregation Technologies
=#7/"52-"1KPK=#"1
;2-"2/KPK;"/
;*/#KPK;*/#*/7",*-
N2/O,42KPKNO4
=""*/-2JKPK=""J
=11*4KPK=11*4,7"21
?-4KPK?-4*/#*/7"2<
=11-KPK=11*4,7",*-
;*KPK;*5#7-J
Q*0-"KPKQ"
R/*1KPKR/*"!2/1
KKKKKKKRRSKPKR7/(2T02KKK'''''
U*/,KPK>2<
Conf dential
i 23

24. Large-Scale Aggregation Technologies
L21"70/7-"KPKL1"/-"
L21"70/7-"KPKL21"07/7-"
V*1#KPKV*1#,"7)
R,))7/<1KPKR,)),7/<1
N7)*-KPKN)-
R0..2"KPKR0..2""
;2-"2/KPK;"/
=#7/"52-"1KPK=#"1
R*0",T02KPKR"T
W2&2)2/1KPKW2&)2/1
;)27-2/1KPK;)-/1
KKKKKQ7/32"KPKQ3"8K'''''
X/7+2-KPKYZL2,))JK[
Conf dential
i 24

25. Kragen O'Reilly?
Conf dential
i 25

26. Large-Scale Deduping
• Specialized data compression & folding techniques
• Eliminate redundant entities - endpoints and authority pages
• Improves precision & recall
• Enables real-time dedupe and crosswalks
Conf dential
i 26

27. Shared Foundational Data
! Commoditization of data
! Head attributes for people, places, things decreasing in value
! hCard data value driven to zero (visual of local data being
identical on thousand of apps)
! Entertainment: IMDB exposed all their data for non-
commercial use (link to site map)
! Yet, there are still lots of errors in foundation data – thus
need “living” model
Conf dential
i

28. LA Neighborhoods: Another Crowdsourcing Example
! LA Times started with 87
neighborhoods based on census
tracts
! Incorporated 650+ user maps
! Ended with 114 neighborhoods for
LA City
! Added additional 158
neighborhoods for LA County
Conf dential
i

29. Ownership & Rights: LA Neighborhoods:
! Terms of Service:
Creative Commons
Attribution,
Noncommercial, Share-
Alike license
! Can share and remix as
long as it’s for
noncommercial uses,
attributed to the LA
Times, and shared
under the same terms
Conf dential
i

30. Evolving “Buy” Model
! Data Marketplaces (“itunes of data?”)
! Data Search Engines
! Microformats / Semantic Web Markups / Other
Standards
! Electronic Forms of T&Cs
Conf dential
i

31. Summary: Road to the Information Singularity
! Rise in community storage and access
! New common schemas and standards
! Def nitive, accountable sources of “open” data
i
! Trends towards sharing of foundational data
! 'Buy' models based on unique data, novel access
methods, SLAs, value-added services
Conf dential
i 31

32. Thank you!
Questions......
Gil Elbaz
@factual
@gilelbaz