ISWC 2010 presentation. Mo

1. Javier D. Fernández, Miguel A. Martínez-Prieto, Claudio Gutiérrez Compact Representation of Large RDF Data Sets for Publishing and Exchange

3. Syntaxes oriented mainly to represent documents

4. RDF/XML, N3, Turtle, JSON, etc.

5. Document-centric data-centricview

6. Redundancy

7. No structure(chunks)

8. Lackof metadata

9. sequentiality of theinformation

10. Use?

11. examples:

12. Billion Triple 2010 (~3200M triples, 318 gzippedchunks, ~27GB)

13. Uniprot (~845M, 12 gzippedchunks, ~23GB)Pag 2 Image: renjithkrishnan / FreeDigitalPhotos.net

14. Real World example: Billion Triple 2010 Where is the metadata? Who did publish this? Do I have all the data? ? [318] PUBLICATION EXCHANGE RDF RDF RDF gzip RDF RDF RDF gzip [318] basicoperations Pag 3

16. Metadata

17. Compactness

18. Efficient exchange

19. RDF compression

20. Basic data operations Pag 4 Image: jscreationzs / FreeDigitalPhotos.net

22. Phylosophy of publication and exchange,

23. Compact RDF representation

24. basedon 3 maincomponents: Header, Dictionary and TriplesPag 5

25. HDT Overview Pag 6

27. Source and providerinformation

28. Publication data

29. Data set statistics

30. Otherinformation

31. Information required to retrieve and process the represented data

32. Location/s, format/s, encoding/s, etc.Pag 7

33. Header use ? Header Header [318] HDT HDT RDF RDF RDF HDT HDT RDF RDF RDF HDT HDT Dictionary &Triples [318] Dictionary &Triples Pag 8

34. Header in Practice http://purl.org/HDT/hdt# SWP SCOVO, SDMX, hdt Void, DublinCore, etc hdt Pag 9

35. DBPedia example Pag 10

36. DBPedia Header Pag 11

37. DBPedia Header Pag 12

39. the number of triples of G in which s occurs as subject

40. deg−(G), deg−(G)

41. partialout-degree, deg− −(s, p)

42. the number of triples of G in which s occurs as subject and p as predicate

43. deg− −(G), deg− −(G)

44. labeledout-degree, degL−(s)

45. the number of different predicates (labels) of G with which s is related as a subject in a triple of G

46. degL−(G), degL−(G)

47. subject-object ratio, αs−o

48. the proportion of common subjects and objects in the graph G

49. αs−o =|SG∩OG| / |SG∪OG|

50. Symmetrically, in-degrees: deg+(o), deg+(G), etc.Pag 13

51. DBPedia example out-degree(page1) = 4 partialout-degree(page1,#label) = 2 labeledout-degree(page1)=3 in-degree(page3) = 2 partial in-degree(page3,#broader) = 2 labeled in-degree(page3)=1 out-degree(page2) = 2 labeledout-degree(page2)=2 Pag 14

52. HDT Pag 15

54. Currently, in RDF formats:

55. namespaces and prefixes

56. Currently, in Triple Stores:

57. assigns a unique ID to each element in the data setHeader Dictionary Pag 16

59. (1) Common subject-objects

60. (2) The non common subjects

61. (3) The non common objects

62. (4) Predicates

63. List of strings matching the mapping of the four subsets, in order from (1) to (4).

64. A reserved character is appended to the end of each string and each vocabulary to delimit their size.Pag 17

65. Dictionary in Practice Pag 18

66. Dictionary in Practice. Header configuration Pag 19

67. HDT Pag 20

69. Compact Triples Predicates: Objects: subject 1 subject 2 subject 3 1 2 6 . 1 3 2 . 2 1 3 . 2 2 4 . 2 2 5 . 2 4 1 . 3 3 2 . 1 2 6; 3 2 . 2 1 3; 2 4, 5; 4 1 . 3 3 2 . 2 6; 3 2 2 3 0 1 2 4 0 3 0 1 2 4 030 3 0 1 2 3 0 3 0 1 1 3 2 4, 5; 4 1 3 2 Adjacency Lists Splitting Subject Grouping 6 0 2 0 3 04 501020 6 0 2 0 3 04 50 6 0 2 0 3 04 5010 6 0 6 0 2 0 6 0 2 0 3 0 Compact Triples Pag 22

70. Predicates: Objects: Predicates Objects subject 2 subject 3 subject 1 Sp 2 3 1 2 4 3 Bp Bitmap Triples 2 301 2 4 03 0 2 301 2 4 030 2 301 2 4 03 0 0 01 0 0 0 1 0 1 2 3 01 2 4 03 0 6 2 3 4 5 1 2 Bitsequence-based reorganization So 6020304501020 6020304501020 6020304501020 01 01010 010 10 1 Bo Compact Triples Bitmap Triples Pag 23

72. take advantage of the structure indexed in Bp and Bo

73. accessible by fast rank/ select operations.

74. sequence S of length n drawn from an alphabet Σ = {0,1}:

75. ranka(S,i): counts the occurrences of a symbol a ∈ {0,1} in S[1,i].

76. selecta(S,i): finds the i-th occurrence of symbol a ∈ {0,1} in S.Pag 24

77. HDT Operations Over Bitmaps Triples Algorithm 1. Check&Find operation for a triple (s,p,o). Thedistribution of lists assures an average cost in O (degL−(G) + deg−−(G)) Pag 25

78. Triples in Practice. Header configuration Pag 26

79. HDT Bitmap Triples Compression (for exchange) HDT-Plain HDT-Compress Textcompression: gzip, bz2, PPM specificcompression: Huffman (S), RRR (B) Pag 27

80. HDT Bitmap Triples Compression Results Pag 28

81. HDT Bitmap Triples Compression Results Uniprot Pag 29

83. Subject-object JOINs resolution can profit from the common naming in the dictionary, as the elements are correctly and quickly localized in the top IDs.

84. Algorithm 1 can response basic ASK queries of SPARQL for patterns (s,p,o), (s,?p,?o) and (s,p,?o).

85. Algorithm 1 can response basic CONSTRUCT query of SPARQL for simple WHERE patterns (s,p,o), (s,?p,?o) and (s,p,?o).The resultant is a RDF HDT graph.Note: The S-P-O Adjacency List order is assumed. The Algorithm1 and the response patterns vary for alternative representations S-O-P AL, P-S-O, P-O-S, O-P-S AL and O-S-P AL. Pag 30

87. lack of structure, metadata information and native operations over the data

88. HDT addressestheseproblems (producer, consumer)

89. Header, Dictionary, and Triples

90. Triples PracticeImplementation

91. Compact (HDT-Plain)

92. Compress (HDT-Compress, outperforms universal compressors)

93. Check&Find (indexedaccess)Pag 31

95. Open-source (soon at http://hdt.dcc.uchile.cl/)

96. RDF native storage

97. Dynamic structures on secondary memory

98. Solve SPARQL joins

99. Multi-Index (size tradeoff)

100. Extensions

101. N-Quads

102. Sparql EndpointsPag 32

103. W3C Member Submission Pag 33 Image: renjithkrishnan / FreeDigitalPhotos.net

104. Thanks for your attention. http://www.rdfhdt.com rdfhdt@gmail.com Javier D. Fernández, Miguel A. Martínez-Prieto, Claudio Gutiérrez