Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion     .                        FAST’12         NCCloud: Applying...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion. Outline               Background                       Backgr...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Background     ...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Proposed Soluti...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Proposed Soluti...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Proposed Soluti...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Proposed Soluti...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Proposed Soluti...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Erasure Codes  ...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Reed-Solomon Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Problem of Trad...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Problem of Trad...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Problem of Trad...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Problem of Trad...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Regenerating Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Regenerating Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Regenerating Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Background Erasure Codes Regenerating Codes. Regenerating Co...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. the Paper’s Contributi...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. F-MSR Key Idea        ...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Upload       Give...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Upload       Uplo...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Download      Dow...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Repair       6 st...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Repair       .   ...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Repair       Solu...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Repair       Solu...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Repair       Cost...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   the Paper’s Contributions F-MSR Code. File Repair       Cost...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Repair Traffic Analysis Cost Analysis Experiments. Repair Traffi...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Repair Traffic Analysis Cost Analysis Experiments. Cost Analys...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Repair Traffic Analysis Cost Analysis Experiments. Cost Analys...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Repair Traffic Analysis Cost Analysis Experiments. Experiments...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Repair Traffic Analysis Cost Analysis Experiments. Response Ti...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion   Repair Traffic Analysis Cost Analysis Experiments. Response Ti...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion. Conclusion & Unsolved Problems      Conclusion:          Prop...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion. Conclusion & Unsolved Problems      Unsolved problems:       ...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion            A.G. Dimakis, P.B. Godfrey, Y. Wu, M.J. Wainwright,...
1. Background – 2. Solution – 3. Evaluation – 4. Conclusion            In USENIX Annual Technical Conference (USENIX ATC),...
Upcoming SlideShare
Loading in …5
×

my presentation of the paper "FAST'12 NCCloud"

1,493 views

Published on

Published in: Technology, Business
  • I also like the www.HelpWriting.net site. They helped me
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Finally found a service provider which actually supplies an essay with an engaging introduction leading to the main body of the exposition Here is the site ⇒⇒⇒WRITE-MY-PAPER.net ⇐⇐⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • To get professional research papers you must go for experts like ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • A professional Paper writing services can alleviate your stress in writing a successful paper and take the pressure off you to hand it in on time. Check out, please ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • This information is rather useful smile emoticon Thanks smile emoticon By the way, HelpWriting.net provides efficient writing services smile emoticon
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

my presentation of the paper "FAST'12 NCCloud"

  1. 1. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion . FAST’12 NCCloud: Applying Network Coding for the Storage Repair in a Cloud-of-Clouds . Yuchong Hu, Henry C. H. Chen, Patrick P. C. Lee, Yang Tang presented by Shuai YUAN NTHU LSA lab . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 1/20
  2. 2. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion. Outline Background Background Erasure Codes Regenerating Codes Solution the Paper’s Contributions F-MSR Code Evaluation Repair Traffic Analysis Cost Analysis Experiments Conclusion . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 2/20
  3. 3. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Background Cloud storage is an emerging service model for remote backup and data synchronization. Single-cloud storage raises concerns: Cloud outage Vendor lock-ins: Costly to switch cloud providers . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 3/20
  4. 4. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Proposed Solution Multi-cloud storage Deploy a proxy between users and multiple clouds Stripe data across multiple clouds . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 4/20
  5. 5. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Proposed Solution Problem: Multi-cloud storage repair The proxy reads the essential data pieces from other surviving clouds, reconstructs new data pieces, and writes these new pieces to a new cloud. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 4/20
  6. 6. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Proposed Solution Problem: Multi-cloud storage repair Table : Monthly price plans (in US dollars) for Amazon S3 (US Standard), Rackspace Cloud Files and Windows Azure Storage, as of September, 2011. Amazon S3 Rackspace Azure Storage (per GB) $0.14 $0.15 $0.15 . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 4/20
  7. 7. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Proposed Solution Problem: Multi-cloud storage repair Table : Monthly price plans (in US dollars) for Amazon S3 (US Standard), Rackspace Cloud Files and Windows Azure Storage, as of September, 2011. Amazon S3 Rackspace Azure Storage (per GB) $0.14 $0.15 $0.15 Replication is expensive! e.g. To achieve double-fault tolerance, we need 2 replicas. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 4/20
  8. 8. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Proposed Solution Problem: Multi-cloud storage repair Table : Monthly price plans (in US dollars) for Amazon S3 (US Standard), Rackspace Cloud Files and Windows Azure Storage, as of September, 2011. Amazon S3 Rackspace Azure Storage (per GB) $0.14 $0.15 $0.15 Replication is expensive! e.g. To achieve double-fault tolerance, we need 2 replicas. ; Use erasure code instead. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 4/20
  9. 9. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Erasure Codes Given a message/file of k equal-size native chunks, encode it into n code chunks (n > k) Optimal erasure codes: any k out of the n code blocks/chunks are sufficient to recover the original message/file. i.e. can tolerate arbitrary (n − k) failures. We call this (n,k) MDS (maximum distance separable) property. e.g. Reed-Solomon Codes [6] Near-optimal erasure codes: require (1 + ε)k code blocks/chunks to recover (ε > 0). i.e. cannot tolerate arbitrary (n − k) failures. e.g. Local Reconstruction Codes (used in Windows Azure Storage System) [4] . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 5/20
  10. 10. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed Solomon Codes (RAID-6) [6] Achieve double-fault tolerance Reed-Solomon Codes: additional data size = M Relication: additional data size = 2M (2 replicas) . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 6/20
  11. 11. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] How do Reed-Solomon Codes work? Given file F : divide it into k equal-size native chunks: F = [Fi ]i=1,2,...,k . Encode them into (n − k) code chunks: C = [Ci ]i=1,2,...,n−k . Use Encoding Matrix EM(n−k)×k to produce code chunks: CT = EM × F T Ci is the linear combination of F1 , F2 , . . . , Fk . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  12. 12. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] In our case, ( ) F = ( B A ) 1 1 EM = (1 2) ( ) 1 1 A CT = × ( 1 2 ) B A+B = A + 2B . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  13. 13. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] We can rewrite it in another equivalent representation: Let P = [Pi ]i=1,2,...,n = [F1 , F2 , . . . ,) k , C1 , C2 , . . . , Cn−k ] be the ( F I n chunks in storage, EM ′ = , Here EM   1 0 ... 0 0 1 . . . 0   I = . . . . . . .. . . . . 0 0 ... 1 then PT = EM ′ ) F T ( T× F = CT . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  14. 14. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] In our case,   ( ) 1 0 I 0 1 EM ′ = = 1 1  EM 1 2 PT = EM ′ × F T   1 0 ( ) 0 1 =  × A 1 1 B 1 2  A  B  =   A+B  A + 2B . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  15. 15. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] Why Reed-Solomon Codes have MDS property? EM ′ is the key of MDS property! . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  16. 16. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] . Theorem (necessary and sufficient condition) . Every possible k × k submatrix obtained by removing (n − k) rows from EM ′ has full rank. equivalent expression of full rank: rank = k non-singular . . . . Alternative view: Consider the linear space of P = [Pi ]i=1,2,...,n = [F1 , F2 , . . . , Fk , C1 , C2 , . . . , Cn−k ], its dimension is k, and any k out of n vectors form a basis of the linear space. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  17. 17. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] Reed-Solomon Codes uses Vandermonde Matrix V as EM   1 1 1 ... 1  1 2 3 ... k     12 2 2 3 2 ... k2  V =   . . . .. .   . . . . . . . .  . 1(n−k) 2(n−k) 3(n−k) . . . k (n−k) . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  18. 18. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] So matrix EM ′ is:   1 0 0 ... 0  0 1 0 ... 0     0 0 1 ... 0     . . . .. .   . . . . . . . . .     0 0 0 ... 1  EM =  ′  1    1 1 ... 1   1 2 3 ... k     12 22 32 ... k2     . . . .. .   . . . . . . . . .  1 (n−k) 2(n−k) 3(n−k) . . . k (n−k) . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  19. 19. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Reed-Solomon Codes (RAID-6) [6] Remark: All arithmetic operations in Galois Field GF(2x ). Then every number is less than 2x . For more details of Galois Field, please refer to [7] and a implementation tutorial [5]. EM ′ satisfies MDS property. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 7/20
  20. 20. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Problem of Traditional Erasure Codes (e.g. R-S Codes) . Definition (Repair traffic) . the amount of outbound data being read from other surviving clouds during the single-cloud failure recovery. . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 8/20
  21. 21. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Problem of Traditional Erasure Codes (e.g. R-S Codes) . Definition (Repair traffic) . the amount of outbound data being read from other surviving clouds during the single-cloud failure recovery. . Why do we only consider outbound traffic without taking inbound traffic into account? . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 8/20
  22. 22. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Problem of Traditional Erasure Codes (e.g. R-S Codes) . Definition (Repair traffic) . the amount of outbound data being read from other surviving clouds during the single-cloud failure recovery. . Why do we only consider outbound traffic without taking inbound traffic into account? Amazon S3 Rackspace Azure Date transfer in (per GB) free free free Date transfer out (per GB) $0.12 $0.18 $0.15 . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 8/20
  23. 23. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Problem of Traditional Erasure Codes (e.g. R-S Codes) . Problem of Reed-Solomon Codes . Need to obtain otain size of the whole file to repair. ; Repair traffic is high! . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 8/20
  24. 24. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Regenerating Codes [1] [2] [3] Goal: Minimize repair traffic while maintaining same fault tolerance as erasure codes ; recover faster than erasure codes Idea: Do not reconstruct the whole file as in erasure codes Instead, read only the chunks (smaller than whole file) that are needed to recover the lost chunks Build on network coding: network bandwidth is more critical resource compared to disk access. encode chunks in storage nodes . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 9/20
  25. 25. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Regenerating Codes [1] [2] [3] An example: . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 9/20
  26. 26. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Regenerating Codes [1] [2] [3] . Problem of Regenerating Codes . In our case, the storage nodes provided by cloud venders: Support read/write Do not have encoding/decoding capabilities . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 9/20
  27. 27. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Background Erasure Codes Regenerating Codes. Regenerating Codes [1] [2] [3] . Problem of Regenerating Codes . In our case, the storage nodes provided by cloud venders: Support read/write Do not have encoding/decoding capabilities . Next we will see how this paper of NCCloud solves these problems. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 9/20
  28. 28. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. the Paper’s Contributions Build NCCloud, a proxy-based storage system that applies regenerating codes in multiple-cloud storage Design goals: Propose an implementable design of functional minimum-storage regenerating (F-MSR) code Support basic read/write operations and the repair function Preserve storage overhead as in MDS codes, while reducing repair traffic Implement and evaluate NCCloud in real storage setting focus on double-fault tolerance (k = n − 2) focus on single-fault recovery . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 10/20
  29. 29. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. F-MSR Key Idea non-systematic: don’t keep original data as in systematic codes. functional repair: don’t need to exactly regenerate the failed chunks, only require the repaired system maintains MDS property. Let the proxy do the encoding/decoding. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 11/20
  30. 30. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Upload Given file F : divide it into k(n − k) equal-size native chunks: F = [Fi ]i=1,2,...,k(n−k) . Encode them into n(n − k) code chunks: P = [Pi ]i=1,2,...,n(n−k) . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 12/20
  31. 31. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Upload Upload process: Encode Produce Encoding Matrix (Use Vandermonde Matrix) EM = [ECV1 , ECV2 , . . . , ECVn(n−k) ] = [αi,j ]n(n−k)×k(n−k) Then produce code chunks: PT = EM × F T For i = 1, 2, . . . , n(n − k) Pi = ECVi × F T ∑k(n−k) = j=1 αi,j Fj ECVi : encoding coeffient vector of Pi . All arithmetic operations are in Galois Field GF(28 ). Duplicate encoding matrix to all nodes as metadata.. . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 12/20
  32. 32. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Download Download process: Download all the chunks from any k of n clouds Multiply inverted encoding matrix with downloaded chunks F T = P T × EM −1 . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 13/20
  33. 33. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Repair 6 steps: . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 14/20
  34. 34. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Repair . Simulations . Consider multiple rounds of permanent node failures for different values of n. In each round, we randomly pick a node to permanently fail and trigger a repair. . . Simulation result . If the loop of Steps 2 to 5 is repeated over 10 times ; bad repair Only checking the MDS property, we see a bad repair very quickly: after no more than 7 and 2 rounds for n = 8 and n = 12, respectively. . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 14/20
  35. 35. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Repair Solution: Two-phrase checking MDS property check: Current repair maintains MDS property Repair MDS property check: Next repair for any possible failure maintains MDS property . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 14/20
  36. 36. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Repair Solution: Two-phrase checking . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 14/20
  37. 37. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Repair Cost of two-phrase checking k MDS property check: enumerating Cn subsets of n nodes to see if each of their corresponding encoding matrices forms a full rank. Repair MDS property check: for any failed node (out of n nodes), we collect any one out of (n − k) chunks from the other (n − 1) surviving nodes to reconstruct, therefore the cost is n(n − k)(n−1) Cn . k Return to .. Unsolved Problems . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 14/20
  38. 38. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion the Paper’s Contributions F-MSR Code. File Repair Cost of two-phrase checking k MDS property check: enumerating Cn subsets of n nodes to see if each of their corresponding encoding matrices forms a full rank. Repair MDS property check: for any failed node (out of n nodes), we collect any one out of (n − k) chunks from the other (n − 1) surviving nodes to reconstruct, therefore the cost is n(n − k)(n−1) Cn . k Return to .. Unsolved Problems We have to check more times for the current repair, but bad repair will be rare in the future iterative repairs. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 14/20
  39. 39. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Repair Traffic Analysis Cost Analysis Experiments. Repair Traffic Native data size M k(n − k) native chunks of size M/k(n − k) k−1 Repair Traffic: M/k(n − k) × (n − 1) = M/k × (1 + ) n−k 1 For k = n − 2, Repair Traffic: M/2 × (1 + ) n−2 limn→∞ Repair Traffic = M/2 . Save the repair traffic by close to 50% when n is large. . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 15/20
  40. 40. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Repair Traffic Analysis Cost Analysis Experiments. Cost Analysis Table : Monthly price plans (in US dollars) for Amazon S3 (US Standard), Rackspace Cloud Files and Windows Azure Storage, as of September, 2011. Amazon S3 Rackspace Azure Storage (per GB) $0.14 $0.15 $0.15 Date transfer in (per GB) free free free Date transfer out (per GB) $0.12 $0.18 $0.15 PUT,POST (per 10K requests) $0.10 free $0.01 GET (per 10K requests) $0.01 free $0.01 . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 16/20
  41. 41. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Repair Traffic Analysis Cost Analysis Experiments. Cost Analysis Metadata of F-MSR Metadata size = 160B; file size = several MBs Overhead due to GET requests during repair Assuming S3 plan in Sep 2011, n = 4, k = 2, file size = 4MB Conventional repair: 0.427% F-MSR repair: 0.854% . Overhead cost is low. . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 16/20
  42. 42. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Repair Traffic Analysis Cost Analysis Experiments. Experiments NCCloud deployment Single machine connected to a cloud-of-clouds n = 4, k = 2 Coding schemes Reed-Solomon-based RAID-6 vs. F-MSR Metric Response time Cloud environments: Local cloud: OpenStack Swift Commercial cloud: multiple containers in Azure . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 17/20
  43. 43. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Repair Traffic Analysis Cost Analysis Experiments. Response Time: Local Cloud F-MSR has higher response time due to encoding/decoding overhead. F-MSR has slightly less response time in repair, due to less data download . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 18/20
  44. 44. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion Repair Traffic Analysis Cost Analysis Experiments. Response Time: Commercial Cloud No distinct response time difference, as network fluctuations play a . . . . . . . . . . . . . . . . . . . . bigger role in actual response time. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 19/20
  45. 45. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion. Conclusion & Unsolved Problems Conclusion: Propose an implementable design of F-MSR: Preserve storage cost. Use less repair traffic. Do not require storage nodes to have encoding capabilities. Build NCCloud, which realizes F-MSR Source code: http://ansrlab.cse.cuhk.edu.hk/software/nccloud/ . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 20/20
  46. 46. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion. Conclusion & Unsolved Problems Unsolved problems: Repair costs is high when n and k are large: As we mentioned before (Click .. here ), F-MSR uses two-phrase checking, which consumes a lot of checking costs in the current repair phrase. Just as Reed-Solomon codes use Vandermonde Matrix to ensure MDS property, a better algorithm is still seeking to replace the check-after-trying approach. The reason why F-MSR chooses to download chunks from all (n − 1) nodes for repairing a file comes from an argument in [1]: The more nodes we download chunks from, the lower repair traffic is. However, [1]’s conclusion is based on a homogeneity model, and NCCloud’s multi-cloud solution is actually a heterogeneous environment. Such a basis may be invalid. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 20/20
  47. 47. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion A.G. Dimakis, P.B. Godfrey, Y. Wu, M.J. Wainwright, and K. Ramchandran. Network coding for distributed storage systems. Information Theory, IEEE Transactions on, 56(9):4539–4551, 2010. A.G. Dimakis, K. Ramchandran, Y. Wu, and C. Suh. A survey on network codes for distributed storage. Proceedings of the IEEE, 99(3):476–489, 2011. A. Duminuco and E. Biersack. A practical study of regenerating codes for peer-to-peer backup systems. In Distributed Computing Systems, 2009. ICDCS’09. 29th IEEE International Conference on, pages 376–384. IEEE, 2009. C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin. Erasure coding in windows azure storage. . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. presented by Shuai YUAN FAST’12: NCCloud 20/20
  48. 48. 1. Background – 2. Solution – 3. Evaluation – 4. Conclusion In USENIX Annual Technical Conference (USENIX ATC), 2012. J.S. Plank et al. A tutorial on reed-solomon coding for fault-tolerance in raid-like systems. Software Practice and Experience, 27(9):995–1012, 1997. I.S. Reed and G. Solomon. Polynomial codes over certain finite fields. Journal of the Society for Industrial & Applied Mathematics, 8(2):300–304, 1960. B. Sklar. Reed-solomon codes. Downloaded from URL http://www. informit. com/content/images/art. sub.–sklar7. sub.–reed-solomo-n/elementLinks/art. sub.–sklar7. sub.–reed-solomon. pdf,(unknown pub date), pages 1–33, 2001. . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . . . . .. .. .. presented by Shuai YUAN FAST’12: NCCloud 20/20

×