NetApp, Inc. · NASDAQ:NTAP
NetApp — Technical Presentation (2013)
25 slides from NetApp's technical presentation, 2013
Slide 1: Rangoli: Space management in deduped environments
Rangoli: Space management in deduped environments P.C. Nagesh and Atish Kathpal Advanced Technology Group, NetApp, India 1
Slide 2: Outline
Outline What is the space management problem ? Intuition behind our solutions Evaluation and Summary 2
Slide 3: Space management Objectives
Space management Objectives Back end volumes as data containers Cluster architecture depiction from OpenStack Low Free Space Ensure adequate free space on volumes 3
Slide 4: Space management problem
Space management problem Logical View Physical View Volume metadata Data Ref block count 6 9 10 12 Freeing up a deduped volume is hard! 4
Slide 5: Illustrative example
Illustrative example Which files to Space move? Reclamation Logical View 6 10 6 10 1 2 2,4 2,4 10 10 3 6 1,2,3 21 5 4 How do you reclaim 50 GB free space?
Slide 6: Outline
Outline What is the space management problem ? Intuition behind our solutions Evaluation and Summary 6
Slide 7: Intuitive solutions and alternatives
Intuitive solutions and alternatives Dedupe unaware strategy Low space reclamation, too many unnecessary side effects 1 2 3 6 Intutive solution 5 4 Move shared files Together. Naïve- du : Pick good “migration bins” Migrate files with most unique content 7
Slide 8: Side effects
Side effects Physical Space bloat Due to loss of disk sharing Percentage increase in physical space consumption Migration utility Bandwidth wastage Amount of reclaimed space per 100 bytes of data transfer It is a source centric strategy 8
Slide 9: Rangoli: Solution overview
Rangoli: Solution overview Fingerprint Database Inode FBN Fingerprint 1. Compute disk sharing relationships 1 3 a23b1234 – Graphical representation 2 5 234c1234 2. Identify groups of highly shared files – Good migration bins 3. Compute and report the exact metrics 1,2,3 – PSB and Migration Utility 6
Slide 10: Outline
Outline What is the space management problem ? Intuition behind our solutions Evaluation and Summary 10
Slide 11: Evaluation
Evaluation Evaluation objectives Comparison against alternate strategies Datasets from diverse workloads VM images Home directories Engineering document repositories NetApp Confidential - Internal Use Only 11
Slide 12: Migration Utility (VMDK dataset)
Migration Utility (VMDK dataset) Higher the better – More space reclamation per unit of data migration 100 90 80 70 Migration Utility (%) 60 50 Naïve-du MinHash 40 Rangoli 30 20 10 0 1 5 10 20 Space Reclamation (%) NetApp Confidential - Internal Use Only 12
Slide 13: Physical space bloat (Debian dataset)
Physical space bloat (Debian dataset) Lower the better – Less percentage increase in physical space consumption 40 35 30 25 PSB (%) 20 Naïve-du MinHash 15 Rangoli 10 5 0 5 10 20 30 Space Reclamation (%) NetApp Confidential - Internal Use Only 13
Slide 14: Summary
Summary Inferences: – Rangoli offers a scalable solution for space reclamation in deduped environments Better than alternatives by upto 35X sometimes. Future work: – Explore destination aware strategies – Combine space reclamation with other desired features such as load balancing and performa
Slide 15: Acknowledgements
Acknowledgements Our thanks to Gaurav Makkar, Kaladhar Voruganti, Kiran Srinivasan, Parag Deshmukh and our anonymous reviewers for the several insights and valuable feedback received This work was done as part of Independent Research Project at Advanced Technology Group, NetApp at Bangalore, Ind
Slide 16: NetApp
NetApp’ © 2013 NetApp, nc. All rights resowed. 16
Slide 17: Scalability
Scalability 28 minutes of end to end running time for the largest dataset tested – Synthetic dataset of 4TB size with 12million files and 85% dedupe. – Running times on a Laptop grade machine NetApp Confidential - Internal Use Only 17
Slide 18: Solution details
NetApp’ Solution details © 2013 NetApp, nc. All rights resowed.
Slide 19: Step 1: FPDB Processing
Step 1: FPDB Processing Amount of File Id disk sharing 1 4 2 3 3 3 4 5 5 6 Algorithm : – A linear scan of the fingerprint sorted FPDB – Output is the bipartite graph Inode FBN Fingerprint – Time is linearly proportional to the data set size 1 3 a23b12349870 2 5 a23b12349870
Slide 20: Step 2: Migration binning
Step 2: Migration binning Algorithm: – Seek partitions with minimal edge Amount of cuts File Id disk sharing – Offers good but not necessarily 1 4 optimal partitions 22 3 – Time is dependent on the complexity of disk sharing in the 3 3 dataset 4 5 given by the number of edges 5 – Quick union fin
Slide 21: Step 3: Compute the metrics
Step 3: Compute the metrics Algorithm: – Disk sharing within a bin contributes to savings in data 1 2 migration 3 6 – Data sharing across migration bins contributes to losses 5 4 We can compute the metrics for any arbitrary bins too. The metrics computed are actuals, not estimates
Slide 22: Datasets and experimental details
Datasets and experimental details Real world datasets from different workloads Synthetic datasets for testing scalability Dataset Size No of Dedup Files Home Directory 74 GB 78 K 49 % Debian 261 GB 448 59 % VMDK 2.4 TB 2.4 K 62 % EngWebBurt 1.3 TB 4M 51 % Synthetic 1 2.6 TB 8M 77% Synthetic 2 4
Slide 23: Physical space bloat
ri Physical space bloat NetApp’ Debian HomeDir @ Naive-du ( MinimumHash ®@ MinHashApprox @Rangoli mm Naive-du 1) MinimumHash ™ MinHashApprox &@ Rangoli 35 3.0 s s § 30 B25 ro) ao 25 y 82.0 § 20 5 715 B = = 1.0 $10 $* = = 35 3 o5 oe. s 0.0 ale aL | hb 1% 5% 10% 20% 30% 50% 1% 5% 10% 20% 30% 50% % Vol
Slide 24: Migration Utility
ri Migration Utility NetApp’ HomeDir @Naive-du (MinimumHash ® MinHashApprox % Rangoli Debian MinimumHash ®@ MinHashApprox @ Rangoli @ Naive-du WM MULL 1% 5 LLL. e s e ca N 3 $ xe og Se 3 & 2 & ¢ nd @ Secz SE Ra Zs g ao8 ROS ec s0es a win Ect BUE 5 3 gS E xs LLL § 3 > 2 Zan anon n 3 Gs 28 SR GS 3 3 3
Slide 25: Scalability
Scalability Scales to large datasets – ~30 minutes for 4TB and 12M file dataset with 85% dedupe Dataset Size No of Dedup Total running Files time in parallel mode Home Directory 74 GB 78 K 49 % 24 sec Debian 261 GB 448 59 % 1 min VMDK 2.4 TB 2.4 K 62 % 13 min EngWebBurt 1.3 TB 4M 51 % 9 min Synthe