NetApp, Inc. · NASDAQ:NTAP

NetApp — Technical Presentation (2008)

15 slides from NetApp's technical presentation, 2008

1. Slide 1: CLUEBOX: A Performance Log Analyzer for Automated Troubleshooting

   Tag line, tag line CLUEBOX: A Performance Log Analyzer for Automated Troubleshooting Niranjan Thirumale et al. NetApp, Inc. WASL 2008

2. Slide 2: The Problem

   The Problem  Complex system deployments have ad-hoc but effective mechanisms for: – Data collection – Monitoring – Alerting  …but do not have good analysis tools  What’s available: detailed performance counters. Can we mine that to: – Characterize and identify workload? – help in the performance

3. Slide 3: The Environment

   The Environment Gigabit Ethernet File System Software FC Fabric (FC) (NFS, CIFS, iSCSI) Aggregates RAID groups Clustered Storage Flexible Volumes Controller Fiber Channel Loop Snapshot Copies Disk Shelves Replica Services © 2008 NetApp. All rights reserved.

4. Slide 4: CLUEBOX Workflow

   CLUEBOX Workflow © 2008 NetApp. All rights reserved.

5. Slide 5: Workload Identification

   Workload Identification Principle Feature Analysis Entire set of counters Reduced set of counters Random Forest Classification Classification into workloads + Ranking of counters by importance to classification Cluster on increasing subsets of ranked counters {1, 2, 3}, {1, 2, 3, 4}, {1, 2, 3, 4, 5}

6. Slide 6: Workload Identification: Evaluation

   Workload Identification: Evaluation  SIO – 6 workloads, each with varying read/write ratios and sequential/random I/O ratios  Cthon – 5 workloads, each exercising different sets of file system metadata operations  PostMark – 2 workloads, each operating on different number of files and different f

7. Slide 7: Workload Identification: How Many Counters Do We Really Need?

   Workload Identification: How Many Counters Do We Really Need? © 2008 NetApp. All rights reserved.

8. Slide 8: Workload Identification: Results

   Workload Identification: Results S1: Read 80%, Seq 80% S2: Read 20%, Seq 80% S3: Read 30%, Seq 90% S4: Read 30%, Seq 10% S5: Read 80%, Seq 90% S6: Read 80%, Seq 10% © 2008 NetApp. All rights reserved.

9. Slide 9: Workload Identification: Results

   Workload Identification: Results  Known workloads training data contains: – S5: SIO, Seq Reads = 72%, Seq Writes = 18%, Random Reads = 8%, Random Writes = 2% – P2: PostMark, # Files = 10^5, File Size = 1KB to 100KB  Test workload: SIO, Seq Reads = 72%, Seq Writes = 8%, Random Reads = 18%, Random W

10. Slide 10: Workload Identification: Results

    Workload Identification: Results  Known workloads training data contains – C2: Cthon Test 8 (symlink, readlink), # files = 1000, #symlinks = 1000 – S1: SIO, Seq Reads = 64%, Seq Writes = 16%, Random Reads = 16%, Random Writes = 4%  Test workload: Cthon Test8 with different parameters © 2008 NetApp

11. Slide 11: Anomaly Detection

    Anomaly Detection  User: “the performance of the system was fine at Time T. I haven’t changed a thing since then, but now the latencies are very high” Run the cluster medoids and Workload at T through RF and get proximities Closest Workload Predict counter values and diff with measured values Rank

12. Slide 12: Anomaly Detection

    Anomaly Detection  A handful of top-ranked counters were enough  Anomalies in counters + Non-anomalous counters were both important  An expert scanning this list diagnosed the root- cause rapidly – Conducive to a rule-based system at this point? © 2008 NetApp. All rights reserved.

13. Slide 13: Anomaly Detection: fsck-like load

    Anomaly Detection: fsck-like load Counter Deviation from  Readahead counters have Prediction dropped, so not a replication Ifnet:e0:total_packets -99.4% load Processor0:hard_switches -63.6%  Absence of anomaly in cp_phase_times counter, so System:cpu_busy 305.6% no high amount of destaging from ca

14. Slide 14: Anomaly Detection: Local I/O load

    Anomaly Detection: Local I/O load Counter Deviation  system:cpu_busy higher than from predicted indicative of some Prediction other load Testaggr:cp_read_blocks 129%  ifnet:e0:total_packets is not different, so not a network I/O Ifnet:e0:total_packets 8.2% load  No other significant System:cpu_bu

15. Slide 15: What Next

    What Next  Test CLUEBOX on customer environments – Real performance problems – Real workloads  Incorporate CLUEBOX into NetApp’s performance log analysis workflows – AutoSupport Database – Support Case Resolution – Duplicate case/bug report Identification  Figure out how to avoid re-training if s

NetApp — Technical Presentation (2008) on 6ix