Slide 1: NVIDIA
© NVIDIA
Slide 2: Accelerating Intelligence
Accelerating Intelligence Bill Dally, Chief Scientist and Senior Vice President of Research GTC China, December 14, 2020
Slide 3: Safe Harbor: Forward-Looking Statements
SAFE HARBOR Forward-Looking Statements Except for the historical information contained herein, certain matters in this presentation including, but not limited to, statements as to: the performance, benefits, abilities and impact of our products and technology; the availability of our products and te
Slide 4: CUDA libraries and platforms
; : Ine CUDA nd | cuSowver | IVA DRIVE ROBOTICS NVSHMEM I t t if
Slide 5: NVIDIA A100
NVIDIA A100 NVIDIA Ampere Architecture 3rd Gen Tensor Cores World’s Largest 7nm Chip Faster, Flexible, Easier to Use 54B Transistors, HBM2 20x AI Performance with TF32 New Sparsity Acceleration New Multi-Instance GPU 3rd Gen NVLink and NVSwitch Harness Sparsity in AI Models Optimal Utilization with
Slide 6: NVIDIA A100 Detailed Specs
NVIDIA A100 Detailed Specs TRANSISTOR COUNT 54 Billion DIE SIZE 826 mm2 FP64 CUDA CORES 3,456 FP32 CUDA CORES 6,912 TENSOR CORES 432 STREAMING MULTIPROCESSORS 108 FP64 9.7 TeraFLOPS FP64 TENSOR CORE 19.5 TeraFLOPS FP32 19.5 TeraFLOPS TF32 TENSOR CORE 156 TeraFLOPS | 312 TeraFLOPS* BFLOAT16 TENSOR CO
Slide 7: Structural Sparsity Brings Additional Speedups
Structural Sparsity Brings Additional Speedups BERT LARGE INFERENCE 2X Faster Execution 1.5x 1x Dense Matrix Sparse Matrix A100 A100 A100 Tensor Core Sparsity STRUCTURED SPARSITY Half the values are zero Skip half of the compute and mem fetches Compute up to 2x rate vs non-sparse BERT Large Inferenc
Slide 8: DGX A100
9x Mellanox ConnectX-6 VPI 200 Gb/s Network Interface Dual 64-core AMD Rome CPU 1TB RAM DGX A100 The Universal AI System 8x NVIDIA A100 GPUs Data Analytics, Training, and Inference 6x NVIDIA NVSwitches 4.8 TB/s Bi-directional Bandwidth 600 GB/s GPU-to-GPU Bandwidth 15TB Gen4 NVME SSD
Slide 9: Rack-Scale Infrastructure
4-node 8-node DGX POD DGX POD Rack-Scale Infrastructure Building an AI Center of Excellence with DGX POD Built on DGX A100 DGX POD More Attainable than Ever with DGX A100 Experience a Faster Start with Building Flexible AI Infrastructure Proven Architectures, with Leading Storage Partners Up to 40 P
Slide 10: The Selene Supercomputer
The Selene Supercomputer NVIDIA’s DGX SuperPOD Deployment #5 on TOP500 (63 PetaFLOPS HPL @ 24GF/W) #5 on Green500 (20.5 GigaFLOPS/watt) Fastest Industrial System in the U.S. — 1+ ExaFLOPS AI One of the Fastest and Most Efficient Supercomputers on the Planet — Built in Under a Month
Slide 11: TOP500 and GREEN500 rankings
#5 #5 #1 TOP500 GREEN500 GREEN500 SELENE SELENE DGX SUPERPOD 8 #1 #1 OF TOP 10 ACADEMIA INDUSTRIAL NVIDIA+MLNX FRONTERA SELENE #1 #1 #1 USA CHINA EUROPE SUMMIT TAIHULIGHT JÜLICH
Slide 12: Specialized Instructions Amortize Overhead
Specialized Instructions Amortize Overhead Operation Energy (pJ) 1 Overhead2 HFMA 1.5 2000% HDP4A 6.0 500% HMMA 110 22% IMMA 160 16% 1 - Energy numbers from 45nm process 2 - Overhead is instruction fetch, decode, and operand fetch – 30pJ
Slide 13: Single-chip Inference Performance
Single-chip Inference Performance 317x in 8 Years 1400.00 1248.00 A100 1200.00 1000.00 Int 8 TOPS 800.00 600.00 400.00 261.00 P100 200.00 K20X M40 125.00 Q8000 3.94 21.20 V100 6.84 0.00 April August December May September February June October 2012 2013 2014 2016 2017 2019 2020 2021
Slide 14: MLPerf Training Benchmarks Relative Speedup
MLPerf Training Benchmarks Relative Speedup Commercially Available Solutions | Speedup Over V100 Huawei Ascend TPUv3 V100 A100 3x 2.5x 2.4x 2.4x 2x 2x 2x 1.9x 1.6x 1.5x 1.2x 1x 1x 1x 1x 1x 1x 1x 1x 1x 0.9x 0.7x 0x X XX XX XX XX XX XX Image Classification NLP Object Detection Reinforcement Object Det
Slide 15: NVIDIA Tops MLPerf Data Center Inference Benchmarks
NVIDIA Tops MLPerf Data Center Inference Benchmarks A100 is up to 237x Faster than the CPU OFFLINE SERVER Per-accelerator Normalized to T4 Per-accelerator Normalized to T4 Xilinx U250 (Available) Intel Cooper Lake (Preview) Xilinx U250 (Available) Intel Cooper Lake (Preview) NVIDIA T4 (Available) NV
Slide 16: NVIDIA Tops MLPerf Edge Inference Benchmarks
NVIDIA Tops MLPerf Edge Inference Benchmarks Leadership Performance for Edge Servers with T4, A100, and Edge Devices with Jetson SINGLE-STREAM MULTI-STREAM Per-accelerator Normalized to Jetson AGX Xavier Per-accelerator Normalized to Jetson AGX Xavier Centaur NVIDIA Jetson AGX Xavier NVIDIA T4 NVIDI
Slide 17: Slide 17
Slide 18: RTXDI Off
RTXDI OFF
Slide 19: RTXDI On
RTXDI ON 66 Light Primitives, with IES Profiles
Slide 20: RTXGI On / Off
RTXGI ON RTXGI OFF
Slide 21: Lighting Only
LIGHTING ONLY RTXGI ON RTXGI OFF
Slide 22: NVIDIA DLSS 2.0
NVIDIA DLSS 2.0 A New Era of AI-powered Computer Graphics 1440P VS Ray Tracing 4K Supercomputer Rendered 16K Ground Truth
Slide 23: Native 4K vs DLSS 4K
108 141 NATIVE 4K DLSS 4K
Slide 24: Native 4K vs DLSS Off/On
108 60 141 100 NATIVE 4K DLSS OFF 4K DLSS ON
Slide 25: Real-time Path Tracing
Real-time Path Tracing Opportunity for New Fixed-function Hardware Roadmap Many-light Sampling BSDF Sampling or Path Guiding Terminate Paths into Lighting Approx. Efficient Long Specular Paths (RTXGI, Neural Light Cache, PSTF, …) through Glass/Mirrors etc. Adaptive Sampling Random Numbers DL Denoisi
Slide 26: Direct Neural Image Synthesis
Direct Neural Image Synthesis Random Numbers in → Image out Director: David Luebke
Slide 27: Exploding Model Complexity
Exploding Model Complexity 30,000x in 5 Years | Now Doubling Every 2 Months 1.E+04 GPT-3 1.E+03 Megatron-BERT Megatron-GPT2 Turing NLG 1.E+02 PetaFLOPS - Days GPT-2 1.E+01 BERT 1.E+00 ResNet 1.E-01 AlexNet 1.E-02 1.E-03 2012 2013 2014 2016 2017 2018 2020
Slide 28: Single-chip Inference Performance
Single-chip Inference Performance 317x in 8 Years 1400.00 1248.00 A100 1200.00 1000.00 Int 8 TOPS 800.00 600.00 400.00 261.00 P100 200.00 K20X M40 125.00 Q8000 3.94 21.20 V100 6.84 0.00 April August December May September February June October 2012 2013 2014 2016 2017 2019 2020 2021
Slide 29: Generative adversarial network
Fake Image Latent Variable Generator Discriminator Real or Fake? Real Images
Slide 30: StyleGAN
StyleGAN Latent z є ζ Latent z є ζ Synthesis network g Noise Normalize Normalize Const 4x4x512 Fully-connected Mapping Network ƒ Style + B A AdaIN PixelNorm FC Conv 3x3 Conv 3x3 FC + B Style PixelNorm FC A AdaIN 4x4 4x4 FC PixelNorm Upsample FC Conv 3x3 Conv 3x3 FC Style + B PixelNorm FC A AdaIN Con
Slide 31: NVIDIA AI Video Compression
Sender Keyframe Webcam Keypoint Extraction Receiver Keyframe fh Neural Network Keypoints Output NVIDIA Al Video Compression
Slide 32: NVIDIA AI Video Compression
Sender Keyframe Webcam Keypoint Extraction Receiver Keyframe fh Neural Network Keypoints Output NVIDIA Al Video Compression
Slide 33: Slide 33
Slide 34: NVIDIA Jarvis Multimodal Conversational AI Services Framework
NVIDIA Jarvis Multimodal Conversational AI Services Framework PRE-TRAINED MODEL RE-TRAIN NVIDIA JARVIS Transfer Learning Dialog Manager NeMo Multi-Speaker Gesture Chatbot Look to Talk Transcription Recognition Service Maker NLU Vision NVIDIA GPU CLOUD NVIDIA AI TOOLKIT Speech JESSICA: What will you
Slide 35: Jarvis demo
jarvis Go an open grassland on a cloudy day
Slide 36: Megatron-CNTRL story generator
&* Applications a file Edit View Places System @ gz Search Terminal Tabs Help Terminal mpatwary@cpu0727-... % mpatwary@selene-lo... 3 mpatwary@selene-lo... % mpatwary@circe-logi... Welcome to Megatron-CNTRL, a controllable story generator Please give a start sentence. Press Enter to exit generation
Slide 37: Recommenders: The Personalization Engine of the Internet
Recommenders: The Personalization Engine of the Internet Item User Item User Embedding Embedding DIGITAL CONTENT E-COMMERCE 2.7 Billion 2 Billion Monthly Active Users Digital Shoppers O(109) Candidate O(102) O(10) Recommended Items Ranking Generation Items SOCIAL MEDIA DIGITAL ADVERTISING 3.8 Billio
Slide 38: NVIDIA Merlin
NVIDIA Merlin Democratizing Large-scale Deep Learning Recommenders 1TB CRITEO DATASET ETL DATA TRAINING INFERENCE LOADER 3hr 0(1000) 0(10) CANDIDATE RANKING HugeCTR GENERATION NVTabular NVTab 0(Billions) 1hr EMBEDDINGS User Query RAPIDS RAPIDS cuDNN Triton 2 min 3 min ETL Training DATA LAKE develope
Slide 39: Single-chip Inference Performance
Single-chip Inference Performance 317x in 8 Years 1400.00 1248.00 A100 1200.00 1000.00 Int 8 TOPS 800.00 600.00 400.00 261.00 P100 200.00 K20X M40 125.00 Q8000 3.94 21.20 V100 6.84 0.00 April August December May September February June October 2012 2013 2014 2016 2017 2019 2020 2021
Slide 40: AI Deep Learning Inference Workflow
AI Deep Learning Inference Workflow Two-part Process Implemented by Multiple Personas Data Scientist | ML Engineer ML Engineer Trained MODEL Models OPTIMIZATION Optimize for Multiple Constraints for High Perf. Inference App Developer MLOps, DevOps Query AI Application INFERENCE Result SERVING Scaled
Slide 41: Triton Inference Server
Triton Inference Server Open-source Software for Scalable, Simplified Inference Serving Dynamic Batching (Real time, Batch, Stream) Per Model Scheduler Query Queues Standard Flexible Model AI Application Result HTTP/gRPC Loading (All, Selective) … Multiple GPU & CPU Model Backends Store Custom Metri
Slide 42: Triton Model Ensemble for Conversational AI
Triton Model Ensemble for Conversational AI Language Model Feature Extraction Acoustic Model Decoder AUDIO IN AI Application NLP – Q&A Waveform Speech Synthesis Generator AUDIO & TEXT OUT Orchestrates Models & Data Movement in the Pipeline | Transparent to AI Application
Slide 43: NVIDIA Clara Discovery
NVIDIA Clara Discovery SEARCH O(1060) Chemical Compounds RAPIDS O(109) GENOMICS STRUCTURE DOCKING SIMULATION IMAGING Biological Drug Target Candidate O(102) Schrodinger Clara Parabricks CryoSPARC, Relion AutoDock Clara Imaging NAMD, VMD, OpenMM RAPIDS AlphaFold RAPIDS MELD MONAI NLP Literature Real
Slide 44: AI Turbocharging Structure, Screening, and Simulation
AI Turbocharging Structure, Screening, and Simulation Turning Months into Minutes Alphafold DeepChem AI AI Minutes New New Deep Learning Deep Learning CryoSparc Protein Structure Prediction TorchANI Virtual Drug Screening HPC + AI HPC + AI Days 12x 100x 12 Days vs 5 Months Screen 10 Million Drugs Fo
Slide 45: NVIDIA Clara Parabricks
NVIDIA Clara Parabricks Improving Accuracy, Speed, and Scale of Computational Genomics NVIDIA CLARA PARABRICKS PIPELINES GPU ACCELERATED VARIANT CALLERS EXECUTION TIME IN HRS GERMLINE SOMATIC RNA-SEQ 40 30 LIBRARIES Alignment Variant Calling Variant Processing 20 Preprocess Joint Genotyping Quality
Slide 46: Slide 46
Slide 47: Slide 47
Slide 48: Manipulating Unknown Objects
Manipulating Unknown Objects Most Existing Work Assumes 3D Models of Objects for Manipulation Train Deep Networks in Sim to Generate Variety of Successful Grasps on Unknown Objects [Mousavian-Eppner-Fox: ICCV-19], [Danielczuk-Mousavian-Eppner-Fox: 2021] | [Murali-Mousavian-Eppner-Paxton-Fox: ICRA-20
Slide 49: Slide 49
Slide 50: Slide 50
Slide 51: End-to-End Platform for AV + IX Development
End-to-End Platform for AV + IX Development COLLECT DATA TRAIN MODELS SIMULATE DRIVE AV DRIVE IX DRIVE RC
Slide 52: World Class Neural Networks
World Class Neural Networks Perception | Mapping | Planning | Interior Sensing OBSTACLES DISTANCE TIME TO COLLISION FREE SPACE LANES LIDAR HIGH BEAM PARKING CAMERA BLINDNESS INTERSECTION TRAFFIC LIGHTS GESTURES/POSE GAZE PREDICTION (RNN}
Slide 53: NVIDIA DRIVE with New Orin and Ampere
NVIDIA DRIVE with New Orin and Ampere 5W to 2,000 TOPS — One Programmable Architecture ADAS L2+ L5 Windshield NCAP Autopilot Robotaxi 10 TOPS, 5W 200 TOPS, 45W 2,000 TOPS, 800W SINGLE SCALABLE ARCHITECTURE — SOFTWARE COMPATIBLE
Slide 54: Single-chip Inference Performance
Single-chip Inference Performance 317x in 8 Years 1400.00 1248.00 A100 1200.00 1000.00 Int 8 TOPS 800.00 600.00 400.00 261.00 P100 200.00 K20X M40 125.00 Q8000 3.94 21.20 V100 6.84 0.00 April August December May September February June October 2012 2013 2014 2016 2017 2019 2020 2021
Slide 55: Efficient DL Inference Accelerators
Efficient DL Inference Accelerators MAGNet DEEP LEARNING DL HARDWARE MODELS ACCELERATOR RC18: 9 TOPS/W, 0.32-128 TOPS MAGNet: 29 TOPS/W DL Inference DL Inference Accelerator Accelerator Generator (7nm) 36-die MCM in 16nm [VLSI 2019, HotChips 2019, JSSC 2020, MICRO 2019] | [ICCAD 2019]
Slide 56: MAGNet
MAGNet AXI-In Weights Input Activation CrossPE-Accumin AXISlave WEIGHT BUFFER INPUT BUFFER PE PE PE R R R Address Address Weight Collector Generator Generator PE PE PE R R R VectorSize Controller Buffer Buffer Manager Manager Wb Ab Wb Ab Wb Ab Global Controller X X X PE PE PE R R R NLanes Vector MAC
Slide 57: Dataflow Options
Dataflow Options Weight Buffer Weight Buffer Input Buffer Input Buffer Temporal Temporal weight reuse Weight Collector weight reuse Weight Collector Vector MAC Vector MAC Vector MAC Vector MAC Vector MAC Vector MAC Temporal partial Accum. Buffer Accum. Collector Temporal partial sum reuse sum reuse
Slide 58: Multi-level Dataflows
Multi-level Dataflows Weight Stationary (WS) Output Stationary (OS) Reduce Accum. Reduce Weight Buffer Accesses Buffer Accesses Weight Buffer Weight Buffer Input Buffer Input Buffer Temporal Temporal Weight Collector Weight Collector weight reuse weight reuse Vector MAC Vector MAC Vector MAC Vector
Slide 59: Multi-level Dataflows
Multi-level Dataflows VectorSize=16, IAPrecision=8, Wprecision=8 Input Buffer Weight Buffer Accumulation Buffer Datapath 100% 80% 70 fJ/MAC 60% 35 fJ/OP 29 TOPS/W 40% 20% 0% WS OS WS-LOS OS-LWS
Slide 60: Long Reach Signaling
Long Reach Signaling ELECTRICAL DGX OPTICAL DGX NVHS Si Photonics 8pJ/b 0.3m 4pJ/b 20m-100m
Slide 61: Photonics Link
Photonics Link Electrical Drivers Electrical Receivers LASER COMB SOURCE TX RX 8-10 Laser Lines Micro-rings Tuned to Laser Lines Micro-rings Tuned to Laser Lines 100ghz Separation on Single Fiber On-off Modulated at 25gbps Each Channel Coupled to Drop Port Modulated Light Detected by Photodetector I
Slide 62: Co-packaged Photonics
Co-packaged Photonics Optical Engine (OE) OE OE HBM HBM s GPU s OE Switch OE 24 NVLINKs OE OE 4.8Tbps per Direction OE 24 Laser Fibers 24 TX Fibers DRA M GPU EIC EIC 24 RX Fibers Switch Microbump TSMC Interposer PIC PIC Bump ~5mm x ~10mm Organic Package Organic Package Ball PCB PCB
Slide 63: Co-packaged Photonics: System Concept
Co-packaged Photonics System Concept GPU TRAY NVSWITCH TRAY
Slide 64: Legate Programming System
Legate Programming System Couple Convenient Notation with Accelerated Libraries via an Advanced Runtime System try: import legate.numpy as np except: import numpy as np FAMILIAR DOMAIN-SPECIFIC INTERFACES Legate def cg_solve(A, b): x = np.zeros(A.shape[1]) r = b - A.dot(x) p = r DATA-DRIVEN TASK RUN
Slide 65: Scalable Execution
Scalable Execution Achieve High-performance Execution at Any Scale with Sufficient Data JETSON NANO DGX A100 DGX SUPERPOD
Slide 66: Example: Jacobi Iteration
Example: Jacobi Iteration 103 Legate CPU NumPy 102 1.5 GB 400 GB Legate GPU Intel (ML) NumPy matrix matrix Dask Array Tuned CuPy 101 Throughput (iterations/s) A = np.random.rand (N,N) 100 b = np.random.rand (N) x = np.zeros (b.shape) d = np.diag (A) 10-1 R = A – np.diag(d) 13860 19600 27566 39204 55
Slide 67: Example: Jacobi Iteration
Example: Jacobi Iteration 103 Legate CPU NumPy 102 Legate GPU Intel (ML) NumPy Dask Array Tuned CuPy 101 Throughput (iterations/s) 100 10-1 13860 19600 27566 39204 55696 78766 111392 157532 222784 (1 Sockets) (2 Sockets) (4 Sockets) (8 Sockets) (16 Sockets) (32 Sockets) (64 Sockets) (128 Sockets) (2
Slide 68: CUDA libraries and platforms
; : Ine CUDA nd | cuSowver | IVA DRIVE ROBOTICS NVSHMEM I t t if
Slide 69: NVIDIA
NVIDIA