Slide 1: Efficient Video Perception through AI
January 19, 2021 @QCOMResearch Efficient Video Perception through AI Qualcomm Technologies, Inc.
Slide 2: Agenda
Agenda • The role of video in our lives • What is video perception & what makes it challenging • Our research toward efficient video perception • Forward looking video perception research 2
Slide 3: A picture is worth a thousand words
A picture is worth a thousand words Out of all the five senses, vision is arguably the most important 3
Slide 4: A minute of video has more than 1,000 pictures
A minute of video has more than 1,000 pictures 4
Slide 5: How video came to be
How video came to be From performing arts and still photos to high-quality video Greek Sanskrit Polynesian Chrono- Motion Color Internet video theater plays hula photography pictures analog TV broadcasting Early ages 1650s 1880s 1890s 1900s 1920s 1938 1990s 1995 8K HDR Chinese Motion video shadow Ro
Slide 6: The scale of video being created and consumed is massive
The scale of video being created and consumed is massive 1M Minutes of video 82% Of all consumer 76 Minutes per day watching 8B Average daily 300 Hours of video are crossing the internet internet traffic is video on digital devices video views uploaded every per second online video by US adults on F
Slide 7: Increasingly, video is all around us
Smartphone Sports Video conferencing Autonomous vehicles Smart factories Extended reality Smart cities Video monitoring Increasingly, video is all around us — providing entertainment, enhancing collaboration, and transforming industries 7
Slide 8: Video perception
Video Making Systems Understand Developing Any compute Recognizing patterns, perception mathematical representations, platform, including SoCs, CPUs, identities, objects, scenes, context, relations, Making systems models, algorithms, GPUs, TPUs, NPUs, and DSPs compositions, changes, motions, actions
Slide 9: Video perception challenges
Data Implementation challenges challenges Diversity in Volume of video data visual data (training/testing) Quality of data acquisition Video Platform limitations perception Availability of challenges Task annotated datasets diversity What makes video perception challenging? 9
Slide 10: The challenge of AI workloads
The challenge of Constrained mobile AI workloads environment Very compute Must be thermally intensive efficient for sleek, ultra-light designs Large, complicated neural network models Power and thermal Complex concurrencies efficiency are essential Requires long battery life for all-day use for on-d
Slide 11: Making video perception ubiquitous
Making video perception ubiquitous Solving additional Robustness Adaptability Scalability key challenges to Robust to data variations Adaptable to different domains Scaling up and down, take video perception from IoT to the data center from the research lab to broad commercial deployment 11
Slide 12: Key concepts for efficient video perception
Leverage Make Temporal Early redundancy Key concepts decisions By reusing what for efficient By dynamically changing is computed before • Learning to skip regions video perception the network architecture per input frame • Recycling features • Early exiting • Frame exiting Efficiently running on-dev
Slide 13: Deep learning basics
Deep learning Feed-forward CNN basics Fully-connected layer of neurons Computable AI Convolutional layer Pooling Data driven learning 2 • Supervised, unsupervised, semi/self/weak supervised, adversarial output Training data Image Feature Neural network map Vectorized features • Convolutional neural
Slide 14: Learning to skip redundant computations
Learning to skip redundant computations Video frames are heavily correlated frame t frame t+10 residual The residual frame, the difference between two consecutive frames, contains little information “Skip-convolutions for efficient video processing” (submitted 2021) in most regions Limit the computa
Slide 15: Skip-convolution
Convolutional layer Skip- convolution Conv Input Output features features A convolutional layer with a skip gate that masks out negligible residuals A convolution at a frame can be written as the previous frame’s convolution plus the Previous frame convolution of the residual output features No comp
Slide 16: Determining the gate for a skip convolution
Determining the gate for a skip convolution Can we learn it? Non-trainable Trainable Based on thresholding Based on a tiny gating the norm of the network that predicts convolved residual gate probabilities Approximate the norm of output Implement as a convolution with without computing it a single o
Slide 17: Learned gate requires very low computational overhead
Learned gate requires very low computational overhead Gate network is an additional output channel No computation Computation Cout Feature map Win Wout Skip convolutions Gating scores Gate mask “Skip-convolutions for efficient video processing” (submitted 2021) Hardware friendly implementation enfor
Slide 18: Predicted gating masks reduce computations
Predicted gating masks reduce computations Example masks show where computations can be skipped Gating masks become more selective at deeper Larger block-wise structured gates are more layers, concentrating on task specific regions selective when training with different sizes Computation No computat
Slide 19: Learning to skip reduces compute and maintains accuracy
Learning to skip reduces 0.65 Object detection compute and maintains accuracy 0.6 Average precision SpotNet Results for object detection on Deep Feature Flow video object detection dataset EfficientDet Ours-EfficientDet 0.55 3x- 5x speed-up over state-of-the-art 0.5 0.1 1 10 100 1000 GMAC “Skip-conv
Slide 20: Learning to skip reduces compute for human pose estimation
Learning to skip reduces compute for human pose estimation 10.2 10.2 10.2 0.4 10.2 0.4 10.2 0.4 10.2 0.5 Results for human pose estimation GMACs without skip convolutions GMACs with skip-convolutions 10.2 10.2 10.2 0.7 10.2 0.8 10.2 0.7 10.2 1.1 “Skip-convolutions for efficient video processing” (su
Slide 21: Learning to skip is complementary to model compression
Learning to skip is 0.96 Pose estimation complementary Percentage of correct keypoints to model 0.95 compression 0.94 Results for human 0.93 pose estimation on HRNet-w32 video human action dataset L0 0.92 W-SVD S-SVD Ours-Skip 0.91 2.5x-8x Ours-Skip+S-SVD speed-up over 0.9 HRNet 0 2 4 6 8 10 GMAC “S
Slide 22: Recycling features saves compute
Recycling features saves compute Instead of computing deep features repetitively, compute once and recycle Deep features remain relatively stationary Compute deep features once and over time — they have lower spatial resolution recycle — reuse from past frame Shallow features are more responsive to
Slide 23: Example of a feature recycling network
Example of a feature recycling network Making the best use of the deep features at hierarchical scales Compute Frame t Frame t (Key frame) H Frame t+1 Transition Transition Transition C Ψ Next key frame Time Transition Convolution Convolution Based Multi-scale Stride=2, Convolution Block Ψ Decoder B
Slide 24: Example of a feature recycling network
Example of a feature recycling network Most frames require a light branch; some frames require a heavy branch Compute Frame t Frame t (Key frame) H Frame t+1 Transition Transition Transition C Ψ Branch Decision Frame t+1 Feature Feature Feature Transform Transform Transform H Next key frame + + C Ψ
Slide 25: Recycling features saves compute
Recycling features saves compute Instead of computing deep features repetitively, compute once and recycle Raw video obtained from Cityscapes Benchmark: https://www.cityscapes-dataset.com/ “Time-sharing networks for efficient semantic video segmentation” (submitted 2021) 25
Slide 26: Feature recycling reduces compute and latency
HRNet w18 v2 Enhanced Net Model efficiency GMACs On-device latency (ms/frame) Feature recycling 77.9 74.0 reduces compute and latency 17.0 78% GMAC reduction 26.0 65% latency reduction Semantic segmentation example Input: 2048x1024 RGB video Memory traffic Output: 2048x1024, MB read MB write 19 obje
Slide 27: Early exiting a neural network saves compute
Early exiting a neural network saves compute Exploit the fact that not all input examples require models of the same complexity Complex Very large, computationally intensive examples models are needed to correctly classify Simple Very small and compact models can achieve very examples high accuracie
Slide 28: Early exiting at the earliest possible NN layer for video
Early exiting at the earliest possible NN layer for video Exit NN layer Frame t-1 Frame t Exit NN layer G1: Conv Gating based on temporal Feature Feature FC Classifier G2: agreement from t-1 from t Gating based on frame complexity G1 Not Continue G2 Not Continue similar confident Similar Confident A
Slide 29: Early exiting applies to most neural network backbones
Classification on image dataset 0.945 0.94 0.935 Top-1 accuracy Early exiting 0.93 applies to most RANets1 neural network 0.925 RANets2 backbones RANets3 0.92 MSDNet Ours-MSDNet DenseNet 0.915 Ours-DenseNet ResNet Ours-ResNet 0.91 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 MAC (x108) Early exiting reduces
Slide 30: Early exiting for object classification
Early exiting for object Object classification on video street scene dataset classification 0.8 0.795 0.79 Top-1 accuracy 2.5x less MACs while 0.785 maintaining 0.78 ResNet34 accuracy ResNet34-Uniform2 Ours-ResNet34-G1 0.775 Ours-ResNet34-G1/G2 0.77 4 6 8 10 12 14 16 18 20 GMAC “FrameExit: Condition
Slide 31: Frame exiting also applies to action recognition tasks
Frame exiting also applies to action recognition tasks Policy Function Frame Frame feature feature Frame feature Frame feature Frame feature Frame feature Gating module Accumulated Accumulated Accumulated Accumulated Feature Pooling Feature Pooling Feature Pooling Feature Pooling 2-layer MLP Shared
Slide 32: Frame exiting improves accuracy and reduces compute
Frame exiting improves accuracy and reduces compute Video activity dataset Video action dataset Methods mAP (%) GFLOPS Top-1 (%) GFLOPS Resnet Policy AdaFrame 71.5 79.0 — — Function LiteEval 72.7 95.1 61.0 99.0 ListenToLook 72.3 81.4 — — SCSampler 72.9 41.9 70.8 41.9 AR-Net 73.8 33.5 71.7 32.0 Frame
Slide 33: Frame exiting for video classification
Frame exiting for video Video classification on video activity dataset classification 80 mean Average Precision (mAP) 75 1.3x-5x 70 AdaFrame5 less GFLOPs while ARNet-ResNet maintaining ARNet-EfficientNet accuracy 65 L2L-Mnet LiteEval Ours-ResNet Ours-EfficientNet 60 5 25 45 65 85 105 GFLOPs “FrameEx
Slide 34: Future work in video perception
Advance Develop efficient existing conditional video neural network compute techniques solutions Unsupervised / semi- supervised learning Efficient sparse Learning to skip regions convolutions Recycling features Personalization Future work Early exiting in video Multi-task networks Frame exiting per
Slide 35: Our perception research is much broader than video
Research Areas Applications Computer vision XR Leading high-impact Camera 3D Sensing research efforts in perception Machine Autonomous vehicles learning RF Sensing Wi-Fi core Personalization Inventing technology Fingerprint ASP enablers for important applications Mobile Biometrics IOT Data driven mo
Slide 36: Video perception is crucial for understanding the world and making devices smarter
Video perception is crucial for understanding the world and making devices smarter We are conducting leading research and development in video perception We are making power efficient video perception possible without sacrificing accuracy 36
Slide 37: Questions?
www.qualcomm.com/ai www.qualcomm.com/news/onq Questions? @QCOMResearch Connect with Us https://www.youtube.com/qualcomm? http://www.slideshare.net/qualcommwirelessevolution 37
Slide 38: Thank you
Thank you Follow us on: For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog Nothing in these materials is an offer to sell any of the References in this presentation to “Qualcomm” may mean Qualcomm components or devices referenced herein. Incorporated, Qualcomm Technologies,