Nxp Semiconductors N.v. · NASDAQ:NXPI
Nxp Semiconductors — Technical Presentation (September 2023)
54 slides from Nxp Semiconductors's technical presentation, September 2023
Slide 1: Deep learning at the edge
Deep learning at the edge NXP Tech Day - Sep. 2023 Gianfranco Comune S r. F i e l d Ap p l i c a t i o n s E n g i n e e r - Ar r o w E l e c t ro n i c s NXP Dedicated Specialist - Processing and Operating Systems – AI/ML – Embedded Boards Gianfranco.comune@arrow.com Arrow Proprietary and Confident
Slide 2: Arrow Electronics, Inc.
Arrow Electronics, Inc. A global technology provider of products, services, and solutions Serving original equipment Serving channel ecosystems of value-added resellers, Serving original equipment manufacturers, manufacturers, and embedded system integrators, service providers, telcos, ISVs, indepen
Slide 3: Sensor to Sunset Solutions
Sensor to Sunset Solutions Markets Served: ✓ Military Marketing eCommerce ✓ Aerospace Components ✓ Medical Engineering Supply Chain Support Management ✓ Commercial Or Or ✓ IoT Start Here Here… Here ✓ Industrial Global ✓ Automotive Manufacturing Distribution Logistics ✓ Lighting Cloud Reverse ✓ Telec
Slide 4: i.MX 6, 7, 8, 9 Series IoT Secure Elements
U N I Q U E LY P O S I T I O N E D T O D E L I V E R F U L L S Y S T E M S O L U T I O N S COMPREHENSIVE SCALABLE PROVEN TRUSTED ECOSYSTEM CONNECTIVITY MCU & MPU PORTFOLIO SECURITY SOLUTIONS PARTNERS PORTFOLIO i.MX 6, 7, 8, 9 Series IoT Secure High-performance, multimedia Elements MPUs LayerscapeTM
Slide 5: Amplify market deployment with NXP's scalable edge processing continuum
AMPLIFY MARKET DEPLOYMENT WITH NXP’S SCALABLE EDGE PROCESSING CONTINUUM Scalable/reusable software, tools and solutions Based on Arm® technology With Industrial-grade reliability, security, longevity CONFIDENTIAL 5
Slide 6: Definitions and Concepts
Definitions and Concepts
Slide 7: Machine Learning??
Machine Learning?? What do those dots represent?? What are the inputs to this??? What is the output?? 7
Slide 8: AI & ML
AI & ML ❖ Artificial intelligence (AI) • Science involved in creating computer systems trained to “sense” its environment, make decisions, and take action. • large sets of data • Is a huge field which includes many other scientific disciplines ➢ building machines that can make decisions similar to h
Slide 9: ML & DL
ML & DL Deep [machine] learning: A subset of machine learning • modeled loosely on the neural pathways of the human brain. • Based on the concept of Artificial or Deep Neural Networks (ANNs / DNNs) • Deep refers to the “many” layers between the input and output layers, but not the defining feature.
Slide 10: Simplified Classical ML Model
Simplified Classical ML Model 10
Slide 11: Feature Extraction on Classical ML Model
Feature Extraction on Classical ML Model Spatial Filter (i.e. Sobel) Typically not efficient for imaging Can be very efficient with low volumes of data (1D sensors, some audio) 11
Slide 12: Simplified Neural Network Model
Simplified Neural Network Model Input Output Layer Hidden Layer(s) Layer ‘y’ pixels 0.01 Armadillo Dog ‘x’ pixels 0.05 0.93 Cat 0.01 Dodo Node Value ax=ReLU(w1a1+w2a2+w3a3….-b) Bias Nodes Weights (wx) Model 12
Slide 13: Vision Based DL Applications
VISION BASED DL Applications Classification Instance Image Classification with Localization Object Detection Segmentation Apple Apple Apples and Orange Apples and Orange Facial Detection Facial Recognition Excellent also with complex audio applications (i.e. NLP) Person’s face Kurt 13
Slide 14: What is AI?
What is AI? Classical Machine Learning Reinforcement Learning https://www.guru99.com/artificial-intelligence-tutorial.html 14
Slide 15: Machine Learning -> NOW YOU KNOW!!!
Machine Learning -> NOW YOU KNOW!!! 15
Slide 16: Models & Frameworks
MODELS & FRAMEWORKS
Slide 17: Models & Framework
Models & Framework ≈ MODEL or ≈ FRAMEWORK (or Format or Implementation Theory / concept (i.e. library) 17
Slide 18: Machine Learning Models
Machine Learning Models • Models are a mathematical / conceptual representation • Algorithm + data -> a specific algorithm with given ***data*** to reach a decision − Data is input + parameters • Neural Network model • made up of many nodes -> “neurons” - organized into multiple layers − Each layer
Slide 19: Frameworks
Frameworks • There are several popular model frameworks in use today. TensorFlow Framework offered by Google Keras Higher level API, usually built on top of TensorFlow PyTorch Framework offed by Facebook • TensorFlow Lite/micro: Faster & smaller than TensorFlow. Supports a subset of the most common
Slide 20: More Options for Deploying Trained NN Models
More Options for Deploying Trained NN Models • OpenCV: Image processing, video encoding/decoding, video analysis, object detection, and processing of neural networks. Utilizes Arm core / NEON for acceleration by default; can target GPU via OpenCL for direct hardware access. • Arm NN: Open-source sof
Slide 21: Universal Framework Translators
Universal Framework Translators There are also universal framework “formats” like ONNX and NNEF • Convert different frameworks into a “standard” format 21
Slide 22: Real Life Implementations
REAL LIFE IMPLEMENTATIONS TENSORFLOW ▪ TENSORFLOW APIs ▪ TENSORFLOW MODEL SUPPORT KERAS ▪ KERAS APIs ▪ https://keras.io/ ONNX ▪ ONNX MODEL ZOO 22
Slide 23: Performance & Optimization
Performance & Optimization
Slide 24: Model Accuracy
Model Accuracy • Every time an inference is run, most models will give a confidence percentage. − This is why last (or one of last) layers in a model is often “softmax” which converts final result to be between 0 and 1. − Always < 1 • Every model has limitations. • There’s no one size fits all or pe
Slide 25: Models are Not Perfect
Models are Not Perfect Model decoder ring Full Name: MobileNet_v1_1.0_224 MobileNet – model used v1 – version of model 1.0 – “size” of model 224 – pixel size of images used for training (224x224) COURTESY OF NXP 25
Slide 26: Transfer Learning
Transfer Learning • Gathering and labeling a dataset as large as ImageNet and designing a complicated neural network is very time consuming, difficult, and expensive. ➢ This is where Transfer Learning steps in! • Replacing the output layer or multiple last layers with layers representing the new tas
Slide 27: Training Accuracy
Training Accuracy AWOW 95.41% 90.41% 85.41% 80.41% 75.41% 70.41% 65.41% 60.41% 55.41% Training Accuracy Comparison for 300 examples per digit =< = With Transfer Learning ———= Without Transfer Learning eeeeee Original MNIST-Trained Model 123 45 67 8 9 101112 13 1415 16 17 18 19 20 21 22 23 24 25 26 2
Slide 28: Quantization
Quantization • Quantization is transforming floating point weights into integer weights − Reduces model size (i.e., by 4x If going from 32-bit FP to 8-bit Int) − Fixed point math much quicker than floating point − Usually results in little loss of accuracy − Uses min/max of floating-point values and
Slide 29: Things That Affect Model Accuracy
Things That Affect Model Accuracy • Quality & Quantity of input training data • Model Structure and Training Method • Quality of input test data • Efficiency of model conversion for running on embedded system − Quantization, Pruning, Compression, etc. 29
Slide 30: Sparse Matrix and Zero-valued Operations
Sparse Matrix and Zero-valued Operations • Node Value ax=ACTIVATION(w1a1+w2a2+w3a3….+b) • Sparse matrix – a matrix with many 0s. • Inputs/Weights of “0”! −These should not be scheduled ▪ Saves power and time −These should not be fetched and cached ▪ Saves power, time and memory space • Encoding sche
Slide 31: Pruning
Pruning • Pruning is removing unused or low importance weights and biases from a neural network − Recommended to retrain model after pruning • Note: benchmarks are often run on heavily quantized and pruned models. It’s important to know this when comparing to ensure apples-to-apples results • Other
Slide 32: Summing It All Up!!
SUMMING IT ALL UP!! 1. ML/DL is nothing more than millions of complex vector/matrix multiplications • Similar to DSP concepts • Just with many millions of parameters (input, intermediates, weights, biases) 2. The “Feature” extraction process defines the technique to be used 3. A model is an algorith
Slide 33: ML/DL Implementation
ML/DL Implementation
Slide 34: Training / Common Concepts
TRAINING / COMMON CONCEPTS TRAINING TYPES • Supervised - labeled datasets to train algorithms • Unsupervised – uses algorithms to analyze and cluster unlabeled datasets • Semi-supervised – Hybrid of both TRAINING ISSUES • Underfitting – Model is too simple; low accuracy • Overfitting – Model is too
Slide 35: Machine Learning Development for embedded (Supervised)
Machine Learning Development for embedded (Supervised) Trial & Error Big Iron, but better on Embedded on big Iron embedded system System USER DATA TRAINING VALIDATION TESTING USER FIELD DATASET DATASET DATASET DATA TRAIN VALIDATE TEST DEPLOYMENT INFERENCE BASE TRAINED TRAINED DEPLOYED MODEL MODEL CU
Slide 36: Training Phase
1 -> Training Phase ➢ Training a model is very compute and time intensive ➢ Typically done on ”big iron” a.k.a. data center servers ➢ Involves trial and error on parameters (i.e. weights and biases) used in calculations ➢ Due to randomness in the initial parameters, the result can be slightly differ
Slide 37: Inference Phase
2 -> Inference Phase • Inference is using a model to perform predictions on user data • 3 mainstream trends 1) Upload an image to cloud and evaluate on cloud platform – Requires network bandwidth – Latency issues – Cloud compute costs – Potential privacy and security concerns 2) Evaluate image on em
Slide 38: CNN Deep Dive
\/ | Five Years Out
Slide 39: CNN Model(s) details for Computer Vision
CNN MODEL(S) details for Computer Vision Only one output!! (…) CLASSES Fully Connected 39
Slide 40: AlexNet: A revolution in DL in 2012
ALEXNET: A revolution in DL in 2012 Caffe AlexNet TensorFlow AlexNet TensorFlowLite AlexNet ONNX AlexNet ALEXNET Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks 40
Slide 41: AlexNet: but it's so old!!!
ALEXNET: but it’s so old!!! GoogleNet ResNET-50 VggNET-16 Inception 41
Slide 42: Deep Learning: is it only for vision???
Deep Learning: is it only for vision??? ***NOTE *** Classical Machine Learning works very well with sensory applications on the edge. 42
Slide 43: Hardware Deep Dive
HARDWARE DEEP DIVE
Slide 44: Why Machine Learning is Moving to the Edge
Why Machine Learning is Moving to the Edge Better User Safeguard Privacy Experience Higher Reliability Enable Real Time Increase Security Reduce Cloud Costs Including: Bandwidth, Processing, Storage 44 44
Slide 45: Machine learning use cases, data inputs and compute accelerators
Machine learning use cases, data inputs and compute accelerators Cortex-M 4x Cortex-A53 GPU Neural Processing Unit (NPU) 0 0.25 0.5 1 2 3 (TOPS) Face and still image Multi-object surveillance Vision Multi-face recognition, Live video face and recognition, person (people, cars, animals) object detect
Slide 46: Footnote on Hardware
FOOTNOTE ON HARDWARE CPU (SIMD)* GPU DSP VPU NPU / TPU Highly programmable / Very programmable / Programable but very Limited programmability – Limited programmability Capable of many types of good number of type of limited – MACs typical specialized functionality for – specialized ops opps Computer
Slide 47: NXP ML Solutions
\/ | Five Years Out
Slide 48: i.MX RT1170/1160 Crossover MCU
i.MX RT1170/1160 CROSSOVER MCU • Key Features ▪ Arm Cortex-M7 processor, 1 GHz, 32KB/32KB L1 Cache, 512KB TCM ▪ Arm Cortex-M4 processor, 400MHz, 16KB/16KB L1 Cache, 256KB TCM ▪ 2MB on-chip SRAM (including TCM for CPU core) ▪ Parallel LCD Display up to WXGA (1280x800) @60fps ▪ 8/16/24-bit Parallel Ca
Slide 49: MCX
System ontrol ore Platform Interfaces 0x P lexcomm MCX ys. D T P C ingle DDpower supply P D/ D DCDC converter Core D I D T witch Trust one rm Cortex p to 0 P P z I D D P xC N D upports T PI I C xI C u D C P thernet rm Cortex Core Platform and Accelerators Clock eneration nit p to 0 z HMI xP s • DSP
Slide 50: i.MX 8M Plus Applications Processor
i.MX 8M Plus Applications Processor Feature Highlights: • Quad Arm® Cortex-A53 up to 1.8 GHz i.MX 8M Plus • Arm v8 fully 64-bit capable, 512KB L2 cache (ECC) • Arm Cortex-M7 up to 800MHz with 512KB RAM (ECC) • Voice Acceleration co-processor: HiFi 4 DSP up to 800MHz with 256KB RAM (ECC) • Machine Le
Slide 51: i.MX 93
11 mm x 11 mm pkg i.MX 935x/933x 14x14 mm TBC System Clock Main CPU External DRAM Oscillator x16 LPDDR4X (Inline ECC) 2x Cortex-A55 i.MX 93 PLLs 32 kB I-cache NEON 32 kB D-cache 64 kB L2 Cache FPU 256 kB L3 Cache (ECC) 1x/2x Arm Cortex-A55 @ 1.7 GHz • Arm v8.2 Fully 64-bit capable, 384kB total L2+L3
Slide 52: NXP Implementations
ASW root@imx8mpev! total 14M -rwxr-xr-x 1 -Fwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -Pwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -Pwxr-xr-x 1 -rwxr-xr-x 1 -Pwxr-xr-x 1 -rwxr-xr-x 1 -PWxr-xP-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -Pwxr-xr-x 1 -rwxr-xr-x
Slide 53: eIQ ML SDK
elQ ML SDK NXP SIG ML Sofware Development Emvronment Inference Engines and Libraries for Neural Network Model Deployment Af OormnNn DeepView|(! DeepView!{! “© TensorFlow| ite for Microcontrollers TensorFlow! ite Osc.tow the ONNX iMX RT1064 iMX RT600 MX RT6OO i.MX 8M Plus i.MX 8M Plus i.MX 8M Plus i.
Slide 54: Thank You
Thank You Gianfranco.comune@arrow.com Russell Pead – rpead@arrow.com Jim Picariello – Jim.picariello@Arrow.com 54
NXP Semiconductors — Technical Presentation (September 2023) on 6ix