Slide 1: Explore the intelligent sensor processing unit (ISPU)
Explore the intelligent sensor processing unit (ISPU) ST Public
Slide 2: Artificial intelligence journey
Artificial intelligence journey ST Public
Slide 3: Migrating processing at the edge
Migrating processing at the edge MCU computing In-sensor AI computing Sensor + MCU Sensor with ISPU Intelligence in the MCU Intelligent sensor processing unit ISPU + Sensor MCU MCU Sensor MCU standalone or hosted in the sensor package ISPU integrated in the sensor ASIC Standard Programmable MCU runs
Slide 4: Challenges for sensors in edge AI
Challenges for sensors in edge AI Ability to integrate intelligent processing in a tiny piece of silicon Optimal power (uW) per performance capabilities Limited logic and memory storage Technology architecture Ecosystem compatibility Advanced tools for increased productivity 4 ST Public
Slide 5: One can’t fit all, ISPU fits many
One can’t fit all, ISPU fits many MCU + sensor rPU* ISPU rPU+ ISPU + MCU Sensor MCU MCU Sensor Sensor Parameters Benefits Benefits Benefits Optimized for Flexibility Ad hoc coding inertial data Integrated computing Low power rPU adds few uA cell, MCU in standby with sensor wake up Data transfer Loca
Slide 6: Evolution of system-based AI
Evolution of system-based AI Today ISPU + MCU ISPU integrated in the sensor ASIC ISPU + MCU Sensor ISPU Intelligent sensor processing unit 6 ST Public
Slide 7: What is ISPU?
What is ISPU? ST Public
Slide 8: ISPU: what and why
ISPU: what and why Highly specialized DSP* for machine learning and processing Unique solution for TinyML with machine learning (ML), binary neural network (BNN), and processing capabilities Super tiny silicon Lowest power consumption IoT node in the market with AI at the edge Productivity: empowers
Slide 9: Integrating brains into sensors
Integrating brains into sensors ISPU Programmable core ultralow power Intelligent sensor processing unit Processes data from internal (accelerometer, gyroscope, temperature) & external sensors MEMS MCU + ISPU ISPU toolchain allows developers ISPU integrated in the sensor ASIC to program in C languag
Slide 10: What’s inside?
What’s inside? DSP* for real-time processing and artificial intelligence MEMS sensor 3-axis accelerometer & 3-axis gyroscope Raw data from MEMS sensor Sensor hub to collect data from additional external sensors (up to 4) ISPU locally process data in a few µW ISPU core Only meaningful Standard packag
Slide 11: ISPU: the architecture
ISPU: the architecture DSP for real-time processing and artificial intelligence SPI / I²C Accelerometer Sensor Core Gyroscope Sensor hub I²C Temperature Program Data Interface memory memory registers ISPU core 32 KB 8 KB ISPU RAM RAM 11 ST Public
Slide 12: ISPU: play with it!
ISPU: play with it! DSP for real-time processing and artificial intelligence Small area: enhanced 32-bit RISC RAM based: Harvard architecture in 8 kilogates 40 KB (program + execution) Binary neural network convolution Full precision: Floating point unit accelerator: patented by ST Fast interrupt re
Slide 13: ISPU core: efficient, small, and fast DSP
ISPU core: efficient, small, and fast DSP x2 performance / area efficiency than best competitors ISPU Pipeline stage 4 Instruction/data No conflicts Harvard architecture Multiplier 16 >32 Tiny FPU Yes Area 8 KGE Branch shadow 1 to 2 CoreMark 70 µW/MHz Clock frequency 10MHz 13 ST Public
Slide 14: ISPU added value
ISPU added value Ultra low power consumption • Efficiency of the embedded DSP (digital signal processing) • Very low data exchange with external MCU (MCU stay in sleep mode most of the time) Ultra low latency • Processing / decision taken directly in the sensor Easily programmable with C language or
Slide 15: ISPU products
ISPU products ST Public
Slide 16: Be inspired by the endless possibilities of ISPU
Be inspired by the endless possibilities of ISPU Personal electronics & consumer IoT Industrial IoT • ISM330IS • LSM6DSO16IS • ISM330ISN* *ISM330ISN is supported in NanoEdge AI Studio 16 ST Public
Slide 17: Discover many real-world examples based on ISPU
Discover many real-world examples based on ISPU Calibration Anomaly detection Complex 3D space orientation gestures 17 17 ST Public
Slide 18: Find inspiration to create your personal application
Find inspiration to create your personal application AI at the edge with ultralow power 6-axis IMU for consumer market A completely new level of capabilities and detection accuracy in human activity recognition applications: • Consumer health • Gesture recognition • Activity recognition • Motion tra
Slide 19: A new way to approach the Industry 5.0
A new way to approach the Industry 5.0 AI at the edge with ultralow power 6-axis IMU for industrial market Higher detection accuracy, always on monitoring in anomaly detection applications • Home alarms • Robotics • Condition monitoring A completely new level of capabilities and detection accuracy i
Slide 20: ISPU products
ISPU products For real-time processing and artificial intelligence Optimization Power consumption vs. performance Standard package 3 x 2.5 x 0.83 mm Efficiency • FS: ± 2000 dps, ± 16 g Floating Point, binary NN, hybrid binary NN • Idd 0.59 mA (combo mode) • Axel noise 70µg/√Hz • Gyroscope noise 3.8m
Slide 21: How to program ISPU?
How to program ISPU? ST Public
Slide 22: Free tools for smart sensors with ISPU
Free tools for smart sensors with ISPU ISPU can be configured by different tools available in the ST Edge AI Suite Bring your Bring your OR own data own model AutoML Model Ideation Data Benchmark Deploy End-to-end flow optimization High Speed NanoEdge AI ST Edge AI Case studies STM32Cube.AI Code exa
Slide 23: ISPU software ecosystem
ISPU software ecosystem Who How Which tool ST Edge AI Core AI expert Command Line Interface (CLI) ISPU toolchain MEMS Studio Application developer Desktop tool NanoEdge AI Studio Evaluation engineer Cloud tool ST Edge AI Developer Cloud Take inspiration Use cases Model zoo in ST Edge AI Suite AI exp
Slide 24: ISPU programming steps
ISPU programming steps Edge AI journey C-programming journey ISPU neural network conversion ISPU programming Integrate & Optimize Generate External tool compile for NN model creation C code .json/.h file Upload 1 2 3 & run Eclipse-based IDE MEMS Studio (GUI) MEMS Studio (GUI) e.g. STM32CubeIDE ST Ed
Slide 25: C-Compiler: how to evaluate the ISPU?
C-Compiler: how to evaluate the ISPU? Products DIL 24 adapter Eval boards & kits Software & package LSM6DSO16IS and DIL 24 adapter STM32 Nucleo: MEMS Studio ISM330IS • NUCLEO-F401RE ST Edge AI Core • X-NUCLEO-IKS4A1 (consumer) ST Developer Cloud • X-NUCLEO-IKS02A1 (industrial) X-CUBE-ISPU Profession
Slide 26: ISPU: there is an AI solution for every need
ISPU: there is an AI solution for every need Use cases Company’s profile Other ML Anomaly detection Neural networks classifications Embedded developers • No dataset available • No dedicated AI team Team with AI expertise • Dataset available • AI team ISPU toolchain ISPU toolchain 26 ST Public
Slide 27: ISPU easily programmed with NanoEdge AI Studio
ISPU easily programmed with NanoEdge AI Studio ON THE PC ON THE ISPU 1 Create the library, ONCE. 2 Use the library, MANY TIMES. Create and embed a self-learning engine Standalone PC For anomaly detection, the model is solution (Win/Linux) self-trained at the edge. 27 ST Public
Slide 28: ST partnership for ISPU
ST partnership for ISPU ST Public
Slide 29: Partnership with Neuton.AI for a tiny AI algorithm in the ISPU
Partnership with Neuton.AI for a tiny AI algorithm in the ISPU Neuton.AI – No-code TinyML platform for ISPU Neuton.AI allow to build and deploy ultra-small neural networks directly into ISPU Fixed-point signal processing and inference engine gives 10x advantage in size and processing time when runni
Slide 30: Solutions leveraging Neuton.AI
Solutions leveraging Neuton.AI Human-machine Gesture recognition interface Human activity Machine fault Asset tracking and recognition classification monitoring 30 ST Public
Slide 31: 3 simple steps to make an AI-driven ISPU
3 simple steps to make an AI-driven ISPU Import .csv file of data 1 collected from ISPU Build automatically tiniest 2 neural networks Generate compiled library 3 for ISPU and run inference on sensor 31 ST Public
Slide 32: Neuton.AI business model with ST
Neuton.AI business model with ST Evaluation → Zero gravity • Use of ML platform is free of charge (user to cover GCP services) • Only compiled library as output of the online tool Production → Enterprise plan • End-to-end machine learning based services from data collection to ISPU deployment • Use
Slide 33: ISPU: application examples
ISPU: application examples ST Public
Slide 34: X-CUBE-ISPU
X-CUBE-ISPU Include libraries, source code examples and templates X-CUBE-ISPU application User PC application software Application-level demonstration Libraries Examples Template Place for future libraries Utilities Used as building blocks Modify & rebuild Write your own C for final application exam
Slide 35: ISPU application libraries
ISPU application libraries Accelerometer calibration Gyroscope calibration calibration Magnetometer Calibration Accelerometer Gyroscope Magnetometer algorithm calibration calibration calibration Sensor fusion 6x Sensor fusion 9x Sensor fusion Sensor fusion 6x Sensor fusion 9x Fall detection Wrist ti
Slide 36: Easily enhance your application with ISPU
Easily enhance your application with ISPU Current consumption Accelerometer calibration 11 µA Gyroscope calibration 26 µA Magnetometer calibration 21 µA Wrist tilt 4 µA Man down 14 µA Wrist - human activity recognition 27 µA Sensor fusion - 6 axis 226 µA Sensor fusion - 9 axis 370 µA 36 ST Public
Slide 37: ISPU Sensor fusion
ISPU Sensor fusion Sensor fusion - 6 axis 226 µA Power consumption Use case Demo overview The inertial sensors with ISPU (LSM6DSO16IS / • Sensor fusion algorithm running on the ISPU ISM330IS) can be used for 6-axis sensor • No processing required from MCU fusion Demo setup Configuration & power • Te
Slide 38: ISPU Sensor fusion
ISPU Sensor fusion Sensor fusion - 9 axis 370 µA Power consumption Use case Demo overview LIS2MDL is connected to the sensor hub of the • Sensor fusion algorithm running on the ISPU inertial sensors with ISPU (LSM6DSO16IS / • No processing required from MCU ISM330IS). It can be programmed for 9-Axis
Slide 39: ISPU Accelerometer calibration
ISPU Accelerometer calibration Accelerometer calibration 11 µA Power consumption Use case Demo overview • The inertial sensors with ISPU • Calibration algorithm running on the ISPU (LSM6DSO16IS / ISM330IS) can be • No processing required from MCU programmed to calibrate the accelerometer Demo setup
Slide 40: ISPU Gyroscope calibration
ISPU Gyroscope calibration Gyroscope calibration 26 µA Power consumption Use case Demo overview The inertial sensors with ISPU (LSM6DSO16IS / • Calibration algorithm running on the ISPU ISM330IS) can be programmed to calibrate the • No processing required from MCU gyroscope Demo setup Configuration
Slide 41: ISPU Magnetometer calibration
ISPU Magnetometer calibration Magnetometer calibration 21 µA Power consumption Use case Demo overview LIS2MDL is connected to the sensor hub of the • Calibration algorithm running on the ISPU inertial sensors with ISPU (LSM6DSO16IS / • No processing required from MCU ISM330IS). It can be programmed
Slide 42: ISPU Wrist tilt
ISPU Wrist tilt Wrist tilt 4 µA Power consumption Use case Demo overview • The inertial sensors with ISPU • Wrist tilt algorithm running on the ISPU (LSM6DSO16IS / ISM330IS) can be • No processing required from MCU programmed to detect the wrist movement. • 2 different events can be identified by th
Slide 43: ISPU Fall detection
ISPU Fall detection Fall detection 14 µA Power consumption Use case Demo overview LPS22HH is connected to the sensor hub of • Fall detection algorithm running on the ISPU the inertial sensors with ISPU (LSM6DSO16IS / • No processing required from MCU ISM330IS). It can be programmed to detect a Fall
Slide 44: ISPU Wrist - Human activity recognition
ISPU Wrist - Human activity recognition Wrist - human activity recognition 27 µA Power consumption Use case Demo overview • The inertial sensors with ISPU • Wrist HAR algorithm running on the ISPU (LSM6DSO16IS / ISM330IS) can be • No processing required from MCU programmed for wrist HAR • The ISPU i
Slide 45: ISPU Pedometer
ISPU Pedometer Pedometer 16 µA Power consumption Use case Demo overview • The inertial sensors with ISPU • Pedometer algorithm running on the ISPU (LSM6DSO16IS / ISM330IS) can be • No processing required from MCU programmed to count the number of steps performed by the user having the device on the
Slide 46: Sliding DFT
ISPU Sliding DFT Sliding discrete Fourier transform 7 to 913 µA Power consumption Use case Demo overview The inertial sensors with ISPU (LSM6DSO16IS / • Pedometer algorithm running on the ISPU ISM330IS) can be programmed to decompose a • No processing required from MCU signal into its frequency comp
Slide 47: ISPU Vibration severity
ISPU Vibration severity Vibration severity as per ISO10816 680 µA Power consumption Use case Demo overview The inertial sensors with ISPU (ISM330IS) can • Vibration severity algorithm (ISO10816) be programmed to filter one accelerometer axis running on the ISPU according to ISO10816 and compute the
Slide 48: Anomaly detection with NanoEdge AI
Anomaly detection with NanoEdge AI Possibility to divide the tasks between the sensor and the MCU Edge ERROR 1 OK ERROR 2 NOK ERROR 3 ERROR 4 Sensor Pump MCU Cloud 48 ST Public
Slide 49: Find out more at www.st.com/ISPU
Find out more at www.st.com/ISPU © STMicroelectronics - All rights reserved. ST logo is a trademark or a registered trademark of STMicroelectronics International NV or its affiliates in the EU and/or other countries. www.st.com/trademarks For additional information about ST trademarks, please refer