Slide 1: How sensors with machine learning core bring power-efficient AI applications to the edge
How sensors with machine learning core bring power-efficient AI applications to the edge
Slide 2: Outline
Outline 1 5 Artificial intelligence 1 and machine learning 5 ST Partner: Arduino® 2 6 Products with machine 2 learning core 6 MLC application demos 3 7 Key steps behind GitHub: MLC configuration 3 machine learning core 7 examples 4 8 4 Form factor tools and GUI 8 Resources 2
Slide 3: Artificial intelligence and machine learning
Artificial intelligence and machine learning
Slide 4: Adding intelligence to make sensorization sustainable
Adding intelligence to make sensorization sustainable Cloud Smart sensors process the data at the edge 110101101111010101111010100011011 01100110001011 Only a reduced, meaningful set of data is 1101011011101101010010110100111 transferred to the network 11010010101010 1101011011101101010101100110 The
Slide 5: Smart sensors are becoming even smarter
Smart sensors are becoming even smarter Edge processing Ultralow power consumption Advanced security Small form factor 5
Slide 6: What is AI?
What is AI? The evolution of AI Artificial intelligence Any technique that enables a Early artificial Machine learning computer to mimic human behavior intelligence stirs excitement Machine learning begins to flourish Deep learning Subset of AI. Algorithms and Deep learning methodologies that improv
Slide 7: Machine learning: why do we need it?
Machine learning: why do we need it? An example of difficult program When a complex task or problem involves • How to recognize the handwritten digits? a large amount of data and lots of • Very difficult to define the rules! variables, but no existing formula or • What makes all these numbers to be
Slide 8: Three components of machine learning
Three components of machine learning The only goal of machine learning is to predict results based on incoming data Data DATA science FEATURES DATA: the more diverse the data, the better the result. It's extremely tough to collect a good collection of data (usually called a dataset) Data ML Classica
Slide 9: Standard vs machine learning algorithm approach
Standard vs machine learning algorithm approach Standard programming Machine learning “A priori” approach “Empirical” approach Log of input data to Desired output from the system the system Design algorithm specific for the given problem General ML model trained for the specific problem 9
Slide 10: Using AI algorithms implies two phases
Using AI algorithms implies two phases Training Inference AI models are produced using Model deployment historical datasets Inference is the process of using a trained machine and a training engine / framework learning model to make predictions or decisions based on new, unseen data. Dataset Trainin
Slide 11: Products with machine learning core
Products with machine learning core
Slide 12: Smart sensors
Smart sensors We create the new generation of sensors to allow developers exploiting their potential while improving the overall system efficiency • Leveraging machine learning techniques from the world of A.I. • Enabling edge computing • Reducing power consumption at both sensor and system levels •
Slide 13: Decision-making capabilities closer to data source: sensors with MLC
Decision-making capabilities closer to data source: sensors with machine learning core (MLC) MLC is a hard-coded engine inside the sensor based on decision tree logic: • optimal system energy efficiency (few µA) • granting maximum data privacy Sensor data Computation Decision tree block Acceleromete
Slide 14: Machine learning core (MLC): What it is
Machine learning core (MLC) What it is MLC is an in-sensor classification engine based on decision tree logic Sensor with machine learning core Motion sensing Main Sensor data Computation block Decision tree processor Accelerometer Filters Meta-classifier External Gyroscope Digital Pre-defined Featu
Slide 15: Machine learning core (MLC): The blocks
Machine learning core (MLC) The blocks Sensor data Computation block Decision tree A wide set of inputs is available: • Sensor data can be filtered • The decision tree is a predictive model built with a 2nd order IIR filter from training data. Outputs of the computation • Accelerometer block are inp
Slide 16: Machine learning core: Features & performance
Machine learning core Features & performance New features Generation 1 Generation 2 User advantages Performance MLC rate increase to enable MLC rates ODR = 13, 26, 53, 104 Hz ODR = 15, 30, 60, 120, 240 Hz addressing new apps (spike detection) MLC with external sensor Full processing of high-resoluti
Slide 17: Machine learning solutions in accelerometers and IMUs
Machine learning solutions in accelerometers and IMUs Automotive Industrial Personal electronics Inertial measurement units (IMUs) Inertial measurement units (IMUs) Inertial measurement units (IMUs) ASM330LHHX ISM330BX LSM6DSV320X LSM6DSV80X ASM330LHHXG1 ASM330LHB LSM6DSV16X ISM330DHCX LSM6DSV16BX A
Slide 18: From low-power sensor to low-power system
From low-power sensor to low-power system Machine learning core (MLC) for real edge computing enables high system flexibility Machine learning Machine learning @ microcontroller @ sensor FSM MLC Sensing up to 16 up to 8 Sensing & + MLC algo Raw data Power optimization ML data • Higher computation po
Slide 19: System level benefits using sensors with MLC
System level benefits using sensors with MLC Activity recognition Activity Recognition in software in sensors with MLC Sensor with Sensor SoC/MCU MLC SoC/MCU Real-time data Interrupt Parameters Benefits in using MLC LSM6DSOX current Similar consumption on LSM6DSOX MIPS MCU/SoC Offload SoC / MCU Traf
Slide 20: Key steps behind machine learning core
Key steps behind machine learning core
Slide 21: Machine learning solutions in sensors: New developer model approach
Machine learning solutions in sensors New developer model approach Shorter development time and better accuracy with machine learning techniques (decision trees) Machine learning core configuration Operating mode Capture data Label data Build decision tree Embed decision tree Process new data • Acce
Slide 22: Example of sensors MLC programming
Example of sensors MLC programming 10 to 1000-time energy saving by running MLC on Sensor vs. MCU/AP Improved detecting accuracy How it works in 5 simple steps and with an intuitive use case: 1 2 3 4 5 User defines classes to be Label data and select filters Build the decision tree Program the decis
Slide 23: Current consumption: AI in MCU vs. MLC in sensor
Current consumption: AI in MCU vs. MLC in sensor Only 4µA additional current consumption to run Activity Recognition with MLC Activity recognition library (MotionAR) Activity recognition library (MotionAR) running in software in MCU running inside the LSM6DSOX sensor In this scenario, the MCU wakes
Slide 24: Form factor tools and GUI
Form factor tools and GUI
Slide 25: Machine learning solutions in sensors: The ecosystem
Machine learning solutions in sensors The ecosystem A complete suite to create ML applications in sensors Programming with ST tools and ST partners Getting start with ST development kit st.com/mlc Examples for motion recognition and context recognition Videos, training material, in products campaign
Slide 26: ST toolbox for machine learning core (MLC) in the sensor
ST toolbox for machine learning core (MLC) in the sensor Machine learning core configuration Operating mode Capture data Label data Build decision tree Embed decision tree Process new data • Accelerometer • Filters • Classification • DT implementation • Real time test WHAT • Gyroscope • Features • R
Slide 27: ST Partner: Arduino with machine learning core
ST Partner: Arduino® with machine learning core
Slide 28: Arduino ecosystem for MLC
Arduino ecosystem for MLC Arduino® RP2040 Arduino® Nicla Vision • Arduino® RP2040 mounts the LSM6DSOX 6 axis • Arduino® Nicla Vision mounts the LSM6DSOX 6 axis sensor with MLC sensor with MLC • Build the MLC with the Arduino® environment: • Build the MLC with the Arduino® environment: • https://www.
Slide 29: MLC applications
MLC applications
Slide 30: Explore MLC examples and resources and get inspired
Explore MLC examples and resources and get inspired Consumer MLC examples are available online 6D position recognition, activity recognition, gym on GitHub and through the case studies section activity recognition, head gestures of the ST Edge AI Suite Industrial 6D position recognition, motion inte
Slide 31: Get inspired by MLC applications
Get inspired by MLC applications Drilling machine status detection Static inclinometer Industrial Fan rack monitoring Cobot movement and anomaly detection Human activity recognition Gym activity recognition Personal Smart kettle Smart access electronics Gesture recognition Yoga position recognition
Slide 32: LSM6DSV32X Activity recognition
LSM6DSV32X Activity recognition Activity recognition for wrist applications LSM6DSV32X Use case Demo overview MLC embedded in LSM6DSV32X inertial • MLC with 3 classes sensor is programmed to detect 3 different • MLC running on MEMS ASIC classes using the MLC: • No pre-processing required from MCU •
Slide 33: ISM330BX Movement and anomaly detection
ISM330BX Movement and anomaly detection Movement and anomaly detection in industrial cobots ISM330BX Use case Demo overview MLC embedded in ISM330BX inertial • MLC with 3 classes sensor is programmed to detect 3 different • MLC running on MEMS ASIC classes using the MLC: • No pre-processing required
Slide 34: Industrial IMUs Drilling machine status detection
Industrial IMUs Drilling machine status detection ISM330DHCX / Drilling machine status detection ISM330BX Use case Demo overview • MLC embedded in ISM330DHCX / • MLC with 3 classes ISM330BX inertial sensor is • MLC running on MEMS ASIC programmed to detect the status of • No pre-processing required
Slide 35: ISM330DHCX Drilling machine status detection
ISM330DHCX Drilling machine status detection Drilling machine status detection ISM330DHCX Use case Demo overview • MLC embedded in ISM330DHCX • MLC with 3 classes inertial sensor is programmed to detect • MLC running on MEMS ASIC the status of the drilling machine. • No pre-processing required from
Slide 36: IIS2ICLX Static inclinometer
IIS2ICLX Static inclinometer Static inclinometer IIS2ICLX Use case Demo overview • MLC embedded in IIS2ICLX inertial • MLC with 3 classes sensor is programmed to detect the • MLC running on MEMS ASIC truck bed positioning • No pre-processing required from MCU • 3 different classes can be identified
Slide 37: Human activity recognition
LSM6DSOX – LSMDSV16X / 32X Human activity recognition Human activity recognition LSM6DSOX Use case Demo overview • MLC embedded in LSM6DSOX inertial • MLC with 5 classes sensor is programmed to detect • MLC running on MEMS ASIC Human Activity Recognition in • No pre-processing required from MCU diff
Slide 38: Gym activity recognition
LSM6DSOX – LSMDSV16X / 32X Gym activity recognition Gym activity recognition LSM6DSOX Use case Demo overview • MLC embedded in LSM6DSOX inertial • MLC with 4 classes sensor is programmed to detect gym • MLC running on MEMS ASIC activity by identifying 4 common • No pre-processing required from MCU a
Slide 39: Smart kettle
LSM6DSOX – LSMDSV16X / 32X Smart kettle Smart kettle LSM6DSOX Use case Demo overview • MLC embedded in LSM6DSOX inertial • MLC with 3 classes sensor is programmed to detect the • MLC running on MEMS ASIC status of a boiling kettle • No pre-processing required from MCU • The MLC identifies 2 stats sc
Slide 40: LSM6DSOX Smart access
LSM6DSOX Smart access Smart access LSM6DSOX Use case Demo overview • MLC embedded in LSM6DSOX inertial • MLC with 4 classes sensor is programmed to detect the • MLC running on MEMS ASIC check a safety box status • No pre-processing required from MCU • The MLC identifies 4 stats scenarios: • “Steady”
Slide 41: SensorTile.box Baby crying detection
SensorTile.box Baby crying detection Baby crying detection Use case MLC for vehicle detection combined with fusion of ST MEMS microphones and STM32 AI to create a fully-trained neural network detecting if there is a baby crying in the room. • Detect a baby crying in car left unattended in a parking
Slide 42: ASM330LHHX Vehicle monitoring
ASM330LHHX – ASM330LHHXG1 Vehicle monitoring Vehicle monitoring ASM330LHHX Use case Demo overview • MLC embedded in ASM330LHHX • MLC with 4 classes inertial sensor is programmed to detect • MLC running on MEMS ASIC different vehicle scenario • No pre-processing required from MCU • The MLC identifies
Slide 43: LSM6DSOX Yoga position detection
LSM6DSOX Yoga position detection Yoga position detection LSM6DSOX Use case Demo overview MLC embedded in LSM6DSOX inertial • MLC with 13 classes sensor is programmed to detect 13 • MLC running on MEMS ASIC different classes using the MLC: • No pre-processing required from MCU • Boat pose • Plank • B
Slide 44: ISM330DHCX Fan rack monitoring
ISM330DHCX Fan rack monitoring Fan rack monitoring ISM330DHCX Use case Demo overview MLC embedded in ISM330DHCX inertial • MLC with 3 classes sensor is programmed detect 3 states: • MLC running on MEMS ASIC • “Fan_on” state • No pre-processing required from MCU • “Fan_off” state • “Fan_fault” state
Slide 45: Resources
Resources
Slide 46: Resources
Resources MEMS sensors ecosystem for machine Learning MEMS sensors community Design Tips for Machine Learning algorithms generation Automatic filters and features selection for Machine Learning Core in MEMS Studio Deploy in-sensor AI in your IoT infrastructure with a cloud-based tool 46
Slide 47: Takeaways
Takeaways We create the new generation of sensors to allow developers exploiting their potential by improving the overall system efficiency leveraging Machine Learning techniques ST has all the building blocks for the machine learning core for tools and software Our constantly expanding development
Slide 48: Find out more at www.st.com
Find out more at www.st.com © STMicroelectronics - All rights reserved. ST logo is a trademark or a registered trademark of STMicroelectronics International NV or its affiliates in the EU and/or other countries. www.st.com/trademarks For additional information about ST trademarks, please refer to ww