Slide 1: Agile Software Design Verification And Validation (V&V) For Automated Driving
F2020-VES-017 Agile Software Design Verification And Validation (V&V) For Automated Driving Yixiao Li1, Yutaka Matsubara1, Daniel Olbrys2, Kazuhiro Kajio2, Hiroaki Takada1 1Nagoya University 2Toyota Research Institute - Advanced Development, Inc.
Slide 2: Verification and Validation (V&V)
Verification and Validation (V&V) V&V must be performed to meet the safety requirements • Required by the development process standard in the automotive domain (e.g. ISO 26262) • Verification: check whether the design can meet the requirements • Validation: check whether the implementation can work
Slide 3: Automated Driving System (ADS)
Automated Driving System (ADS) ADS is much more complex than the traditional system • Massive data: HD dynamic maps, sensors (e.g. camera, radar, LiDAR) • Software: soft/hard real-time tasks, parallelized with many threads, machine learning • Hardware: multi-core CPUs, many-core GPUs, deep learning
Slide 4: Simulation-based V&V
Simulation-based V&V Simulation-based V&V can find design problems earlier • Create a model from the requirements and design during the verification phases • Generate the approximations of system behavior using a simulator • Check the requirements during the verification without testing on the real
Slide 5: Our Proposed Framework
Our Proposed Framework An automized framework to model the ADS software efficiently User (Developer) provides design & trace as input. Our framework provides scripts to automatically create a simulation model. User can verify the current design and/or explore potential improvements on a simulator.
Slide 6: Input: The Design Model
Input: The Design Model Provide necessary information to schedule the application threads • Required parameters can be effortlessly collected from the design specifications • Design configuration files (e.g. ARXML, JSON) can also be used to fill some parameters automatically. Global Parameters CPU C
Slide 7: Input: The Application Trace Data
Input: The Application Trace Data Provide the execution time information of each thread • Traced on an environment with specification (e.g. CPU, GPU) close to the real target • Application running in Docker container on a Linux-based AUTOSAR Adaptive Platform • Tracer: LTTng (Linux Trace Toolkit Nex
Slide 8: Input: The Application Trace Data
Input: The Application Trace Data The trace data can be very huge • Difficult to handle directly • a 90-sec trace of our applications has over 60 million events Extract necessary timestamps to ease the procedure of creating a timing model • CPU timestamps: the active CPU time • GPU timestamps: the t
Slide 9: Create the Timing Model
Create the Timing Model A timing model is a design model with per-thread timing information • A thread with GPU usage consists of multiple parts with different timing characteristic • E.g. a thread accesses GPU twice during each execution: Create an execution time distribution for each part from the
Slide 10: Create the Timing Model
Create the Timing Model The traced discrete distributions can optionally be fitted using statistical models • E.g. Gaussian distribution • Pros: values not included in the trace can also be generated • Cons: bad-fitting models can lead to poor predictability • Currently, the user is responsible for
Slide 11: Create the Simulation Model
Create the Simulation Model A simulation model consists of the necessary files to execute on a simulator • Generated from the timing model • Use INCHRON chronSIM simulator in our paper Features of INCHRON chronSIM simulator • Project (IPL) file in XML format, easy to handle by external scripts • Off
Slide 12: Create the Simulation Model
Create the Simulation Model The basic task model of chronSIM lacks some features for our application • Trigger messages cannot be queued • GPU usage is not supported Extend the simulation task model implementation • chronSIM allows us to add C source files to override the default task behaviors in t
Slide 13: Evaluation
Evaluation An industrial case study • An ADS prototype on AUTOSAR Adaptive Platform from TRI-AD, Inc. • Use 7 CPU cores and include 50 threads in total • 30 threads created by the ADS software, 20 threads created by external libraries, 7 threads using GPU • Trace environment CPU Intel Core i7-8700K,
Slide 14: Evaluation
Evaluation Event chain E2E (end-to-end) latency • Simulate a representative event chain (Camera data output → Multiple threads with different processing algorithms → Result output) • chronSIM simulator can create a histogram for the end-to-end latency Identify a design problem • A significant gap wa
Slide 15: Evaluation
Evaluation CPU usage comparison • Per-thread values: very close (Δ < 0.1% for each thread except #1,#2) • Per-core values: prior cores (1~3) higher in simulation, latter cores (4~7) higher in trace Comparison of per-core CPU usage Comparison of per-thread CPU usage (sorted by absolute value of Δ)
Slide 16: Evaluation
Evaluation Per-core CPU usage differences • Reason: system threads in the trace environment frequently migrate applications threads to other cores • System threads relate to many factors (e.g. devices, kernel, services…), and thus are difficult to simulate • Instead, we simulate the interference by
Slide 17: Conclusion Future work
Conclusion • Propose an automized framework for modeling the ADS applications based on the design and trace data • Support commercial simulator chronSIM for design verification and exploration • Evaluation results show high accuracy of the simulation results • Simulation-based V&V enables the early/