In the dynamic landscape of automotive testing, data integrity is paramount for meaningful insights. This week, we delve into the nuances of data quality checks, specifically concerning automotive test data sourced from Engine/Chassis Dynos, Vehicle ECUs, Sensors, and external measurement instruments, often in the form of machine-generated hex and tabular data.
Various data acquisition sources (Data loggers) have unique conventions and user defined configurations for recording data, necessitating data quality checks, sanitization, and uniformization before effective data analysis. Traditionally, Python scripting has been the go-to method for routine data cleaning, addressing known issues like eliminating blank spaces and data gaps. However, manual deployment of these scripts can be time-consuming and prone to oversights.
DriveTech's StellarAi platform takes a different approach, offering a fresh perspective on automating data quality checks. Going beyond routine cleaning, StellarAi automates the entire process, addressing both known and potential unknown issues in the data.
StellarAi adopts a holistic approach, employing two fundamental types of data checks: Syntactic and Semantic Checks. Syntactic Checks ensure that data is well-formed and correctly organized. It smartly identifies data from different tests within a single file, facilitating precise segregation for accurate analysis. Semantic Checks, the second layer of scrutiny, focus on extracting crucial information from the dataset, enhancing the accuracy of data analysis. These checks include:
- Detecting and cleaning up empty rows
- Detecting and cleaning up columns with no headings/names
- Detecting anomalies in recordings (very high values, abrupt changes)
- Detecting threshold breaches to tag the file/test as failed
- Detecting stuck values – values that do not change for one or more files
- Automatically detecting standard parameters for analysis, using a Word Cloud model
- Automatically detecting DAQ frequency
- Detecting the presence of timestamps
- Differentiation between Message files & Signal files
The findings from these checks are recorded separately by StellarAi and serve as a basis for data analysis, leaving the original data intact. The automated checks run in the background, saving users precious time spent on writing and running Python scripts.
StellarAi not only detects previously unknown issues with data but also customizes definitions for each OEM's toolchain and workflow; enhancing its value proposition for various use cases. In summary, StellarAi streamlines & automates data quality assurance, providing a comprehensive solution that not only saves time but also significantly enhances the efficiency and overall quality of automotive test data analysis.
