Have you ever driven a car with ADAS features like Adaptive Cruise Controll and/or Lane Keeping Assistant? Scary, right? It takes some time to trust the system and relax. Imagine being responsible for the error-free functioning of such a system. In this interview you will learn how this challenge is being tackled by Codelab’s ADAS testing team.
Q: How does the ADAS testing look nowadays?
A: Today we have powerful tools to simulate road environment and we can check most of the situations that ADAS can face on the road. As testers we get access to big amount of documentation for automotive testing for example: ISTQB automotive software tester documents that can give us a lot of information how we should test ADAS systems and requirements which need to be covered by test scenarios.
ADAS testing is full of automatic tests runned in specialistic tools which are able to simulate situation on road, e.g. CarMaker. We are able to design and implement different situations which could happen in the real live.
Q: What was the goal of your ADAS testing activities in the last project?
A: One of the projects that we were involed in, was to develop a secondary emergency system that would take control over the car when the primary system breaks down. Our team was working on development of automated tests for Software Qualifications Tests (SWE.6 level according to Automotive SPICE). We simulated behaviour of the car in CarMaker by IPG and checked whether the car behaves according to the SW requirements specification. Additionally we implemented (in Jenkins and Python) the whole testing infrastructure that allowed to execute tests and provide their results in automated way. Such approach sped up the testing process and helped fixing SW problems in a faster time.
Q: How did you get it done? Can you describe your guidelines/process that helped you to achieve this goal for ADAS systems?
A: To make all of that running we were working according to the following process / steps:
- Software and System (SYS.2 & SWE.1) requirements definition and review
- Test cases definition in Polarion and automation in CarMaker by IPG
- Realization of full scope of SWE.5 & SWE.6 according to aSPICE, incl.:
- Quality reviews
- Quality gate assessment
- Process definition (Software Test Plan preparation) and improvements
- Test results analysis and problem reporting
- Fault Injection toolset creation
- ADAS KPIs and Safety Score verification
- Autonomous driving features analysis and tests execution
- ADAS algorithms and drives simulation in CarMaker by IPG
Q: What are the main features which need to be tested?
A: There are a bunch of features we usually test, such as but not limited to:
- Nominal Behavior (keeping dedicated deceleration and lane)
- Follow Mode (will follow the vehicle in front and keep the appropriate distance)
- Cut in/out (vehicle using the road of the Ego vehicle)
- Road Behavior testing: simulating: Lane Change Initiation / Completion / Abortion
- Emergency Brake (simulating emergency braking in an emergency in order not to lead to a collision, or to lead to a collision with the lowest possible force)
- System Degradation
- Mode Manager
- Manual Risk Maneuver (cascade testing)
- TrajectoryValidator (testing the trajectory of the vehicle)
- Vulnerable Road Users (VRU) – motorcycle, cyclist, pedestrian, animal on the road
- Risk Zone
Q: Which levels of ASIL where tested?
A: The main level of ASIL we have tested was ASIL‑D responsible for the active safety system cooperating with the steering and braking system of the vehicle and the automated driving system.
Q: What kind of tools have been used and what kind of advantages do these tool have?
A: As a team we worked in SCRUM methodology and we used JIRA and Confluence as main “scrum” tools. Those two allowed us to fully manage a Product and Sprint Backlog and additionally store the whole project knowledge (manuals, theory, implementation examples, reports, etc).
JIRA and Confluence are very good, intuitive tools and working with them was pleasure.
Additional tool that was used for managing a project documentation was Polarion. This is a pretty popular tool that supports work in automotive projects according to the ASPICE rules. In this tool we stored such documents like system requirements specification, SW requirements specification, architecture design and also created by us software test specifications. This tool helped in making a bilateral tracebility, however working with this tool was very annoying and sometimes very slow. Probably pushing all of those to the Atlassian package would help a lot.
All our tests were implemented in CarMaker – that tool gives us possibility to simulate road environment. Thanks to CarMaker we were able to: use different types of cars as traffic and we got the possibility to simulate their behavior, use other traffic participants (bikes, motorcycles, people, animals), set road properties such as: number of lines, type (straight, curve), lines (solid, dotted), set different types of weather. CarMaker also gives us possibility to see or record our tests that was very helpful during further reporting of problems and analysis.
Q: Did you implement automated tests which verified the ADAS solutions?
Yes, actualy we have all test cases automated. It is the fundamental of the testing nowadays also in ADAS. In last project we had automated Jobs on Jenkins which verified the tests results after software upgrade.
We used Python scripts for development of test automation, test report creation, test analysis and video analysis. Automation of (our input/output date) communication with Polarion system was implemented by Polarion API.
Q: How was ISO 26262 implemented in the test procedures?
A: ISO 26262 in test procedures was implemented by using the ASIL‑D motor risk classification (which is a major part of this standard). This standard dealt with the functional safety requirements for various electrical and electronic systems. The tested safety systems were responsible for assessing the risk that appeared on the road and the appropriate counteractively undesirable effects of a collision. The goal was the safety of all road users.
As testers we were involved in two phases of safety lifecycle: Product concept and Product development. We used the following methods of the ISO 26262:
- Test design techniques – we used equivalence partitioning, boundary value analysis
- Techniques of the test execution – we simulate road environment and we check behavior of our system
- Test environments – We were using Software in the Loop (SiL)
- Static test techniques – e.g. we were involved in reviewing requirements
Q: What was the biggest challenge which you faced in ADAS testing?
A: To fully understand the requirements and operation of the system while estimating the scope of work quarterly in advance.
Q: What are the upcoming trends to testing ADAS systems?
A: In our opinion, there will be better and more powerful simulation tools. We would see it like predefined road situations, even based on some statistics, law regulations and then the test engineer would focus only on correct measures. Such road situations could be used as a reference to prove that the behaviour of the car is correct.