Introduction to Visual Inspection AI

Before image alt After image alt

The field of visual inspection has grown enormously in the insurTech industry in recent years thanks to the increased availability of AI computer vision solutions which can automate a large part of the process. Using mobile phone apps or mobile websites, users are able to perform damage assessment remotely and receive a quote immediately, which facilitates the process and makes the user experience much less stressful. The field is already expanding beyond vehicle damage assessment to other types of properties, such as real estate.

However, training visual inspection solutions using AI is not without its challenges, and it requires a continuous replenishment of high-quality and diverse data. Especially with the expansion of such services into different geographies, it’s essential to have representative data from each location, annotated precisely according to each insurer’s taxonomy.

Challenges and best practices

Based on our extensive experience annotating data for vehicle visual inspection, we are sharing below some of our best practices and tips on how to ensure your AI project will be a success:

Challenge

Best Practice

Each auto insurer and manufacturer may have a different taxonomy of how car damages are classified and assessed. In addition, the assessment of car damages is a very complex process because not all damages may be shown on the surface of the car. The detection of damages may not be accurate or precise enough, especially if the image quality is low or there are unusual types of damages. In addition, damage severity can be a very subjective notion, while damage area may be difficult to estimate based on just an image or a video.

In preparation for the annotation process, we work with clients to define very clear instructions for how to annotate damages. For example, when annotating a group of scratches close together: should they be labeled separately or as a group? Whenever there is a missing part, like a mirror, should the empty space where it is supposed to be located be labeled?


1. Take special care of the taxonomy of damages, severity can be very subjective
2. Be careful with zoomed in images, where the location of the damage is unclear
3. Make sure all types of damages are included: flat tyre, missing window, rust

In vehicle inspection datasets, there is a big challenge to reach equal distribution of different types of damages. For example, it’s easy to collect images of deformed bumpers, broken tail lights or scratched doors, but images of roof damages or missing windows may be both difficult to acquire in large enough quantities and difficult to detect in general. In addition, when scaling your solution to different markets, you need to take into account the variety of local brands and makes which are most commonly used.

Using our adversarial example collection services, once you have identified underrepresented classes or failure modes for your model, we are able to collect specific instances which represent difficult or challenging setups and environments. Does your model perform worse at dusk or at night time? We will be able to collect images specifically meant to cover these edge cases. Does your model perform worse in Eastern Europe, where many cars are older than what is usually represented in Western-centric datasets, or perhaps they include Soviet-made cars like Ladas and Moskvich? We can collect the necessary data in order to make sure your model is as accurate as possible in different locations.

During deployment, damage assessment AI is supposed to provide an evaluation of damages almost immediately, but there is a considerable risk of providing an inaccurate estimate if the model certainty is low or if the car at hand has an unusual damage or is a fraud attempt.

Our best practice in these cases is to hire a human-in-the-loop team and train them according to the specific requirements of each insurer. They cover different geographic locations and time zones in order to provide 24/7 coverage, and are able to perform real-time handling of edge cases and alerts using simple API requests. In this way, you get a guaranteed second verification layer which will help to avoid wrong estimations.

 

This is why it’s really important to start with the right training data and to adopt a human-in-the-loop approach in order to continuously improve the visual inspection models that you are using! This includes real-time edge case handling in order to make sure all damages are detected and assessed properly before calculating the repair cost.

High quality ground truth metadata

Edge case handling

Do you want to learn more about our approach and how it improves your model’s performance? 

Model validation circle icon from humans in the loop with girl and robot with question on purple globe background

Types of annotation for AI damage detection

Below we are featuring several different use cases of image and video annotation for vehicle visual inspection using AI and some of the best practices for each one:

Before image alt After image alt

Damage detection

For the purposes of detecting damages on vehicles, you need to collect a diverse dataset of different damage types and to annotate them using a polygon or a brush. Bounding boxes are not recommended because they will not provide the necessary precision and they will create overlaps which may confuse your model. 

Damage assessment

For each type of damage that has been annotated, the dataset for damage assessment needs to include a classification of the type of damage: eg scratch, broken part, chipped paint, broken glass, etc. In addition, tags can be added to estimate the severity: eg low, medium, high. 

Before image alt After image alt
Before image alt After image alt

Car part detection

Car part detection is a necessary parallel step to damage detection, in order to detect exactly which part is the damaged one. This helps also to understand how large the damaged area is compared to the total area of the car part and may help to calculate the exact area in cm2.

Car position detection

When analyzing photos or videos, it’s important to estimate the exact position of the camera relative to the car, in order to know which side of the vehicle has been damaged. This can be performed as a tag to each image with an estimate of the car rotation in degrees.

Before image alt After image alt

Damage detection

For the purposes of detecting damages on vehicles, you need to collect a diverse dataset of different damage types and to annotate them using a polygon or a brush. Bounding boxes are not recommended because they will not provide the necessary precision and they will create overlaps which may confuse your model. 

Damage assessment

For each type of damage that has been annotated, the dataset for damage assessment needs to include a classification of the type of damage: eg scratch, broken part, chipped paint, broken glass, etc. In addition, tags can be added to estimate the severity: eg low, medium, high. 

Car part detection

Car part detection is a necessary parallel step to damage detection, in order to detect exactly which part is the damaged one. This helps also to understand how large the damaged area is compared to the total area of the car part and may help to calculate the exact area in cm2.

Car position detection

When analyzing photos or videos, it’s important to estimate the exact position of the camera relative to the car, in order to know which side of the vehicle has been damaged. This can be performed as a tag to each image with an estimate of the car rotation in degrees.

Tools we love

Here are some of our tips and recommendations on the best tools we’ve used for this type of annotation which can hopefully be useful for anyone working on document parsing or processing models.

CVAT is a very easy to use open source tool which will provide you with all the basic functionalities needed to annotate your dataset: from bounding boxes and polygons to video annotation with interpolation.

V7 is a more advanced tool which offers great auto-labeling features for bounding boxes and polygons and will speed up your annotation process significantly. 

Blogs

Best Annotation Tools for 2022

Found your humans – now all you need is the right tool for the job? Here’s our review of the most popular tools for 2021!

Image in the blog article 10 of the best open-source annotation tools for computer vision
Image in the blog article 10 of the best open-source annotation tools for computer vision

How to use a human-in-the-loop for visual inspection AI

When applying visual inspection solutions across geographies, it is a challenging task to keep up with data drift and to handle examples that the model has not been trained on or which are underrepresented in its training data. Therefore it’s essential to use human input on a continuous basis, not just for the initial training of your models. Here are some of the ways in which humans can be plugged into the entire MLOps cycle:

Before image alt After image alt
  1. Dataset collection: our humans in the loop on the ground can collect datasets with images and videos of damaged vehicles from around the world, depending on where your model will be applied 
  2. Ground truth annotation: in order to train your initial models, we offer full dataset annotation from scratch in batches: anything from vehicle position and part detection to bounding box, polygon and semantic segmentation for damage detection and severity assessment 
  3. Output validation with active learning: once you’ve trained an initial model, we can use it in order to pre-annotate a large part of the dataset, which will both increase the speed of the annotators and the impact of their work, by setting up an active learning workflow and prioritizing instances where your model is least certain
  4. Adversarial example collection: once you’ve trained an initial model, we can expand your core dataset with additional difficult and challenging edge cases, such as unusual car brands and makes, unusual view angles, or rare types of vehicle damages, such as broken or missing windows, depending on the failure modes of your model during testing
  5. Real-time edge case handling: once you have a model in deployment, our humans-in-the-loop are available 24/7 to handle potential visual inspection edge cases that appear in real time or close to real time, using a simple API request and sending the correct response in seconds in order to ensure a second layer of verification for your model’s most critical responses

Wondering who is annotating your data?

When you are hiring a company to help you with your annotation needs, you frequently never meet the workers who are labeling your data. We want to change this and present to you the inspiring stories of our annotators!

Image of a HITL annotator Lana

Does this sound like something you’d like to try out? Get in touch with us and we’d be happy to schedule a free trial so as to explore how we can best help you with your visual inspection annotation needs!

Get In Touch

We’re an award winning social enterprise powering the AI solutions of the future