It is widely acknowledged that AI systems may exhibit unreliability in high-risk decision-making, particularly when confronted with novel, unforeseen, or anomalous data that was not included in their training datasets. This is attributable to the fact that AI models are trained to learn from specific data and subsequently generate predictions based on this data. When they encounter data that is markedly different from their training datasets, their capacity to make accurate predictions is potentially compromised. Consequently, in high-stakes decision-making contexts involving human lives or significant financial investments, exclusive reliance on AI systems may be inadequate. Self-driving car manufacturers like Tesla have been raising awareness about the importance of outliers for a long time, given how critical these could be for the performance of autonomous vehicles. 

Feature image in the blog article "Edge case handling in real time with a human-in-the-loop: why and how?" Unidentified object detected

Frequently these outliers are the most difficult to address because they may be underrepresented or completely missing from the training data. Given that current AI systems rely heavily on pattern recognition, the long tail of outliers has been frequently recognized as the main challenge for Artificial Intelligence nowadays.

This is a problem which affects not just the self-driving cars industry. For any AI system which is tasked with making a high-stakes decision, errors may be critical. And in many of these cases, real-time intervention of human operators may be required in order to address outliers.

When is edge case handling needed?

There are many use cases in which AI systems have to make important decisions and mistakes can be very dangerous. These include:

Edge case handling on weapon detection ai

CCTV systems: be it for intruder detection, shoplifter detection, or weapon detection, such systems are extremely important but must have the minimum amount possible of false negatives and must also avoid too many false positives in order not to trigger false alarms too frequently and cause frustration in users.

Cameras for wildfire monitoring: Similarly to the previous case, such systems must detect all instances of fire and smoke with great reliability and avoid missing any fire while also not producing alert fatigue.

wildfire edge case handling ai
Edge case handling patient monitoring

Patient monitoring: In hospitals and homes for the elderly, it’s very important to use privacy-preserving solutions which can help ensure the safety of patients and to detect patients who have fallen out of their beds or who need support. Nurses may receive alerts but it’s important not to overwhelm them with too many.

Manufacturing: In manufacturing processes, there may sometimes be errors in the position of products, in their barcode locations, or particular defects. This is important to detect and address in real time in order not to halt the entire production line, and this is where a human operator may come in handy.

edge case handling manufacturing
Edge case handling agriculture

Agriculture: Robots in agriculture such as smart picking arms, smart tractors, and smart herbicide spraying solutions may sometimes be blocked by unexpected obstacles on the ground. In order not to block the robot for a long time, a human intervention may be needed.

Logistics: In environments such as warehouses and shipping centres, drones and robots are used to ease human work, but they may come across unexpected barriers and obstacles as well. A human operator can help them navigate more challenging situations.

logistics edge case handling
delivery robots edge case handling

Delivery robots: similar to previous cases, delivery robots may have to face the challenges of navigating the real world, with unexpected surprises and changes in their itinerary and environment. This is when a human can step in to support them.

Document processing: even in more mundane use cases such as document parsing, there may be a need for human intervention, in case an outlier is detected and a document cannot be fully processed due to unreadable text. This is important for cases such as invoice and receipt processing, customs documentation, etc.

document processing edge case handling

How does edge case handling work?

In all of the above use cases, AI systems may not have been trained on a comprehensive enough dataset to know how to handle all exceptions. During inference, it’s a best practice to plug in a human operator who can handle the strange cases which are flagged by the system using custom logic-based triggers or active learning approaches, for example:

Positive instance detected: Especially in surveillance cases such as wildfire or weapon detection, you may want to have all alerts reviewed by a human.

positive instance detection edge case handling
Unidentified object detection edge case handling

Unidentified object detected: In use cases such as delivery and logistics robots, whenever they face an unknown obstacle, they can pass the alert on to a human.

Low model certainty: When an instance is detected and categorised but the prediction certainty is below a certain threshold, the case may be sent to a human arbiter to make a final judgement.

low model certainty edge case handling
person fallen down edge case handling

An instance out of the usual distribution detected using active learning: Even if the model has a high certainty about categorising it, it may be worth having it checked by a human so as to avoid costly false negatives.

These instances are then sent to a human operator who has to handle them within a certain SLA. For example, at Humans in the Loop we have projects where the human response has to be immediate within seconds of seeing the alert, while other projects have an expected turnaround time of 24 hours when the use case is not very high-risk. We also offer dedicated teams, which provide coverage 24/7 and develop expertise overtime by learning the intricacies of your data. 

The human response may be collected using a consensus among two or more users in order to increase the accuracy, and humans can also be rated based on how they compare to other operators or to gold standard data. 

The responses of the humans are both used to override the predictions of the AI system, and to retrain the model. In this way, you are closing the loop of your MLOps pipeline!

Illustration of image verification

Real-time edge case handling

Start your project with edge case handling in real time. Check out our services for real-time outlier detection and handling

Interested in exploring a real-time human-in-the-loop for your use case? Schedule a call with our team through the link below:

Get In Touch

We’re an award winning social enterprise powering the AI solutions of the future