How Computer Vision Technologies Build our Virtual Technicians of the Future

Share on facebook
Share on twitter
Share on linkedin
Share on email
Share on google
Improving Virtual Technician Performance With Deep Learning

Over the past few years, deep learning has driven significant improvements in computer vision accuracy and performance. Deep learning, the most advanced form of AI, enables independent learning of massive data sets. Unlike classic methods in which a human expert needs to define features (rules and attributes), deep learning can learn straight from data without human intervention, with  minor guidance (supervised learning) or without guidance at all (unsupervised learning).  In some fields, deep learning achieves far greater results than classic computer learning methods.

Let’s take a look at how we can use some of these computer vision technologies to build our virtual technicians of the future.

Object Recognition

Object recognition is a technology that enables finding and recognizing objects within images or videos. Object recognition includes several tasks, such as:

  • classifying that the image has a specific object
  • localizing the object in the picture
  • distinguishing the object from other objects
  • identifying parts within the object.

Since 2015, deep-learning-based object recognition has achieved amazing results, with the error rate dropping below 5% (the human level). This means that in several fields, today’s machines can recognize objects even better than human beings can!

This incredible accuracy makes object recognition a core technology for the future computer vision-powered virtual technician. To diagnose an Internet connectivity problem, for example, a virtual technician  needs to identify the router (and recognize the specific model), its parts (such as indication LEDs and back panel), and its cables; it also needs to localize all these objects to understand their context in order to diagnose the problem.

Image 1: Identifying Modem and Printer with Object Recognition

Image to text

Image to text is the process of using ‘deep semantic alignments’ to create textual descriptions. The technology allows the network to describe what it sees on an image in a simple sentence.

A machine develops this capability by:

  • recognizing objects and their locations within an image
  • converting this information into text
  • creating a meaningful contextual sentence to describe the image.

Using this technology, customers will be able to upload images of their equipment, after which a virtual technician with computer vision will  automatically describe in a sentence what it sees. For example, “A DLINK 5323 modem with a red light and ADSL cable disconnected”.

Such a description may help a human agent in a contact center diagnose the problem more quickly. Alternatively, it could also help by using Natural Language Processing (NLP) to automatically retrieve textual explanations of how to solve a problem.

Image 2: Textual description for a TV screen issue

Visual Similarity

Visual similarity enables finding similar images to the image that is provided to the machine. This capability is essential, as sometimes it is hard to explain an image in words, and the easiest way to a solution starts with  finding similar ones.

Visual search engines and websites such as Pinterest use visual similarity  to provide their users with similar images (with related objects, colors, or patterns). Visual similarity will allow virtual technicians to use computer vision to take images of a technical issue and search for similar issues within massive visual data sets of captured technical cases.

Image 3: Finding similar issues with a specific router

Motion Detection

Motion detection is a technology that allows tracking moving objects in videos in real time. Motion detection is a fundamental capability in autonomous cars for ‘pedestrian detection’ capabilities; and in the security domain to identify moving persons detected by security cameras.

Motion detection is a key component of the future virtual technician, enabling it to provide instructions and feedback to customers in real time . For example, in order to instruct a customer on how to install a printer, a virtual technician needs to identify the customer’s hand movements and provide him with real-time instructions accordingly: for instance, “Now put the papers in the tray”, “Now hold the network cable. Not that one, the other one”.

Image 4: Motion detection in troubleshooting process

Face Recognition

Face recognition is recognizing faces within images. This is one of the most common tasks in computer vision. Deep learning has made significant improvements to machines’ facial recognition capability, especially in challenging lighting conditions, angles, and backgrounds.

“Good morning Mrs Brown,” chirps the virtual technician

With facial recognition, a virtual technician will be able to perform the critical task of recognizing customers, and greeting them accordingly:

“Hello Mrs. Brown, I see that you’re having trouble with your washing machine”.

Companies can also use facial recognition to offer biometric warranties, ensuring that customers get service for their devices without forcing them to save receipts and warranty documents.

Is Computer Vision the Way Forward for Virtual Technicians?

The progress of computer vision with deep learning will foster the creation of  ‘artificial eyes’ for the virtual technician of the future, and help us, the consumers, manage the growing number of smart devices that we’re setting up in our smart homes.

In our next article in this series, we will detail how we can actually use computer vision to bring the virtual technician of the future into the present with object recognition.

Liad Churchill, VP Product Marketing

Liad Churchill, VP Product Marketing

Artificial Intelligence and Deep Learning expert, Liad Churchill, brings depth of knowledge in marketing smart technologies.


How Visual Assistance Resolves the Challenges of Delivering A Personalized Tenant Experience
Business Operations

Benefits of Remote Visual Assistance Technology For Property Managers and Tenants

Find out how property management organizations are leveraging remote support to enhance tenant experience as part of multiexperience journeys.

How to Optimize Customer Service Costs with Visual Assistance
Business Operations

How to Optimize Customer Service Costs with Visual Assistance

Quality customer service is expensive, but you can’t cut it! Optimize customer service costs and offer better service with visual assistance.

How Augmented Reality is Poised to Disrupt Field Services: Four Scenarios
Business Operations

Augmented Reality Field Service: The Enhancement Across Four Operating Models

Learn how AR-driven visual assistance plays a crucial role in field service improvement in four different operating scenarios