Video Analytics Solution Using Amazon Sagemaker

The Challenge

The client manufactures floor cleaning machines, operating at different customer centres. In the face of increasing complaints around machines not performing, the client wanted to have detailed logs of machine performance across all locations. This would allow them to validate or refute these customer claims, and hence prevent unwarranted expenditure on recalls and repairs.

The Proposed Solution

A  video analytics algorithm capable of identifying the machine and verifying its performance at given locations was proposed. The approach was focussed on recording real-time footage of the machines operating at different customer locations and then automatically analyzing the video feed to identify and verify if the machines are performing as expected.

The Srijan team worked to create a PoC of this solution, focusing on delivering it in two phases:

Phase 1 - Building the right machine-learning model and deployment 

The first part of the solution involved designing and training a machine learning algorithm that could accurately identify the client’s floor cleaning machine from a video feed with proposed location. Typically, the video feed would be peppered with other objects and people. So the algorithm’s ability to accurately recognize/identify the cleaning machine is crucial.

To facilitate this, the Srijan team:

• Designed a model based on Convolutional Neural Network (CNN), using Tensorflow as their deep learning framework of choice. CNN is an often used deep learning algorithm when it comes to analyzing video imagery.
• Curated images of the client’s floor cleaning machine from the internet, product brochures, and screen grabs from YouTube videos. Since, the acquired data was insufficient, it was then increased with data augmentation technique, which worked to render existing images into different profiles. For example: create a left profile by extrapolating from an existing image of the machine in right profile.
• Normalized image data to ensure all collected images are in the same format and dimensions, to feed into the model. .
• Tagged each image to define the area within which the machine was located, so the model can consistently learn to recognize the machine.
• Fed the pre-processed data into the model for training to spot the cleaning machines in a video feed.
• After successful testing the model, it was deployed as Endpoint in AWS Sagemaker.

AWS Solution at Play

Amazon SageMaker

Amazon Sagemaker is a fully managed platform to build, train and deploy machine learning models. For this particular project, Srijan used SageMaker to create the algorithm, as well as for data augmentation and training. The AWS-optimized Tensorflow on SageMaker allowed for quick and efficient scaling of training data to create a more accurate machine learning model.

Amazon SageMaker also enabled automatic tuning of the model across repeated training and validation runs, thus creating a more accurate algorithm.

Phase 2 - Analyze Video Feed

Once the endpoint was in place, the next phase involved:

• Collecting the CCTV video stream from various client locations and dropping it in an Amazon S3 bucket.
• Dropping the file in S3 triggers the AWS Lambda function.
• The Lambda function hit the Sagemaker endpoint with necessary parameters.The endpoint processed and analyzed the video feed to accurately identify cleaning machines at every 5 seconds interval, along with the timestamp and site location using video analysis techniques.
• These sightings were made into a JSON file with timestamps and locations, to create a complete log of a machine cleaning or failing to clean at a particular customer location or site.
• This file was again stored in an Amazon S3 bucket.

Amazon Solutions at Play

Amazon S3

A highly scalable, available and secure object storage service, Amazon S3 was used in this project to both store the incoming video streams, as well as the logs of machine performance in a JSON file format.

AWS Lambda

AWS Lambda is a serverless computing platform that can run code in response to event triggers. It also automatically manages all the resources required to manage and scale your code. Here, the lambda function acts as a middleware between the API and the S3 bucket. It invokes the Sagemaker endpoint with required parameters.

Here, a new video feed is being dropped into the S3 bucket which triggers the Lambda function. It invokes the API endpoint (Endpoint on Sagemaker), with required parameters like bucket information and target bucket. The endpoint will process/analyze the video feed to identify machine and extract information like site name along with the timestamp. Finally, it creates the JSON file with the extracted information.

Business Benefits

• Validating the claims made by the customer/client for the machines that are failing to clean the given sites.
• Helps in perceiving more insights about the behavior of the machines, which later helps in root cause analysis for the odd machine behavior and improve the product.
• Helps in being more proactive in asset performance assessment and maintenance, instead of being reactive

Srijan is an Advanced Consulting Partner for Amazon Web Services (AWS). It is currently working with enterprises across media, travel, retail, technology and telecom to drive their digital transformation, leveraging a host of AWS solutions. Srijan's expert team of certified AWS engineers are working with machine learning and natural language understanding to create interesting enterprise chatbots for diverse industry use cases. 

Looking to develop an effective enterprise bot ecosystem? Just drop us a line and our team will get in touch.