Anomaly detection for product quality control

Anomaly detection can be useful in a wide range of applications, such as product quality control. For example, products can be evaluated for anomalies before being forwarded down a supply chain and/or provided to consumers. Automated anomaly detection systems have been developed in industry, in which machines make determinations of presence or absence of anomalies by processing object images acquired through digital cameras or sensors. Anomaly detection makes it possible to prevent deficient products from moving along supply chains, improve product quality control, increase economic efficiency, and optimize manufacturing process, among other uses.

Implementations of the present disclosure are generally directed to anomaly detection of objects based on images. More particularly, implementations of the present disclosure are directed to an anomaly detection system that processes images of objects using machine learning (ML) models to detect anomalies. In some examples, the anomaly detection system of the present disclosure can be used for quality control of products.

In some implementations, actions include receiving an image of an object; obtaining a reconstructed image by processing the image through a ML model; obtaining a gradient difference image by comparing the image to the reconstructed image; generating an output image at least partially by suppressing non-significant regions representing non-significant anomalies from the gradient difference image using a non-significant suppression (NSS) map; determining whether an anomaly is depicted in the output image; and in response to determining that an anomaly is depicted in the output image, sending an alert indicating that the object is defective. Other implementations of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other implementations can each optionally include one or more of the following features. In some implementations, the actions include generating the NSS map by: obtaining a set of reconstructed images by processing a set of images using the ML model; comparing each image in the set of images with a respective reconstructed image in the set of reconstructed images to obtain a difference map including a set of difference components; and generating the NSS map based on selecting difference components that satisfy a difference threshold. In some implementations, the difference threshold is determined based on a set of defect images, each defect image representing a sample of the object including one or more defects.

In some implementations, the NSS map is initially generated using a set of images, each image in the set of images being absent any defect anomaly. In some implementations, the image depicts a side of the object and the ML model and the NSS map are each specific to the side.

In some implementations, the actions include extracting a region of interest (ROI) for the image included in the one or more images; and adjusting the image based on the ROI prior to obtaining the reconstructed image. In some implementations, the ROI is extracted based on a predefined range in a hue saturation value (HSV) color-space of the image.

In some implementations, the output image is further generated by removing representation of debris from the output image, removing the representation of debris including removing any connected component having a size that is less than a size threshold. In some implementations, the ML model includes a convolutional neural network (CNN)-based auto-encoder.

The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations in accordance with implementations of the methods provided herein.

The present disclosure further provides a system for implementing the methods provided herein. The system includes one or more processors, and a computer-readable storage medium coupled to the one or more processors having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations in accordance with implementations of the methods provided herein.

It is appreciated that methods in accordance with the present disclosure can include any combination of the aspects and features described herein. That is, for example, apparatus and methods in accordance with the present disclosure are not limited to the combinations of aspects and features specifically described herein, but also may include any combination of the aspects and features provided.

The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the present disclosure will be apparent from the description, drawings, and claims.

Like reference numbers and designations in the various drawings indicate like elements.

Implementations of the present disclosure are generally directed to anomaly detection of objects based on images. More particularly, implementations of the present disclosure are directed to an anomaly detection system that processes images of objects using machine learning (ML) models to detect anomalies. In some examples, the anomaly detection system of the present disclosure can be used for quality control of products.

In some implementations, actions include receiving an image of an object; obtaining a reconstructed image by processing the image through a ML model; obtaining a gradient difference image by comparing the image to the reconstructed image; generating an output image at least partially by suppressing non-significant regions representing non-significant anomalies from the gradient difference image using a non-significant suppression (NSS) map; determining whether an anomaly is depicted in the output image; and in response to determining that an anomaly is depicted in the output image, sending an alert indicating that the object is defective.

To provide further context, and as introduced above, anomaly detection can be useful in a wide range of applications, such as product quality control. Automated anomaly detection systems have been developed in industry, in which machines make determinations of presence or absence of anomalies by processing object images acquired through digital cameras or sensors. In anomaly detection, an example system includes using a ML model to detect presence or absence of anomalies. However, traditional automated anomaly detection systems have technical deficiencies. For example, the ML model may not be able to differentiate non-significant anomalies from real defects. For example, a detected anomaly may be due to a certain part of the object being reflective, and, as a result an image of the object is determined to depict an anomaly, when an anomaly is not actually present. That is, an anomaly is detected, but is not a real defect in the object. Because the ML model can have difficulty recognizing such non-significant anomalies that do not count as defects, the prediction results from the ML model may include a relatively high false positive rate. This results in consumption of resources to address objects that are actually non-defective.

In view of the foregoing, and as introduced above, implementations of the present disclosure are directed to an anomaly detection system that can reduce false positives by suppressing the non-significant anomalies and removing other debris/noises. In some implementations, one or more images of an object are received. A region of interest (ROI) can be extracted from an image. A reconstructed image of the ROI can be obtained by restoring the ROI of each image using a ML model. Presence or absence of an anomaly is determined from the image by comparing the restored image with the received image. For example, areas within the image can indicate presence of an anomaly. However, it is unclear whether the anomaly is a true positive (true defect) or a false positive (noise and/or other non-significant anomalies that do not count as defects). To distinguish whether a detected anomaly is a true positive or a false positive, implementations of the present disclosure apply a non-significant suppression (NSS) map. In some examples, the NSS map is generated using image samples that are absent anomalies (true defects). In some implementations, debris that may be present on the object can be removed by removing connected components whose sizes are less than a size threshold. An output image is provided and is used to determine the presence or absence of anomalies in the object. As described in further detail herein, the anomaly detection system of the present disclosure can detect real defects of the object with higher accuracy and lower false positive rate than traditional anomaly detection systems.

depicts an example systemthat can execute implementations of the present disclosure. The example systemincludes a computing device, a back-end system, and a network. In some examples, the networkincludes a local area network (LAN), wide area network (WAN), the Internet, or a combination thereof, and connects web sites, devices (e.g., the computing device), and back-end systems (e.g., the back-end system). In some examples, the networkcan be accessed over a wired and/or a wireless communications link.

In some examples, the computing devicecan include any appropriate type of computing device such as a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), a cellular telephone, a network appliance, a camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or an appropriate combination of any two or more of these devices or other data processing devices.

In the depicted example, the back-end systemincludes at least one server system, and data store. In some examples, the at least one server systemhosts one or more computer-implemented services that users can interact with using computing devices. For example, the server systemcan host systems for training ML models of an anomaly detection system in accordance with implementations of the present disclosure.

In the example of, a cameraand an objectare depicted. The cameracan by any appropriate type of camera (e.g., video camera) that generates images representing objects, such as the object. In the context of the present disclosure, the cameracan generate images as digital data representing the object. The cameracan capture images of every side of the object, such as front, back, left, right, top, and bottom sides of the object. In some example, multiple camerasinstalled in different angles can be provided to capture images of every side of the object. In some examples, the objectcan be rotated, so that the cameracan capture images of every side of the object. In accordance with implementations, images can be processed by an anomaly detection system to determine whether the object, as represented within the images, includes any anomal-y/ies. In some examples, the anomaly detection system is executed on the computing device. In some examples, the anomaly detection system is executed in the back-end system.

depicts an anomaly detection systemin accordance with implementations of the present disclosure. In the example of, the anomaly detection systemincludes a ROI extraction module, an anomaly detection module, a NSS module, and a debris removal module. As described in further detail herein, the anomaly detection systemreceives an input imageand processes the input imageto generate an output image. The output imagecan represent one or more anomalies of an object depicted in the input image, if any.

In further detail, the anomaly detection systemcan receive the input imagefrom one or more cameras. The input imagecan include an image of an object from a respective side (e.g., top, bottom, left, right, front, back). In general, the input imageis a digital image represented as data, the data providing values for each of a plurality of pixels that the input imageis composed of. The input imagecan be provided using any appropriate color-space. Example color-spaces can include, without limitation, red, green blue (RGB) and hue, saturation, value (HSV). In some examples, and with respect to RGB, each pixel includes respective values for a red channel, a green channel, a blue channel, and an alpha channel, where alpha represents an opaqueness of the respective pixel (e.g., referred to as the RGBa format). In some examples, the input imageis processed to convert the pixel values to another color-space. For example, the input imagecan be processed to convert from RGB to HSV. In some examples, in HSV, the hue value represents a color of a respective pixel and is expressed as a number from 0 to 360 degrees (e.g., red falls between 0 and 60 degrees, yellow falls between 61 and 120 degrees, green falls between 121-180 degrees, cyan falls between 181-240 degrees, blue falls between 241-300 degrees, and magenta falls between 301-360 degrees). In some examples, the saturation value represents an amount of gray in the color of the respective pixel and can be provided in a range of 0 to 1, where 0 is gray and 1 is a primary color. In some examples, the value in HSV represents the brightness of the color of the respective pixel and can be provided in a range of 0 to 100 percent, where 0 is completely black and 100 is the brightest.

In some implementations, the input imagecan be processed by the ROI extraction moduleto determine a ROI. In some examples, the ROI represents a region within the input imagethat the object is depicted within. In some examples, the ROI is determined based on a predefined range in the HSV color-space. For example, any pixel having a hue between 0 and 60, a saturation between 0.1 and 0.2, and a value between 0.3 and 0.4 is included in the ROI. In some examples, after determining the ROI, the input imageis adjusted (e.g., cropped) to largely depict the extracted ROI of the input image. For example, adjusting can eliminate entities (e.g., reflections, shadows) around the object that can negatively impact anomaly detection. The (adjusted) input image is processed by the anomaly detection module.

In some implementations, after determining the ROI based on the HSV of the input image, the image data (e.g., RGB value) of the adjusted input image can be normalized using mean and variance under ImageNet setting. The normalized (adjusted) input image can be processed by the anomaly detection module.

In some implementations, the anomaly detection modulecan include a ML model that processes the input image(e.g., the adjusted input image) to obtain a reconstructed image. In some examples, the ML model is provided as an auto-encoder (AE) that is trained to learn normal (non-defective) features of the object based on training data that includes images depicting non-defective (non-anomalous) examples of the object. In some examples, the AE can be a convolutional neural network (CNN)-based AE. In some examples, the AE can be a network that combines visual geometry group (VGG) network and U-net. For example, the U-net encoder is replaced with the VGG network, e.g., the combination of encoder of VGG and decoder of U-net. In image reconstruction, the ML model attempts to reconstruct features of the input imagetowards non-anomalous features in the resulting reconstructed image. For example, anomalous features are attempted to be reconstructed as non-anomalous features of the object to obtain the reconstructed image.

In general, the ML model is iteratively trained, where, during an iteration, one or more parameters (e.g., weights) of the ML model are adjusted, and an output is generated based on the training data. In some example, the ML model (e.g., AE) uses ImageNet pertained weights to accelerate the model training process. In the context of the present disclosure, the training data includes images depicting non-defective (non-anomalous) examples of the object. For each iteration, a loss value is determined based on a loss function. The loss value represents a degree of accuracy of the output of the ML model. The loss value can be described as a representation of a degree of difference between the image input to the ML model and the reconstructed image output by the ML model (e.g., reconstruction loss). In some examples, if the loss value does not meet an expected value (e.g., is not equal to zero), parameters of the ML model are adjusted in another iteration of training. In some instances, this process is repeated until the loss value meets the expected value.

In some implementations, a ML model is provided for each side of the object. For example, a set of images can be provided for the object, each image capturing a respective side of the object (e.g., top, bottom, left, right, front, back). Each image is processed through a respective ML module that is trained for the respective side of the object.

In some implementations, a gradient difference image is generated based on the reconstructed image and the input image(e.g., adjusted input image). For example, the reconstructed image is compared to the input imageto calculate a difference (e.g., gradient difference) there between. This is described in further detail herein with reference to.

The ML model is trained to be sensitive enough to capture any potential anomalies of the object represent in the input image. As a result, the gradient difference image can include a mix of true positives (true defects) and false positives (noises and other non-significant anomalies that do not count as defects). To remove any false positives, the gradient difference image is processed by the NSS moduleto suppress any non-significant anomalies. In some examples, a non-significant anomaly is an anomaly that is represented in the reconstructed image that does not qualify as an actual anomaly. Examples can include, but are not limited to, reflections (e.g., from a camera flash) and dirty spots. The NSS modulecan include a NSS map that provides representations of non-significant anomalies (that do not count as defects) in images (e.g., images in the training data used to train the ML model) depicting non-defective (non-anomalous) examples of the object. The NSS modulecan suppress non-significant regions that include such non-significant anomalies by subtracting the NSS map from the gradient difference image to provide a NSS output image. The NSS output image is absent non-significant anomalies. This is described in further detail herein with reference to.

In some implementations, a NSS map is provided for each side of the object. For example, a set of gradient difference images can be provided for the object, each gradient difference image corresponding to a respective side of the object (e.g., top, bottom, left, right, front, back). Each gradient difference image is processed using a respective NSS map that represents non-significant regions of a respective side.

In some implementations, the debris removal modulecan further remove the debris (such as noises) that do not count as defects to reduce false positives. In some examples, the debris can be removed from the NSS output image based on a size of connected components depicted in the NSS output image. For example, contours of the connected components in the NSS output image can be analyzed to calculate their sizes. If the size of a connected component is less than a size threshold, the connected component is removed from the NSS output image as debris. In other words, removing the debris includes removing connected components whose sizes are less than a size threshold. The output imageis provided from the debris removal module.

In accordance with implementations of the present disclosure, the anomaly detection systemcan process the output imageto determine whether any anomal-y/-ies that represent defects in the object (e.g., the anomalies are actual defects in the object). The anomalies representing actual defects can be determined based on that one or more pixel values of the output image(with NSS and debris removed) satisfy (e.g., beyond) a threshold value. The anomaly detection systemcan selectively send an alert based on the detection result. The alert can indicate that the object includes at least one defect. In some examples, the anomaly detection systemcan display the alert in a user interface. The user interface can include various output interfaces, such as a speaker or a screen. In some examples, the alert can include an image of the object that highlights the detected defect(s).

depict example images for anomaly detection using NSS in accordance with implementations of the present disclosure. The example ofrepresents a difference between an input imageand a reconstructed imageto provide a gradient difference image. The example ofincludes a difference between the gradient difference imageand a NSS mapto provide an output image.

In further detail, as shown in, the input imageis the extracted ROI in the input image. The reconstructed imageis the reconstructed image of ROI. The gradient difference imagecan be calculated by subtracting the reconstructed imagefrom the input image. As discussed above, the gradient difference imagecan include a mix of true positives (true defects) and false positives (noises and other non-significant anomalies that do not count as defects).

In some implementations, the input imageand the reconstructed imagecan be iteratively downscaled by half on each dimension to obtain image pyramids that contains a set of images with lower resolution. For example, the downscaled images can be at different scales, such as an original scale, a half scale (original scale/2), and a quarter scale (original scale/4). As a result, a set of images (under different scales) for the input imagecan be obtained. Similarly, a set of images (under different scales) for the reconstructed imagecan be obtained. At each scale, the horizontal and vertical gradient values of the two sets of images can be calculated. In some examples, the gradient values can be calculated using image processing filter methods, such as Prewitt filters. Based on the horizontal and vertical gradient values, the degree of difference between the input imageand the reconstructed imageunder each scale can be obtained. The degree of difference at different scales can be combined to obtain a weighted combination (such as a weighted average). The weighted combination can be the gradient difference image.

As shown in, the non-significant anomalies are removed from the gradient difference imageto reduce false positives. The non-significant anomalies are included in the NSS map. By subtracting the NSS map(including the non-significant anomalies) from the gradient difference image, a NSS output imageis obtained. The NSS output imageis an image that is absent non-significant anomalies. As a result, the false positives caused by non-significant anomalies can be removed from the gradient difference image. The NSS output imagecan be further processed using the debris removal moduleto remove debris, as discussed above.

depicts a NSS map generation systemin accordance with implementations of the present disclosure. In the example of, the NSS map generation systemincludes a trained AE module, a difference map generator module, and a NSS map generator module. As described in further detail herein, the NSS map generation systemreceives a set of imagesand processes the set of imagesto generate a NSS map. In some examples, the set of imagesincludes images of samples of the object that are absent any defect anomalies (non-defective samples). However, the set of imagescan include non-significant anomalies that do not count as defects. The NSS map generation systemcan generate the NSS mapto account for such non-significant anomalies represented in the set of images. In some examples, the set of imagesare used as training data to train the ML model of the anomaly detection moduleof.

The set of imagesare processed by the trained AE modulethat restores each image in the set of imagesto obtain a respective reconstructed image in a set of reconstructed images′. The difference map generator modulecan compare each image in the set of imagesand its respective reconstructed image in the set of reconstructed images′ to determine a difference map for each image. In some examples, the difference map generator modulecan average the difference maps of the set of images to obtain the difference map for the object. The difference map can include a set of difference components at a pixel level.

The NSS map generator modulecan generate the NSS mapthat includes the non-significant anomalies (that do not count as defects) based on selecting difference components that satisfy a difference threshold. For examples, the NSS map generator modulecan determine whether the difference components included in the difference map satisfies the difference threshold. The difference threshold can be a predetermined value, such as a value in the range of [0, 1]. For instance, a difference threshold of 0.3 indicates that there is a 30% difference in the pixel level. If a difference component satisfies (at or larger than) the difference threshold, the difference component is included in the NSS map, otherwise, the difference component is not included in the NSS map. For example, for each pixel on the difference map, if its value is larger than the difference threshold, the pixel is included in the NSS map, otherwise, the pixel is set to zero. As a result, the NSS mapcan include the differences representing anomalies in the image samples of the non-defective object. Such anomalies are thus non-significant anomalies that do not count as defects.

In some implementations, the difference threshold used by the NSS map generator modulecan be initially a human-selected value and then updated as objects are identified as defective using the anomaly detection systemof. For example, and as described in further detail herein, the difference threshold can be updated based on images that are input to the anomaly detection systemand that are determined to depict defective objects.

depicts a threshold determination systemin accordance with implementations of the present disclosure. In the example of, the threshold determination systemincludes the trained AE moduleand a difference threshold module. As described in further detail herein, the threshold determination systemreceives a set of imagesand processes the set of imagesto determine a difference threshold (Δ). In some examples, the set of imagesincludes images of samples of the object that have one or more defect anomalies (defective samples).

The set of imagesare processed by the trained AEthat reconstructs each image to provide a respective reconstructed image in a set of reconstructed images′. The difference threshold modulecan compare each image in the set of imagesto its respective reconstructed image in the set of reconstructed images′ to determine a set of differences at the pixel level. The minimum difference among the set of differences can be determined as the difference threshold (Δ). After obtaining the difference threshold, the NSS map generation systemofcan use the updated difference thresholdin the NSS map generator moduleto update the NSS map. The updated NSS mapcan be used by the NSS moduleoffor anomaly detection as described herein.

depicts an example processthat can be executed in implementations of the present disclosure. In some examples, the example processis provided using one or more computer-executable programs executed by one or more computing devices. The example processofrepresents anomaly detection in accordance with implementations of the present disclosure.

An image is received (). For example, and as described herein, the image can be an image of an object captured by a camera (e.g., the cameraof). For example, the image can depict a front, a back, a left, a right, a top, or a bottom side of the object. A ROI is determined (). For example, and as described herein, the ROI can be extracted based on a predefined range in HSV space, the ROI represents a region within the image that the object is depicted within. In some examples, the image is adjusted (e.g., cropped), such that only the ROI is depicted within the image.

One or more anomalies are detected (). For example, and as described herein, the image (e.g., adjusted image) can be processed using a ML model (e.g., AE) to obtain a reconstructed image. In image reconstruction, the ML model attempts to reconstruct features of the input imagetowards non-anomalous features in the resulting reconstructed image. After obtaining the reconstructed image, the image and the reconstructed image are compared to calculate a difference (e.g., a gradient difference) there between. For example, a gradient difference image is obtained based on the comparison (e.g., subtracting the reconstructed image from the image). The gradient difference image can include any potential anomalies. Among the potential anomalies, there can be true positives (true defects) and false positives (noise and other non-significant anomalies that do not count as defects).

One or more non-significant regions are suppressed (). For example, and as described herein, the gradient difference image can be further processed to remove potential false positives, such as non-significant regions that include non-significant anomalies that are not considered as defects. The non-significant anomaly regions can be represented in a NSS map. The one or more non-significant anomaly regions can be suppressed by subtracting the NSS map from the gradient difference image. As a result, a NSS output image is obtained.

Debris is removed (). For example, and as described herein, any debris on the object and depicted in the NSS output image can be removed to reduce false positives. In some examples, the debris can be removed from the NSS output image based on the size of connected components included in the NSS output image. For example, the contours of the connected components in the NSS output image can be analyzed to calculate their sizes. If the size of a connected component is less than a size threshold, the connected component is removed from the NSS output image as debris. In other words, removing the debris includes removing connected components whose sizes are less than a size threshold.

An anomaly image is output (). For example, and as described herein, the anomaly image is an output image that is absent any non-significant anomalies and debris. Any anomalies included in the anomaly image are actual defects of the object. A detection result of defect anomaly is determined and an alert is selectively sent (). If the anomaly image includes anomalies after the non-significant anomalies and debris being removed, the anomalies can be determined as defect anomalies (actual defects) of the object. An alert can be selectively sent based on the detection result. The alert can indicate that the object is defective. In some examples, the anomaly detection system can display/provide the alert in a user interface. The user interface can include various output interfaces, such as a speaker or a screen.

Implementations of the present disclosure achieve one or more technical improvements. For example, by suppressing non-significant anomalies and other debris/noises that do not count as defects from detected anomalies, the anomaly detection system of the present disclosure can reduce false positives, increase defective anomaly detection accuracy, and thus improve product quality control. Furthermore, the ML learning model of the present disclosure is trained using a set of images that are absent defect anomalies. The set of images is of a limited size (e.g., including 200-300 non-defective image samples). The dataset for model training of the present disclosure is significant reduced comparing to typical defect detection algorithms that need thousands of defect samples for each defect type. Implementations of the present disclosure can improve resource usage efficiency and data processing efficiency. Implementations of the present disclosure can provide quality inspection at a relatively fast speed (e.g., faster than human).

Implementations and all of the functional operations described in this specification may be realized in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations may be realized as one or more computer program products (i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus). The computer readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “computing system” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or any appropriate combination of one or more thereof). A propagated signal is an artificially generated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to suitable receiver apparatus.