Systems and Methods for Performing Line Clearance and Monitoring

The present application relates generally to the use of imaging systems and image analysis algorithms to identify unexpected items on or near a manufacturing line. More specifically, the present application relates to systems and methods for performing line clearance and monitoring in biopharmaceutical processes and applications.

There exists a multitude of manufacturing processes that require the reconciliation between input materials of a process with output materials of the process. The procedure for achieving this reconciliation is generally known as line clearance. Line clearance is a prominent issue particularly on biopharmaceutical manufacturing lines, and conventionally involves a standardized procedure for ensuring that equipment and work areas are free of products, documents, and materials from a previous process (e.g., manufacturing line run). Broadly, line clearance procedures help operators prepare for the next scheduled process and avoid mislabeling or cross-contamination of finished products.

However, conventional line clearance procedures suffer from numerous drawbacks. Namely, conventional line clearance procedures involve operators manually clearing packaging lines after each lot, and manually inspecting all areas of the manufacturing line to ensure no components or materials have remained at the conclusion of a process. This conventional procedure is time consuming, can require two-person verifications, and generally raises safety and ergonomic concerns for the human operators involved. Moreover, as both the physical inspection operations and the documentation completion of conventional procedures is almost entirely manual, these conventional procedures often introduce a significant amount of human error. As a result, these conventional line clearance procedures inevitably delay subsequent manufacturing operations, place operators in compromising/dangerous positions within the manufacturing line to conduct the manual inspections, yield mislabeled or cross-contaminated products from manual error, and/or result in hazardous conditions when the manual inspections miss or otherwise overlook stray objects in the manufacturing line.

Accordingly, there is a need for line clearance systems and methods for performing line clearance and monitoring in biopharmaceutical processes and applications that enables operators to easily, efficiently, and safely monitor and clear manufacturing lines during and after live operations.

Generally, the systems and methods of the present disclosure may include a camera system that may be placed on a benchtop, or on a manufacturing line, to remotely view and record video and images of the manufacturing line and surrounding areas on a network. The camera(s) may run a motion detection or machine learning (ML) algorithm to record and save a video when anomalous events (e.g., falling objects or stationary objects) occur outside of expected regions, and can notify users/operators in real-time. For example, the systems and methods of the present disclosure may provide immediate notification(s) of dropped or dislodged product(s) that travel through manufacturing lines, and may provide video/image evidence of the final location of the dropped or dislodged product, which decreases downtime and increases overall line clearance quality. The live/real-time video and recorded video may be accessed within enterprise networks, manufacturing networks, and/or private cloud servers, and the recorded video may be stored for historical reference/records. Further, the systems and methods of the present disclosure are modular, such that any number of devices may be used on a single manufacturing line, and these devices may be coordinated using on-premises or remote computer systems. The systems and methods of the present disclosure may include multiple cameras installed at select locations in, near, around, and/or otherwise proximate to the manufacturing line to provide a large field of view (FOV) corresponding to the manufacturing line. The systems and methods of the present disclosure may also enable a user/operator to view the processes and line clearance operations associated with the manufacturing line in real-time through live camera feeds.

Overall, the systems and methods of the present disclosure may yield significant advantages over conventional techniques, at least including: (1) substantial (e.g., approximately 60%) reduction in time spent performing line clearance, monitoring, and reconciliation; (2) increased manufacturing line production time/up-time (e.g., approximately 20 days per year); (3) increase in safety and a corresponding reduction in ergonomic issues due to elimination of manual inspections of hard-to-reach and/or otherwise hazardous areas within a manufacturing line; and (4) fewer issues/deviations in manual clearance and documentation operations due to a universal reduction in human-introduced error.

In particular, aspects of the present disclosure provide a computer-implemented method for performing line clearance and monitoring, comprising: receiving, by one or more processors, a first set of images of a manufacturing line during a run-time operation of the manufacturing line, the first set of images representing a first field of view (FOV) that is oriented to capture objects while falling from the manufacturing line; receiving, by the one or more processors, a second set of images of the manufacturing line during the run-time operation of the manufacturing line, the second set of images representing a second FOV that is different from the first FOV and oriented to capture objects positioned below the manufacturing line; analyzing, by the one or more processors, the first set of images and the second set of images to identify (i) a falling object within the first FOV or (ii) a stationary object within the second FOV; and responsive to identifying the falling object or the stationary object, causing, by the one or more processors, a display to present a notification, wherein the notification includes an image of the falling object or the stationary object.

In some aspects, generating the notification further comprises: responsive to identifying the falling object or the stationary object, triggering, by the one or more processors, a recording of multiple images from either the first set of images or the second set of images, each image of the multiple images depicting the falling object or the stationary object; and causing, by the one or more processors, a display to present the notification, wherein the notification includes the recording.

In certain aspects, the computer-implemented further comprises: masking a portion of the first set of images or the second set of images prior to analyzing the first set of images or the second set of images, the portion of the first set of images or the second set of images corresponding to one or more moving components of the manufacturing line.

In some aspects, generating the notification further comprises: generating the notification substantially in real-time for display at the user computing device in response to identifying the falling object or the stationary object, wherein the notification comprises at least one of: (i) an email message, (ii) a text message, or (iii) a line monitoring application alert.

In certain aspects, analyzing the first set of images and the second set of images may further comprise analyzing, by the one or more processors, the first set of images by applying a first algorithm and the second set of images by applying a second algorithm to identify (i) the falling object within the first FOV or (ii) the stationary object within the second FOV. In these aspects, both the first algorithm and the second algorithm may be, for example, a motion detection algorithm or a ML algorithm/model. Additionally, in these aspects, the first algorithm may be (i) a motion detection algorithm or (ii) a machine learning (ML) algorithm trained with a plurality of training data comprising a plurality of training images representing the manufacturing line, and wherein the ML algorithm is configured to receive image data of the manufacturing line as input and to output an anomaly score corresponding to a confidence level associated with detection of the falling object or the stationary object; and the second algorithm is (i) the motion detection algorithm or (ii) the ML algorithm. Further in these aspects, the computer-implemented may further comprise: training the ML model using the plurality of training images representing the manufacturing line, wherein the plurality of training images represent the manufacturing line operating (i) without a falling object within the first FOV and (ii) without a stationary object within the second FOV. Moreover, in these aspects, the ML algorithm may be at least one of (i) an anomaly detection algorithm, (ii) an image classification algorithm, or (iii) an object detection algorithm.

Another aspect of the present disclosure provides a computer system for performing line clearance and monitoring including, one or more processors; and a program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to perform the method of any one of the previous aspects.

Further aspects of the present disclosure provide a tangible, non-transitory computer-readable medium storing executable instructions for performing line clearance and monitoring, that when executed by one or more processors of a computer system, cause the computer system to perform the method of any one of the previous aspects.

In accordance with the above, and with the disclosure herein, the present disclosure includes improvements in computer functionality or in improvements to other technologies at least because the present disclosure describes that, e.g., line clearance and monitoring systems, and their related various components, may be improved or enhanced with the disclosed methods, computer systems, and tangible, non-transitory computer-readable mediums that provide more accurate, efficient, and safer performance of line clearance and monitoring procedures. That is, the present disclosure describes improvements in the functioning of a line clearance and monitoring system itself or “any other technology or technical field” (e.g., the field of line clearance and monitoring) because the disclosed methods, computer systems, and tangible, non-transitory computer-readable mediums improve and enhance operation of line clearance and monitoring systems by introducing imaging devices incorporating multiple algorithms that are specifically configured to monitor/analyze active manufacturing line operations and thereby eliminate errors and inefficiencies typically experienced over time by line clearance and monitoring systems lacking such methods, computer systems, and tangible, non-transitory computer-readable mediums. This improves over the prior art at least because such previous systems are error-prone, as they lack the ability to accurately, consistently, or efficiently analyze line clearance and perform line monitoring.

In addition, the present disclosure includes applying various features and functionality, as described herein, with, or by use of, a particular machine, e.g., imaging devices, computing systems, and/or other hardware components as described herein.

Moreover, the present disclosure includes effecting a transformation or reduction of a particular article to a different state or thing, e.g., transforming or reducing the error rate and/or the down-time of a manufacturing line from a non-optimal or error state to an optimal state as a result of accurate line clearance and monitoring based on real-time video and/or image analysis by multiple algorithms specifically configured to analyze particular areas of the manufacturing line.

Still further, the present disclosure includes specific features other than what is well-understood, routine, conventional activity in the field, or adding unconventional steps that demonstrate, in various embodiments, particular useful applications, e.g., analyzing, by applying a first algorithm, the first set of images to identify a falling object within the first FOV; analyzing, by applying a second algorithm, the second set of images to identify a stationary object within the second FOV; and generating a notification for display at a user computing device, wherein the notification includes an image of the falling object or the stationary object.

Additional advantages of the presently disclosed techniques over conventional approaches of line clearance and monitoring will be appreciated throughout this disclosure by one having ordinary skill in the art. The various concepts and techniques introduced above and discussed in greater detail below may be implemented in any of numerous ways, and the described concepts are not limited to any particular manner of implementation. Examples of implementations are provided below for illustrative purposes.

1 FIG.A 1 FIG.A 1 FIG.A 100 160 100 100 100 110 150 160 162 110 160 150 170 170 170 100 100 110 160 162 150 is a simplified block diagram of an example systemA for performing line clearance and monitoring in biomanufacturing process machinery, which, for example, may produce a drug product. In some embodiments, the systemA includes standalone equipment, though in other embodiments the systemA is incorporated into other equipment. At a high level, the systemA includes components of a computing device, one or more training image data sources, the biomanufacturing process machinery, and one or more imaging devices. In, the computing device, the biomanufacturing process machinery, and the training image data sourcesare communicatively coupled via a network, which may be or include a proprietary network, a secure public internet, a virtual private network, and/or any other type of suitable wired or wireless network(s) (e.g., dedicated access lines, satellite links, cellular data networks, combinations of these, etc.). In embodiments where the networkcomprises the Internet, data communications may take place over the networkvia an Internet communication protocol. In some aspects, more or fewer instances of the various components of the systemA than are shown inmay be included in the systemA (e.g., one instance of the computing device, ten instances of the biomanufacturing process machinery, ten instances of the imaging devices, two instances of the training image data sources, etc.).

100 160 100 160 100 It is worth noting that while the systemA is illustrated as including the biomanufacturing process machinery, one of ordinary skill in the art will understand that the present techniques and components of the systemA may be applied to performing line clearance and monitoring in other processes or fields. For example, instead of the biomanufacturing process machinery, the present techniques and components of the systemA may be applied to manufacturing in food/beverage, automotive, electronic, chemical, and/or other industries.

160 160 The biomanufacturing process machinerymay include a single biomanufacturing process machine, or multiple biomanufacturing process machines that are either co-located or remote from each other and are suitable for producing biological products, such as drug products. The biomanufacturing process machinerymay generally include physical devices configured for use in producing (e.g., manufacturing) biological products (e.g., drug products), such as filling devices, agitating devices, starwheels or other vessel conveyances, and so on.

160 110 170 160 110 160 160 160 The biomanufacturing process machinerymay, in some embodiments, be connected with the computing deviceeither via the network, or directly, allowing for at least some of the functionality of the biomanufacturing process machineryto be controlled by the computing device. In some embodiments, the biomanufacturing process machinerymay be capable of receiving instruction directly from a user (e.g., the biomanufacturing process machinerymay be manually-configurable). For example, in some embodiments, the biomanufacturing process machinerymay receive instructions directly from a user to control operation (e.g., start or stop operation).

162 160 160 160 162 160 162 110 170 160 110 160 162 110 110 162 160 162 160 160 The imaging devicesmay be included in the biomanufacturing process machinery(e.g., integrated into the biomanufacturing process machinery) or may be external devices connected to and/or otherwise located proximate to the biomanufacturing process machinery. The imaging devicesmay be used to collect video/image data inside, outside, and/or around the biomanufacturing process machinery. The imaging devicesmay provide the video/image data to, for example, the computing device(e.g., via the network). The video/image data may be any suitable data type, such as real-time video data of a manufacturing line included as part of the biomanufacturing process machinery, single image frames of the manufacturing line, and/or any other suitable data type or combinations thereof. The video/image data may be collected or provided automatically, or in response to a request. For example, a user of the computing devicemay wish to monitor the manufacturing line in the biomanufacturing process machineryover a period of time. In response, one or more of the imaging devicesmay collect and provide the video/image data of the manufacturing line to the computing deviceover the period of time, and/or may transmit a live video stream of the manufacturing line to the computing deviceover the period of time or a portion thereof. In some aspects, the imaging devicesmay collect video/image data in response to the biomanufacturing process machineryoperating. For example, the imaging devicesmay begin collecting video/image data when the biomanufacturing process machineryis powered on/begins operation and may continue collecting video/image data until the biomanufacturing process machineryis powered off/ends operation.

160 160 160 160 160 110 170 160 160 160 130 The biomanufacturing process machinerymay include one or more devices (not shown) used in manufacturing of biological products (e.g., drug products, as discussed in the Background Section). The biomanufacturing process machinerymay be configured to be controllable via manual or automated inputs. In some embodiments, the biomanufacturing process machinerymay be configured to receive such control inputs locally, such as via a user input device local to the biomanufacturing process machinery. In some embodiments, the biomanufacturing process machineryis configured to receive control inputs remotely, such as from the computing device(e.g., via the network). The control inputs may include operation instructions, such as instructing the biomanufacturing process machineryto power on/begin operation. In some aspects, the biomanufacturing process machinerymay end operation in response to one or more of: (i) the biomanufacturing process machinerycompleting production of biological product (e.g., a full batch of drug product is finished), (ii) an instruction from the line monitoring applicationrelated to line clearance and monitoring, or (iii) receiving a manual instruction to end operation.

150 160 110 162 162 100 150 110 150 110 The training image data sourcesgenerally include training video/image data that may correspond to (e.g., may have been collected during performance of) one or more biomanufacturing processes for producing one or more biological products using the biomanufacturing process machinery. The training video/image data may represent: (i) manufacturing line components, (ii) a manufacturing line floor area, (iii) a manufacturing line interior (e.g., gaps between components, etc.), and/or other suitable areas or portions of areas related to the manufacturing line. Further, the training video/image data may have been collected (by computing deviceor another device/system) using image device(s)or other, similar sensors. In some aspects, the training video/image data includes image data corresponding to each component and region of the manufacturing line. As such, the imaging device(s)may have a collective field of view (FOV) that includes each component and/or region of the manufacturing line, such that the models and algorithms described herein may be trained and/or otherwise configured to analyze subsequent run-time image data of any component or region of the manufacturing line based on the training video/image data. In some embodiments, the systemA may omit the training image data sources, and instead receive the training video/image data locally, such as via user input at the computing device(e.g., a user providing a portable memory drive with the training video/image data). In some examples, the training video/image data includes video/image data that does not include any unexpected and/or otherwise rogue items (e.g., product containers) that is combined (or mixed) with second video/image data that includes an item. In these examples, the training image data sourcesor the computing devicemay augment (or combine) the first video/image data and the second video/image data using techniques such as Poisson image blending, seamless cloning, or the like.

110 110 110 110 120 122 124 126 128 The computing devicemay include a single computing device, or multiple computing devices that are either co-located or remote from each other. The computing deviceis generally configured to input video/image data over a period of interest to at least one model/algorithm (e.g., trained using training video/image data) to analyze the video/image data to identify a falling object within a first FOV and/or a stationary object within a second FOV. Components of the computing devicemay be interconnected via an address/data bus or other means. The components included in the computing devicemay include a processing unit, a network interface, a display, a user input device, and a memory, discussed in further detail below.

120 128 110 120 The processing unitincludes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in the memoryto execute some or all of the functions of the computing deviceas described herein. Alternatively, one or more of the processors in the processing unitmay be other types of processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.).

122 162 160 150 170 122 The network interfacemay include any suitable hardware (e.g., front-end transmitter and receiver hardware), firmware, or software configured to use one or more communication protocols to communicate with external devices or systems (e.g., the imaging devices, the biomanufacturing process machinery, the training image data sources, etc.) via the network. For example, the network interfacemay be or include an Ethernet interface.

124 126 124 126 124 126 110 The displaymay use any suitable display technology (e.g., LED, OLED, LCD, etc.) to present information to a user, and the user input devicemay be a keyboard or other suitable input device. In some aspects, the displayand the user input deviceare integrated within a single device (e.g., a touchscreen display). Generally, the displayand the user input devicemay combine to enable a user to interact with graphical user interfaces (GUIs) or other (e.g., text) user interfaces provided by the computing device(e.g., for purposes such as notifying users of line clearance/monitoring actions, etc.).

128 110 128 130 120 100 130 132 134 136 138 140 132 140 130 130 132 136 132 140 130 110 110 The memoryincludes one or more physical memory devices or units containing volatile or non-volatile memory, and may or may not include memories located in different computing devices of the computing device. Any suitable memory type or types may be used, such as read-only memory (ROM), solid-state drives (SSDs), hard disk drives (HDDs), etc. The memorymay store instructions for one or more software applications included in a line monitoring applicationthat can be executed by the processing unit. In the example systemA, the line monitoring applicationincludes a data collection unit, a model training unit, a user interface unit, a unexpected item detection unit, and a notification unit. The units-may be distinct software components or modules of the line monitoring application, or may simply represent functionality of the line monitoring applicationthat is not necessarily divided among different components/modules. For example, in some embodiments, the data collection unitand the user interface unitare included in a single software module. Moreover, in some embodiments, the units-may be distributed among multiple copies of the line monitoring application(e.g., executing at different components in the computing device), or among different types of applications stored and executed at one or more devices of the computing device.

132 132 132 150 136 126 132 162 136 126 110 132 110 162 132 The data collection unitis generally configured to receive data (e.g., video/image data, operator instructions, etc.). In some embodiments, the data collection unitreceives the training video/image data (e.g., including historical video/image data of a plurality of instances of the biomanufacturing process and corresponding historical video/image data) of a biomanufacturing process for producing a biological product. The data collection unitmay receive the training video/image data via, for example, the training image data sources, user input received via the user interface unitwith the user input device, or other suitable means. In some embodiments, the data collection unitmay receive video/image data via, for example, the imaging devices, user input received via the user interface unitwith the user input device, or other suitable means. In some embodiments, the computing devicemay receive at, for example, the data collection unitan indication that a biomanufacturing process has begun and one or more components of the computing devicemay begin monitoring video/image data provided, e.g., by the imaging devices. In some aspects, the data collection unitmay apply pre-processing to received video/image data, for example, resizing, re-orienting, color balancing, etc. applied to one or both of training video/image signals or video/image signals, wherein the training video/image signals and the video/image signals respectively correspond to the training video/image data and the video/image data.

134 100 110 162 The model training unitis generally configured to generate, train, or apply a model. The model may be any suitable model for analyzing video/image data to identify falling or stationary objects. In some embodiments, and as discussed further below, the model may be trained using at least some of the systemA, or, in some embodiments, the model may be pre-trained (i.e., trained prior to being obtained by the computing device). The model may be trained using training video/image data that represent: (i) manufacturing line components, (ii) a manufacturing line floor area, (iii) a manufacturing line interior (e.g., gaps between components, etc.), and/or other suitable areas or portions of areas related to the manufacturing line. In some aspects, the model may include a statistical model, a rules-based model, or other suitable models or combinations thereof to analyze images captured by the imaging device(s)by performing, for example, motion detection on the video/image data. Accordingly, in these aspects, the model may include any suitable image processing algorithm, such as a motion detection algorithm, a dithering algorithm, a feature detection algorithm, a seam carving algorithm, a segmentation algorithm, and/or any other suitable image processing algorithm or combinations thereof.

134 150 130 In other embodiments, the model includes a machine learning model. For example, the model may employ a neural network, such as a convolutional neural network or a deep learning neural network. Other examples of machine-learning models in the model are models that use support vector machine (SVM) analysis, K-Nearest neighbor analysis, naïve Bayes analysis, clustering, reinforcement learning, or other machine-learning algorithms or techniques. Machine learning models included in the model may identify and recognize patterns in training data in order to facilitate making predictions for new data. The model training unitmay train the model using the training video/image data that may be received from the training image data sources. Of course, generally speaking, the line monitoring applicationmay include any ML models, statistical models, rules-based models, and/or any other suitable models/algorithms in any suitable combination to identify falling objects and/or stationary objects in video/image data.

130 150 In particular, when at least one of the models included as part of the line monitoring applicationis a machine learning model, the model may be universal (i.e., applicable to all circumstances), or may be more specific (i.e., different models for different circumstances). The machine learning model may be trained using a supervised or unsupervised machine-learning program or algorithm. The machine-learning program or algorithm may employ a neural network, which may be a convolutional neural network (CNN), a deep learning neural network, or a combined learning module or program that learns in two or more features or feature datasets in a particular areas of interest. The machine-learning programs or algorithms may also include regression analysis, support vector machine (SVM) analysis, decision tree analysis, random forest analysis, K-Nearest neighbor analysis, naïve Bayes analysis, clustering, reinforcement learning, and/or other machine-learning algorithms or techniques or combinations thereof. In some embodiments, due to the processing power requirements of training machine learning models, the selected model may be trained using additional computing resources (e.g., cloud computing resources) based upon data provided by external sources (e.g., the training image data sources). The training data may be unlabeled, or the training data may be labeled, such as by a human. Training of the machine learning model may continue until at least one model of the machine learning model is validated and satisfies selection criteria to be used as a predictive model for identifying a falling object and/or a stationary object within video/image data. In one embodiment, the machine learning model may be validated using a second subset of the training data to determine algorithm accuracy and robustness. Such validation may include applying the machine learning model to the second subset of training data to identify a falling object and/or a stationary object within video/image data in the second subset of the training data. The machine learning model may then be evaluated to determine whether the machine learning model performance is sufficient based upon the validation stage predictions. The sufficiency criteria applied may vary depending upon the size of the training data available for training, the performance of previous iterations of machine learning models, or user-specified performance requirements.

To be most effective, the ML model(s) may be computationally inexpensive to allow for real or near-real-time identification of a falling object and/or a stationary object within video/image data (e.g., be capable of processing and classifying live video/image data at the edge—meaning, by the device itself—or capable of sending video/image data to the cloud for processing in real-time). Within the computational constraint driven by the device specifications, it is generally preferred that the ML model(s) maximize predictive power. Because falling objects and/or stationary objects are uncommon and the impact of detection is substantial (e.g., biomanufacturing process stoppage/delay), the ML model(s) may preferably have exceptionally strong predictive power to avoid false positive events (erroneously identifying a falling object and/or a stationary object in video/image data).

130 Generally, CNNs are well-suited for machine vision applications due to their pattern recognition capabilities. As will be appreciated, CNNs differ from standard multi-layer perceptrons (MLPs) by using convolutional layers wherein matrices of numbers commonly referred to as filters are convolved with an input image to generate a tensor representing a new image with an arbitrary number of channels. This new tensor can be subsequently convolved with a new set of filters in another convolutional layer, producing yet another tensor. The process repeats for each layer defined in the CNN. In a typical classification task, the final output of a CNN is a vector set representing the predicted likelihood of each class. The filters of the CNN can be trained and selected based on recognizing distinct patterns such as edges, corners, or shapes. Thus, in certain aspects, the ML model(s) included as part of the line monitoring applicationmay be or include a CNN configured to identify a falling object and/or a stationary object within video/image data.

136 136 124 126 134 136 126 160 162 136 136 In any event, the user interface unitis generally configured to receive user input. In one example, the user interface unitmay generate a user interface for presentation via the display, and receive, via the user interface and user input device, user-input training video/image data to be used by the model training unitwhen training the model. In another example, the user interface unitmay receive, via a user interface and user input device, inputs to start operation of the biomanufacturing process machineryor the imaging devices. The user interface unitmay also be used to display information. For example, the user interface unitmay be used to display an indication of a falling object or a stationary object represented in the video/image data.

138 134 110 138 132 138 132 138 138 The unexpected item detection unitmay also apply or access the model trained by the model training unit(or otherwise obtained by the computing deviceas a pre-trained model) and/or another model/algorithm (e.g., a motion detection algorithm) when analyzing the video/image data to identify a falling object and/or a stationary object. In some embodiments, the unexpected item detection unitbegins analyzing video/image data in response to the data collection unitreceiving video/image data. The unexpected item detection unitmay monitor video/image data as it is collected by the data collection unitin real-time, in near-real-time (i.e., with some buffer), or asynchronously (i.e., after the video/image data is fully collected over a period of interest). It should be understood that when the unexpected item detection unitis referred to as identifying a falling object and/or a stationary object, this also includes detecting that an object has fallen from the manufacturing line and/or is lying stationary on a floor or other surface proximate to the manufacturing line (as the unexpected item detection unitmay monitor in real-time).

140 140 136 140 110 140 160 138 128 160 The notification unitis generally configured to notify a user when a falling object and/or a stationary object has been identified, and/or to notify the user where the falling object and/or the stationary object is located with respect to the manufacturing line. The notification unitmay coordinate with the user interface unitto display a notification. The notification unitmay send an electronic message (e.g., e-mail, text, etc.) with a notification to a user of the computing deviceor an external computing device. In some aspects, the notification unitmay send control signals to stop operation of the biomanufacturing process machineryif a falling object and/or a stationary object is detected by the unexpected item detection unit. In some aspects, the notification may be stored (e.g., in the memory), possibly along with other data (such as operation data) related to the biomanufacturing process machinerythat may be useful in diagnosing the cause of the falling object and/or the stationary object.

130 110 130 138 130 110 134 132 140 170 In some aspects, some or all of the functionalities of the line monitoring applicationmay be provided by a third-party (i.e., not on the computing device). For example, the machine learning model and/or the other algorithms/models may be hosted by a third-party and the line monitoring applicationmay access the machine learning model and/or the other algorithms/models remotely by sending data (e.g., the video/image data) and receiving data (e.g., an identification of a falling object and/or a stationary object). In such example, the functionality of the unexpected item detection unitmay be hosted by the third-party. Turning to a different example, the machine learning model may be trained by a third-party and the line monitoring applicationmay receive the machine learning model remotely from the third-party (e.g., by the computing devicereceiving one or more elements of the machine learning model, such as weights or architecture). In such example, the functionality of the model training unitmay be hosted by the third-party. In other examples, one or more instances of functionality of any of the units-may be hosted by a third-party, on, for example, a remote server accessible via the network.

1 FIG.B 100 100 100 100 100 170 1 170 5 100 170 170 170 depicts various exemplary system configurationsB for performing line clearance and monitoring in biopharmaceutical processes and applications, in accordance with various aspects disclosed herein. Generally, the exemplary system configurationsB may correspond to various configurations of several components included in the example systemA, and/or may include fewer or additional components, as described herein. Each of these configurationsB may enable the actions described herein for performing line clearance and monitoring, such that some components of each of the configurationsB may be disposed proximate to a manufacturing line (e.g., imaging devicesA), while other components may not need to be disposed proximate to the manufacturing line (e.g., operator workstationsA). In particular, the exemplary system configurationB may include three distinct configurations: an Internet of Things (IoT) configurationA, a full cloud-based configurationB, and a local computing configurationC.

170 170 1 170 2 170 3 170 4 170 5 170 1 170 1 In the IoT configurationA, the system may generally include a set of imaging devicesA, a set of computing devicesA, a network switchA, a cloud-based platformA, and operator workstationsA. The set of imaging devicesAmay include any suitable type and/or number of imaging devices configured to capture video/image data corresponding to a manufacturing line or surrounding areas (e.g., a floor area proximate to the manufacturing line) before, during, and/or after operation. Due to the differences in lighting, available space, and other imaging parameters at various locations near and around the manufacturing line, these various locations may require different imaging devices to capture video/image data that is useful for the subsequent analysis described herein to perform line clearance and monitoring. Thus, the set of imaging devicesAmay include a variety of cameras and lens hardware that is specifically configured to monitor and capture video/image data specific part(s)/area(s) of the manufacturing line.

170 1 These set of imaging devicesAmay include, without limitation, standard FOV cameras with variable or fixed zoom, specialized wide angle (e.g., 180°+) lenses/cameras, gyroscopic style cameras, and/or any other suitable imaging device type or combinations thereof. Generally, the standard FOV cameras may be configured to capture video/image data corresponding to general observations of various stations/equipment in a manufacturing line. The specialized wide angle/area of observation cameras may be specifically configured to observe and capture video/image data corresponding to areas of larger physical volumes, such as floor space under the manufacturing line and associated equipment. Gyroscopic style cameras may be configured to capture video/image data that may correspond to observation regions in tighter spaces in and between equipment that other imaging devices are unable to properly capture. In addition to the cameras and lenses themselves, there may be a local camera/image processor to perform computations on images or videos and perform analysis.

170 2 170 1 170 2 170 1 170 3 170 2 170 4 170 2 The set of computing devicesAmay be or include one or more IoT devices, and may be communicatively coupled with one or more other devices (e.g., the set of imaging devicesA). For example, the set of computing devicesAmay include an interface for connecting to an imaging device (which may be one or more of the set of imaging devicesA), and may connect to and/or otherwise interact with a network switchAconfigured to transmit data between the set of computing devicesAand the cloud-based platformA. The set of computing devicesAmay be chosen for any suitable reason, such as for technical specifications enabling recording and storing of live video/image data while providing remote access and a simple user interface.

170 4 170 4 171 1 4 170 4 171 1 4 171 1 171 2 171 3 171 4 170 4 170 2 170 3 170 4 Generally, the cloud-based platformAmay be or include any suitable cloud-based computing platform, such as, for example, Amazon Web Services (AWS). The cloud-based platformAmay also include a plurality of web-based servicesA-A, that may perform a variety of services corresponding to the video/image data and/or notifications resulting therefrom. For example, in aspects where the cloud-based platformAis AWS, the plurality of web-based servicesA-Amay include, without limitation, AWS IoT CoreA, Amazon CloudWatchA, Storage Service (S3)A, and Amazon CognitoA. In some aspects, the cloud-based platformAmay receive video/image data from the set of computing deviceAvia the network switchA, and the platformAmay apply various algorithms/models to the video/image data to identify a falling object and/or a stationary object within the video/image data.

170 4 170 4 170 4 170 5 170 4 124 170 5 Further in these aspects, if the cloud-based platformAsuccessfully identifies the falling object and/or the stationary object within the video/image data, the platformAmay generate and/or cause a notification to be displayed to a user/operator that includes an image from the video/image data of the falling object and/or the stationary object. For example, the cloud-based platformAmay generate a notification by aggregating each image from the video/image data of the falling object and/or the stationary object, and transmitting the aggregated images to the operator workstationAfor display to the user/operator. Additionally, or alternatively, the cloud-based platformAmay simply cause a display (e.g., display, display of operator workstationA) to present a notification that includes each image from the video/image data of the falling object and/or the stationary object.

170 4 170 5 170 5 170 1 170 170 5 170 In particular, the cloud-based platformAmay generate and/or cause the notification to be displayed at an operator workstationAfor review by the user/operator. The operator workstationAmay be a computing device/system (e.g., a supervisory control and data acquisition (SCADA) system) that may be communicatively coupled to and/or otherwise configured to control operation of one or more components of the manufacturing line that is monitored by the set of imaging devicesA, and the IoT configurationA, more generally. Specifically, the operator workstationAmay be configured to communicate and coordinate activities and/or processes between the IoT configurationA and the manufacturing line equipment on operations such as timing, equipment operation, start/stop/hold/restart commands, and/or any other suitable commands or combinations thereof.

170 170 170 171 1 171 2 170 170 2 170 170 3 171 1 4 171 1 2 171 1 171 2 171 3 171 4 171 1 171 2 171 1 171 2 130 170 2 170 170 170 170 3 170 170 4 170 5 Broadly speaking, the full cloud-based configurationB includes many similar components to the IoT configurationA, with several differences. Namely, the full cloud-based configurationB utilizes additional cloud-based servicesBandB, relative to the IoT configurationA, to account for the lack of the set of computing devicesAincluded as part of the IoT configurationA. In particular, and in aspects where the cloud-based platformBis AWS, the plurality of web-based servicesA-AandB-Bmay include, without limitation, AWS IoT CoreA, Amazon CloudWatchA, Storage Service (S3)A, Amazon CognitoA, Amazon KinesisB, and Amazon EC2B. The Amazon KinesisBand Amazon EC2Bweb services may generally host an application (e.g., line monitoring application) that is configured to perform and/or may otherwise independently perform video/image data processing that would otherwise be performed by the set of computing devicesAin the IoT configurationA. Otherwise, the full cloud-based configurationB may include a similar or the identical set of imaging devicesA, the network switchAconnecting the set of imaging devicesA directly to the cloud-based platformA, and the operator workstationA.

170 170 170 170 170 1 170 2 170 3 170 4 170 170 170 1 170 1 170 5 170 5 Similarly, the local computing configurationC may include similar components to both the IoT configurationA and the full cloud-based configurationB, with several differences. More specifically, the local computing configurationC includes a local computing deviceCthat may be configured to perform some/all of the video/image data aggregation, processing, and notification generation/transmission that is performed by some combination of the set of computing devicesA, the network switchA, and/or the cloud-based platformAin the IoT configurationA and the full cloud-based configurationB. Thus, the local computing deviceCmay receive live/real-time streaming video/image data from the set of imaging devicesA, analyze the video/image data in accordance with the various line clearance and monitoring operations/actions described herein, generate notifications corresponding to the video/image data analysis, transmit the notifications and/or the video/image data to the operator workstationAfor display to a user/operator, and/or cause a display of the operator workstationAto present a notification including video/image data of the falling object and/or the stationary object to a user/operator.

100 170 170 170 As a consequence of these various exemplary system configurationsB, it should be understood that the some/all of the processing steps/actions performed as part of line clearance and monitoring operations described herein may be performed remotely (e.g., in a configuration similar or identical to the IoT configurationA or the full cloud-based configurationB) and/or locally (e.g., in a configuration similar or identical to the local computing configurationC).

1 FIG.C 1 FIG.B 1 FIG.C 100 100 100 170 100 181 2 181 2 100 180 1 180 2 180 3 180 4 180 5 180 6 depicts another exemplary system configurationC for performing line clearance and monitoring in biopharmaceutical processes and applications, in accordance with various aspects disclosed herein. Generally speaking, the exemplary system configurationC may correspond to any of the various exemplary system configurationsB illustrated in, and more specifically, may correspond the IoT configurationA. In particular, the exemplary system configurationC generally illustrates how an IoT based architecture may function as a complementary system to the manufacturing line (e.g., manufacturing linesAandB). The data flow illustrated inbroadly includes data/notifications/commands corresponding to user management, event logging, video/image data transmission and recording, email and text message dispatching, and system operating commands. The exemplary system configurationC includes two manufacturing observation regionsAandA, an IoT upload pointA, a cloud-based notification/command storage serviceA, a local computing networkA, and a user/operator accountA.

180 1 180 2 181 2 181 2 181 1 181 1 181 3 181 3 181 4 181 4 181 5 181 5 181 1 181 1 181 1 181 1 162 170 1 More specifically, the two manufacturing observation regionsAandAmay each include multiple production/manufacturing components (e.g., included as part of the manufacturing linesA,B), as well as components configured to monitor the production/manufacturing components. These monitoring components include a set of imaging devicesAandB, a set of IoT topicsAandB, a cloud-based video/image data storage serviceAandB, and real-time operator alertsAandBthat may be generated as a result of video/image data captured by the set of imaging devicesAandB. The set of imaging devicesAandBmay be similar or identical to the imaging devices (e.g., imaging devices, set of imaging devicesA) described herein.

181 3 181 3 181 2 181 2 181 3 181 3 181 3 181 3 181 2 181 2 181 1 181 1 The IoT topicsAandBmay generally include operational commands related to the manufacturing linesAandB, such as stop commands, start commands, restart commands, hold commands, and the like. The IoT topicsAandBmay also include and/or otherwise generate/transmit notifications corresponding to motion alerts (e.g., associated with falling objects), device status (e.g., device stopped/halted), and/or other suitable notifications or combinations thereof. Moreover, the IoT topicsAandBmay generally organize the sets of commands and notifications by particular manufacturing lines (e.g., manufacturing linesAandB), batch identifications (e.g., particular batches of products and/or particular products), imaging device identifications (e.g., particular imaging device of the set of imaging devicesAandB), timestamps, and/or by any other suitable identifier/metric or combinations thereof.

181 4 181 4 181 1 181 1 181 4 181 4 181 2 181 2 180 3 181 3 181 3 180 3 181 4 181 4 180 6 180 5 181 3 181 3 The cloud-based video/image data storage serviceAandBmay generally receive video/image data from the set of imaging devicesAandB. More specifically, the cloud-based video/image data storage serviceAandBmay receive video/image data representative of motion events and/or other events taking place on or near the manufacturing linesAandB. Thus, when the IoT upload pointAreceives a notification from the IoT topicsAandBcorresponding to a motion alert, the IoT upload pointAmay also retrieve and/or otherwise receive the video/image data from the cloud-based video/image data storage serviceAandBfor any further processing, storage, and/or to transmit the video/image data to the user/operator accountAthrough the local computing networkA. Of course, in certain aspects, the notification received from the IoT topicsAandBmay include the video/image data.

181 3 181 3 180 6 181 5 181 5 181 1 181 1 181 5 181 5 130 As part of the notifications generated/transmitted as a result of the IoT topicsAandB, the user/operator accountAmay also receive the real-time operator alertsAandB, as a result of the video/image data captured by the set of imaging devicesAandB. These real-time operator alertsAandBmay be or include text messages, email messages, line monitoring application messages (e.g., messages received via the line monitoring application, as executed on a user/operator computing device), and/or any other suitable type of message or combinations thereof.

180 1 180 2 180 3 180 3 181 3 181 3 180 4 180 3 180 5 180 5 180 3 180 5 180 6 180 6 180 5 181 5 181 5 As mentioned, the commands, notifications, video/image data, and/other data generated and/or stored in the two manufacturing observation regionsAandAmay be transmitted to the IoT upload pointAfor further processing, storage, and/or transmission/routing to relevant components. For example, the IoT upload pointAmay transmit notifications and commands received from the IoT topicsAandBto the cloud-based notification/command storage serviceAfor storage. The IoT upload pointAmay also forward notifications, commands, video/image data, and/or any other data to the local computing networkAfor further processing, storage, or user/operator interaction. When the local computing networkAreceives the data from the IoT upload pointA, the local computing networkAmay route the data to an appropriate user/operator accountA. Through the user/operator accountA, the associated user/operator may view any/all notifications, commands, video/image data received from the local computing networkAand/or as a direct real-time operator alertAandB.

180 6 181 1 181 1 181 2 181 2 181 2 181 2 181 2 181 2 180 6 More specifically, the user/operator accountAmay enable the associated user/operator to analyze the data, and generally respond to the data. In certain aspects, the components configured to monitor the production/manufacturing components (e.g., set of imaging devicesAandBetc.) may be directly integrated with the manufacturing linesAandB, such that these components may communicate and coordinate with the manufacturing equipment of the manufacturing linesAandBon operations such as timing, equipment operation, start/stop/hold/restart commands, and the like. Therefore, in these aspects, the monitoring components may directly influence the operation of and/or otherwise control the manufacturing equipment of the manufacturing linesAandB, and the user/operator may view commands executed by the monitoring components as data uploaded to the user/operator accountA.

1 FIG.C 1 FIG.C 1 FIG.C 181 1 181 1 181 3 181 3 181 2 181 2 180 6 181 2 181 2 Alternatively, in some aspects and as illustrated in, the components configured to monitor the production/manufacturing components (e.g., set of imaging devicesAandB, IoT topicsAandB, etc.) may be configured in an add-on style of architecture that does not directly communicate with the manufacturing equipment of the manufacturing linesAandB. In these configurations, the monitoring components may require additional operator input through the user/operator accountAto execute control commands (e.g., timing, equipment operation, start/stop/hold/restart commands). Thus, the configuration illustrated inmay require significantly less integration time than the directly integrated configuration described previously, and may be substantially more modular, such that the configuration illustrated inmay readily apply to various manufacturing lines (e.g., manufacturing linesAandB).

2 FIG.A 200 202 202 204 206 202 202 208 202 204 208 202 204 110 208 204 206 204 204 204 depicts an example implementationof an imaging devicedisposed within a manufacturing line to perform line clearance and monitoring, in accordance with various aspects disclosed herein. Generally, the imaging devicemay be configured to capture real-time video/image data of a floor areathat is within a FOVof the imaging device. More generally, the imaging devicemay be a portable device placed as needed within a system, and/or may be integrated within and/or affixed to the manufacturing line equipment. In this manner, the imaging devicemay be positioned, oriented, and configured to capture video/image data that may include, for example, a stationary object that has fallen into the floor areafrom the overhanging manufacturing line equipment (referenced herein collectively as). The imaging devicemay continually capture video/image data of the floor area, and this video/image data may be streamed or periodically uploaded to a processing device (e.g., computing device) for analysis. When an object falls from the manufacturing line equipment(or elsewhere) onto the floor area, such that the object is within the FOV, the live stream video data and/or real-time image data may capture the moment when the object lands in the floor area. Accordingly, the systems and methods of the present disclosure may determine when and where the object landed on the floor area, and may subsequently perform actions sufficient to clear the object from the floor area, as necessary.

202 220 224 222 222 222 222 222 222 222 2 FIG.A 2 FIG.B 2 FIG.B Imaging devices identical to and/or similar to the imaging deviceinmay be positioned at a plurality of locations throughout a manufacturing line to capture video/image data corresponding to any relevant area of the line. For example,depicts an example implementationof a plurality of imaging devicesA-F disposed throughout manufacturing line equipmentto perform line clearance and monitoring, in accordance with various aspects disclosed herein. As illustrated in, the manufacturing line equipmentmay include multiple stationsA-F, wherein manufacturing components are positioned and configured to perform operations/processes that result in the manufacture of a particular product. Each stationA-F may perform specific operations that contribute a portion of the overall manufacturing process, such that an unfinished product may enter stationA and become incrementally completed at each stationA-F until a finished product exits stationF.

222 224 222 222 224 222 220 224 222 224 222 224 222 224 222 222 222 222 2 FIG.A At each stationA-F, the corresponding imaging deviceA-F may capture video/image data corresponding to the specific manufacturing line equipmentcomponents located at the respective stationA-F. For example, the imaging deviceB may capture video/image data corresponding to the specific components located at stationB. Moreover, it should be understood that while the example implementationillustrated indepicts a single imaging device (e.g., imaging devicesA-F) at each stationA-F, there may be multiple imaging devicesA-F at each stationA-F. In this manner, the multiple imaging devicesA-F may capture video/image data corresponding to multiple different FOVs, and may thereby provide a more complete perspective of the manufacturing line equipmentand the surrounding areas to more effectively perform line clearance and monitoring. For example, a first imaging device (e.g., imaging deviceC) positioned at stationC may be positioned/oriented to include equipment/manufacturing components that are part of the manufacturing line equipmentwithin the FOV of the first imaging device, such that the first imaging device captures video/image data corresponding to the equipment/manufacturing components. A second imaging device positioned at stationC may be positioned/oriented to include a floor area surrounding equipment/manufacturing components that are part of the manufacturing line equipmentwithin the FOV of the second imaging device, such that the second imaging device captures video/image data corresponding to the floor area surrounding the equipment/manufacturing components.

3 FIG.A 300 130 300 130 302 304 304 302 304 304 304 130 302 304 302 304 depicts an example line clearance and monitoring analysis actionperformed as part of the execution of a line monitoring application (e.g., line monitoring application), in accordance with various aspects disclosed herein. The example line clearance and monitoring analysis actiongenerally includes the line monitoring applicationreceiving an initial imageof a surrounding area of a manufacturing line, and a subsequent imageof the surrounding area that includes a stationary objectA. For example, the initial imagemay represent the area surrounding the manufacturing line at a first time instance, where there is no stationary object on the floor or general area surrounding the manufacturing line. At a second time instance, an unexpected objectA may fall from the manufacturing line and/or otherwise fall through the area surrounding the manufacturing line to land on the floor. Accordingly, the subsequent imagemay feature the stationary objectA, and the line monitoring applicationmay record and/or otherwise store the initial imageand the subsequent imagealong with the timestamps corresponding to the first time instance and the second time instance, during which the initial imageand the subsequent imagewere captured.

130 302 304 130 302 304 304 304 130 302 304 304 304 302 304 306 130 302 304 306 302 304 306 When the line monitoring applicationreceives the initial imageand the subsequent image, the applicationmay execute one or more algorithms on the initial imageand the subsequent imageto identify the stationary objectA represented in the subsequent image. In particular, the line monitoring applicationmay execute a motion detection algorithm on the initial imageand the subsequent imageto identify the stationary objectA represented in the subsequent image. The motion detection algorithm may include subtracting the initial imagefrom the subsequent image, and may thereby generate the subtracted image. Generally, if there are multiple pixels that exceed a specified threshold, then an event will be recorded by the line monitoring application, and the image data (e.g., initial image, the subsequent image, and/or the subtracted image) may be saved for operator review. In certain aspects, executing the motion detection algorithm may also include applying additional filters to either the initial imageand/or the subsequent imageto generate the subtracted image.

130 130 This line monitoring applicationmay also execute the motion detection algorithm to identify falling objects that are captured by imaging devices with high frame capture rates. In general, executing the motion detection algorithm with high frame capture rate imaging devices may result in the line monitoring applicationhaving a higher probability of detecting fast moving objects (e.g., falling objects). In certain aspects, the motion detection algorithm may also include a masking feature that will enable the motion detection algorithm to ignore areas of the captured images that may reliably include movement (e.g., a moving conveyor belt), while still enabling the motion detection algorithm to detect objects that leave the area of the conveyer belt.

170 170 170 As previously mentioned, in addition to using motion detection algorithms, other algorithms may be used to perform line clearance and monitoring operations, such as algorithms/models that utilize machine learning (ML) and artificial intelligence (AI). In certain aspects, these AI and ML models may be trained for each imaging device individually, and may be specifically tailored for the specific perspective and FOV the imaging device has on the manufacturing line. Of course, captured images may be processed using these AI/ML models on a local processor (e.g., in local computing configurationC), a centralized server located on premise, and/or in a cloud-based server environment (e.g., IoT configurationA or full cloud-based configurationB), but training these AI/ML algorithms is traditionally difficult as a result of requiring images that display a line clearance problem (e.g., stray containers, vials, or syringes) in the FOV. While it is simple/straightforward to obtain images of manufacturing line equipment during normal operation, it is substantially more difficult for conventional techniques to introduce a line clearance issue (e.g., stray containers) during normal operation, as doing so would venture outside of standard operating procedures written for many manufacturing processes (e.g., FDA approved manufacturing processes). To overcome this difficulty experienced by conventional systems, the present techniques may apply image augmentation techniques that augment images of stray containers to images of manufacturing line equipment taken during normal manufacturing.

3 FIG.B 320 130 320 130 322 322 324 324 130 322 324 130 324 130 324 324 304 To illustrate,depicts an example line clearance and monitoring analysis actionperformed as part of the execution of a line monitoring application (e.g., line monitoring application), in accordance with various aspects disclosed herein. The example line clearance and monitoring analysis actiongenerally includes the line monitoring applicationreceiving an input imageof a component of a manufacturing line, and augmenting the input imagewith an unexpected itemA (e.g., a vial) to generate an augmented image. Generally, the line monitoring applicationmay augment the input imagewith the image of the unexpected itemA utilizing image augmentation techniques including, for example and without limitation, Poisson image blending, seamless cloning, and/or other suitable image augmentation techniques or combinations thereof. When the line monitoring applicationgenerates the augmented image, the line monitoring applicationmay then use the augmented imageto train an AI/ML model to identify falling objects (e.g., the unexpected itemA) and/or stationary objects (e.g., stationary objectA).

3 FIG.C 3 FIG.C 3 FIG.B 340 130 340 130 324 324 324 324 324 130 130 324 324 304 130 More specifically,depicts an example line clearance and monitoring analysis actionperformed as part of the execution of the line monitoring application, in accordance with various aspects disclosed herein. The example line clearance and monitoring analysis actiongenerally represents training inputs and training outputs for training an AI/ML model, as trained and executed by the line monitoring application. The training inputillustrated inis the augmented imageof, so the training inputalso includes the unexpected itemA that has been augmented into the training inputby the line monitoring application. Thus, the line monitoring applicationmay input the training inputinto an AI/ML model to train the AI/ML model to identify falling objects (e.g., the unexpected itemA) and/or stationary objects (e.g., stationary objectA). In certain aspects, the AI/ML model that may be trained by the line monitoring applicationmay be or include an anomaly detection model, an image classification model, an object detection model, and/or any other suitable model/algorithm or combinations thereof.

324 342 342 342 324 324 324 342 324 130 As a result of inputting the training inputinto the AI/ML model, the model may output training outputs that may be presented as part of a line monitoring graphical user interface (GUI). For example, the line monitoring GUIincludes a training outputA that indicates the AI/ML model correctly identified the falling object within the training input. Additionally, the AI/ML model may output a score related to the identification of the falling object within the training input. This score may be an anomaly score that generally reflects the confidence with which the AI/ML model identified the anomaly (e.g., the falling object) within the training inputas the training outputA. The AI/ML model may condition identification of a falling object and/or a stationary object within training data (e.g., the training input) and/or live data (e.g., data captured during normal operation of a manufacturing line) on an identification threshold stored in the line monitoring application. In certain aspects, the identification threshold may be adjusted/set by the user/operator during training and/or before execution of the AI/ML model during normal operation of the manufacturing line.

3 FIG.D 3 FIG.D 3 FIG.D 360 130 364 1 2 364 1 364 1 2 364 1 2 364 1 364 1 2 360 130 362 362 360 130 362 362 depicts yet another example line clearance and monitoring analysis actionperformed as part of the execution of the line monitoring application, in accordance with various aspects disclosed herein. It should be appreciated that, whileshows the heatmap portionsA-A,B, andC-Cin grayscale shading and/or a patterning, the heatmap portionsA-A,B, andC-Care in some embodiments portrayed using color-coding. Regardless, in the example line clearance and monitoring analysis actionof, the line monitoring applicationmay receive a first imagethat features a portion of a manufacturing line during normal operation with an identified unexpected objectA. In this example line clearance and monitoring analysis action, the line monitoring applicationmay have executed a motion detection algorithm and/or an AI/ML model on the first imageto identify the unexpected objectA.

362 130 362 364 130 364 362 130 362 364 1 364 2 364 1 364 2 364 352 364 1 364 364 364 1 364 1 364 2 364 364 130 364 364 3 FIG.D Regardless, upon identification of the unexpected objectA, the line monitoring applicationmay access video/image data from some/all of the other imaging devices captured at the same or similar timestamp as the first image, and may execute the motion detection algorithm and/or an AI/ML model on the video/image data to identify any additional unexpected objects. In certain aspects, and as illustrated in the example line monitoring GUIof, the line monitoring applicationmay retrieve and analyze data from three additional imaging devices in order to generate a plurality of heatmaps corresponding to identified unexpected objects within and/or around the manufacturing line. The first GUI imageA may correspond to the first imagewhen the line monitoring algorithmapplies an algorithm configured to generate a heatmap graphical overlay on the first imageto generate the heatmap portionsAandA. These heatmap portionsAandAmay correspond to portions of the first GUI imageA that may include an unexpected object, and the unexpected objectA may be represented in the first heatmap portionA. Similarly, the second and third GUI imagesB andC may include multiple heatmap portionsB,C, andCthat also correspond to portions of the GUI imagesB andC that may include an unexpected object. The line monitoring applicationmay analyze the fourth GUI imageD, and may not detect any unexpected objects, such that the fourth GUI imageD may not include a heatmap graphical overlay.

364 362 364 130 364 1 364 2 362 362 364 364 364 362 362 In certain aspects, the heatmap graphical overlay may also indicate historical regions of the respective FOVs represented by the images of the example line monitoring GUIthat have included identified unexpected objects. Accordingly, the first imagemay influence the historical heatmap graphical overlay represented by the first GUI imageA by causing the line monitoring applicationto update the location and/or the depth of color/patterning/etc. representing the heatmap portionsAandAbased on the identified unexpected objectA within the first image. Moreover, the heatmap graphical overlay included as part of the first GUI imageA, the second GUI imageB, and the third GUI imageC may indicate areas within the manufacturing line that may have been the cause of an unexpected object (e.g., unexpected objectA) within an image (e.g., first image).

4 FIG. 402 130 124 402 130 124 130 402 402 402 130 124 depicts an example user interface, which may be presented by the line monitoring application (e.g., line monitoring application) via a display (e.g., display), that includes notificationsA-E to the user, in accordance with various aspects disclosed herein. Generally, the line monitoring applicationmay transmit notifications to and cause the notifications to be presented to a user/operator at a display (e.g., display) in response to one of the applied algorithms/models identifying an unexpected object (e.g., a falling object and/or a stationary object) within video/image data representative of the manufacturing line and/or areas surrounding the manufacturing line. For example, the line monitoring applicationmay transmit one or more notificationsA-E including a link to real-time video/image data that includes the unexpected object to be displayed as part of the user interface. When the user interacts (e.g., clicks, taps, swipes, etc.) with the link included in the notificationA-E, then the line monitoring applicationmay cause the displayto display the video/image data including the unexpected object.

402 402 130 130 130 4 FIG. In certain aspects, each of the notificationsA-E may correspond to different events wherein an unexpected object was identified in video/image data of the manufacturing line and/or areas surrounding the manufacturing line. While illustrated inas text messages, some or all of the notificationsA-E may additionally or alternatively transmitted by the line monitoring applicationas an email message and/or as a message within the line monitoring applicationthat the user may access by initiating the line monitoring application. In this manner, the user/operator may receive notifications that enable the user to access and analyze real-time video/image data corresponding to an identification of an unexpected object, and to take corrective action, such as executing and/or approving control commands to stop/halt operation of the manufacturing line in order to conduct line clearance operations.

5 FIG. 3 3 FIGS.A-C 500 500 100 120 130 160 500 500 502 504 506 508 510 is a flow diagram depicting an example methodfor performing line clearance and monitoring in biopharmaceutical processes and applications, in accordance with various aspects disclosed herein. The methodmay be implemented by one or more components of the systemA-C, such as the processing unitwhen executing instructions of the line monitoring application, and possibly also the biomanufacturing process machinery(which may be operating a biomanufacturing process). The methodmay be or include analysis that is the same as or similar to the example line clearance and monitoring analysis actions performed in. The example methodmay generally include the following elements: (1) receiving a first set of images (block), (2) receiving a second set of images (block), (3) analyzing, by applying a first algorithm, the first set of images to identify a falling object within the first FOV (block), (4) analyzing, by applying a second algorithm, the second set of images to identify a stationary object within the second FOV (block), and (5) generating a notification for display at a user computing device (block).

500 502 500 504 500 The methodmay include receiving a first set of images of a manufacturing line during a run-time operation of the manufacturing line (block). The first set of images may represent a first FOV that is oriented to capture objects while falling from the manufacturing line. The methodmay also include receiving a second set of images of the manufacturing line during the run-time operation of the manufacturing line (block). The second set of images may represent a second FOV that is different from the first FOV and oriented to capture objects positioned below the manufacturing line. In some aspects, the methodmay further comprise capturing the first set of images and the second set of images by at least one of: (i) a variable zoom imaging device, (ii) a fixed zoom imaging device, (iii) a wide angle imaging device, and/or (iv) a gyroscopic imaging device.

500 506 500 508 502 508 170 4 The methodmay further include analyzing the first set of images and the second set of images to identify (i) a falling object within the first FOV or (ii) a stationary object within the second FOV (block). The methodmay further include, responsive to identifying the falling object or the stationary object, causing a display to present a notification, wherein the notification includes an image of the falling object or the stationary object (block). In certain aspects, the falling object and the stationary object are a same object, such that the falling object in the first FOV is the same object as the stationary object in the second FOV. Further, in some aspects, the processors performing one or more of the actions included in blocks-may be cloud-based processors (e.g., hosted on cloud-based platformA).

In some aspects, generating the notification further comprises: responsive to identifying the falling object or the stationary object, triggering, by the one or more processors, a recording of multiple images from either the first set of images or the second set of images, each image of the multiple images depicting the falling object or the stationary object; and causing, by the one or more processors, a display to present the notification, wherein the notification includes the recording.

500 In certain aspects, the methodfurther comprises: masking a portion of the first set of images or the second set of images prior to analyzing the first set of images or the second set of images, the portion of the first set of images or the second set of images corresponding to one or more moving components of the manufacturing line.

In some aspects, generating the notification further comprises: generating the notification substantially in real-time for display at the user computing device in response to identifying the falling object or the stationary object, wherein the notification comprises at least one of: (i) an email message, (ii) a text message, or (iii) a line monitoring application alert.

500 In certain aspects, analyzing the first set of images and the second set of images may further comprise analyzing, by the one or more processors, the first set of images by applying a first algorithm and the second set of images by applying a second algorithm to identify (i) the falling object within the first FOV or (ii) the stationary object within the second FOV. In these aspects, both the first algorithm and the second algorithm may be, for example, a motion detection algorithm or a ML algorithm/model. Additionally, in these aspects, the first algorithm may be (i) a motion detection algorithm or (ii) a machine learning (ML) algorithm trained with a plurality of training data comprising a plurality of training images representing the manufacturing line, and wherein the ML algorithm is configured to receive image data of the manufacturing line as input and to output an anomaly score corresponding to a confidence level associated with detection of the falling object or the stationary object; and the second algorithm is (i) the motion detection algorithm or (ii) the ML algorithm. Further in these aspects, the methodmay further comprise: training the ML model using the plurality of training images representing the manufacturing line, wherein the plurality of training images represent the manufacturing line operating (i) without a falling object within the first FOV and (ii) without a stationary object within the second FOV. Moreover, in these aspects, the ML algorithm may be at least one of (i) an anomaly detection algorithm, (ii) an image classification algorithm, or (iii) an object detection algorithm.

130 138 In some aspects, the first algorithm and the second algorithm are included as part of a line monitoring application (e.g., line monitoring application); and analyzing the first set of images and the second set of images may be performed by an unexpected item detection unit (e.g., unexpected item detection unit) executing instructions comprising the first algorithm and the second algorithm.

402 In certain aspects, the notification (e.g., notificationsA-E) may include a heatmap image that comprises a heatmap portion superimposed over the image of the falling object or the stationary object. The heatmap portion may be positioned over the falling object or the stationary object within the image. In certain instances, the notification may include multiple images, and the heatmap image may be multiple heatmap images. In these instances, multiple heatmap portions may be superimposed over the multiple images, such that the notification may include multiple images from the first set of image data and/.or the second set of image data, as well as multiple heatmap images.

500 500 1 4 FIGS.- In some aspects, the methodmay be performed either entirely by automation, e.g., by one or more processors (e.g., a CPU or GPU) that execute instructions stored on one or more non-transitory, computer-readable storage media (e.g., a volatile memory or a non-volatile memory, a read-only memory, a random-access memory, a flash memory, an electronic Attorney erasable program read-only memory, or one or more other types of memory). More generally, the methodmay use any of the components, processes, or techniques of one or more of.

Some of the figures described herein illustrate example block diagrams having one or more functional components. It will be understood that such block diagrams are for illustrative purposes and the devices described and shown may have additional, fewer, or alternate components than those illustrated. Additionally, in various aspects, the components (as well as the functionality provided by the respective components) may be associated with or otherwise integrated as part of any suitable components.

Some aspects of the disclosure relate to a non-transitory computer-readable storage medium having instructions/computer-readable storage medium thereon for performing various computer-implemented operations. The term “instructions/computer-readable storage medium” is used herein to include any medium that is capable of storing or encoding a sequence of instructions or computer codes for performing the operations, methodologies, and techniques described herein. The media and computer code may be those specially designed and constructed for the purposes of the aspects of the disclosure, or they may be of the kind well known and available to those having skill in the computer software arts. Examples of computer-readable storage media include, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media such as optical disks; and hardware devices that are specially configured to store and execute program code, such as ASICs, programmable logic devices (“PLDs”), and ROM and RAM devices.

Examples of computer code include machine code, such as produced by a compiler, and files containing higher-level code that are executed by a computer using an interpreter or a compiler. For example, an aspect of the disclosure may be implemented using Java, C++, or other object-oriented programming language and development tools. Additional examples of computer code include encrypted code and compressed code. Moreover, an aspect of the disclosure may be downloaded as a computer program product, which may be transferred from a remote computer (e.g., a server computer) to a requesting computer (e.g., a computer or a different server computer) via a transmission channel. Another aspect of the disclosure may be implemented in hardwired circuitry in place of, or in combination with, machine-executable software instructions.

As used herein, the singular terms “a,” “an,” and “the” may include plural referents, unless the context clearly dictates otherwise. This description, and the claims that follow, should be read to include one or at least one and the singular also includes the plural unless expressly stated or it is obvious that it is meant otherwise. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

As used herein, the terms “approximately,” “substantially,” “substantial,” “roughly” and “about” are used to describe and account for small variations. When used in conjunction with an event or circumstance, the terms can refer to instances in which the event or circumstance occurs precisely as well as instances in which the event or circumstance occurs to a close approximation. For example, when used in conjunction with a numerical value, the terms can refer to a range of variation less than or equal to ±10% of that numerical value, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%. For example, two numerical values can be deemed to be “substantially” the same if a difference between the values is less than or equal to ±10% of an average of the values, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%.

Additionally, amounts, ratios, and other numerical values are sometimes presented herein in a range format. It is to be understood that such range format is used for convenience and brevity and should be understood flexibly to include numerical values explicitly specified as limits of a range, but also to include all individual numerical values or sub-ranges encompassed within that range as if each numerical value and sub-range is explicitly specified.

While the techniques disclosed herein have been described with primary to particular operations performed in a particular order, it will be understood that these operations may be combined, sub-divided, or re-ordered to form an equivalent technique without departing from the teachings of the present disclosure. Accordingly, unless specifically indicated herein, the order and grouping of the operations are not limitations of the present disclosure.