Camera Alignment Using Semi-Transparent Reference Image for Asset Inspection Systems and Methods

This application is a continuation of International Patent Application No. PCT/US2024/030439 filed May 21, 2024 and entitled “CAMERA ALIGNMENT USING SEMI-TRANSPARENT REFERENCE IMAGE FOR ASSET INSPECTION SYSTEMS AND METHODS,” which claims priority to and the benefit of U.S. Provisional Patent Application No. 63/504,037 filed May 24, 2023 and entitled “CAMERA ALIGNMENT USING SEMI-TRANSPARENT REFERENCE IMAGE FOR ASSET INSPECTION SYSTEMS AND METHODS,” all of which are incorporated herein by reference in their entirety.

The present invention relates generally to asset inspection and, more particularly, to image-based inspection of assets using reference images.

In industrial environments such as manufacturing facilities or other locations, there is often a need to inspect various assets such as machines, electronics, or other devices. In many cases, the assets may be temperature-sensitive and therefore required to operate at temperatures within expected tolerances to facilitate ongoing reliable functionality. For example, if an asset exhibits a temperature that is too high or too low, this may indicate a fault in need of repair.

Various conventional techniques exist for monitoring assets. In some cases, large numbers of sensors or fixed camera systems may be installed throughout a facility. However, such implementations can require significant investments in infrastructure and may be cost prohibitive. Moreover, the fixed nature of such implementations can limit their ability to monitor all relevant assets in a given environment. In other cases, a user may be required to manually inspect the assets. However, this approach can be subject to human error as it puts the responsibility on the user to properly monitor the condition of the asset repeatedly. Accordingly, there is a need for an improved approach to asset monitoring.

According to various embodiments of the present disclosure, a method includes capturing, by a camera, a live image of an asset under inspection. The method further includes receiving, at the camera, a manipulation to align the camera relative to the asset based on a comparison between the live image and a reference image of the asset. The method further includes capturing, by the camera, an adjusted live image of the asset aligned with the reference image.

According to various embodiments of the present disclosure, a system includes a camera. The camera is configured to capture a live image of an asset under inspection. The camera is configured to receive a manipulation to align the camera relative to the asset based on a comparison between the live image a reference image of the asset. The camera is configured to capture an adjusted live image of the asset under inspection aligned with the reference image.

According to various embodiments of the present disclosure, a method includes capturing, by a camera, a live image of an asset under inspection. The method further includes preparing a blended image using the live image and a semi-transparent reference image of the asset, wherein the blended image comprises a first representation of the asset from the live image and a second representation of the asset from the semi-transparent reference image. The method further includes receiving, at the camera, a manipulation to align the camera relative to the asset in response to a review of an alignment of the first and second representations of the asset relative to each other in the blended image.

According to various embodiments of the present disclosure, a system includes a camera. The camera is configured to capture a live image of an asset under inspection. The camera is configured to prepare a blended image using the live image and a semi-transparent reference image of the asset, wherein the blended image comprises a first representation of the asset from the live image and a second representation of the asset from the semi-transparent reference image. The camera is configured to receive a manipulation to align the camera relative to the asset in response to a review of an alignment of the first and second representations of the asset relative to each other in the blended image.

The scope of the invention is defined by the claims, which are incorporated into this section by reference. A more complete understanding of embodiments of the present invention will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.

Embodiments of the present invention and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

Embodiments of the present disclosure provide systems and methods for asset inspection. A reference image of an asset may be used to capture a similar image of the asset repeatedly, speed up inspection, and draw correct conclusions regarding the status or health of the asset. The reference image may be a thermal and/or visible light image of the asset, such as under normal state or conditions.

During inspection of the asset, the reference image may be presented to the user, such as while the user is in front of the asset. For example, the reference image may be presented together with a live image of the asset, such as to compare the live image to the reference image (e.g., for feedback to align the images together and/or to assess a status of the asset). For instance, a manipulation may be provided to the camera to align the camera relative to the asset based on a comparison between the live image and the reference image. Presenting the reference image together with the live image may support the taking of similar images of the asset every time, which enables trending. For example, such configurations may ensure that the camera is roughly the same distance and angle towards the asset, otherwise the two images would be dissimilar.

To speed up inspection and provide decision support (e.g., to assist the user in making the correct conclusions in the field), the live image can inherit properties from the reference image, or vice versa. For example, the reference image can be prepared before being transferred to the camera. For instance, measuring tools, such as measuring spots or boxes, can be placed on areas of interest, and/or color palette, level and span, among other image properties, can be adjusted to make the areas of interest clearly visible in the image, among other properties or characteristics. In short, any kind of preparation can be applied to the reference image before transferring to the camera. The selection and preparation of the reference image can be done by the same user doing the inspection, or it can be done as guidance by a more experienced user. In embodiments, the reference image may inherit a characteristic of the live image, such as when the user changes any setting of live image. In such embodiments, the reference image may be changed correspondingly.

1 FIG. 100 101 198 101 illustrates a block diagram of an inspection systemcomprising a portable deviceand a remote systemin accordance with an embodiment of the disclosure. In some embodiments, portable device, which may be referred to as an imaging system or simply a camera, may be implemented, for example, as a handheld camera system, a small form factor camera system provided as part of part of and/or an attachment to a personal electronic device such as a smartphone, or as another device.

101 194 194 190 101 102 190 192 101 192 1 FIG. Portable devicemay be positioned to receive infrared radiationA and/or visible light radiationB from a scene(e.g., corresponding to a field of view of portable device) in an environment(e.g., a workplace, warehouse, industrial site, manufacturing facility, or other environment). In various embodiments, scenemay include one or more physical assets(e.g., temperature-sensitive machines, equipment, electronics, or other devices) of interest which may be captured in thermal images and/or visible light images by portable device. Although a single example assetis illustrated in, any desired number of assets may be inspected in accordance with the techniques of the present disclosure.

101 103 110 110 168 170 172 174 176 178 179 180 182 101 As shown, portable deviceincludes a housing(e.g., a camera body graspable by a user), a thermal imaging subsystemA, a visible light imaging subsystemB, a logic device, user controls, a memory, a communication interface, a machine readable medium, a display component, a position sensor, other sensors, and other components, or any combination thereof. Such embodiments are illustrative only, and portable devicemay include other components facilitating the operations described herein.

110 110 194 194 190 Thermal imaging subsystemA and visible light imaging subsystemB may be used to capture thermal images and visible light images in response to infrared radiationA and visible light radiationB, respectively, received from scene.

110 158 160 162 164 166 194 158 160 194 162 194 164 Thermal imaging subsystemA may include an apertureA, filtersA, optical componentsA, a thermal imagerA, and a thermal imager interfaceA. In this regard, infrared radiationA passing through apertureA may be received by filtersA that selectively pass particular thermal wavelength ranges (e.g., wavebands) of infrared radiationA. Optical componentsA (e.g., an optical assembly including one or more lenses, additional filters, transmissive windows, and/or other optical components) pass the filtered infrared radiationA for capture by thermal imagerA.

164 190 194 164 190 164 166 168 172 172 Thermal imagerA may capture thermal images of scenein response to the filtered infrared radiationA. Thermal imagerA may include an array of sensors (e.g., microbolometers) for capturing thermal images (e.g., thermal image frames) of scene. In some embodiments, thermal imagerA may also include one or more analog-to-digital converters for converting analog signals captured by the sensors into digital data (e.g., pixel values) to provide the captured images. Thermal imager interfaceA provides the captured images to logic devicewhich may be used to process the images, store the original and/or processed images in memory, and/or retrieve stored images from memory.

110 158 160 162 164 166 110 110 Visible light imaging subsystemB may include an apertureB, filtersB, optical componentsB, a visible light imagerB, and a visible light imager interfaceB. It will be appreciated that the various components of visible light imaging subsystemB may operate in an analogous manner as corresponding components of thermal imaging subsystemA with appropriate technology for capturing visible light images.

110 110 Moreover, although particular components are illustrated for each of thermal imaging subsystemA and visible light imaging subsystemB, it will be understood that the illustrated components are provided for purposes of example. As such, greater or fewer numbers of components may be used in each subsystem as appropriate for particular implementations.

168 168 101 168 172 176 172 176 168 101 Logic devicemay include, for example, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device configured to perform processing operations, a digital signal processing (DSP) device, one or more memories for storing executable instructions (e.g., software, firmware, or other instructions), and/or any other appropriate combinations of devices and/or memory to perform any of the various operations described herein. Logic deviceis configured to interface and communicate with the various components of portable deviceto perform various method and processing steps described herein. In various embodiments, processing instructions may be integrated in software and/or hardware as part of logic device, or code (e.g., software and/or configuration data) which may be stored in memoryand/or a machine readable medium. In various embodiments, the instructions stored in memoryand/or machine readable mediumpermit logic deviceto perform the various operations discussed herein and/or control various components of portable devicefor such operations.

172 Memorymay include one or more memory devices (e.g., one or more memories) to store data and information. The one or more memory devices may include various types of memory including volatile and non-volatile memory devices, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory, fixed memory, removable memory, and/or other types of memory.

176 168 176 101 101 101 176 101 101 Machine readable medium(e.g., a memory, a hard drive, a compact disk, a digital video disk, or a flash memory) may be a non-transitory machine readable medium storing instructions for execution by logic device. In various embodiments, machine readable mediummay be included as part of portable deviceand/or separate from portable device, with stored instructions provided to portable deviceby coupling the machine readable mediumto portable deviceand/or by portable devicedownloading (e.g., via a wired or wireless link) the instructions from the machine readable medium (e.g., containing the non-transitory information).

168 178 178 101 168 178 168 172 178 101 178 168 Logic devicemay be configured to process captured images and provide them to display componentfor presentation to and viewing by the user. Display componentmay include a display device such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, and/or other types of displays as appropriate to display images and/or information to the user of portable device. Logic devicemay be configured to display images and information on display component. For example, logic devicemay be configured to retrieve images and information from memoryand provide images and information to display componentfor presentation to the user of portable device. Display componentmay include display electronics, which may be utilized by logic deviceto display such images and information.

170 170 178 170 178 168 170 178 170 User controlsmay include any desired type of user input and/or interface device having one or more user actuated components, such as one or more buttons, slide bars, knobs, keyboards, joysticks, and/or other types of controls that are configured to generate one or more user actuated input control signals. In some embodiments, user controlsmay be integrated with display componentas a touchscreen to operate as both user controlsand display component. Logic devicemay be configured to sense control input signals from user controlsand respond to sensed control input signals received therefrom. In some embodiments, portions of display componentand/or user controlsmay be implemented by appropriate portions of a tablet, a laptop computer, a desktop computer, and/or other types of devices.

170 101 In various embodiments, user controlsmay be configured to include one or more other user-activated mechanisms to provide various other control operations of portable device, such as auto-focus, menu enable and selection, field of view (FoV), brightness, contrast, gain, offset, spatial, temporal, and/or various other features and/or parameters.

179 101 102 192 179 179 168 101 102 Position sensormay be implemented as any appropriate type of device used to determine a position (e.g., location) of portable devicein environment(e.g., in an industrial facility containing assetsto be monitored). For example, in various embodiments, position sensormay be implemented as a global positioning system (GPS) device, motion sensors (e.g., accelerometers, vibration sensors, gyroscopes, and/or others), depth sensing systems (e.g., time of flight cameras, LiDAR scanners, thermal cameras, visible light cameras, and/or others), antennas, other devices, and/or any combination thereof as desired. In some embodiments, position sensormay send appropriate signals to logic devicefor processing to determine the absolute and/or relative position of portable devicein environment.

101 180 Portable devicemay include various types of other sensorsincluding, for example, temperature sensors and/or other sensors as appropriate.

168 166 179 180 170 198 174 174 174 174 174 Logic devicemay be configured to receive and pass images from thermal and visible light imager interfacesA-B, additional data from position sensorand sensors, and control signal information from user controlsto one or more external devices such as remote systemthrough communication interface(e.g., through wired and/or wireless communications). In this regard, communication interfacemay be implemented to provide wired communication over a cable and/or wireless communication over an antenna. For example, communication interfacemay include one or more wired or wireless communication components, such as an Ethernet connection, a wireless local area network (WLAN) component based on the IEEE 802.11 standards, a wireless broadband component, mobile cellular component, a wireless satellite component, or various other types of wireless communication components including radio frequency (RF), microwave frequency (MWF), and/or infrared frequency (IRF) components configured for communication with a network. As such, communication interfacemay include an antenna coupled thereto for wireless communication purposes. In other embodiments, the communication interfacemay be configured to interface with a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, and/or various other types of wired and/or wireless network communication devices configured for communication with a network.

101 In some embodiments, a network may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, the network may include the Internet and/or one or more intranets, landline networks, wireless networks, and/or other appropriate types of communication networks. In another example, the network may include a wireless telecommunications network (e.g., cellular phone network) configured to communicate with other communication networks, such as the Internet. As such, in various embodiments, portable deviceand/or its individual associated components may be associated with a particular network link such as for example a URL (Uniform Resource Locator), an IP (Internet Protocol) address, and/or a mobile phone number.

101 182 Portable devicemay include various other componentssuch as speakers, displays, visual indicators (e.g., recording indicators), vibration actuators, a battery or other power supply (e.g., rechargeable or otherwise), and/or additional components as appropriate for particular implementations.

101 198 101 198 101 101 174 101 198 101 198 198 199 172 198 192 198 197 198 198 101 1 FIG. Although various features of portable deviceare illustrated together in, any of the various illustrated components and subcomponents may be implemented in a distributed manner and used remotely from each other as appropriate. For example, remote systemmay be implemented with any of the various components of portable device. Remote systemmay communicate with portable deviceto send and receive data therewith, perform remote processing for portable device, and/or other tasks (e.g., through appropriate communication interfacesof portable deviceand/or of remote system). For example, in some embodiments, thermal images, visible light images, position data, and/or additional information obtained by portable devicemay be communicated to remote systemfor further processing and/or storage. In this regard, remote systemmay include a database(e.g., maintained in an appropriate memoryof remote system) used for storage and recall of various images and/or other information to monitor historical temperatures of assets. In embodiments, remote systemand portable device may communicate over a network. For example, remote systemmay be implemented as a cloud-based system, although other configurations are contemplated. In various embodiments, remote systemmay include any of the various components of portable deviceas appropriate.

2 FIG. 200 192 200 101 200 101 200 110 110 170 178 101 200 illustrates an imaging system (e.g., a camera) configured to capture an image of an asset under inspection (e.g., asset) for comparison against a reference image, in accordance with an embodiment of the disclosure. Cameramay be similar to portable devicedescribed above, such that camerais a particular implementation of portable device. For example, cameramay include thermal imaging subsystemA and/or visible light imaging subsystemB, user controls, display component, and other components of portable device, described above. Although described as a thermal and/or visible light imaging system, cameramay be or include any other type of imaging system.

2 FIG. 200 210 192 192 200 210 192 192 210 214 192 178 210 214 200 210 214 198 Referring to, camerais configured to capture a live imageof an asset under inspection (e.g., asset). For example, during an inspection of asset, cameramay capture live imagefor use in monitoring a condition of asset, such as to detect faults, failures, or undesired operating conditions of asset. In embodiments, live imagemay be presented together with a reference imageof asset. For example, display componentmay be configured to display live imageand reference imagesimultaneously for viewing by a user of camera, although other implementations are contemplated, as described below. For instance, in some implementations, live imageand reference imagemay be displayed on a remote display component, such as on a display component of remote system, although other configurations are contemplated.

214 192 210 214 192 192 192 192 210 214 192 214 192 192 192 192 Reference imagemay be any image used to identify an inspection condition of assetbased on a comparison with live image. For example, reference imagemay be an image of assetitself, such as an image taken by an installer during installation of asset, an image taken by a manufacturer during manufacture of asset, or any other image of assettaken at any time prior to live image. In some embodiments, reference imagemay be an image of a similar asset and not of assetitself. For instance, reference imagemay be an image of another device/equipment of the same model as asset(e.g., a standard image of asset model, the same asset at another location, etc.) or an image of another device/equipment having properties and/or a configuration similar to asset(e.g., a prior model of asset, a comparable model of asset, etc.).

214 214 200 199 198 214 200 214 192 192 Reference imagemay be provided in many ways. For example, reference imagemay be provided (e.g., to camera) by an image database maintained by a server (e.g., by databaseof remote system). In some embodiments, reference imagemay be taken by a second camera different than camera. For example, as noted above, reference imagemay be taken by the installer during installation of asset, by the manufacturer during manufacture of asset, or by another person or device.

214 100 210 170 214 192 192 214 192 214 192 214 192 214 214 In embodiments, reference imagemay be selected or identified (e.g., by a user, by system, etc.) for use in comparing against live image. For example, using user controls, a user may select, from among multiple images, an image to use as reference image, such as toggling between various prior images of asset. In this manner, the user may toggle between a time series of images of asset, such as to provide additional decision support. Such embodiments may also enable a trend plot of temperature values to be presented for a measurement tool, which may provide additional decision support. In embodiments, reference imagemay be selected automatically, or at least selected by default, based on a user setting. For instance, the user setting may include at least one of a “last asset inspection” setting, a “first image taken of the asset” setting, or a “last image associated with a similar time or environmental condition of the live image” setting, although other configurations are contemplated. The “last asset inspection” setting may select, as default, the last inspection image of assetas reference image. The “first image taken of the asset” setting may select, as default, the earliest image taken of assetas reference image. The “last image associated with a similar time or environmental condition of the live image” setting may select, as default, the latest image of assettaken during a similar time of day and/or year (e.g., morning, afternoon, fall, October, etc.) and/or similar environmental conditions (e.g., ambient temperature, etc.) as reference image, such as for assets whose temperature may vary over the year. Depending on the application, the selection of reference imagecan be done by the same user doing the inspection, or the selection can be done as guidance by a more experienced user.

2 FIG. 210 214 210 214 210 214 200 210 214 214 210 210 210 214 214 210 210 214 As shown in, live imageis displayed adjacent (e.g., side-by-side) reference imageto facilitate a comparison between live imageand reference image. For example, live imageand reference imagemay be presented together to allow a user (e.g., a user of camera, a remote user, etc.) to compare live imageand reference imageside-by-side. In embodiments, reference imagemay be displayed picture-in-picture with live image, such as at a corner of live image, although other configurations are contemplated. In embodiments, the display characteristics of live imageand reference imagemay be adjusted by user preference. For instance, a user may adjust the size and position of reference imagerelative to live image, as desired, to aid in comparing live imageto reference image.

200 100 214 210 210 214 192 214 230 232 214 200 214 In embodiments, cameraand/or systemmay apply a characteristic associated with reference imageor live imageto live imageor reference imageprior to displaying the images. The characteristic may include temperature measuring functions, properties that affect the appearance of the image, and/or properties that affect the temperature reading of asset. For example, reference imagemay be prepared to include measuring tools (e.g., temperature measurement box(es), temperature measurement spot(s), etc.) placed on areas of interest and/or by adjusting image characteristics (e.g., color palette, temperature span settings, thermal brightness (level) settings, etc.) in a way to make areas of interest clearly visible. In embodiments, the characteristic may include other image properties/characteristics, such as various image parameters/properties and/or other data associated with the image (e.g., ambient temperature, time, user, camera type/model, location, position, etc.). Such examples are illustrative only, and any kind of preparation can be applied to reference imagebefore transferring to camera. Depending on the application, the preparation of reference imagecan be done by the same user doing the inspection, or the preparation can be done as guidance by a more experienced user.

210 214 210 230 232 214 192 210 210 214 210 214 214 210 210 214 Live imagemay inherit the characteristics of reference imagedescribed above. For example, live imagemay inherit any or all measurement functions (e.g., temperature measurement box(es), temperature measurement spot(s), etc.) and their placement from reference image, leading to an efficient inspection of assetas live imageis automatically prepared with the correct measuring tools in place. Additionally, or alternatively, live imagemay inherit the image characteristics and/or properties of reference image, such as color palette, temperature span, level, emissivity, distance to object, ambient temperature, etc. In this manner, the system may ensure that a user is looking at images with the same visual presentation to aid in inspection (e.g., to ensure an “apples-to-apples” comparison). For example, with the same properties controlling the presentation of the images, something that appears warmer in one image will be warmer. Although live imageis described as inheriting a characteristic of reference image, in embodiments, reference imagemay inherit a characteristic of live image(e.g., should the user change any setting of live image, reference imageis changed correspondingly).

210 214 192 210 214 200 192 200 200 192 210 214 200 210 214 200 210 214 100 198 Comparison between live imageand reference imagemay facilitate the taking of similar images of assetevery time. For example, comparing live imageto reference imagemay ensure that camerais roughly the same distance and angle towards assetduring inspection; otherwise, the two images may not look similar or capture the same information. To that end, cameramay be configured to receive a manipulation to align camerarelative to assetbased on a comparison between live imageand reference image. The manipulation may adjust at least one of a position, an angle, or a field of view of camerato align live imagewith reference image. Depending on the application, the manipulation may be performed by the user of camera, such as in real time based on user comparison of live imageto reference image, or the manipulation may be performed by another device (e.g., a robot operated by systemand/or remote system), although other configurations are contemplated.

210 214 200 192 214 192 192 192 200 199 192 214 192 Once live imageand reference imageare (or appear) similar, cameramay capture an adjusted live image of asset, the adjusted live image aligned with reference image. The adjusted live image may then be used to determine a status, condition, or operational state of asset. For example, the adjusted live image may display or otherwise identify a fault, failure, or undesired operation condition of asset, or that assetis operating satisfactorily. The adjusted live image may be stored (e.g., in camera, in database, etc.) for use in future inspections of asset. For example, the adjusted live image may be used as reference imagein future inspections of asset.

210 214 110 200 192 110 192 200 210 214 Depending on the application, live image, reference image, and the adjusted live image may be thermal images, such as captured by thermal imaging subsystemA. In embodiments, cameramay be configured to capture a visible light live image of asset(e.g., as captured by visible light imaging subsystemB) and receive a visible light reference image of asset. In such embodiments, the manipulation of camerato align live imagewith reference imagemay be based on a comparison between the visible light live image and the visible light reference image.

200 200 200 210 214 In some embodiments, cameramay be configured to process the thermal live image and the visible light live image to provide a combined live image. Cameramay also process the thermal reference image and the visible light reference image to provide a combined reference image. In such embodiments, the manipulation of camerato align live imagewith reference imagemay be based on a comparison between the combined live image and the combined reference image. The combined live image and the combined reference image may be generated using various thermal plus visible light combining techniques as further discussed herein.

3 5 FIGS.- 3 FIG. 300 310 178 300 310 300 310 316 320 300 310 330 300 334 300 illustrate diagrams of various example comparisons between a live image of an asset under inspection and a reference image, in accordance with an embodiment of the disclosure. Referring to, a live imagemay be displayed together with a reference imageon a display component (e.g., display component). As shown, live imageand reference imagemay be presented side-by-side. Each of live imageand reference imagemay include data related to when the image was captured (e.g., image capture information). In embodiments, the system may provide an indicationreminding the user to compare live imagewith reference image. In embodiments, the system may include a first executable controlthat causes the system to accept live image, and a second executable controlthat causes the system to retake live image, although other configurations are contemplated.

4 FIG. 400 410 400 410 420 424 428 420 Referring to, a live imagemay be displayed together (e.g., side-by-side) with a reference image. As shown, each of live imageand reference imagemay include a temperature measurement box, with a maximum temperatureand a minimum temperaturein the boxindicated (e.g., below the images).

5 FIG. 500 510 500 178 200 510 178 178 500 510 500 510 520 524 528 Referring to, a live imagemay be displayed prominently relative to a reference image. For example, live imagemay fill a substantial portion of display componentof camera, with reference imagetaking up a smaller portion of display component. As shown, the system may be configured to identify, such as highlight, (e.g., on display component) a detected difference between live imageand reference image. For instance, the system may visually identify temperature differences of certain portions or elements in live imagecompared to reference image. In embodiments, temperature differences may be highlighted differently based on the delta amount, such as with a first indication(e.g., a first color) for small temperature differences (e.g., less than 5° C.), a second indication(e.g., a second color) for moderate temperature differences (e.g., between 6° C. and 14° C.), and a third indication(e.g., a third color) for large temperature differences (e.g., greater than 15° C.), although other configurations are contemplated. In embodiments, an alarm may be provided based on the detected difference exceeding a threshold, such as if a detected temperature difference exceeds a temperature limit, among other examples.

6 FIG. 6 FIG. 192 192 600 192 illustrates a diagram of an identifying of a detected anomaly of an asset under inspection, in accordance with an embodiment of the disclosure. Referring to, the system may be configured to identify detected faults, failures, or other anomalies of asset. For example, based on data received from live and reference images of asset, the system may identify and flag (e.g., automatically) inspection issues, such as via a boxor another indication. As shown, the identification of faults or failures may be highlighted in a visible light image to facilitate user identification of faulty components or equipment, although other configurations are contemplated. For example, some types of problems can be clearly seen in a visible light image but not in a thermal image. The user may toggle between thermal and visible light images of asset. In such embodiments, measurement functions and their results can be inherited from a thermal reference image to the visible light live image, even if the user is using the visible/visual portion of the image. The measurement functions and their results may also be presented on the visible light image with correct location and measurement results.

7 FIG. 7 FIG. 700 700 700 168 101 200 198 700 100 192 illustrates a flow diagram of a processof comparing a live image of an asset under inspection against a reference image, in accordance with an embodiment of the disclosure. In this regard, processmay operate in relation to any of the various live images, reference images, and/or combined images discussed herein and/or illustrated in the various drawings of the present disclosure. In embodiments, processmay be performed by logic deviceof portable device(e.g., camera) and/or remote system. In some embodiments, processmay be performed during runtime operation of inspection systemto permit real-time inspection of one or more assets (e.g., asset). Note that one or more operations inmay be combined, omitted, and/or performed in a different order as desired.

710 700 200 192 192 200 192 200 192 170 In block, processincludes receiving (e.g., by camera) an identification of an asset to be inspected. For example, assetmay be flagged as needing inspection, such as during routine inspections of one or more assets in a warehouse, on an industry floor, etc. In embodiments, the identification of assetto be inspected may be based on at least one of a predetermined inspection route, a detected position of camerarelative to asset(e.g., GPS positioning), a communication between cameraand asset(e.g., near-field communication (NFC), wireless communication, Bluetooth communication, etc.), or user input. For example, during routine inspections, the user may provide an indication (e.g., via user controls, voice control, etc.) to proceed to the next asset for inspection. In embodiments, the use of an inspection route may be the same or similar to that disclosed in U.S. Provisional Patent Application No. 63/003,111, filed Mar. 31, 2020, and International Patent Application No. PCT/US2021/025011 filed Mar. 30, 2021, all of which are hereby incorporated by reference in their entirety.

715 700 200 210 110 110 192 In block, processincludes capturing (e.g., by camera) a live image (e.g. live image) of the asset under inspection. For instance, thermal imaging subsystemA and/or visible light imaging subsystemB may be used to capture a thermal live image and/or a visible light live image of asset, such as in a manner as described above.

720 700 In block, processincludes identifying a reference image of the asset based on a user selection and/or a setting. The user setting may include at least one of a “last asset inspection” setting, a “first image taken of the asset” setting, or a “last image associated with a similar time or environmental condition of the live image” setting, as described above. Such implementations are exemplary only, and other configurations are contemplated.

725 700 200 725 199 198 200 192 192 192 192 In block, processincludes receiving (e.g., by camera) the reference image of the asset. Blockmay include receiving the reference image from an image database maintained by a server (e.g., databaseof remote system). The reference image may be taken by a second camera different than camera. For example, the reference image may be an image taken by an installer during installation of asset, an image taken by a manufacturer during manufacture of asset, or any other image of assettaken at any time prior to live image. The reference image may be an image of assetitself, or an image of a different asset. The reference image may be a thermal reference image or a visible light reference image.

700 730 730 730 In some embodiments, processmay include optional blockwherein thermal and/or visible images may be combined to provide combined live images and/or combined reference images comprising thermal image content and visible light image content. For example, blockmay include processing a thermal live image and a visible light live image to provide a combined live image, processing a thermal reference image and a visible light reference image to provide a combined reference image, and/or other processing. In some embodiments, the processing performed in blockmay include any of the various techniques set forth in U.S. Pat. Nos. 8,520,970, 8,565,547, 8,749,635, 9,171,361, 9,635,285, and/or 10,091,439, all of which are hereby incorporated by reference in their entirety. In some embodiments, such processing may include, for example, contrast enhancement processing (e.g., also referred to as MSX processing, high contrast processing, and/or fusion processing), true color processing, triple fusion processing, alpha blending, and/or other processing as appropriate.

700 Such combined live images and/or combined reference images may be used as the live images and/or reference images in other blocks of processdescribed herein to facilitate convenient review of such combined images and ease of alignment by a user or by another device. For example, in some embodiments, such combined images may permit a high resolution visible light features to be discerned simultaneously with low resolution thermal features.

700 732 732 In some embodiments, processmay include optional block, where the live image and/or the reference image is adjusted to compensate for detected environmental conditions and/or device operating conditions. For example, one or more sensors may monitor ambient temperate, device temperature, humidity, and/or other conditions of the environment, which may affect camera operation and/or image capture/characteristics. In such embodiments, blockmay compensate for the detected conditions, such that the live and reference images are similar for comparison purposes.

735 700 200 In block, processincludes receiving (e.g., at camera) a manipulation to align the camera relative to the asset based on a comparison between the live image and the reference image of the asset. The manipulation may adjust at least one of a position, an angle, or a field of view of the camera to align the live image with the reference image. The manipulation may be performed by the user of the camera, or by another device (e.g., a robot, a machine, etc.), as described above. The manipulation and comparison may be based on thermal imagery, visible light imagery, or combined thermal and visible light imagery, as noted above.

740 700 230 232 192 In block, processincludes applying a characteristic associated with the reference image or the live image to the live image or the reference image. The characteristic applied may include temperature measuring tools or functions (e.g., temperature measurement box(es), temperature measurement spot(s)) placed on areas of interest, properties that affect the appearance of the image (e.g., color palette, span, level), and/or properties that affect the temperature reading (e.g., emissivity, distance to object, ambient temperature) of asset, among other characteristics.

745 700 178 101 200 198 2 5 FIGS.- In block, processincludes displaying the live image and the reference image simultaneously on a display component for viewing by a user. For example, the live and reference images may be displayed on display componentof portable device/cameraand/or a display component of a remote device (e.g., of remote system, a smartphone, etc.). The live and reference images may be displayed side-by-side, picture-in-picture, vertically stacked, or in other configurations. For example, the live and reference images may be displayed as shown in, described above.

750 700 750 168 101 200 198 750 5 6 FIGS.- In block, processincludes identifying a detected difference between the live image and the reference image on the display component. Blockmay include visually highlighting, flagging, or otherwise noting differences between the live image and the reference image, such as identified by the user. In embodiments, the differences between the live image and the reference image may be detected using a processor (e.g., logic deviceof portable device/cameraand/or remote system), such as via a neural network running a machine learning algorithm or other artificial intelligence. Blockmay include boxing or otherwise isolating the detected difference, such as in a manner as explained with reference to, described above.

755 700 200 110 110 192 192 In block, processincludes capturing (e.g., by camera) an adjusted live image of the asset aligned with the reference image. For example, once the live image and the reference image are (or at least appear) similar, thermal imaging subsystemA and/or visible light imaging subsystemB may capture a thermal live image and/or a visible light live image of assetaligned with the reference image, such as in a manner as described above. In embodiments, the adjusted live image may be used as a reference image in future inspections of asset.

In view of the present disclosure, it will be appreciated that various techniques are provided to facilitate alignment of live images with reference images to permit comparable and useful images to be repeatedly captured of an asset under inspection. Repeated capture of comparable and useful images of the asset may speed up inspection and lead to correct conclusions regarding the status or health of the asset. For example, the live image may be presented together with a reference image to facilitate quick identification of any differences in the live image from the reference image, such as a change in temperature. To aid inspection and increase efficiency, the live image can inherit properties from the reference image, or vice versa, such that the images appear similar for the comparison.

715 700 735 200 700 745 732 735 700 200 735 200 200 In some embodiments, block(e.g., the capture of the live image) may be repeated and may continue throughout processsuch that the live image is repeatedly updated to provide a current live image (e.g., a plurality of successive live images). Also in some embodiments, block(e.g., the manipulation of camera) may be repeated and may continue throughout process. Also in some embodiments, block(e.g., the display of the live image and the reference image) may be performed between blocksandand may be repeated and may continue throughout processsuch that a user of cameramay simultaneously view the live image and the reference image while performing the manipulation of block. As a result, the user may repeatedly view the displayed current live image and reference image while manipulating camerato further align camerarelative to the asset.

Additional embodiments are also contemplated. Although live images have been primarily illustrated and discussed as being positioned adjacent to (e.g., side-by-side or otherwise) reference images, other display techniques may be used. For example, reference images may be partially or fully superimposed on live images to provide a blended image that aids a user in aligning a camera relative to an asset under inspection. Such an approach can substantially assist the user in aligning a camera to achieve a desired distance, angle, and orientation relative to an asset under inspection to match reference images and therefore capture images from repeatable vantage points to provide useful images for comparison with previously captured images and future captured images.

In this regard, by superimposing a reference image over a live image, the user may manipulate the camera until the live image substantially matches the reference image (e.g., until the size, orientation, position, and/or other features of the asset in the live image substantially match the reference image) such that the blended image exhibits little to no misalignment (e.g., offset) in the live image representation of the asset and the reference image representation of the asset. Moreover, by using a superimposition-based alignment, a user may review a single blended image comprising the superimposed reference image and live image (e.g., rather than switching the user's gaze between different images that may be simultaneously but separately displayed adjacent to each other or otherwise). As a result, the user may review such a blended image and align the camera with the more rapidly and more accurately.

168 101 198 8 14 FIGS.- To further aid in such superimposition-based alignment, reference images may be presented in a semi-transparent manner. In some embodiments, semi-transparent reference images may be prepared by processing existing opaque reference images (e.g., by logic deviceof portable deviceand/or by remote system). In some embodiments, a user may select a transparency amount (e.g., predetermined or in realtime) to selectively adjust the amount of transparency exhibited by the semi-transparent reference images. Such an approach permits users to achieve improved alignment and ease of use as different transparency amounts may be more beneficial under various operating conditions and/or environmental conditions. Such features can be further appreciated in relation to.

8 FIG. 9 FIG. 10 FIG. 800 810 900 800 910 800 900 For example,illustrates a live imageof an asset under inspection (e.g., depicted in image portion),illustrates a semi-transparent reference imageof the same asset as live image(e.g., depicted in image portion), andillustrates live imagein relation to semi-transparent reference image, in accordance with embodiments of the disclosure.

10 FIG. 10 FIG. 900 800 800 900 810 910 As shown in, reference imagemay be superimposed over live image(e.g., imagesandare shown displaced in relation to each other infor illustration purposes to aid the reader in discerning the relative alignment of the images in the X-Y plane). As a result, when blended together, image portionsanddepicting the asset under inspection may be easily viewed in relation to each other.

11 FIG. 1100 800 900 1100 178 200 170 In this regard,illustrates a blended imageprepared using live imageand reference image, in accordance with an embodiment of the disclosure. For example, blended imagemay be displayed on display componentof camerawith additional user controlsand other information as appropriate.

11 FIG. 810 910 1100 810 910 1100 200 810 910 800 As shown in, image portionsandfor the asset are both illustrated in blended imageand are slightly offset in relation to each other for illustration purposes (e.g., image portionsandare slightly translated in both the X axis and the Y axis relative to each other). Accordingly, while a user views blended image, the user may manipulate camerato further align image portionsandif desired until they substantially overlap each other with little or no offset. When so aligned, live imagemay be captured, stored, and/or otherwise maintained for future reference to aid in the inspection and review of the asset.

200 900 200 170 200 In some embodiments, cameraoptionally receives a user selection of a transparency amount (e.g., the opacity) to be used for semi-transparent reference image. For example, in some embodiments, a user of cameramay operate user controlsand/or other portions of camerato select the transparency amount. In some embodiments, the transparency amount may be identified by a transparency percentage (e.g., 25 percent, 50 percent, 75 percent, or others) or other appropriate metric.

800 900 800 800 900 In some embodiments, live imagemay be a fully opaque image (e.g., having a transparency of 0 percent). In such embodiments, superimposing semi-transparent reference imageover opaque live imagewill result in some of opaque live imagebeing visible behind (e.g., through) semi-transparent reference image.

1100 900 800 900 1100 800 1100 800 900 1100 For example, in blended image, semi-transparent reference imagehas a transparency amount of 50 percent and opaque live imagehas a transparency amount of 0 percent. In this case, semi-transparent reference imagecontributes 50 percent to blended imageand opaque live imagecontributes a remaining 50 percent to blended image. As a result, live imageand semi-transparent reference imageare equally visible in blended image.

12 FIG. 1200 178 200 900 800 900 1200 800 1200 900 800 1200 As another example,illustrates a blended image(e.g., as displayed by display componentof camera), in accordance with an embodiment of the disclosure. Here, semi-transparent reference imagehas a transparency amount of 25 percent and opaque live imagehas a transparency amount of 0 percent. In this case, semi-transparent reference imagecontributes 75 percent to blended imageand opaque live imagecontributes a remaining 25 percent to blended image. As a result, semi-transparent reference imagemay be substantially more visible than live imagein blended image.

13 FIG. 1300 178 200 900 800 900 1300 800 1200 900 800 1300 As another example,illustrates a blended image(e.g., as displayed by display componentof camera), in accordance with an embodiment of the disclosure. Here, semi-transparent reference imagehas a transparency amount of 75 percent and opaque live imagehas a transparency amount of 0 percent. In this case, semi-transparent reference imagecontributes 25 percent to blended imageand opaque live imagecontributes a remaining 75 percent to blended image. As a result, semi-transparent reference imagemay be substantially less visible than live imagein blended image.

800 900 Thus, it will be appreciated that the proportions of live imageand semi-transparent reference imageexhibited by the resulting blended image may be selectively adjusted as desired by a user to achieve a preferred representation as may be appropriate for various conditions or user preferences. Various techniques may be used to provide transparency such as alpha blending, alpha channel processing, pixel-specific transparency, and/or other techniques as appropriate.

Although user-selectable transparency amounts have been discussed, other approaches are also contemplated. For example, a predetermined transparency amount (e.g. 50 percent or other amount) may be used in place of the user selection.

14 FIG. 14 FIG. 1400 800 900 1400 1400 168 101 200 198 1400 100 192 1400 700 illustrates a flow diagram of a processof providing a blended image prepared using live imageand a semi-transparent reference image, in accordance with an embodiment of the disclosure. In this regard, processmay operate in relation to any of the various live images, reference images, and/or combined images discussed herein and/or illustrated in the various drawings of the present disclosure. In embodiments, processmay be performed by logic deviceof portable device(e.g., camera) and/or remote system. In some embodiments, processmay be performed during runtime operation of inspection systemto permit real-time inspection of one or more assets (e.g., asset). Note that one or more operations inmay be combined, omitted, and/or performed in a different order as desired. In various embodiments, any of the features of processmay be combined with, substituted for, and/or otherwise used with any of the features of process.

1410 1415 1420 1425 710 715 720 725 Blocks,,, andmay be performed as discussed with regard to blocks,,, and, respectively.

1430 200 1435 168 900 1410 168 1425 172 199 168 198 In block, cameraoptionally receives a user selection of a transparency amount as discussed. In block, logic deviceprepares semi-transparent reference imageaccording to the transparency amount selected in block. For example, in some embodiments, logic devicemay process an opaque reference image (e.g., having a transparency amount of 0 percent) previously received in blockto adjust its transparency amount. In some embodiments, the reference image may already exhibit a predetermined transparency amount (e.g., 50 percent or another amount) as stored by memoryand/or database, or as pre-processed by logic deviceand/or remote system.

1440 168 1100 1200 1300 800 900 1440 900 800 1445 200 178 In block, logic deviceprepares a blended image (e.g., any of blended images,,, or others depending on the selected transparency amount) using live imageand semi-transparent reference image. Various techniques may be used to perform blending such as alpha blending, Poisson blending, gradient mixing, and/or other techniques as appropriate. In some embodiments, the blending performed in blockcomprises superimposing semi-transparent reference imageover live image. In block, cameradisplays the resulting blended image on display componentfor review by a user.

1450 1400 200 200 800 900 In block, processincludes receiving (e.g., at camera) a manipulation to align camerarelative to the asset based on a review (e.g., by a user and/or machine-based processing) of the blended image. The manipulation may adjust at least one of a position, an angle, or a field of view of the camera to align live imagewith semi-transparent reference imageas shown in the blended image.

1455 1400 200 1450 755 700 In block, processincludes capturing (e.g., by camera) an adjusted (e.g., updated) live image following the manipulation performed in blockas similarly discussed in blockof process.

1460 168 1440 1465 200 178 200 200 In block, logic deviceprepares an updated blended image using the updated live image and the semi-transparent reference image as similarly discussed in block. In block, cameradisplays the resulting updated blended image on display componentfor review by a user. Thus, it will be appreciated that by preparing and updated blended images while the user manipulates camera, the user may conveniently and efficiently align camerato capture live images that are aligned with reference images.

700 1400 Any or all of the operations of processand/or processmay be repeated for a plurality of assets, a plurality of images (e.g., live images, adjusted live images, updated live images, reference images, blended images, updated blended images, and/or others discussed herein) as appropriate.

900 800 1435 1430 1440 Other implementations are also contemplated. For example, in some embodiments, reference imageand live imagemay be selectively blended (e.g., using alpha blending or otherwise) according to a particular transparency amount to achieve a blended image without requiring the transparency preprocessing of block(e.g., blockmay be followed by block).

In view of the present disclosure, it will be appreciated that blending semi-transparent reference images with live images may permit a user to easily discern misalignments and offsets between the images during asset inspection operations. As a result, asset data (e.g., thermal images and associated measurements) may be more consistently recorded and potential problems associated with the assets may be more quickly located and identified.

Where applicable, various embodiments provided by the present disclosure can be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and/or software components set forth herein can be combined into composite components comprising software, hardware, and/or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and/or software components set forth herein can be separated into sub-components comprising software, hardware, or both without departing from the spirit of the present disclosure. In addition, where applicable, it is contemplated that software components can be implemented as hardware components, and vice-versa.

Software in accordance with the present disclosure, such as program code and/or data, can be stored on one or more computer readable mediums. It is also contemplated that software identified herein can be implemented using one or more general purpose or specific purpose computers and/or computer systems, networked and/or otherwise. Where applicable, the ordering of various steps described herein can be changed, combined into composite steps, and/or separated into sub-steps to provide features described herein.

Embodiments described above illustrate but do not limit the invention. It should also be understood that numerous modifications and variations are possible in accordance with the principles of the present invention. Accordingly, the scope of the invention is defined only by the following claims.