Metrology 3D Scanning System and Method

This application is a continuation-in-part of U.S. patent application Ser. No. 18/183,767, entitled “Metrology 3D Scanning System and Method” and filed Mar. 14, 2023, which claims priority to U.S. Provisional Patent Application No. 63/343,435, entitled “Metrology 3D Scanning System and Method” and filed May 18, 2022, the entire disclosures of which are hereby incorporated by reference herein.

This invention was made with government support under Advanced Robotics for Manufacturing Institute Subaward Agreement No. ARM-TEC-19-04-F04 awarded by the U.S. Army Contracting Command. The government has certain rights in the invention.

This disclosure relates generally to metrology three-dimensional scanning systems and methods and, more particularly, to metrology 3D scanning systems and methods for generating scanning path and motion plans for specific objects.

Currently, when 3D measurement is performed for metrology applications, such as quality inspection, well-defined (accurate, precise, stable, etc.) fixturing and target object placement is needed. When a sensor is used manually to inspect target objects, it takes significant time and effort by the human operator to move the sensor to all the desired locations to collect measurement data. This manual process is very slow and error prone, and thus robotic automation of sensing is desired. However, to robotically automate sensing, fixturing scheme(s) have to be developed to hold target object(s), robot program(s) have to be created by human recording each location that a robot has to visit one-by-one to preempt collision during the robot's motion across multiple robot locations based on human expertise or by testing the robot program very slowly while force-stopping the robot motion in anticipation of collision. Furthermore, the skill set needed to perform this robotic programming is hard to find from someone who knows how to perform metrology, and thus a seemingly great robot program may not result in a reasonable measurement outcome, or the desired locations from the perspective of a metrology expert may not be accomplishable by a robot.

In accordance with one exemplary aspect of the present invention, a metrology three-dimensional (3D) scanning system configured to generate scanning path and motion plans for specific objects includes a computer memory, one or more processors communicatively coupled to the computer memory, a device, a robotic element, and a metrology 3D scanning application (app). The metrology 3D scanning app includes computing instructions configured to execute on the one or more processors and, when executed by the one or more processors, causes the one or more processors to: record preliminary object data as the device is operated; generate a preliminary scan path based on the preliminary object data for operating the robotic element within an operating environment; execute instructions to move the robotic element along at least a portion of the preliminary scan path and record, by the one or more processors into the computer memory, preliminary scan data including at least a subset of dimension data defining at least a target object; generate a metrology scanning path plan and a motion plan for the robotic element based on the preliminary scan data; and execute instructions to move the robotic element according to the metrology scanning path plan and the motion plan for scanning the target object, wherein the metrology scanning path plan and the motion plan, when implemented for scanning the target object, causes the robotic element to move within the operating environment.

In further accordance with any one or more of the foregoing exemplary aspects of the present invention, a metrology 3D scanning system may further include, in any combination, any one or more of the following preferred forms.

In one preferred form, the device is operated to define a collision zone in a proximity to the target object. The collision zone includes a 3D area that the robotic element should avoid physical entry into when operating within the operating environment and the preliminary scan path is generated for operating the robotic element within the operating environment while avoiding the collision zone.

In another preferred from, the metrology 3D scanning app is configured to generate a localized 3D model of the target object by aligning the subset of dimension data as defined in the preliminary scan data to a 3D model of the target object.

In another preferred from, the 3D model includes a 3D reference design.

In another preferred from, the preliminary object data/preliminary device scan path data and/or the preliminary scan path and/or the preliminary scan data and/or the metrology scanning path plan and the motion plan and/or data gathered during execution of the metrology scanning path plan and the motion plan are selected as training data for input into an artificial intelligence (AI) algorithm for training an AI model. The AI model is configured to output a more effective metrology scanning path and motion plan or sufficient metrology data including watertight completeness data for inspecting or manufacturing.

In another preferred from, the metrology 3D scanning app is configured to determine defects of the target object after scanning of the target object and/or determine whether the target object requires maintenance or repair after scanning of the target object.

In another preferred from, the metrology 3D scanning app is configured to create a visual representation of the target object with an indicator of the defects and/or an indicator of an area requiring maintenance or repair.

In accordance with another exemplary aspect of the present invention, a metrology three-dimensional (3D) scanning system configured to generate scanning path and motion plans for specific objects includes a computer memory, one or more processors communicatively coupled to the computer memory, one of a device or an HRI device, a first robotic element, a second robotic element, and a metrology 3D scanning application (app). The metrology 3D scanning app includes computing instructions configured to execute on the one or more processors and, when executed by the one or more processors, cause the one or more processors to: record one of preliminary object data as the device is operated or HRI data as a human operator operates the HRI device; generate a preliminary scan path based on the preliminary object data/HRI data for operating the first robotic element; execute instructions to move the first robotic element along at least a portion of the preliminary scan path and record, by the one or more processors into the computer memory, preliminary scan data including at least a subset of dimension data defining at least a target object; generate a metrology scanning path plan and a motion plan for the second robotic element based on the preliminary scan data; and execute instructions to move the second robotic element according to the metrology scanning path plan and the motion plan for scanning the target object, wherein the metrology scanning path plan and the motion plan, when implemented for scanning the target object, causes the second robotic element to move within an operating environment.

In further accordance with any one or more of the foregoing exemplary aspects of the present invention, a metrology 3D scanning system may further include, in any combination, any one or more of the following preferred forms.

In one preferred form, the device is operated to define a collision zone in a proximity to the target object. The collision zone includes a 3D area that the first robotic element should avoid physical entry into when operating and the preliminary scan path is generated for operating the first robotic element while avoiding the collision zone.

In another preferred form, the collision zone includes a virtual near-net shape augmented reality (AR) bounding box that is generated via hand or gazing by the human operator using the HRI device and the HRI device includes an AR device.

In another preferred form, the metrology 3D scanning app is configured to generate a localized 3D model of the target object by aligning the subset of dimension data as defined in the preliminary scan data to a 3D model of the target object.

In another preferred form, the HRI data or preliminary object data/preliminary device scan path data and/or preliminary scan path and/or the preliminary scan data and/or the metrology scanning path plan and the motion plan and/or data collected during execution of the metrology scanning path plan and the motion plan are selected as training data for input into an artificial intelligence (AI) algorithm for training an AI model. The AI model is configured to output a more effective metrology scanning path and motion plan or sufficient metrology data including watertight completeness data for inspecting or manufacturing.

In another preferred form, the metrology 3D scanning app is configured to determine defects of the target object after scanning of the target object and/or determine whether the target object requires maintenance or repair after scanning of the target object.

In another preferred form, the metrology 3D scanning app is configured to create a visual representation of the target object with an indicator of the defects and/or an indicator of an area requiring maintenance or repair.

In accordance with another exemplary aspect of the present invention, a metrology three-dimensional (3D) scanning system configured to generate scanning path and motion plans for specific objects includes a computer memory, one or more processors communicatively coupled to the computer memory, one of a device or an HRI device, a robotic element, and a metrology 3D scanning application (app). The metrology 3D scanning app includes computing instructions configured to execute on the one or more processors and, when executed by the one or more processors, causes the one or more processors to: record preliminary object data and preliminary device scan path data as a device is operated or HRI data as a human operator operates the HRI device, the preliminary object data/HRI data including at least a subset of dimension data defining at least a target object; generate a metrology scanning path plan and a motion plan for the robotic element based on the preliminary object data/preliminary device scan path data or HRI data; and execute instructions to move the robotic element according to the metrology scanning path plan and the motion plan for scanning the target object, wherein the metrology scanning path plan and the motion plan, when implemented for scanning the target object, causes the robotic element to move within an operating environment.

In further accordance with any one or more of the foregoing exemplary aspects of the present invention, a metrology 3D scanning system may further include, in any combination, any one or more of the following preferred forms.

In one preferred form, the device is operated to define a collision zone in a proximity to the target object. The collision zone includes a 3D area that the robotic element should avoid physical entry into when operating and the metrology scanning path plan and the motion plan are generated for operating the robotic element while avoiding the collision zone.

In another preferred form, the metrology 3D scanning app is configured to generate a localized 3D model of the target object by aligning the subset of dimension data as defined in the preliminary scan data to a 3D model of the target object.

In another preferred form, the HRI data or preliminary object data/preliminary device scan path data, and/or the metrology scanning path plan and the motion plan, and/or data collected during execution of the metrology scanning path plan and the motion plan are selected as training data for input into an artificial intelligence (AI) algorithm for training an AI model. The AI model is configured to output a more effective metrology scanning path and motion plan or sufficient metrology data including watertight completeness data for inspecting or manufacturing.

In another preferred form, the metrology 3D scanning app is configured to determine defects of the target object after scanning of the target object and/or determine whether the target object requires maintenance or repair after scanning of the target object.

In another preferred form, the metrology 3D scanning app is configured to create a visual representation of the target object with an indicator of the defects and/or an indicator of an area requiring maintenance or repair.

In accordance with another exemplary aspect of the present invention, a metrology three-dimensional (3D) scanning system configured to generate scanning path and motion plans for specific objects using human-robot interaction (HRI) comprises a computer memory, one or more processors communicatively coupled to the computer memory, an HRI device, and a robotic element, and a metrology 3D scanning application (app) comprising computing instructions configured to execute on the one or more processors. The metrology 3D scanning app, when executed by the one or more processors, causes the one or more processors to: record HRI data as a human operator operates the HRI device; generate a preliminary scan path based on the HRI data for operating the robotic element within an operating environment; execute instructions to move the robotic element along at least a portion of the preliminary scan path and record, by the one or more processors into the computer memory, preliminary scan data comprising at least a subset of dimension data defining at least a target object; generate a metrology scanning path plan and a motion plan for the robotic element based on the preliminary scan data; and execute instructions to move the robotic element according to the metrology scanning path plan and the motion plan for scanning the target object, wherein the metrology scanning path plan and the motion plan, when implemented for scanning the target object, causes the robotic element to move within the operating environment.

In further accordance with any one or more of the foregoing exemplary aspects of the present invention, a metrology 3D scanning system may further include, in any combination, any one or more of the following preferred forms.

In one preferred form, the human operator operates an HRI device to define a collision zone in a proximity to a target object, wherein the collision zone comprises a 3D area that the robotic element should avoid physical entry into when operating within an operating environment and the preliminary scan path is generated for operating the robotic element within the operating environment while avoiding the collision zone.

In another preferred form, the HRI device comprises an augmented reality (AR) device.

In another preferred form, the AR device renders a 3D representation of the target object on a display of the AR device, and wherein at least a portion of the 3D representation is marked as a missing or incomplete portion by the human operator, and wherein the missing or incomplete portion is added to the subset of dimension data defining the target object.

In another preferred form, the collision zone comprises a virtual near-net shape augmented reality (AR) bounding box that is generated via hand or gazing by the human operator using the HRI device and the HRI device comprises an AR device.

In another preferred form, the metrology 3D scanning app is configured to generate a localized 3D model of the target object by aligning the subset of dimension data as defined in the preliminary scan data to a 3D model of the target object.

In another preferred form, the 3D model comprises a 3D reference design.

In another preferred form, a dilated surface around the target object is determined, wherein the dilated surface is a surface defined where a signed distance field to the target object is equal to a dilation distance, wherein a distance of a coordinate is a minimum distance to any surface of the target object.

In another preferred form, aligning the subset of dimension data to a 3D reference design of the target object further comprises determination of an alignment deviation having an alignment preset threshold defining whether alignment between the subset of dimension data and the 3D reference design resulted in an alignment success or an alignment failure.

In another preferred form, the alignment deviation compared to the alignment preset threshold indicates the alignment failure has occurred, an additional 3D scan of the target object is captured, and the additional 3D scan is used to align to the 3D reference design.

In another preferred form, the alignment deviation compared to the alignment preset threshold indicates the alignment failure has occurred, and the human operator manipulates the HRI device to map points from the subset of dimension data and the 3D reference design to align the subset of dimension data and the 3D reference design.

In another preferred form, scanning of the target object generates an insufficient metrology scan data, and wherein the human operator manipulates the HRI device to collect additional metrology scanning data including generation of watertight mesh.

In another preferred form, the HRI data and the preliminary scan data are selected as training data for input into an artificial intelligence (AI) algorithm for training an AI model and the AI model is configured to output a more effective metrology scanning path and motion plan or sufficient metrology data including watertight completeness data for inspecting or manufacturing.

In another preferred form, the metrology 3D scanning app is configured to determine defects of the target object after scanning of the target object.

In another preferred form, the metrology 3D scanning app is configured to create a visual representation of the target object with an indicator of the defects.

In another preferred form, the metrology 3D scanning app is configured to determine whether the target object requires maintenance or repair after scanning of the target object.

In another preferred form, the metrology scanning path plan and the motion plan are provided to a second robotic element for scanning of a new target object by the second robotic element.

In accordance with another exemplary aspect of the present invention, a metrology three-dimensional (3D) scanning method for generating scanning path and motion plans for specific objects using human-robot interaction (HRI) comprises the steps of: recording, by one or more processors, HRI data as a human operator operates an HRI device; generating, by the one or more processors, a preliminary scan path based on the HRI data for operating the robotic element within an operating environment; executing, by the one or more processors, instructions to move the robotic element along at least a portion of the preliminary scan path and record, by the one or more processors into a computer memory, preliminary scan data comprising at least a subset of dimension data defining at least a target object; generating, by the one or more processors, a metrology scanning path plan and a motion plan for the robotic element based on the preliminary scan data; and executing, by the one or more processors, instructions to move the robotic element according to the metrology scanning path plan and the motion plan for scanning the target object, wherein the metrology scanning path plan and the motion plan, when implemented for scanning the target object, causes the robotic element to move within the operating environment.

In accordance with another exemplary aspect of the present invention, a tangible, non-transitory computer-readable medium stores instructions for generating scanning path and motion plans for specific objects using human-robot interaction (HRI), that when executed by one or more processors cause the one or more processors to: record HRI data as a human operator operates an HRI device; generate a preliminary scan path based on the HRI data for operating the robotic element within an operating environment; execute instructions to move the robotic element along at least a portion of the preliminary scan path and record, by the one or more processors into a computer memory, preliminary scan data comprising at least a subset of dimension data defining at least a target object; generate a metrology scanning path plan and a motion plan for the robotic element based on the preliminary scan data; and execute instructions to move the robotic element according to the metrology scanning path plan and the motion plan for scanning the target object, wherein the metrology scanning path plan and the motion plan, when implemented for scanning the target object, causes the robotic element to move within the operating environment.

Advantages will become more apparent to those of ordinary skill in the art from the following description of the preferred embodiments which have been shown and described by way of illustration. As will be realized, the present embodiments may be capable of other and different embodiments, and their details are capable of modification in various respects. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.

The Figures depict preferred embodiments for purposes of illustration only. Alternative embodiments of the systems and methods illustrated herein may be employed without departing from the principles of the invention described herein.

1 4 FIGS.- 10 60 10 15 40 15 45 15 60 Referring to, an example metrology three-dimensional (3D) scanning systemcan be located within an operating environment, such as an area in a manufacturing facility, a sustainment (e.g., maintenance and repair) facility, a logistics facility (e.g., distribution center), or any other indoor or outdoor space where a physical object's dimension is measured for metrology applications such as quality inspection, and is configured to perform a 3D scanning method to generate scanning path and motion plans for specific objects and can, in some implementations, use human-robot interaction (HRI). Metrology 3D scanning systemgenerally includes a computing device, can include one or more HRI device(s)or other devices such as fix mounted cameras or scanners, robotic devices equipped with cameras or scanners (including, but not limited to, drone mounted cameras or scanners), etc., in communication with computing devicethrough a wired or wireless connection, and one or more robotic element(s)in communication with computing devicethrough a wired or wireless connection. In some implementations, the other device could be a drone with a long-range camera or scanner mounted on it that hovers above operating environmentor an autonomous mobile robot moving around and monitoring various operating environments and can execute a preliminary scan path by moving itself around the part/operating environment. In this case, the drone or autonomous mobile robot's movement/motion can define a preliminary scan path and collect the preliminary object data just using this preliminary scan path. In other implementations, a robotic arm or dexterous robots can be mounted on an autonomous mobile robot and the camera or the scanner can be mounted on the robotic arm. In this case, the preliminary scan path is not only defined by the movement/motion of the autonomous mobile robot, but also the robotic arm's joint motion that gives higher degree of freedom for the preliminary scan path.

15 20 25 20 40 45 35 25 Computing devicecan be any type of computing device, such as a computer, laptop computer, workstation, tablet, etc. and includes: a computer memory, one or more processors, such as CPUs, GPUs, etc., that are communicatively coupled to computer memory, to HRI device/other device, and to one or more robotic element(s); and one or more input/output devices, such as keyboards, displays, touchscreens, etc. Processorscan be on-site processors and/or one or more processors of cloud based servers.

40 30 In the example shown, HRI deviceis an augmented reality (AR) device, but could also be a haptic device, a hand-held object with vision tracked marker or laser tracked object, the robotic element itself manipulated using hand-guided motion planning, or any other HRI device that can be used to provide input that can be used by metrology 3D scanning appto generate scanning paths and motion plans.

45 45 45 Robotic elementis a combination of a motorized system that can move a sensor mounted on the motorized system and/or pick up and move the part being measured in front of sensor(s) when an instruction is provided. For example, robotic elementcould be a 6-axis robotic arm, such as that manufactured by Universal Robots, a 2-axis horizontal turntable, such as that manufactured by Parker, and a multi-line laser 3D scanner by Creaform. Robotic elementmay be able to move and manipulate the sensor around the target object or the target object around sensor. Both target or the sensor can be static or in motion in a given point of time.

30 20 25 30 25 25 50 60 100 30 25 6 FIG. A metrology 3D scanning application (app)can be stored in computer memoryor on any tangible, non-transitory computer-readable medium and can include computing instructions configured to be executed on processor(s)to execute the metrology 3D scanning method. Metrology 3D scanning app, when executed on processor(s), can execute the metrology 3D scanning method and cause processor(s)to perform various steps to generate scanning paths and motion plans for scanning and inspection of a target objectlocated within operating environment. For example, referring to, an example metrology 3D scanning methodthat can be performed by metrology 3D scanning appwhen executed on processor(s)is shown.

105 100 30 15 40 25 20 80 40 80 50 50 80 40 75 50 75 50 45 60 50 60 50 40 75 80 40 40 50 50 30 50 40 2 FIG. At stepof metrology 3D scanning method, HRI data can be received by metrology 3D scanning appon computing devicefrom HRI deviceand processor(s)can record the HRI data, for example in computer memory, as a human operatoroperates HRI device. HRI data can be: (1) outcome data in a form of coordinate locations that define the vertices of rectangular bounding boxes that human operatordrew by loading, grabbing, dragging, dropping, and uniformly or non-uniformly scaling, whereas target objectis encompassed by only one box or by multiple boxes to encompass target objectmore near-net shape (i.e., the surplus volume between the target object's surface and the box is smaller than using just one box, almost like a aggregation of Lego blocks that mimic a realistic object shape); (2) outcome data in a form of scan surface on which scan path(s) (i.e., location that a sensor has to visit and the sequence of the locations) will be generated, whereas the scan surface would contain the sensor's orientation information for each location that is generated on the surface [x, y, z, w, p, r]. For example, human operatorcan operate HRI deviceto define a collision zone(see) around and in proximity to target object. Collision zonepreferably defines a 3D area around target objectthat robotic elementshould avoid physical entry into when operating in operating environmentto avoid collision with target objector other potential objects within operating environment, such as the surface/stand that target objectis positioned on. For example, when HRI deviceis an AR device, collision zonecan include a virtual near-net shape AR bounding box that is generated via hand or gazing by human operatorusing HRI deviceor can be automatically generated using previously existing sensing data collected before the preliminary scan process using HRI device. As an example, rough dimensional information may be collected on target objectfrom 2D cameras (e.g., even security cameras) that can inform the rough size and location and possibly shape of target object. Metrology 3D scanning appmay receive this information and dilate the rough 3D representation of target objectand use the dilated surface as-is as the collision zone or automatically generate the bounding box around the dilated surface particularly if there are missing surface dimensions even after dilating the rough dimensional information. A dilated surface is a surface defined where a signed distance field to the target object is equal to a dilation distance, wherein a distance of a coordinate is a minimum distance to any surface of the target object. When such rough dimensional information from previously existing sensing data is not available and connected to the system, the preliminary scanning process using HRI deviceis used.

45 50 45 45 75 40 75 40 50 50 50 Alternatively, a particular type of sensor may be mounted on robotic element(if so, there may be a tool changing capability where sensors can be swapped in and out) or a static sensor may have the capability to capture the 3D data on target objectfrom far away (i.e., longer stand-off distance and possibly larger field of view), which will allow robotic elementto perform a simple-motion from afar (or even safer if this sensor is statically placed). This sensor could be dedicated for preliminary scanning (instead of using the metrology scanner for the preliminary scanning as well) and will likely be cheaper and possibly faster while capturing lower-accuracy/lower-resolution data. In this case, the user would choose a few locations using robotic elementby hand-guiding it to vertices of collision zone(which is just another HRI devicemanipulation to set up collision zone), and then perform the scan path and motion planning. The case of not performing collision zone setup using HRI devicemost likely will make sense when there is sufficient pre-existing rough 3D representation of target object, which can algorithmically (autonomously) lead to a collision zone (using approaches like dilation). When there is too partial information that has been collected on target object, dilation may not lead to an encompassing dimension of target object, and so such approach would make sense when high-coverage but low-quality data is collected.

40 30 15 25 20 75 50 2 FIG. In other implementations, the other device (e.g., fix mounted camera(s) or scanner(s), robotic device(s) equipped with camera(s) or scanner(s) (including, but not limited to, drone mounted cameras or scanners), etc.) can be used in the place of HRI deviceand preliminary object data and preliminary device scan path data can be received by metrology 3D scanning appon computing devicefrom the device, and processor(s)can record the preliminary object data and preliminary device scan path data, for example in computer memory, as the device is operated. In this case, the preliminary object data does not have to be collected via human-robot interaction. The preliminary device scan path data is the data regarding the path that the device or the device's control system takes (e.g., actuating or rotating segment of the device). The preliminary object data is the actual scan data of the object gathered by the device as the device and/or its control system followed the path represented by the preliminary device scan path data. The device can be operated to define collision zone(see) around and in proximity to target object. In some implementations, fix mounted cameras or scanners can be installed to observe, monitor, or collect data on the operating environment of the robot to collect a preliminary object data. In another form, robotic device(s) equipped with cameras or scanners can move around the operating environment to collect the preliminary object data.

110 30 65 65 45 60 75 65 45 60 75 40 80 45 80 40 80 80 40 30 80 80 45 75 3 5 FIGS.and Once the HRI data/preliminary object data has been recorded, at step, metrology 3D scanning appcan generate a preliminary scan path(see) based on the HRI data/preliminary object data. Preliminary scan pathcan include instructions for operating robotic elementwithin operating environment. For example, if collision zonewere defined, as discussed above, preliminary scan pathcan be generated to operate robotic elementwithin operating environmentwhile avoiding collision zone. For example, using HRI device, human operatorcan generate scan surface(s) on which scan path(s) (i.e., locations that sensor on robotic elementhave to visit and the sequence of the locations) will be generated. Human operatormay wear AR glasses and using his/her hand to generate a 3-dimensional ellipsoid, similar to how bounding boxes are set up by loading, grabbing, dragging, dropping, and uniformly or non-uniformly scaling the vertices of the rectangular bounding box that defines the inner-ellipsoid. While HRI devicemanipulation is similar, the human intent and the approach is different, which results in a different set of HRI data compared to the collision zone setup. As the intent is for human operatorto quickly and simply generate a preliminary scan path, HRI data is used to autonomously generate scan paths on the scan surface that human operatorgenerated using HRI device. Once the location, orientation, size, and shape of the ellipsoid(s) is defined, metrology 3D scanning appmay assume that the center of the ellipsoid defines the vector to which the sensor has to be pointing at when it is at one of the scan locations on the path computationally generated by the system on the scan surface defined by human operator. Additional HRI may be performed and HRI data collected by human operatorsetting multiple center points that the sensor has to be facing at a given scan location, improving the measurement angle from the same scan location. For a very large object, creating an ellipsoid scan surface whose scan locations are facing the center(s) of the ellipsoid may not be optimal for an intuitive user experience, as a very large ellipsoid would have to be drawn using hands (e.g., a large flat surface). In this case, a very similar HRI process and HRI data type with the same device can be used by generating a different shape of scan surface. This scan surface may be 2D (X-Y plane) whose vector angles are all directly perpendicular to its 2D surface. The system may also use a different algorithm type to autonomously generate the scan sequence, such as a simple raster motion (e.g., left to right and then right to left). In all variations of the scan surface and path, robotic elementwon't execute any motion whose target location or in-between location enters collision zonedefined by the bounding boxes.

40 110 30 65 3 5 FIGS.and In other implementations that use another device, rather than HRI device, once the preliminary object data has been recorded, at step, metrology 3D scanning appcan generate preliminary scan path(see) based on the preliminary object data. [The preliminary object data collected from the device will entail the surface dimension data with a reasonable accuracy, sufficient for the metrology 3D scanning app to generate preliminary scan path without collision.

115 30 25 45 45 65 25 20 50 50 50 75 45 65 45 50 50 50 50 45 65 50 50 At step, metrology 3D scanning appexecutes instructions on processor(s)and communicates with robotic elementto move robotic elementalong at least a portion of, and preferably the entire, preliminary scan pathand preliminary scan data is recorded by processor(s)into computer memory. The preliminary scan data preferably includes at least a subset of dimension data defining at least target object. Preliminary scan data can be 3D measurement data of target object, which is expected to be 3D scanned thoroughly using a high-quality/high-resolution sensor using robotic automation with a certain level of autonomy (full or partial). The scan data may be preliminary in that it may only represent subset the overall dimension of target objector it has an insufficient accuracy/resolution to be used as replacement of 3D model to serve as collision zone(i.e., near net shape) by itself or used for the metrology applications, such as comparing the preliminary scan to the engineering model (e.g., 3D CAD) to perform quality inspection. This scan data is also preliminary in that timing-wise, it precedes the metrology scan data, and so the intent of collecting the preliminary data is to save time, minimize manual effort, reduce risk, or improve intelligence in the overall workflow for the human operator to perform metrology scanning with the metrology 3D scanning system. The preliminary scan data can be in various formats including, point cloud of the object surface, mesh of the object surface, parametric representation of the part dimension, 3D point cloud or mesh constructed from 2D image and/or depth sensors, a multitude of 2D dissection data such as X-ray or CT images defining the overall dimension of the object when compiled into one coordinate system. As robotic elementmoves along preliminary scan path, one or more sensors of robotic elementcan capture the subset of dimension data defining target object, for example by reconstructing two-dimensional (2D) data into 3D, capturing a 3D surface, registering 2D dissection images, etc. Typically, full dimension data for target objectis not collected since target objectwill typically be set on a surface/stand for inspection and dimension data for the portion of target objectthat is on surface/stand cannot be obtained. In addition, as robotic elementfollows preliminary scan path, in addition to collecting dimension data of target object, dimension data of surface/stand may also be collected as the lower portion of target objectis scanned.

65 30 45 75 50 50 50 50 50 45 If preliminary scan pathis not available, metrology 3D scanning appcan instruct robotic elementto perform a global “simplified” sensing, which can be accomplished with an industrial 3D scanner, low-resolution/low accuracy depth sensors, or 2D imaging that generate a simple measurement path based on collision zone. This global “simplified” sensing can be considered a preliminary scan and can be accomplished by a stationary sensor from a single location/orientation, a robotically movable sensor moving around target object, moving/rotating target objectin front of a static sensor, or any combination of the foregoing. The global “simplified” sensing process may be iterative, as a first global “simplified” sensing may not capture the full dimensions of target objector may not even capture sufficient coverage of target objectto serve as a “preliminary scan”. For example, a dilation approach could be used to revise/augment a prior global “simplified” sensing. Using the subset of dimensional data obtained in the prior global “simplified” sensing, a dilated surface, which is an artificially generated surface outside of the scan data, with a more smooth, rounded, averaged surface, around target objectcan be determined which is typically is the surface defined where the signed distance field to the target object is equal to the dilation distance, wherein the distance of a coordinate is the minimum distance to any surface of the target. This first attempt at “simplified” sensing allows sensing by the sensor of robotic elementto measure the part from closer to near-net shape as the dilated surface generated from an insufficient “simplified” sensing, which hence would be considered an insufficient preliminary scan. Compared to the previous attempt of “simplified” sensing this next iteration of sensing from the dilated surface will allow a more effective preliminary scan from improved scan distance, angle, point of view, etc.

40 15 50 40 35 15 80 40 15 30 45 55 50 80 40 45 Once the preliminary scan data has been recorded, HRI device, such as an AR device, and/or computing devicecan render a 3D representation of target objecton a display of HRI deviceor an output deviceof computing device. Based on the 3D representation, human operatorcan mark, indicate, or highlight as missing or incomplete at least a portion of the 3D representation using HRI deviceor computing deviceand metrology 3D scanning appcan instruct robotic elementto rescan the missing or incomplete portionand new dimension data from the rescan can be added to the subset of dimension data defining target object. When there is a missing area based on the simulated scanning or actual scanning, human operatorcan localize the missing area by using HRI device. One example would be to generate the scan surface only targeted for the local area with missing data. The other approach would be to identify the location of missing points/mesh by comparing the incomplete 3D scan data, which can be based on preliminary scan data or metrology scan data, to an engineering model (e.g., CAD file) and then sending robot elementto go to that area where no comparison data exists. Alternatively, iterative simplified sensing with improving dilated surface can be another method of accomplishing sufficient preliminary scan data.

30 50 50 45 50 50 50 50 50 50 40 50 50 50 50 80 In addition, metrology 3D scanning appcan be configured to generate a localized 3D model of target object, for example, a 3D representation of target objectlocated and oriented in the coordinate system of robotic element, by aligning the subset of dimension data as defined and recorded in the preliminary scan data to a 3D model of target object. Preferably, the 3D model of target objectis a 3D reference design, such an electronic drawing file of target objector a golden 3D scan of target object, which is a complete and accurate 3D representation of target objectfrom a previous 3D measurement or a metrology effort on a different part with the same design. The alignment can first be attempted using a global best fit alignment. However, this alignment could possibly fail when target objectlacks features. This alignment failure can be recognized and corrected automatically or manually, for example, using HRI device. Aligning the subset dimension data to the 3D reference design of target objectcan include comparing the preliminary scan data to the 3D reference design and determining an alignment deviation, which can be the size of the transformation metrics between the measurement data and the 3D reference design. The alignment deviation can be compared to an alignment preset threshold to define whether alignment between the subset of dimension data and the 3D reference design resulted in an alignment success or failure. Depending on the manufacturing process, there is an expected range of an alignment deviation. There may be a major error in certain sections of target objectwhich would be a manufacturing quality failure, but when target objectis made reasonably well, the global dimensional deviation between the dimension data and the 3D reference design post-alignment would likely be within a reasonably expected range. If the deviation, particularly across various subset of dimensions, exceeds the expected range, a failed alignment could be suspected. When a failed alignment is suspected, the input parameters to the global best fit alignment can be changed to find a new alignment with a different approach (e.g., changing the sequence of rotation or axis of rotation to reach an improved local minimum in the distance function), which may entail a more time consuming and costly computation. This may still not be sufficient particularly when there target objectis featureless or features are relatively insignificant to the global dimension data, and then the HRI data can be used for the alignment. One example would be for human operatorto pick three corresponding points on the dimension data and the 3D reference design.

50 50 50 80 40 80 40 80 80 3 40 80 40 If a comparison of the alignment deviation to the alignment preset threshold indicates that alignment failure has occurred, an additional 3D scan of target objectcan be executed and the dimension data captured and recorded, registered to the previously collected dimension data, and the accrued dimension data from the additional 3D scan of target objectcan be aligned to the 3D reference design of target object. Alternatively, human operatorcan use HRI deviceto map points, meshes, group of points or meshes, selectable area, features, etc., between the subset of dimension data and the 3D reference design to manually align the subset of dimension data and the 3D reference design. The wrongly aligned reference data can be visualized to human operatorthrough HRI deviceand human operatorcan perform alignment. As an example, human operatormay selectcorresponding points or if the specific HRI device, such as AR glasses, don't allow accurate selection of a point, pointing and selecting a target point may define the surrounding dimension data (i.e., group of points or mesh) including the target point to be used for alignment. Alternatively, a similar HRI may be used to select a specific geometric feature to perform alignment, This selection can then be used as the reference data for alignment (i.e., datum driven alignment). If the 3D reference does not contain data for the area for which human operatorwants to use for alignment, HRI devicecan be used to select points or regions in the subset of dimension data and additional scan iterations can be performed to capture missing data for the indicated points or regions.

30 120 40 70 30 70 In some implementations, rather than generating a preliminary scan path and moving a robotic elements along the preliminary scan path to record preliminary scan data, metrology 3D scanning appcan skip these steps and can move to stepand generate a metrology scanning path and a motion plan for a robotic element based on the HRI data or preliminary object data/preliminary device scan path data, rather than the preliminary scan data. When HRI deviceor another device is used to collect HRI data or preliminary object data/preliminary device scan path data using cameras or scanners with reasonably high resolution and accuracy or using advanced algorithms to augment lesser quality scan/image data, the HRI data or preliminary object data/preliminary device scan path data results in a 3D representation of the part that can replace/outperform the preliminary scan data in generating metrology scanning pathand motion plan and that is sufficient for metrology 3D scanning appto generate metrology scanning pathand motion plan without first collecting a preliminary scan.

120 30 25 70 45 65 70 45 50 70 50 65 70 45 45 4 FIG. At step, metrology 3D scanning appcan generate via processor(s)a metrology scanning path(see) and motion plan for a robotic element, such as robotic element, based on the preliminary scan data or based on the preliminary object data and the preliminary device scan path data. In comparison to preliminary scan path, metrology scanning pathprovides a scan path for robotic elementthat can provide a more detailed scan of target object, for example, for inspection purposes. Metrology 3D scanning collects high-accuracy 3D measurement data across the overall dimensions, often at higher-resolution for more accurate representation of the object's shape, or a subset of the overall dimensions, sometimes with different varying tolerance requirement (i.e., allowed maximum deviation from the accurate representation or the 3D reference design) across different sections of the object. Metrology scanning pathis generated to perform the metrology 3D scanning, whose scan location and angle (i.e., the [X,Y,Z,W,P, R]) on the scan path and the motion across the scan path, for the sensors that capture measurement data even during motion, have to be well-defined to be optimally positioned from target object, compared to preliminary scan path. It is common for a sensor to have optimal settings to collect high-accuracy measurement data, often recommended by the OEM, such as optimal stand-off distance range between the scan location to the target dimension, field-of-view within which the target dimension should be, speed in which the sensor should be moving while scanning, and etc. When there are multiple scan locations on metrology scanning path, it is important for robotic elementto operate so its location, angle, and motion keeps being positioned optimally. A subset of this operation may not be reachable due to factors such as anticipated collision (e.g., an optimal scan location being within the collision zone), in which case, the preliminary scan process described above may disregard this subset operation while still being able to capture the sufficient preliminary scan data. On the other hand, for metrology scanning, finding an alternative operation to best-capture the target dimension which was originally planned to be measured by the collision-anticipated subset operation, may be required. Therefore, metrology scanning path typically requires iterative computing to identify optimal if not achievable, suboptimal operation (i.e., robotic elementexecuting locations, angles, and motion) to reach the dimension data appropriate for metrology applications.

70 125 30 25 45 45 60 70 50 30 45 60 70 50 45 70 50 20 Once metrology scanning pathhas been generated, at step, metrology 3D scanning appexecutes instructions via processor(s)and communicates with robotic elementto move robotic elementwithin operating environmentaccording to metrology scanning pathand motion plan to scan target object. In some embodiments, metrology 3D scanning appcan communicate with a second robotic element, different than robotic element, to move the second robot within operating environmentaccording to metrology scanning pathand motion plan to scan target object. As robotic element/second robotic element is moved according to metrology scanning path, metrology data is collected for target objectand stored in computer memory. Metrology data can be dimension data sufficient to be used for metrology applications. This dimension data may be the measurement data as-is or post-processed dimension data for metrology applications. For the quality inspection, the collected metrology data is accurate dimension data that locally meets the dimension-specific tolerance requirements for various dimensions (e.g., point location, line location, distance, flatness, GD&T, and etc.) or meets global tolerance across the subset of dimension data. For reverse engineering/re-engineering, the overall dimension should be collected without significantly missing dimensions as the application is ultimately to remanufacture a physical object starting from the measurement data. Therefore, higher resolution and coverage data is also desirable, so that can the measurement data be saved or converted in a watertight mesh format with minimal human inputs. With some post-processing, the watertight mesh data, which is another form of metrology data, can be directly inputted to a manufacturing process such as a 3D printing to manufacture a new physical object. Often, the metrology scanning app will perform an automated algorithmic qualification of metrology data, by looking at the density and distribution of points or mesh (e.g., for 3D point cloud data, certain characteristics indicate higher-accuracy, such as less distance between the points, more uniformly distributed points in their local proximity, and etc.)

100 65 65 Once metrology 3D scanning methodis complete and metrology scanning pathhas been executed, the metrology data collected during the execution of metrology scanning pathcan be used for a variety of purposes.

30 30 30 80 40 For example, metrology 3D scanning appcould generate sufficient metrology data including watertight mesh. Generating a watertight mesh can mean that the metrology 3D scanning appdirectly outputs the watertight mesh. If metrology 3D scanning appgenerates an incomplete measurement data that can be successfully saved in or converted to watertight mesh, human operatorcan use HRI deviceto select regions with missing dimension data to generate a complete watertight mesh. If some dimensions are still not measurable, the human operator may manually generate connecting surfaces between previously collected mesh to complete watertight mesh.

Another option is that the HRI data or preliminary object data/preliminary device scan path data, and/or the preliminary scan path, and/or the preliminary scan data can be selected as training data and input into an artificial intelligence (AI) algorithm for training an AI model to predict and recommend how alignment between dimension data and the 3D reference design should be performed (e.g., what corresponding point or feature should be selected as datum for alignment) or how the system should output sufficient preliminary scan data. The information contained in the training data on how insufficient preliminary scan data was improved to become sufficient can be trained for an AI model using machine learning techniques such as imitation learning where the system emulates human operator's decision inputted by the HRI device improving the preliminary scan path in the first place. As a result, this AI can autonomously generate or recommend a successful alignment or sufficient preliminary scan path for the same or different object with the same design but positioned differently. By using techniques such as transfer learning or meta learning, the AI model can be trained to generate or recommend a sufficient preliminary scan data for another object with a different design using limited size of training data.

80 45 Another option is that HRI data or preliminary object data/preliminary device scan path data, and/or the preliminary scan path, and/or the preliminary scan data, and/or the metrology scanning path and the motion plan, and/or data gathered during execution of the metrology scanning path plan and the motion plan, can be selected as training data and input into an artificial intelligence (AI) algorithm for training an AI model and the AI model is configured to output a more effective metrology scanning path and motion plan or sufficient metrology data including watertight completeness data for inspecting or manufacturing. Preliminary scan data augmented with an iterative HRI process will result in information on how the system can algorithmically generate sufficient preliminary scan data with complementary input from human operatorto execute an improved metrology scanning. This AI-generated preliminary scan data may at least output metrology scan data, yet it may not result in sufficient metrology data. In which case, a similar learning approach can be used to train an AI model to improve the metrology data using techniques such as imitation learning using the information on how the operator improved an insufficient metrology data (e.g., using HRI device to collect additional dimension data to watertight completeness). Furthermore, machine learning techniques such as reinforcement learning can be used as the iterative nature of preliminary scanning process, where insufficient is punished and augmentation is rewarded until metrology scanning is feasible based on the preliminary scan data. This closed loop of robotic sensing autonomous measuring and human augmenting the imperfect autonomously collected data (supervised autonomy) can be used for AI training to over time accomplish completely intelligent autonomy. Measurement data (the 3D reference design-point cloud or mesh) can be associated with the location/orientation of the robotic element. The human augmentation will be saved in the same format. In the future, when a similarly shaped object (by searching in the database shape to shape or by utilizing metadata such as part name/category), the simplified sensing or the complete 3D measurement program will predict what points or regions/areas will not be easily captured without multiple iterations or human augmentation, and so the AI will preempt the inefficiency by already applying augmentation to those sections.

30 50 50 30 50 30 40 50 Another option is that metrology 3D scanning appcan use the metrology data to determine if there are defects in target object, such as additive manufacturing (AM) quality issues, or if target objectrequires maintenance or repair. Metrology 3D scanning appcan then be configured to create a visual representation of target objectwith an indicator of the defects or the areas that require the maintenance or repair. For example, metrology 3D scanning appcan display the subset of metrology scan data with each point or mesh mapped in different colors based on the degree of defect (i.e., the deviation from the 3D reference design) on the HRI deviceor on a display device of computing device. Once metrology data is collected, each point or mesh represents an accurate dimensional data of the overall or required subset of the target object. This metrology data can be compared to the 3D model after successful alignment, and then the two data can be compared to calculate the deviation as an example between each corresponding mesh or point which exists both on the dimensional data and the localized 3D model. The degree of deviation will be in range and various levels can be expressed in different colors representing varying degree of defect. Alternatively, the deviation itself can be visualized in a vector format as a line linking the two corresponding point or mesh. This results in deviation overlaid on top of the physical target object, enabling the human operator to perform additional operations such as inspection, repair/maintenance, or using HRI device to move the same or another robotic element to perform other tasks based on the deviation.

70 30 Another option is that metrology scanning pathand motion plan determined by metrology 3D scanning appcan be provided to the second robotic element for manipulation of another target object by the second robotic element.

Other options are also possible, such as: (1) comparing the metrology scan data to the reference design of the part to perform inspection of the object, which can be the part or tool/mold; (2) comparing the metrology scan data to the reference design of the part to perform inspection of the mold/tool that made the part; (3) the metrology scan data and the reference design and correct the tool or part (in which case) a very similar HRI process can be used, where the operator can as an example use the AR device to repair the tool by seeing the deviation data presented to provide HRI input to align the scan data with heat map or deviation data on top of the physical object to perform manual or robotic repair; etc. . . . .

While various embodiments have been described above, this disclosure is not intended to be limited thereto. Variations can be made to the disclosed embodiments that are still within the scope of the appended claims.

The detailed description is to be construed as exemplary only and does not describe every possible embodiment since describing every possible embodiment would be impractical. Numerous alternative embodiments may be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.

The following additional considerations apply to the foregoing discussion. Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

Additionally, certain embodiments are described herein as including logic or a number of routines, subroutines, applications, or instructions. These may constitute either software (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware. In hardware, the routines, etc., are tangible units capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

Accordingly, the term “hardware module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where the hardware modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.

Hardware modules may provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple of such hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and may operate on a resource (e.g., a collection of information).

The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules.

Similarly, the methods or routines described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented hardware modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors may be located in a single location, while in other embodiments the processors may be distributed across a number of locations.

This detailed description is to be construed as exemplary only and does not describe every possible embodiment, as describing every possible embodiment would be impractical, if not impossible. A person of ordinary skill in the art may implement numerous alternate embodiments, using either current technology or technology developed after the filing date of this application.

Those of ordinary skill in the art will recognize that a wide variety of modifications, alterations, and combinations can be made with respect to the above described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

The patent claims at the end of this patent application are not intended to be construed under 35 U.S.C. § 112 (f) unless traditional means-plus-function language is expressly recited, such as “means for” or “step for” language being explicitly recited in the claim(s).