Method and system for combining multiple scans into a single data set

This application claims the benefit of Provisional Patent Application No. 63/437,570 filed on Jan. 6, 2023 entitled “Method and System for Combining Multiple Scans into a Single Data Set.”

This invention generally relates to a data processing method of combining multiple data sets and more particularly to a method for combining data from multiple scans of a single environment into a single data set.

There are some environments where you cannot see the entire area which you would like to measure, or you can gather more information about the environment from a different perspective. This means that multiple sets of measurements must be taken from different locations or at different times. The merging of these individual measurements is challenging, error prone, and time consuming. Existing systems rely on line of sight to all measurement positions. When there is an obstruction in the scene existing systems must break the scene into multiple measurements which must then be manually combined. The multiple measurements are error prone for referencing different positions. With existing systems there is no overview or feedback on the accuracy of the combined measurements.

Existing solutions utilize large data sets with thousands or millions of data points to heuristically identify features and points of interest to be used as the key alignment vectors to compute the transformation. These data sets are expensive in the hardware components and data processing to perform the calculations.

As such there still remains a need for an improved method of combining multiple data sets, such as multiple scans of a single environment, that enable scanning in more conditions and large environments, improve efficiency by being able to show augmented data in real (or near real) time, and better works around and reduces errors resulting from obstructions.

An improved method can be obtained for combining multiple data sets of a single subject to form a single representation of the subject, such as an environment. The method uses a small set of alignment points and scanning the environment from a first position, repositioning the scanning system to take a scan of the same environment from a second position, and labeling the same alignment points to the extent they are visible from the second location. The alignment points may be static points of interest in the environment or they may be placed in the environment by the user to denote the location of common reference points. Preferably the alignment points may be chalk, tape, signs or other fiducial markings that are known in advance and can be detected by the scanning system.

The present invention can be better understood by the following discussion of the manufacture and use of certain preferred embodiments.

One preferred embodiment involves the scanning of an outdoor environment, such as a backyard. Such an environment is too large to capture with sufficient detail from a single location since there are often obstacles, such as trees or buildings, that will obstruct part of the environment from a single scan location. The measurement device will be moveable around the measurement scene and will combine the measurement data from the one or more locations into a single set of measurements or schematics.

Another preferred embodiment allows for an environment to add additional measurements and information to be gathered from the scene. Sometimes the measured environment may have some changes, such as a new additional feature, removal of an object or change of materials worth noting and measuring with the device. The measuring device will allow an operator to return to the same location, or a different location within the environment to add measurements at a later time.

In a preferred embodiment, the measurement device is composed of one or more visual capture devices, one or more depth measuring device, one or more positional sensor, a data processing engine, and a command and control application. The measurement device will be mounted on a tripod or other fixed position and will be repositioned around the measurement scene at specific locations. In other embodiments the device will also be hand-held, wheeled or otherwise mobile during the scanning process to move around obstructions. Particularly preferred is the use of the measuring device disclosed in Provisional Application No. 63/437,561 titled “Measuring Device and Method of Use” and filed on the same day as this application and which is incorporated herein by reference.

1 FIG. 20 60 80 30 40 90 100 60 30 90 80 30 70 110 depicts a flow chart of the overall method of a preferred embodiment of the current invention. Generally, in a preferred embodiment, the method involves positioning a scanning system in a first location in the environment to be scanned. In step, a small set of alignment points are identified in the environment that are visible from multiple locations. In step, the environment is scanned from the first location and then, since this is the first scan, in stepthe scanning system is moved to a second location in the environment. In step, the same set of alignment points are labeled from this second location. Since this is not the first scan, as determined in step, the system in stepthen determines a transformation for the second scan to align the alignment points with those of the prior scans. This includes the possibility for manual calibration of orientation and perspective by the user so the same alignment points are always the same relative origin of measurement. If the user determines the transformed alignment is accurate in step, the environment is scanned from the second location in step. If not, the user repositions the measuring device and/or re-labels the alignment points back from the second location at stepand has the system re-determine a transformation for the second scan in step. Once the second scan is completed, the scanner is moved to another location in stepand the process is repeated from stepby labelling the alignment points from the new location. This process is repeated until the user has obtained the desired number of scans of the environment, which will preferably be determined based on the complexity of the environment and the number of obstructions present. At this point, in stepit is determined that no more scans are required and the process proceeds to stepwhere it is completed.

If the user has measured the environment at least once at a previous time and must come back to the same environment again then the same set of alignment points are labeled from a selected location at this new time. The location may be the same as the first scan or any prior scans, or may be from a new location, as long as the alignment points are labeled and the calibrated transformation looks correct.

The alignment points may be static elements that are already present in the environment, such as the corner of a building or patio. Alternatively, the alignment point can be placed in the environment for the purpose of providing common reference points. This might be chalk, tape, signs or other fiducial markings that are known to be detected. Particularly preferred is the use of unique fiducial markings that are known to the scanning system in advance and can be automatically identified as alignment points when taking each of the scans. For example, a sheet containing a high contrast icon or design that would not normally be present in the environment can be used as a fiducial marking.

Preferably, in order to combine the data from multiple datasets with a minimal set of alignment points, all of the selected alignment points are visible from all the locations a scan of the environment will be taken. However, it is possible to use one or more alignment points that are not visible from a subset of the scanning locations, provided that at least a minimum number of alignment points are shared between each of the scans.

The minimum number of alignment points that are visible from each location will be at least 3. Preferably, the number of alignment points are at least 4 or more in order to improve the alignment accuracy and minimize any error from the transformation. If the number of alignment points is less than 3, then there are multiple possible methods to combine the different locations and inaccurate measurements will occur. If fewer than 3 points are visible then the measurement system will display an error to the user to find and label additional alignment points. It is also preferred that the number of alignment points selected are less than 100 and more preferably less than 6. While a larger number of alignment points can be used, it is less effective as it increases the amount of time the user needs to conduct the scan and does not provide a significant improvement in the resulting combined scan.

When the scanning system is first set these alignment point locations are preferably chosen and labeled using a mobile device via an app that uses an application program interface (API) to connect to the software on the scanning system wirelessly. This can be accomplished automatically using image recognition software to identify the preselected fiducial markings that were placed in the environment. Alternatively, the user can manually select the alignment points from an image of the environment as viewed from the first scan location that is depicted on the screen of the mobile device. Once the alignment points are identified, the scanner will record the position (e.g. XYZ coordinates) for each of the alignment points. On the subsequent scans the set (or a subset) of the alignment points are retargeted from the new location and are similarly identified either automatically or by the user. The scanner will now have multiple data sets representing the same collection of alignment points.

The measurement device will use the alignment data sets to attempt to compute a common frame of reference for all data points and a transformation to combine all of the data points. This is accomplished by carrying out iterations of rotational and translational transformations to minimize the difference between the original points and the transformed points. Prior to the transformation each data set will have a different frame of reference. The absolute position of the alignment points will be common between the multiple locations and the data processing engine will match and align the data points by attempting to find common characteristics between the labeled alignment points. The characteristics used for alignment will include, but are not limited to, distance between points, edge length of segments, normal vector alignment, and error distance after transformation. The specifics of the characteristics used for alignment will allow for the small set of alignment data points to be uniquely identified so as to allow the transformation to be determined.

5 FIG. 140 142 144 144 146 138 148 As shown inthe process to determine the common frame of reference will be iterative to calculate a transformation (step), apply the computed transformation (step), and compare the alignment characteristics (step), where the iteration will continue until the transformation alignment is within the prescribed error threshold (as determined at step). During the iterations it is possible for the measurement device and the user to add, modify, or delete the alignment points (stepsand). This can include removal of inaccurate or noisy points which may cause errors in the alignment process. The iteration may also include repositioning or re-measuring a subset of alignment points. When the operation is complete there will be transformation information returned (step) to allow the measurement device to apply to all subsequent measurements within the scene.

All data points collected will utilize the computed transformation information in order to combine the data sets into a common reference frame. In typical cases there may be rotation about all 3 rotation axes (Pitch, Roll, Yaw) as well as translation shift of distance (XYZ). The common reference frame may be chosen as the frame of reference for one of the scans, or a complete independent frame of reference into which all data is translated.

3 FIG. 3 FIG. 2 2 FIGS.A andB 2 FIG.A 2 FIG.B 450 116 118 120 112 114 120 112 114 shows a simple example of a transformation would be rotation about one of the axes. If the scanner is simply turned at aangle in the azimuth between the two scans, the transformation would be a rotation about the azimuth axis and would be calculated as shown in. Specifically, scanwould be rotated 45° per transformationto result in scan. This can also be seen in. In, two scans have resulted in data setand data set.shows the result after a transformation has been applied to data setto combine the data setsand.

1. Real Time Representation

The computed transformation with the common frame of reference is used to adapt all data from the scanning system so that the output information for the user interface and application API is always in the common frame of reference. This means that the labeled points of interest, measurements and other data from the scanner will be shown in such a way that the multiple scans appear as a single data set. This augmentation of the data set will occur in real-time (or near real-time) so that while in the measuring environment the scanning system automation and the user interface can validate the combined data sets for accuracy and validity to ensure the multiple scans look and act as a single continuous scan.

4 FIG. 122 134 126 128 130 132 122 124 126 128 122 124 126 130 128 130 134 1 122 124 126 1 132 2 2 2 128 122 124 126 shows an example of a preferred embodiment of the method being used to scan a backyard that contains a rectangular pool. In this example there are 4 alignment points (,,,). All of the alignment points are not visible from both the first scan positionand second scan positionlocations. Points,,, andare points of interest in this environment, namely the four corners of the rectangular pool. Points,, andare visible from the first scan location, but pointis not visible from the first scan locationdue to the obstruction. Scanbegins with setting the alignment points. Preferably, the visible points of interest (,and) can be labeled in the Scanas well. Then the scanner is physically moved and reoriented to the second scan locationwhere the same alignment points are labeled. After the alignment and registration phase on Scanthe overlay of the data is now combined between thescans so that as Scanlabels the point of interestit is shown in relative position to the previous labeled points (,, and) to form a single unified data set from a common frame of reference.

The above descriptions of certain embodiments are made for the purpose of illustration only and are not intended to be limiting in any manner. Other alterations and modifications of the invention will likewise become apparent to those of ordinary skill in the art upon reading the present disclosure, and it is intended that the scope of the invention disclosed herein be limited only by the broadest interpretation of the appended claims to which the inventors are legally entitled.