A method, apparatus, and system for visualizing nonconformance data for a physical object. An augmented reality application in a portable computing device plots, in a defined coordinate cube, points corresponding to nonconformance locations on the physical object. The augmented reality application determines a sub-set of the points plotted that correspond to a region of the physical object visible in an image of the region of the physical object acquired by the portable computing device at a position of the portable computing device, where the sub-set of the points exclude nonconformance locations occluded from view by a physical object structure of the physical object in the image. The augmented reality application displays the nonconformance data for the sub-set of the points visible in the image in association with a sub-set of the nonconformance locations for the physical object in the image displayed on a display system in the portable computing device.
Legal claims defining the scope of protection, as filed with the USPTO.
1. An augmented reality system configured to visualize nonconformance data for a physical object, such that the augmented reality system comprises: a portable computing device that comprises a sensor system and a display system, such that the portable computing device and the sensor system are configured to generate scan data that describes points in space for the physical object at a phase of manufacture; a scan application configured to generate a point cloud representation of a surface profile of a region of the physical object based upon the scan data; a modeling application that, when executed, generates a model of the region of the physical object in a defined coordinate cube using the point cloud representation for the region of the physical object; a localization application that, when executed, localizes the portable computing device with the model of the region of the physical object in the defined coordinate cube based on a location target, and identifies a position of the portable computing device with respect to the physical object; and an augmented reality application configured such that, when executed the augmented reality application: plots points in the defined coordinate cube corresponding to nonconformance locations on the physical object; determines a sub-set of points plotted in the defined coordinate cube that are visible in an image of the region of the physical object acquired by the portable computing device where the sub-set excludes points determined to be located at a predetermined distance beyond a physical object structure in the image when viewed at the position of the portable computing device and thus occluded from a view from the position of the portable computing device; and displays the nonconformance data for the sub-set of points visible in the image in association with a sub-set of the nonconformance locations on the physical object in the image displayed on the display system in the portable computing device.
2. The augmented reality system of claim 1 , wherein in displaying the nonconformance data for the sub-set of the points plotted in the defined coordinate cube that are visible in the image in association with the sub-set of the nonconformance locations on the physical object in the image displayed on the display system in the portable computing device, the augmented reality application displays the nonconformance data for the sub-set of the points visible in the image in association with the sub-set of the nonconformance locations on the physical object in the image displayed on the display system in the portable computing device without displaying the nonconformance data for the nonconformance locations on the physical object that are occluded from view in the image of the region of the physical object.
3. The augmented reality system of claim 1 , wherein the physical object is an aircraft structure and wherein in determining the sub-set of the points plotted in the defined coordinate cube that are visible in an image of the region of the physical object acquired by the portable computing device at the position of the portable computing device, the augmented reality application determines a distance of each nonconformance location from the aircraft structure displayed that is beyond a predetermined distance that corresponds to the nonconformance location being occluded from view.
4. The augmented reality system of claim 1 , wherein in localizing the portable computing device with the model of the region of the physical object in the defined coordinate cube and identifying a position of the portable computing device with respect to the physical object, the localization application locates a location target for the physical object, uses a target position of the location target to localize the portable computing device with the model of the region of the physical object in the defined coordinate cube, and identifies the position of the portable computing device with respect to the physical object.
5. The augmented reality system of claim 1 , further comprising the augmented reality application configured to: determine whether the portable computing device is greater than a selected distance for a panoramic view; and display the points for all of the nonconformance locations in the defined coordinate cube to provide a panoramic view of all of the nonconformance locations.
6. The augmented reality system of claim 1 , further comprising the augmented reality application configured to filter the nonconformance data based on time information to display at least one of historical nonconformance data or active nonconformance data.
7. The augmented reality system of claim 1 , wherein the nonconformance data for the sub-set of the points is live nonconformance data.
8. The augmented reality system of claim 1 further comprising: the portable computing device comprising an interface configured to receive the nonconformance locations on the physical object.
9. The augmented reality system of claim 1 , wherein the sensor system comprises at least one of a three-dimensional scanner, a camera, a global positioning system receiver, structured light three-dimensional scanner, a lidar system, a laser scanner, a conoscopic holograph system, or a time of flight three-dimensional scanner.
10. The augmented reality system of claim 1 , wherein the portable computing device is selected from a group comprising a mobile phone, a tablet computer, a head mounted device, and smart glasses.
11. The augmented reality system of claim 1 , wherein the physical object is selected from a group comprising a mobile platform, a stationary platform, a land-based structure, an aquatic-based structure, a space-based structure, an aircraft, a commercial aircraft, a rotorcraft, a surface ship, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, an automobile, a power plant, a bridge, a dam, a house, a manufacturing facility, a building, an aircraft structure, a wing, a fuselage section, an engine housing, an engine, a composite panel, a wall, a skin panel, and a chair.
12. An augmented reality system configured to visualize nonconformance data for a physical object such that the augmented reality system comprises: a portable computing device configured to acquire an image of a physical object; an augmented reality application configured to execute the following: plot points in a defined coordinate cube, wherein the points correspond to nonconformance locations on the physical object; determine a sub-set of the points plotted in the defined coordinate cube that correspond to a region of the physical object that is visible in the image acquired by the portable computing device at a position of the portable computing device, wherein the sub-set of the points excludes plotted points determined to be located at a predetermined distance beyond a physical object structure in the image when viewed at the position of the portable computing device and thus occluded from a view from the position of the portable computing device by the physical object structure of the physical object in the image; and display the nonconformance data for the sub-set of the points visible in the image in association with a sub-set of the nonconformance locations for the physical object in the image displayed on a display system in the portable computing device.
13. The augmented reality system of claim 12 , further comprising: the portable computing device comprising a sensor system that is configured to generate scan data that describes points in space for the physical object at a phase of manufacture; a scan application configured to generate a point cloud representation of a surface profile of a region of the physical object using the scan data; a modeling application configured to generate a model of the region of the physical object in the defined coordinate cube using the point cloud representation for the region of the physical object; a localization application configured to: localize the portable computing device with the model of the region of the physical object in the defined coordinate cube based on a location target; and identify a position of the portable computing device with respect to the physical object.
14. The augmented reality system of claim 13 further comprising: a computer system, wherein at least one of the scan application, the modeling application, and the localizing application are located in the computer system.
15. The augmented reality system of claim 13 , wherein at least one of the scan application, the modeling application, and the localization application are located in the portable computing device.
16. A method for visualizing nonconformance data for a physical object, the method comprising: plotting in a defined coordinate cube, by an augmented reality application in a portable computing device, points corresponding to nonconformance locations on the physical object; determining, by the augmented reality application in the portable computing device, a sub-set of the points plotted in the defined coordinate cube that correspond to a region of the physical object that is visible in an image of the region of the physical object acquired by the portable computing device such that the sub-set excludes plotted points determined to be located beyond a predetermined distance that corresponds to a nonconformance location being occluded from view from a physical object structure in the image when viewed at a position of the portable computing device, where the sub-set of the points exclude nonconformance locations that are occluded from the view from the portable computing device by the physical object structure of the physical object in the image; and displaying, by the augmented reality application in the portable computing device, the nonconformance data for the sub-set of the points visible in the image in association with a sub-set of the nonconformance locations for the physical object in the image displayed on a display system in the portable computing device.
17. The method of claim 16 , determining step comprises: determining, by the augmented reality application in the portable computing device, the points plotted within the defined coordinate cube for nonconformance locations are located at the predetermined distance beyond the physical object structure in the image which are occluded from a view from the portable computing device at the position; and excluding, by the augmented reality application in the portable computing device, these points that are the predetermined distance beyond the physical object structure, wherein remaining points form the sub-set of the points.
18. The method of claim 16 further comprising: locating a location target for the physical object; using a target position of the location target to localize the portable computing device with a model of the region of the physical object in the defined coordinate cube based on a location target; and identifying the position of the portable computing device with respect to the physical object.
19. The method of claim 16 , wherein the displaying step comprises: displaying, by an augmented reality application in a portable computing device, the nonconformance data for the sub-set of the points visible in the image in association with the sub-set of the nonconformance locations on the physical object in the image displayed on the display system in the portable computing device without displaying the nonconformance data for the nonconformance locations on the physical object that are occluded from view in the image of the region of the physical object.
20. The method of claim 16 further comprising: determining whether the portable computing device is greater than a selected distance for a panoramic view; and displaying the points for all of the nonconformance locations on the physical object in the defined coordinate cube to provide a panoramic view of all of the nonconformance locations.
21. The method of claim 16 , further comprising: the augmented reality application filtering the nonconformance data based on time information; and displaying at least one of historical nonconformance data or active nonconformance data.
22. The method of claim 16 , wherein the nonconformance data for the sub-set of the points is live nonconformance data.
23. The method of claim 16 , wherein the portable computing device is selected from a group comprising a mobile phone, a tablet computer, a head mounted device, and smart glasses.
24. The method of claim 16 , wherein the physical object is selected from a group comprising: a mobile platform, a stationary platform, a land-based structure, an aquatic-based structure, a space-based structure, an aircraft, a commercial aircraft, a rotorcraft, a surface ship, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, an automobile, a power plant, a bridge, a dam, a house, a manufacturing facility, a building, an aircraft structure, a wing, a fuselage section, an engine housing, an engine, a composite panel, a wall, a skin panel, and a chair.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
October 12, 2018
August 11, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.