10147210

Data Visualization System

PublishedDecember 4, 2018
Assigneenot available in USPTO data we have
Technical Abstract

Patent Claims
21 claims

Legal claims defining the scope of protection, as filed with the USPTO.

1

1. A system comprising: sensors within a facility, wherein the sensors include cameras with fields-of-view encompassing one or more of: an inventory location or overhead looking down; a memory, storing computer-executable instructions; and a hardware processor configured to execute the computer-executable instructions to: access sensor data acquired from the sensors; generate processed data from the sensor data, wherein the processed data comprises information indicative of: a location of a user in the facility, a user identifier associated with the user, and a user touch association between the user and an item; and generate, from the processed data, graphical data for visualizing use of the facility by the user, the graphical data including: two or more images from the sensor data that are stitched together to form a single aggregate image, visual indicia indicative of the location of the user in the facility overlaying the single aggregate image, visual indicia indicative of the user identifier overlaying the single aggregate image, and visual indicia indicative of the user touch association overlaying the single aggregate image.

2

2. The system of claim 1 , wherein the sensor data includes images acquired by the cameras; and further wherein the hardware processor is configured to execute the computer-executable instructions to: perform facial recognition of one or more of the images with an artificial neural network to identify the user appearing in the sensor data; apply a transform to an apparent position of the user identified within the one or more of the images to determine the location of the user in the facility; and process the one or more of the images to: determine a first contour associated with the user; determine a second contour associated with the item; determine the first contour and the second contour are in contact with one another; and generate the user touch association.

3

3. The system of claim 1 , wherein the sensors comprise a touchpoint sensor; and further wherein the hardware processor is configured to execute the computer-executable instructions to: access touchpoint data indicative of a touch to a particular point in space, wherein the touchpoint data is acquired from the touchpoint sensor; generate second processed data indicative of touchpoints based on the touchpoint data; access a planogram image associated with the inventory location proximate to the touchpoints; and generate second graphical data for visualizing use of the facility by the user comprising visual indicia indicative of the touchpoints overlaying the planogram image.

4

4. A system comprising: a memory, storing computer-executable instructions; and a hardware processor configured to execute the computer-executable instructions to: access sensor data acquired from one or more sensors; and generate data for display using the sensor data, wherein the data for display comprises: a graphical representation of at least a portion of a layout of one or more inventory locations of a material handling facility (“facility”); visual indicia of a location of a user in the facility at a first time, wherein the visual indicia of the location of the user in the facility at the first time is presented with respect to the graphical representation; visual indicia indicative of touchpoint data, wherein the touchpoint data is indicative of a location where user interaction with an item is detected; and a graphical representation of at least a portion of the user, acquired by a sensor within an interval of the first time.

5

5. The system of claim 4 , wherein the hardware processor is further configured to execute the computer-executable instructions to: access a plurality of images acquired by a camera; compare at least a subset of the plurality of images to determine unchanging portions of the plurality of images; designate the unchanging portions as background; and generate the graphical representation of the at least a portion of the user by subtracting the background from at least one of the plurality of images.

6

6. The system of claim 4 , wherein the hardware processor is further configured to execute the computer-executable instructions to: generate the graphical representation of the at least a portion of the user by combining a plurality of images, wherein each of the plurality of images is acquired by a different one of a plurality of cameras.

7

7. The system of claim 4 , wherein the hardware processor is further configured to execute the computer-executable instructions to: generate additional data for display comprising: a user identifier associated with the user.

8

8. The system of claim 4 , wherein the hardware processor is further configured to execute the computer-executable instructions to: generate additional data for display comprising: inventory location identifiers associated with the one or more inventory locations.

9

9. The system of claim 4 , wherein the hardware processor is further configured to execute the computer-executable instructions to: generate additional data for display comprising: a planogram of at least a portion of the one or more inventory locations, wherein the touchpoint data is presented with respect to the planogram.

10

10. The system of claim 9 , wherein the hardware processor is further configured to execute the computer-executable instructions to: determine as a touch by the user that a hand of the user was one or more of: proximate to, or in contact with, the item held at the one or more inventory locations.

11

11. The system of claim 4 , wherein the hardware processor is further configured to execute the computer-executable instructions to: determine a touchpoint based on the touchpoint data indicative of a portion of the user being one or more of: proximate to, or in contact with, the item held at the one or more inventory locations; and generate, for the data for display, an indicator comprising one or more lines extending from an image of a head of the user to the touchpoint.

12

12. The system of claim 4 , wherein the sensor data includes three-dimensional data; and the hardware processor is further configured to execute the computer-executable instructions to: determine the location of the user in the facility using a transform indicative of a correspondence between a point in the facility and coordinates in the three-dimensional data.

13

13. The system of claim 4 , wherein the hardware processor is further configured to execute the computer-executable instructions to: determine, based on the sensor data, the item associated with an action of the user using a machine learning system; and receive input data indicative of accuracy of the determination.

14

14. A method comprising: accessing sensor data acquired by one or more sensors in a material handling facility; accessing touchpoint data indicative of a position of one or more touches of a user with respect to an inventory location; generating data for display comprising: a planogram representative of a plurality of inventory locations; and touchpoint indicators presented with respect to the planogram, wherein the touchpoint indicators are indicative of the position of the one or more touches.

15

15. The method of claim 14 , further comprising: generating additional data for display comprising: visual indicia of a location of the user in the material handling facility.

16

16. The method of claim 14 , further comprising: generating additional data for display comprising: a graphical representation of at least a portion of the user, based on an image acquired by a camera within an interval of time of at least a portion of the one or more touches.

17

17. The method of claim 14 , further comprising: generating additional data for display comprising: information indicative of a user identifier associated with the user.

18

18. The method of claim 14 , further comprising: generating second data for display comprising: an aggregate image comprising a plurality of individual images acquired by a plurality of cameras at the material handling facility.

19

19. The method of claim 18 , further comprising: generating third data for display comprising: a tracking view comprising visual indicia of a location of the user in the material handling facility with respect to the aggregate image.

20

20. The method of claim 14 , further comprising: determining an item associated with an action of the user based on a machine learning system; generating additional data for display comprising data indicative of the item; receiving input data indicative of accuracy of the determination; and training the machine learning system using the input data.

21

21. A method comprising: accessing sensor data acquired by one or more sensors in a facility at a first time; generating data for display using the sensor data, the data for display comprising: a planogram representative of one or more inventory locations within the facility; visual indicia of a location of a user in the facility at the first time overlaying the planogram; and visual indicia indicative of a user identifier overlaying the planogram; and presenting historical sensor data acquired by the one or more sensors before the first time, wherein the historical sensor data is indicative of one or more touches by the user at the one or more inventory locations, and wherein the historical sensor data is presented as an overlay to the data for display.

Patent Metadata

Filing Date

Unknown

Publication Date

December 4, 2018

Inventors

NISHITKUMAR ASHOKKUMAR DESAI
FRANK FLORIAN LIBERATO JR.
I-HUNG WANG
DANIEL BIBIREATA
MURALIDHAR KOKA

Want to explore more patents?

Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.

Citation & reuse

Analysis on this page is generated by Patentable — an AI-powered patent intelligence platform. AI-generated summaries, explanations, and analysis may be reused with attribution and a visible link back to the canonical URL below. Patent abstracts and claims are USPTO public domain.

Cite as: Patentable. “DATA VISUALIZATION SYSTEM” (10147210). https://patentable.app/patents/10147210

© 2026 Patentable. All rights reserved.

Patentable is a research and drafting-assistant tool, not a law firm, and does not provide legal advice. Documents we generate are drafts for review by a licensed patent attorney.