Ball Feeding Systems and Methods

This application is a continuation-in-part of and claims priority to U.S. patent application Ser. No. 18/411,744, entitled “Player Monitoring Systems and Methods for Compensating for Network Delay” and filed on Jan. 12, 2024, which is incorporated herein by reference. U.S. patent application Ser. No. 18/411,744 claims priority to U.S. Provisional Patent Application No. 63/479,855, entitled “Systems and Methods for Dispersed Athletic Games” and filed on Jan. 13, 2023, which is incorporated herein by reference. U.S. patent application Ser. No. 18/411,744 also claims priority to U.S. Provisional Patent Application No. 63/471,664, entitled “Systems and Methods for Dispersed Athletic Games” and filed on Jun. 7, 2023, which is incorporated herein by reference.

This disclosure generally relates to player monitoring systems and methods for efficiently processing sensor data in a manner that reduces processing burdens on processing resources of the system. In some embodiments of the disclosure, sensor data from one sensor (e.g., a low-resolution or low-rate sensor) is used to cull sensor data from another sensor (e.g., a high-resolution or a high-rate sensor) so as to reduce the overall amount of data processed for evaluating player performance.

Athletes often spend countless hours training in order to improve their skill level so that they can become more competitive in sporting events, such as basketball games, soccer games, hockey games, and games of other sports. In an effort to assist athletes in improving their skill levels, systems have been developed that track an athlete's performance while training or playing a game and then provide feedback indicative of the performance. Such feedback can then be evaluated for helping the athlete to improve his skill level. As an example, commonly-assigned U.S. Pat. Nos. 7,094,164; 8,948,457; 10,360,685; and 10,010,778, which are incorporated herein by reference, generally describe systems that track various performance metrics of athletes, such as the trajectory of a basketball during a basketball shot, dribble, or pass, to provide information that can be used to evaluate and improve player performance.

People around the world have played sporting games against each other on a single court. Basketball type games played on a single court can include around the world, horse, ghost, knockout, etc. It is a challenge to provide the same experience for basketball shooting games over dispersed courts. For example, in the basketball shooting game of knockout, two or more shooters shoot free throws on a single court, with a second shooter attempting to make a basket before the first shooter in order to “knock out” the first shooter. When the first shooter makes a basket, the second shooter becomes the first shooter, and a third shooter, if participating, becomes the second shooter. In the absence of a third shooter, the first shooter becomes the second shooter and attempts to “knock out” the first shooter by making a basket before the first shooter. In a dispersed game, it is difficult to consistently and accurately track the position of each shooter and to determine the sequence in which the shooters make baskets. Additionally, in dispersed basketball shooting games such as horse, in which a second shooter must emulate a shot made by a first shooter, it is difficult to track the shot criteria of the first shot and verify whether the second shooter has successfully emulated the shot criteria. Due to these and other challenges, very few attempts have been made to develop systems that enable playing sporting games, such as knockout, horse, and others, over dispersed courts.

The embodiments disclosed herein are directed to these and other considerations.

Systems and methods are provided for enabling dispersed game tracking. In one embodiment, the system is configured to enable dispersed basketball shooting games, such as the basketball game of knockout. The system can be configured to identify players on dispersed courts and to track the motion of a basketball on the court to determine when a shot is made. In some embodiments, the system is configured to track the body position of the shooter and the trajectory of the ball in order to identify the type of shot. In the basketball game of horse, the system can determine not only whether a shot was made, but also what type of shot was made according to identified shot criteria (e.g., hook shot, lay-up, dunk, overhand, underhand, etc.). The system can then determine whether a second person playing on a dispersed court makes a second shot using the identified shot criteria. In other embodiments, the system can be configured to track games such as baseball, soccer, golf, tennis, etc. on dispersed courts, allowing players to play remotely. In some embodiments, the system can be configured to track player performance, and provide training feedback to the players. Players can train by themselves, with the system providing real-time feedback, or in other embodiments, the system can facilitate training multiple players simultaneously over dispersed courts.

The system can use one or more sensors, including depth sensors, audio sensors, cameras, light sensors that are enabled to capture data indicative of a person's movements around a game area and, in some embodiments, the motion of a game ball (e.g., basketball, soccer ball, golf ball, tennis ball, etc.) to track movements of the player and the game ball. The system can include at least one processor to analyze the sensor data to determine and evaluate the movements of players and the trajectory of the game ball, including whether the game ball is successfully put through the goal, hoop, etc., depending on the game being tracked. Characteristics of player movements and ball movements can be identified and tracked in order to provide real-time feedback to one or more players on how to improve their shot posture, dribbling, racket swing, kick, throw, etc., based on the game being played.

shows an embodiment of a systemfor dispersed game tracking. For illustrative purposes, the systemwill be described in the context of the basketball shooting game of knockout. However, systemcan be used to track dispersed games of other basketball games, such as around the world, traditional basketball, and horse, and non-basketball games such as tennis, football, soccer, golf, hockey, field hockey, etc., as well as games of other sports. The system may enable players to play a game across dispersed locations in real-time. As shown, systemcan include user device(s)A,B, connected device(s)A,B, server, viewing device, analytics database, and training device(s)A,B communicating over network.

According to some embodiments, the systemmay include additional sensor systems for different types of competitions or training sessions. For example, a football passing, kicking, or punting competition would likely require a larger field of view than a basketball-oriented system. Thus, the football competition might use depth sensor(s) mounted on a drone while the basketball competition might use a 2D image sensor from a mobile phone (although a basketball competition could include the use of other types of sensors or sensor configurations, including depth sensors mounted on a drone as described for embodiments using larger playing spaces). Other types of competitions requiring a larger field of view may include soccer kicking, golf shots, and tennis strokes, although any of the games described herein may use larger fields of view or fields of view of different sizes, as may be desired. Exemplary techniques for monitoring sporting events with drone mounted depth sensors are disclosed in commonly-assigned U.S. Pat. No. 11,450,106 entitled “Systems and Methods for Monitoring Objects at Sporting Events” and filed on Feb. 28, 2022, which is incorporated herein by reference.

The systemcan include user device(s)A,B associated with a respective user. Although systemis illustrated with two user devicesA and user deviceB, it should be understood that there can be any number of user devices within system(e.g., the system can include user deviceA, user deviceB, . . . user device). Reference numbermay be used herein to refer to any of the user devicesA,B of the system. Each user deviceA,B can be correlated to a user participating within a dispersed game being enabled by system. User deviceA,B can be a smart phone, laptop, tablet, or other computing device enabled to communicate with other elements of systemover network.

In some embodiments, each player can use a user deviceA,B to interact with systemand likewise, to receive messages and alerts from systemand/or from other players using system. Each user deviceA,B, can optionally be in operative communication with a deviceA,B, referred to herein as “connected device” indicating that the deviceA,B is connected to another device (e.g., user deviceA,B) for communication (with wires or wirelessly). Although systemis illustrated with two connected devicesA,B, it should be understood that there can be any number of connected devices within system(e.g., the system can include connected deviceA, connected deviceB, . . . connected device). Reference numbermay be used herein to refer to any of the connected devicesA,B of the system. Each connected deviceA,B can be a wearable smart device (e.g., as a smart watch), display device, speaker (e.g., earbud), a wireless headset, a virtual reality headset, an augmented reality headset, etc. Each connected deviceA,B can be configured to allow the player to receive information from systemand other players using systemas well as provide input to systemor send messages to other players using systemwithout the interruption of the dispersed game associated with having to refer to a user deviceduring an active dispersed game being tracked by system. In some embodiments, as will be described in more detail with respect to, user device(s)can include a camera that can provide camera data (e.g., captured images) to systemthat can be used to track the player's motions, including movement about the court and dribbling, and the motions of the basketball, including shooting trajectory of shots taken by a player using systemand trajectory of passes made by a player user system. Systemcan instruct the player associated with user device(e.g., user deviceA) to place the respective user devicein a location at which the camera associated with user devicecan track the player, the basketball, and the basketball hoop, so that the systemcan determine the position of the player on the court, the trajectory of the basketball during a dispersed game, and when and whether the basketball successfully passes through the basketball hoop. In some embodiments, multiple players can play on a single court while playing a dispersed game with other players that are located on one or more courts dispersed from the single court. In such a scenario, a single user device (e.g., user deviceA) may be sufficient to track the multiple players as well as the trajectories of basketball shots taken by the multiple players. The user device(e.g., user deviceA) may use data collected from one or more sensors to distinguish the multiple players from one another to enable the tracking of individual player performance of the multiple players. Each dispersed court can support one or more individual players as described herein.

depicts a playeron a basketball courtshooting a basketballto a basketball hoop. In the embodiment shown by, the player has positioned a user deviceon the side of the court, such as on a bench or other structure or on the ground or floor. The user devicemay have a camera or other type of sensor, and the player, basketball, and hoopmay all be within the field of view of the camera or other sensor. In some embodiments, the camera of the user devicecaptures a video stream defined by a plurality of frames where each frame shows a respective location of the basketballas it travels to and through the hoop. The video images can be evaluated by the user device(or a component in communication with the user device, such as the server) to determine the ball's trajectory as it travels during the basketball shot. As discussed in more detail herein, various characteristics of the shot, including the trajectory, can be determined and tracked by the system. In other embodiments, the user devicemay be at other locations and other devices, such as tracking devices, may be used to capture information about the basketball shots, as will be described in more detail below. As an example, it is possible to mount a tracking deviceabove the hoop(e.g., above, behind, or within a backboard) or at other locations as may be desired.

Referring to, the servercan be in operative communication with other components of system. The serveris at a remote location capable of communicating with other devices of the systemthrough one or more networks.

In some embodiments, the servercan evaluate one or more characteristics of a shot and/or pass executed by a player using system, for example, whether a shot was made into the basket or goal and the trajectory of the ball including shot angle, shot velocity, ball spin, etc. The servercan also track characteristics of the player, including a player's posture during a shot, while blocking a shot of another player, while moving about the court, and while passing a ball, etc. The servercan evaluate a player's body motion and stance by receiving real-time data over networkreceived from the one or more sensors. In other embodiments, these characteristics can be evaluated directly by user device(s)locally without the use of server. In other embodiments, as will be described in more detail below, tracking device(s)can both capture (using integrated sensors) and evaluate characteristics associated with a player's and a ball's motion during a dispersed game without the use of user device(s)or server. In some embodiments, the servercan be in operative communication with analytics database, either directly connected or communicating over network. Analytics databasecan be configured to store data received from the one or more sensors associated with user device(s), tracking device(s), and/or training device(s). Analytics databasecan be configured to also store data generated by the computer vision logic included in user device(s), server, and/or tracking device, as described in further detail with respect to,, and, respectively. From time to time, user device(s), server, and tracking devicemay update the analytics databasewith new data and/or access existing data stored on analytics database.

Viewing deviceis configured to receive image data and/or a video stream from other components of system, including but not limited to user device(s), tracking device(s)and/or training device(s). Reference numbermay be used herein to refer to any of the tracking devicesA,B of the system, and reference numbermay be used herein to refer to any of the training devicesA,B of the system. Viewing devicecan include many of the same components as described with respect toassociated with user device(s). In some embodiments, viewing devicecan be used by spectators to view a game monitored by system, or to view a training session implemented on system. As will be described in further detail with respect to, viewing devicecan receive a video stream in substantially real-time from other components of system. In other embodiments, viewing devicecan receive the video stream delayed by a delay factor that allows the systemto adjust for variable latency between players utilizing systemon dispersed courts. For example, a first player utilizing systemfrom a first location may have a network connection suffering from higher latency than a second player utilizing systemfrom a second location. Accordingly, viewing devicecan be configured to delay the video stream received from the second player in order to synchronize the video feed received from both the first location and the second location. In other embodiments, the video feed can be received asynchronously without adjusting for latency differences between the first location and the second location. In other embodiments, rather than receiving a video feed, still images can be transmitted to the viewing device. Although the systemis described with respect to a single viewing device, there can be any number of viewing devicesA,B,, connected to systemthat are configured to receive image and/or image data from other components of system. In some embodiments, users of viewing devicescan be human coaches that use viewing devicesto analyze player performance and provide real-time training feedback to players utilizing system.

Note that there are various techniques that can be performed to assess latency between networked devices, such as between the serverand any of the user devices, tracking devices, training devices, and viewing devices. As an example, the servermay perform a handshake process, whereby the roundtrip delay between the serverand a user deviceis measured. In this regard, the servermay transmit a ping to the user device, which is configured to immediately transmit a reply to the server. The servermay determine the time taken to receive the reply from the transmission of the ping and calculate the latency between the serverand the user deviceto be about half of the roundtrip delay. This process may be repeated, and the servermay determine an average latency that is used to account for latency, as will be described in more detail below. Similar processes may be performed with other devices,,to estimate the associated network latencies.

Tracking devicecan have components similar to user device(s)and server. In some embodiments, each dispersed court can have one or more tracking deviceslocated thereon that is configured to both capture data using one or more sensors (described in more detail with respect to, below) and analyze the data received to track player and ball movements around the dispersed court. Tracking devicecan also be configured to evaluate characteristics associated with a player's and a ball's motion during a dispersed game without the use of user device(s)or server. In some embodiments, tracking devicecan be implemented as a basketball hoop with integrated sensors, processors, and computer logic configured to enable the functionalities of tracking device. The tracking devicecan also be implemented with one or more sensors that are worn by a player to detect parameters associated with the player, such as player position or motion, or that are imbedded in the ball. In some embodiments, tracking devicemay comprise one or more cameras that capture images of the playing spaces so that players and balls may be tracked. In various other embodiments, tracking devicecan be implemented as a football goal post, soccer goal, hockey goal, etc. with integrated sensors, processors, and computer logic as described above. Although systemis illustrated with two tracking devices, tracking deviceA and tracking deviceB, it should be understood that there can be any number of tracking devices within system(e.g., the system can include tracking deviceA, tracking deviceB, . . . tracking deviceN).

Training devicecan be a device configured to assist with training exercises for players utilizing system. Training devicecan include the same or similar components described with respect to tracking device. In some embodiments, training devicecan be a basketball passing machine that is configured to automatically return a basketball to a player utilizing system. In some embodiments, the training devicecan be configured to be used in other sports games. For example, training devicecan be a baseball pitching machine, a tennis ball machine, a hockey puck machine, etc. In any case, training device can either receive data indicative of a dispersed player from other components of system(e.g., from one of user device(s), server, and/or tracking device(s)) or be configured to identify a player on a dispersed court using its own sensors and computer logic. Subsequent to identifying a dispersed player and his or her position, training devicecan be configured to enhance the training of the dispersed player by returning a basketball to the player subsequent to a shot attempt and/or set up more complex training plays by passing the basketball to a predetermined location on the dispersed court depending on the parameters of the selected training exercise.

Networkcan be of any type, for example, networkcan be a local area network (LAN) or wide area network (WAN), to permit user device(s), connected device, server, viewing device, analytics database, tracking device, and/or training deviceto communicate with one another. In one embodiment, networkcan be the Internet. In one embodiment, networkcan be a mobile device network, for example but not limited to an EDGE network, a 3G network, a 4G network, and/or a 5G network. The networkmay also include any combination of networks as may be desired. As an example, the networkmay include a cellular network and the Internet, where the components of the system(e.g., a user device) includes a cellular transceiver that can be used to access the Internet through the cellular network. In some embodiments, components of systemcan communicate over a conductive medium (e.g., a wire), fiber, or otherwise. In some embodiments, one or more components of the system can communicate directly with one another over a wireless or wired connection.

As shown by, user devicecan include at least one processor, which can incorporate processing hardware for executing instructions stored in the memory. As an example, the processormay include a central processing unit (CPU) a digital signal processor (DSP), a graphic processing unit (GPU) and/or a quantum processing unit (QPU). The processorcan communicate to and drive the other elements within the user devicevia a local interface, which can include at least one bus. The user devicecan have a clock, which can be used to track time and synchronize operations with other components of system, including connected devices, server, viewing devices, tracking devices, and/or training devices.

The user devicecan have a communication module. The communication modulecan include a radio frequency radio (RF) radio or other device for communicating wirelessly with other components of system. The power supplycan be an internal battery, such as a lithium-ion battery or nickel cadmium battery. In some embodiments, power supplycan have an interface that allows it to plug into or otherwise interface with an external component, such as a wall outlet or battery, and receive electrical power from the external component.

As shown in, the user devicecan also include various sensors, such as a camera, a depth sensor, an audio sensor, a light sensor, and a wearable sensor(e.g., an accelerometer). In other embodiments, other types of sensors may be used, such as sensors (e.g., accelerometers or other types of sensors) embedded in the ball or worn by a player. The cameracan be used to record, capture, or obtain images or videos the area surrounding or in proximity to the user device. In one embodiment, the camerais configured to capture two-dimensional (2-D) video images of the dispersed court including images of the object (basketball) being dribbled, passed, or shot, the person performing the dribbling, passing, or shooting, and any other players on the dispersed court. The depth sensorcan be configured to determine a relative distance (with respect to the depth sensor) Of objects in the field of view of the user device. The audio sensorcan be configured to record sound or noise in proximity to the user device. The light sensorcan be configured to sense ambient light in the area surrounding the user device. The wearable sensorcan be configured to sense motion of the player or body part of the player.

The cameracan include one or more CCDs (charge coupled devices) and/or one or more active pixel sensors or CMOS (complementary metal-oxide semiconductor) sensors. The images or videos from cameracan be stored as image datain memory. In one embodiment, the image datamay define frames of the captured images. The image datacan be stored in any appropriate file format, including but not limited to, PNG (portable network graphics), JPEG (joint photographic experts group), TIFF (tagged image file format), MPEG (moving picture experts group), WMV (windows media video), QuickTime, and GIF (graphics interchange format). The sound recordings from audio sensormay be incorporated into the video file from the cameraand stored in image data. If the sound recording from the audio sensoris not part of the video file, then the sound recording can be stored in any appropriate file format, including but not limited to WAV (waveform audio), MP3 (MPEG Layer III Audio), WMA (Windows media audio) and MPEG and saved in image dataor elsewhere in memory.

In one embodiment, for each frame of image data, the depth sensorcan provide a depth map indicating a respective depth for each pixel of the image frame. The depth maps provided by the depth sensorcan be stored as depth datain memory. Note that the depth sensormay be oriented such that the distance measured by the depth sensoris in a direction that is substantially normal to the plane of the 2D coordinate system used by the camera, although other orientations of the depth sensorare possible in other embodiments.

From time to time, the camera logiccan be configured to transmit the image dataand the depth datato server, viewing device, tracking device, and/or training device. The image dataand the depth datamay be analyzed by the user device, server, tracking device, and/or training deviceto track the motion of the players, the motion of the ball, and determine one or more characteristics of the ball motion (e.g., ball velocity, angle, spin, etc.), one or more characteristics of a player's motion (posture, speed, positioning, etc.), and to track a shot, dribble, or pass being made by a player. The image dataand the depth datacan be time-stamped based on the time indicated by the clockin order to indicate when the image dataand the depth datawere obtained. Thus, upon receiving the image data from multiple user device(s), a user device, the server, tracking device, and/or training devicecan determine which image frames from the multiple user deviceswere captured at substantially the same time in order to facilitate tracking of ball and player movement. From time to time, the user devicesmay communicate with each other and/or the other components of systemin order to synchronize their clocks so that a comparison of a timestamp for an image frame from one camera(and/or image sensor) with a timestamp for an image frame from another camera(and/or image sensor) accurately indicates the time difference that the two image frames were captured. The image dataand the depth datamay be presented to a user for analysis or review (e.g., via user device, connected device, server, viewing device, and/or tracking device).

Various types of camerasand depth sensorsmay be used in user device. In some embodiments, user device may be a smartphone device, such as an Apple iPhone™ or Android™ device with integrated camera. In such a user device, the cameraand depth sensorcan be integrated into the same housing. The camerais configured to capture a video stream including frames of video data in which each frame is defined by a plurality of pixels. Each pixel can be associated with two coordinates, an x-coordinate and a y-coordinate, representing a location in 2D space. For each frame, each pixel is assigned a color value (which may include a red component I value, a blue component (B) value, and a green component (G) value) indicative of the color of light received by the camerafrom the location in 2D space corresponding to the pixel's coordinates. Further, for each pixel, the depth sensorcan measure the distance from the depth sensorto the real-world object that is at the pixel's corresponding location in 2D space. The distance (which, as described above, may be in a direction substantially normal to the plane of the 2D coordinate system used by the camera) may be referred to as the “depth” as the corresponding pixel. Using the image datafrom the cameraand the depth datafrom the depth sensor, the location of an object captured by the cameracan be determined in 3D space. That is, for a point on the object, its x-coordinate and y-coordinate from the image dataprovided by the cameraindicate its location along two axes (e.g., the x-axis and y-axis), and the point's depth value from the depth sensor, which may be referred to as the “z-coordinate,” indicates its location along a third axis (e.g., the z-axis). Notably, the coordinate system defined by the three axes is not necessarily relative to gravity. That is, depending on the orientation of the user device, gravity may be in any direction relative to the axes of the coordinate system. Thus, unless a calibration process is performed, the direction of gravity relative to the coordinate system may be unknown. An example of a calibration process for determining the direction of gravity relative to the coordinate system is described by commonly-assigned U.S. Pat. No. 9,734,405, entitled “Systems and Methods for Monitoring Objects in Athletic Playing Spaces” and issued on Aug. 15, 2017, which is incorporated herein by reference.

In some embodiments, the depth sensorhas a wave emitter (e.g., an infrared laser projector or other type of emitter) and a wave sensor for sensing reflections of the energy emitted by the wave emitter. The wave emitter emits infrared radiation at various wavelengths into free space, although radiation at other wavelengths outside of the infrared spectrum (e.g., visible light) may be emitted in other embodiments, and the wave sensor senses the reflected energy to capture a video stream having frames of video data. Each frame of the depth datafrom the depth sensorcorresponds to a respective frame of image datafrom the camera. Further, a pixel of a frame of the depth datacorresponds to (e.g., has the same x and y coordinates) and indicates the depth for at least one corresponding pixel in the image datafrom the camera. In another embodiment, the depth sensorcan use a stereoscopic camera to capture depth data.

In this regard, for a frame of video data captured by the depth sensor, the depth sensorconverts the frame to a depth map by assigning each pixel a new color value (referred to herein as “depth value”) representative of the pixel's depth. Thus, when the depth map is displayed, objects displayed as the same color within the image should be approximately the same distance away from the depth sensor, noting that it is often unnecessary for the depth map to actually be displayed during operation.

As described above, a given pixel of the image datafrom the camerais associated with an x-coordinate and y-coordinate indicative of the pixel's location in 2D space, and the pixel is associated with a depth value from a corresponding pixel in the depth dataprovided by the depth sensorindicative of the pixel's z-coordinate. The combination of the x-coordinate, y-coordinate, and z-coordinate defines the pixel's location in 3D space relative to the coordinate system of the camera. That is, the x-coordinate, y-coordinate, and z-coordinate define the location of the point from which light measured for the pixel was reflected toward the image sensor from an object.

User devicecan include device logicfor generally controlling the operation of the user device, including communicating with the other components of the system. User devicecan optionally include computer vision logicfor processing and analyzing the image dataand depth datafrom the user device, object trackerthat is configured to determine the position and movement of objects, such as a game ball, and players handling the object, and any other persons in the playing area. The device logic, camera logic, computer vision logic, and object trackercollectively will be referred to herein as the “control logic”of device, though the control logicmay have other configurations in other embodiments. User devicecan optionally include evaluation data, which can be data that is used and/or analyzed by device logic, computer vision logic, and/or object trackerto track the motion of an object including dribbling, passing and/or shooting, determine whether a shot has been made, and determine one or more characteristics of a shot or pass. Object tracker, computer vision logic, and evaluation dataare substantially similar to object tracker, computer vision logic, and evaluation data, which are described in more detail with respect to; thus a full description here is omitted for brevity.

is a block diagram of an embodiment of a serverused in the tracking system of. The servercan be implemented as one or more general or special-purpose computers, such as a laptop, hand-held (e.g., smartphone) desktop, or mainframe computer. The servercan include server logicfor generally controlling the operation of the server, including communicating with the other components of the system. The servercan also include object tracker logic, referred to herein as “object tracker,” that can determine the position and movement of an object, such as a ball used for a sporting event, the person handling the object, and any other persons in the athletic playing area. The servercan also include computer vision logicfor processing and analyzing the image dataand depth datareceived from user device(s)(and/or the camera datareceived from tracking device, as described in more detail with respect to). The server logic, the computer vision logic, and the object trackerwill be collectively referred to herein as the “control logic”of the server, though the control logiccan have other configurations in other embodiments. Any portions of the control logiccan be implemented in software, hardware, firmware, or any combination thereof. In the servershown in, the control logicis implemented in software and stored in memoryof the server. Note that the control logic, when implemented in software, can be stored and transported on any non-transitory computer-readable medium for use by or in connection with an instruction execution apparatus that can fetch and execute instructions.

The servercan include at least one processor, which has processing hardware for executing instructions stored in memory. As an example, the processorcan include a central processing unit (CPU), a digital signal processor (DSP), a graphic processing unit (GPU), and/or a quantum processing unit (QPU). The processorcommunicates to and drives the other elements within the servervia a local interface, which can include at least one bus. Furthermore, an input interface, for example, a keypad, keyboard, or a mouse, can be used to input data from a user of the server, and an output interface, for example, a printer, monitor, liquid crystal display (LCD), or other display apparatus, can be used to output data to the user. Further, a communication interfacemay be used to exchange data among the components of the systemor with network, as shown in.

As shown by, sensor data, evaluation data, camera data, and gameplay datacan be stored in memoryat the server. The camera datacan include image dataand depth datafrom user device(s). The sensor datacan include data and measurements from any additional sensors located on a dispersed court, for example, sensors integrated into tracking deviceand/or training device. The camera data, the sensor data, and the evaluation datacan be used and/or analyzed by server logic, the computer vision logic, and the object trackerto track the object including dribbling, passing, shooting, and transition from dribbling to passing, determine whether a shot has been made, and determine one or more characteristics of a shot, pass, dribble, or transition.

The gameplay dataincludes information regarding the gameplay being monitored by the system. Such datacan indicate events that affect the outcome of the gameplay. As an example, in the game of knockout, the gameplay datamay indicate a history of the shots made by the respective player, as well as the time of occurrence of each made shot (e.g., when the ballpasses through the hoop). Thus, the gameplay datacan be analyzed to determine the outcomes of the game and the status of each player during the game, including which player won the game and when the other players are eliminated and/or active. For the game of horse, the gameplay datamay indicate the location, shot type, and outcome of each shot attempted such that the datamay be analyzed to determine the status of each player in the game as well as the winner of the game once the game has concluded. The gameplay datamay indicate other types of events and information about the game being monitored by the systemas may be desired. In addition, gameplay datais shown as being stored by the server, but such datamay be stored and used by other devices of the system, as may be desired. Storing such dataat the device hosting gameplay may be a convenient location of the gameplay data, but the datamay be stored elsewhere as may be desired.

The evaluation datacan include information associated with one or more shot characteristics, one or more passing characteristics, one or more dribbling characteristics, and/or one or more player movements characteristics, for example, the movements associated with particular shot types or the movements associated with particular pass types. The evaluation datacan also include training information such as diagrams and videos that can be transmitted and displayed on a user device, connected device, and/or tracking deviceto provide training instructions on proper shooting form and/or technique, proper dribbling form and/or technique, proper passing form and/or technique, and proper player posture during passing, shooting, and/or dribbling associated with a user. In one embodiment, the testing procedures can be displayed to the user on user device,, connected deviceand/or tracking device, and the object trackercan evaluate a user's performance with respect to the testing procedure in evaluation databased on the user's shooting motion, passing motion, and/or dribbling motion captured in camera data.

The object trackercan receive camera data, sensor data, information from computer vision logicand/or other information relating to the ball and/or the person handling the ball to track the shooting motion, passing motion, and/or dribbling motion to determine one or more characteristics of the shooting motion, passing motion, dribbling motion, and/or transition from passing motion to dribbling motion. Once a characteristic has been determined, the object trackercan compare the determined characteristic to proper characteristic information in evaluation datato score or otherwise evaluate the determined characteristic. The proper characteristics stored in evaluation datacan be preselected parameters or techniques that are associated with a preferred shooting motion, player posture, passing motion, and/or dribbling motion. The proper shooting characteristics may include a predetermined number, such as a predetermined ball velocity or predetermined shooting angle. The proper shooting characteristics may also be defined relative to the body of the person performing the shot (e.g., arm bend angle while shooting). Further, the proper shooting characteristics may be defined relative to actions of the person shooting the ball (e.g., there may be one set of proper shooting characteristics when the person is running and a different set of proper shooting characteristics for when the person is walking or stationary). In still other embodiments, the proper shooting characteristics may be defined as a range (e.g., greater than a predetermined minimum, less than a predetermined maximum, or between a predetermined minimum and a predetermined maximum).

The proper passing characteristics may include a predetermined number such as a predetermined speed, a predetermined pass height or a predetermined number of turnovers. The proper passing characteristics may also be defined relative to the body of the person performing the passing (e.g., the pass height should not exceed the chest of the person passing the ball). Further, the proper passing characteristics may be defined relative to actions of the person passing the ball (e.g., there may be one set of proper passing characteristics when the person is running and a different set of proper passing characteristics for when the person is walking or stationary). In still other embodiments, the proper passing characteristics may be defined as a range (e.g., greater than a predetermined minimum, less than a predetermine maximum, or between a predetermined minimum and a predetermined maximum).

The proper transition characteristics may include a predetermined number such as a predetermined time, a predetermined ball position or a predetermined number of turnovers. The proper transition characteristics may also be defined relative to the body of the person performing the transition (e.g., the ball position should not exceed the chest of the person passing the ball). Further, the proper transition characteristics may be defined relative to actions of the person completing the transition (e.g., there may be one set of proper transition characteristics when the person is dribbling while running and a different set of proper transition characteristics when the person is dribbling while walking or being stationary). In still other embodiments, the proper transition characteristics may be defined as a range (e.g., greater than a predetermined minimum, less than a predetermine maximum, or between a predetermined minimum and a predetermined maximum).

The computer vision logiccan be used to analyze and process the image dataand depth datafrom the user device(s)stored in camera data. The computer vision logiccan extract information from the image dataand depth datain camera datausing models, theories and other techniques to identify or recognize the object to be tracked and one or more participants (including the torso, arms, legs, hands, feet, etc., of the participants) involved in the athletic event associated with the object. The computer vision logiccan use numerous techniques to identify or recognize objects and people such as content-based image retrieval, pose estimation, optical character recognition, 2D code reading, shape recognition, facial recognition, object recognition, pattern recognition and any other appropriate identification or recognition technique. Exemplary techniques for identifying and tracking players are disclosed in commonly-assigned U.S. Pat. No. 11,450,106 entitled “Systems and Methods for Monitoring Objects at Sporting Events” and filed on Feb. 28, 2022, which is incorporated herein by reference.

In one embodiment, the computer vision logiccan perform one or more of the following techniques and/or processes on the image dataand depth datadata from camera data: pre-processing; feature extraction; detection/segmentation; high-level processing; and decision making. The pre-processing of the camera datacan involve the processing of the data to confirm that the data is in the proper form for subsequent actions. Some examples of pre-processing actions can include noise reduction and contrast enhancement. After the camera datahas been pre-processed, the camera datacan be reviewed or analyzed to extract features (e.g., lines, edges, corners, points, textures and/or shapes) of various complexity from the camera data. Next, in the detection/segmentation step, decisions can be made regarding the features and/or regions that are relevant and require additional processing. The high-level processing of the reduced set of camera data(as a result of the detection/segmentation step) involves the estimation of specific parameters (e.g., object size) and classifying of a detected object into categories. Finally, the decision-making step makes a determination of the identity of the detected object or person or indicates that the detected object or person is not known.

The computer vision logiccan identify objects and persons that are present in the camera databy processing the individual images and videos received from a cameraand/or any combined or grouped images and videos based on camera datafrom multiple camerasA,B, . . . ,N (and/or image sensorsand depth sensorsassociated with tracking deviceA,B, . . . ,N). The computer vision logiccan identify objects or persons using labels carried by the objects or persons (e.g., numbers worn on jerseys), facial recognition techniques (if identifying a person), profiling techniques (using the profile of the object or person) or any other suitable recognition techniques.

In one embodiment, the object or person can have a label that is attached or affixed to the object or person and that can be recorded by user device(s)and/or tracking device(s). If the person is carrying a tag, the label can be (but does not have to be) incorporated into the tag carried by the person. The computer vision logiccan identify the label attached to the object or person and then identify the object or person based on stored information in memorycorrelating each label to an object or person. In another embodiment, the computer vision logiccan identify a person using facial recognition or can identify an object or a person by using a distinguishable or identifiable profile or feature of the object or person. For example, the identification of a circular or spherical shape may indicate the presence of the ball in the frame. Similar to the process for identifying an object or person using a label, the computer vision logiccan identify facial features and/or other profiles or features of the object or person in the camera dataand then compare the identified facial features and/or other profiles or features of the asset to stored information in memorycorrelating information on features and/or profiles to an object or person.

The computer vision logiccan send the camera dataand/or information on the identified object or person from analyzing the camera datato the object tracker. The object trackercan use information on the identified object and/or persons from the computer vision logicto determine a dribbling motion for the object and one or more dribbling characteristics associated with the dribbling motion, a transition from a dribbling motion to a passing motion and one or more characteristics associated with the transition, a passing motion with the object, one or more passing characteristics associated with the passing motion, and a shooting motion with the object, and one or more shooting characteristics associated with the shooting motion. In one embodiment, the object trackercan use synchronized and calibrated camera datato determine a dribbling motion and corresponding dribbling characteristics, a dribble-to-pass transition and corresponding transition characteristics, a passing motion and corresponding passing characteristics, and/or a shooting motion and corresponding shooting characteristics. The synchronization and calibration of the camera datacan be done by the computer vision logicor the object tracker.

The synchronization of the camera datamay involve ensuring that the corresponding frames of camera dataprocessed by the computer vision logicor the object trackerfor a given sample were captured substantially at the same time. In this regard, a sample generally refers to data from measurements that were taken substantially at the same time. For example, at a given instant, images of multiple balls over dispersed courts may be captured by multiple user devices(e.g., user deviceA,B, . . . ,) located on the dispersed courts. Further, each ball's position may be calculated from each of the images. Since the position data from the multiple user devicesare based on image data captured substantially at the same time in such example, the measured positions are part of the same sample. In order to determine which frames were captured substantially at the same time, a global time system may be defined.

As an example, the servermay maintain a global time domain and adjust the timestamps from each of the user devicesaccording to the global time domain so that the timestamps are synchronized. That is, images captured at the same time should have the same adjusted timestamp in the global time domain. Alternatively, the server(or other device maintaining a global time domain, such as one or more of user device(s)and/or one or more tracking device(s)) may from time-to-time transmit timing information to the user device(s). The user device(s)may then use such information to adjust their respective clocks so that images having the same timestamps relative to the global time domain from the user device(s)were captured substantially at the same time. Alternatively, the servermay analyze unsynchronized timestamps from the user device(s)and determine which frames were captured substantially at the same time. In such embodiment, the servermay communicate with the user device(s)in a controlled calibration process in order to assess timing differences between the user device(s). As an example, each user devicemay report a current timestamp to the serverin a handshake process, as previously described above for determining a roundtrip delay, so that the servercan determine the respective user device(s) 100 time relative to a global time system maintained by the serveror otherwise. Specifically, by calculating the delay from a device (e.g., a user device) to the server, the server can be configured to use such delay to determine the proper global timestamp to assign a given frame so that the global timestamp indicates the time that the frame was captured by the user device. In other embodiments, other techniques for synchronizing the camera data are possible.

In one embodiment, assume that the serveris configured to maintain the global time domain and that the clocks of the other components, such as user devices, viewing devices, tracking devices, and training devices, are updated from time-to-time to remain synchronous (within an acceptable margin of error) with the clockof the server. In the described embodiment, the clockof each user device(as well as the clocks of the other devices,,) may be configured to maintain a time relatively accurately. As an example, the clockmay be based on a cellular or GPS signal received by the device. As indicated above, the servermay be configured to transmit from time-to-time timing information that may be used to adjust the clockso that the timestamps generated by the clockare relative to the global time domain.

As an example, for a given update, the servermay be configured to transmit the current timestamp generated by the clock, which may be used as the reference for the global time domain. Upon receiving the timestamp, the user devicemay adjust the clocksuch that the timestamps generated by the clockmatch the timestamps generated by the clockof the serverat the same respective time, noting that the devicemay take into account the network latency between it and the server. In this regard, the user devicemay assume that that the current time relative to the global time domain corresponds to the time indicated by the timestamp received from the serverplus the determined network latency between the deviceand server. The user devicemay then appropriately update the clocksuch that the timestamp generated by the clockis relative to the global time domain. Thus, if the clocksandgenerate timestamps at the same time, such timestamps preferably have the same value within an acceptable margin of error.

When multiple user device(s) are present on a court, the calibration of the camera datacan involve the correlation of the pixels in an image frame to a global coordinate system such that the serveris aware of which pixels in different frames from different user device(s) represent the same physical location in space. This may be achieved, for example, by ensuring that the pixels in frames from different user device(s) representing the same location on the court are assigned the same global coordinates. By calibrating the camera data, the object and the person handling the object can be tracked through multiple image frames from different user device(s)since the location of the object and the person handling the object, as defined in the global coordinate system, can be the same in each image frame regardless of the field of view of the user devicecapturing the image frame. Once the camera datais calibrated, the object trackercan track the object through multiple image frames as the object moves into and out of view individual image frames. If one or more user device(s)become misaligned, the calibration process can be repeated to calibrate the misaligned user device(s). Note that the global coordinate system may be referenced to predefined directions (e.g., one reference direction downward toward the center of the earth and another reference direction toward the north pole) such that play at two different locations (e.g., two different cities) can be normalized.

In one embodiment, the object trackercan determine a dribbling motion, a passing motion, and/or a shooting motion by analyzing successive frames of camera datato determine changes in the position and/or depth of the identified object and/or changes in the position of the person preforming the dribbling, passing, and/or shooting motion. Some exemplary techniques of calculating the trajectory of a ball that may be used by object trackercan be found in commonly-assigned U.S. Pat. No. 8,908,922 entitled “True Space Tracking of Axisymmetric Object Flight Using Diameter Measurement” and U.S. Pat. No. 8,948,457 entitled “True Space Tracking of Axisymmetric Object Flight Using Diameter Measurement,” both of which patents are hereby incorporated by reference. By identifying changes associated with the upward and downward trajectories of the object or the person handling the object, the object trackercan determine characteristics associated with the dribbling, passing, and/or shooting motion. In one embodiment, some of the characteristics can be determined using conventional mathematical and physics principles and equations based on trajectory information extracted from the camera data. The determined characteristics can then be stored in memoryand/or scored based on proper characteristics stored in evaluation data.

As an example, the object trackermay analyze the trajectory of the ball and identify a plurality of dribbles. For one or more dribbles, the object trackermay determine a parameter indicative of a dribbling characteristic, such as ball speed, dribble height, repetition rate, type of dribble, etc., and store such parameters for analysis. In some cases, the object trackermay correlate a given parameter with information that can be used to characterize dribbling performance. For example, if a given dribble is performed with the left hand, the parameter determined for the dribble may be correlated in memory with a left-hand identifier. Based on the parameters correlated with such identifier, the object trackermay calculate one or more scores or other statistics indicative of the player's performance with his left hand. As an example, an average repetition rate, ball speed, or dribble height for the player's left hand may be calculated. If a dribble type is identified for a particular dribble, as will be described in more detail below, the parameter determined for the dribble may be correlated in memory with a type identifier indicative of the dribble type. Based on the parameters correlated with such identifier, the object trackermay calculate one or more scores or other statistics indicative of the player's dribbling performance for the identified dribble type. If a particular defender (e.g., a virtual defender, as described below with respect to) can be identified as guarding the player, as will be described in more detail below, the parameter determined for the dribble may be correlated in memory with an identifier that identifies the defender. Based on parameters correlated with such identifier, the object trackermay calculate one or more scores or other statistics indicative of the player's dribbling performance against the defender. In other embodiments, the data can be grouped in other ways in order to provide further insight into the player's dribbling performance relative to certain conditions. Any of the parameters, scores, or other statistics described herein may be reported by the system to indicate one or more dribbling characteristics for the tracked player. Any such parameters, scores, or other statistics may be used in order to calculate an overall or combined assessment of the player's dribbling performance that may be reported.