Multi-mode video event indexing includes determining a quality of object distinctiveness with respect to images from a video stream input. A high-quality analytic mode is selected from multiple modes and applied to video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality, else a low-quality analytic mode is selected and applied to the video input images via a hardware device to determine object activity within the video input images, wherein the low-quality analytic mode is different from the high-quality analytic mode.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method for multi-mode video event indexing, the method comprising: applying background subtraction via a programmable hardware device to a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of video input images from the video stream input; determining a level of the detected foreground object activity with respect to the video input images from the video stream input; selecting from a plurality of video analytics modes and applying an object tracking-based analytic mode via a hardware device to the detected foreground object activity of the video input images to track a foreground object, in response to the determined level of detected quality of the detected foreground object activity meeting a threshold level of quality; selecting from the plurality of video analytics modes and applying a non-object tracking-based analytic mode to the detected foreground object activity of the video input images via a hardware device to determine object movement from extracted foreground object appearance attributes without tracking the foreground object, in response to the determined level of detected quality of the detected foreground object activity not meeting the threshold level of quality; and writing results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.
The invention is a method for indexing video events using multiple analysis modes. It first subtracts the background from a video stream to detect foreground objects. If the amount of foreground activity exceeds a threshold, it uses object tracking to follow the objects. Otherwise, it analyzes the appearance of foreground objects to determine movement without tracking. The results of either tracking or movement determination are then written to an index for later retrieval. A hardware device performs the background subtraction and video analytics.
2. The method of claim 1 , wherein the determining the level of the detected foreground object activity comprises: determining a density of the foreground relative to an entirety of the image; and wherein the threshold level of activity is a density value.
The method described above determines the amount of foreground activity by calculating the density of the foreground objects relative to the entire image. The threshold for selecting object tracking is then a specific density value. The system switches to object tracking if the foreground density is high enough.
3. The method of claim 2 , wherein the selecting the object tracking-based mode further comprises: determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time; determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time; determining an elapsed time between the first and the second times; comparing the elapsed time to a stability time period; and selecting the object tracking-based mode if the elapsed time is less than the stability time period.
This expands on the method where foreground activity is measured by density. To select object tracking, the system checks if the foreground density is above a threshold at one time, and then drops below a lower minimum value later. If the time between these events is short (less than a "stability time period"), the system selects the object tracking mode. This prevents the system from switching to object tracking for objects that disappear quickly.
4. The method of claim 2 , wherein the selecting the object tracking-based mode further comprises: determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity; comparing the time duration to a specified time period; selecting the object tracking-based mode if the time duration is less than the specified time period.
Expanding on the method using foreground density, the system selects object tracking based on how long the density stays above the threshold. If the duration that the foreground density remains above the threshold is less than a specified time period, the system switches to, or remains in, object tracking mode.
5. The method of claim 4 , wherein the density of the foreground is an average of a plurality of individual densities over the specified time period.
In the method described where density is used for mode selection, the foreground density is calculated as an average over a specified time period, rather than a single instantaneous value. This average density is then compared to the threshold to determine which analytic mode to use.
6. The method of claim 5 , further comprising: determining the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.
In the density averaging method, more recent density values are weighted higher than older density values when calculating the average. This provides a more responsive measure of foreground activity and ensures the system responds quicker to recent changes.
7. A computer system for multi-mode video event indexing, the computer system comprising: a processing unit, computer-readable memory and a computer-readable storage system; fourth program instructions to apply background subtraction to a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of video input images from the video stream input; first program instructions to determine the quality of object distinctiveness by determining a level of the detected foreground object activity with respect to the video input images from the video stream input; second program instructions to select from a plurality of video analytics modes and apply an object tracking-based mode to the detected foreground object activity of the video input to track a foreground object, in response to the determined level of detected quality of object distinctiveness meeting a threshold level of quality; third program instructions to select from the plurality of video analytics modes and apply a non-object tracking-based to the detected foreground object activity of the video input images to determine object movement from extracted foreground object appearance attributes without tracking the foreground object, in response to the determined level of detected quality of the detected foreground object activity not meeting the threshold level of quality; and fifth program instructions to write results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index; and wherein the first, second, third, fourth and fifth program instructions are stored on the computer-readable storage system for execution by the processing unit via the computer-readable memory.
The invention is also a computer system that indexes video events using multiple analysis modes. It includes a processor, memory, and storage. Program instructions perform background subtraction to detect foreground objects. If the amount of foreground activity exceeds a threshold, it uses object tracking to follow the objects. Otherwise, it analyzes the appearance of foreground objects to determine movement without tracking. The results of either tracking or movement determination are then written to an index.
8. The computer system of claim 7 , wherein the first program instructions are further to determine the level of the detected foreground object activity as a density of the foreground relative to an entirety of the image; and wherein the threshold level of activity is a density value.
The computer system, as described above, determines the amount of foreground activity by calculating the density of the foreground objects relative to the entire image. The threshold for selecting object tracking is then a specific density value. The system switches to object tracking if the foreground density is high enough.
9. The computer system of claim 8 , wherein the second program instructions are to select and apply the object tracking-based mode by: determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity; comparing the time duration to a specified time period; and selecting the object tracking-based mode if the time duration is less than the specified time period.
This computer system selects object tracking mode based on the time duration that foreground density is greater than or equal to a threshold. If that duration is less than a specified time, object tracking is enabled. The system doesn't switch if the object lingers too long, possibly preventing tracking errors for static foreground objects.
10. The computer system of claim 8 , wherein the second program instructions are to select and apply the object tracking-based mode by: determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time; determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time; determining an elapsed time between the first and the second times; comparing the elapsed time to a stability time period; and selecting the object tracking-based mode in response to determining that the elapsed time is less than the stability time period.
The computer system selects object tracking based on density changes. It checks if density is above a threshold at one time and later drops below a minimum value. If the time between these events is less than a stability period, object tracking is selected. This prevents using object tracking when activity quickly disappears.
11. The computer system of claim 10 , wherein the second program instructions are further to determine the density of the foreground as an average of a plurality of individual densities over the specified time period.
In this computer system, the foreground density is an average of several individual densities calculated over a period of time. This averaged value is used when determining whether to use object tracking or the non-object tracking method.
12. The computer system of claim 11 , wherein the second program instructions are further to determine the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.
When calculating the average density in the computer system, recent density values are given more weight than older values. This makes the system more responsive to recent changes in the video, allowing for quicker switching between tracking modes.
13. A computer program product for multi-mode video event indexing, the computer program product comprising: a computer-readable tangible storage device having computer-readable program code embodied therewith, the computer-readable program code comprising instructions that, when executed by a computer processing unit, cause the computer processing unit to: apply background subtraction to a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of video input images from the video stream input; determine a level of the detected foreground object activity with respect to the video input images from the video stream input; select from a plurality of video analytics modes and apply an object tracking-based analytic mode to the detected foreground object activity of the video input images to track a foreground object, in response to the determined level of detected quality of the detected foreground object activity meeting a threshold level of quality; select from the plurality of video analytics modes and apply a non-object tracking-based analytic mode to the detected foreground object activity of the video input images to determine object movement from extracted foreground object appearance attributes without tracking the foreground object, in response to the determined level of detected quality of the detected foreground object activity not meeting the threshold level of quality; and write results of the tracking of the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.
This invention can also be realized as a computer program product residing on storage. The instructions, when executed, perform background subtraction on video input. Based on foreground activity, it switches between object tracking and analyzing foreground object appearances for movement. Results from either analysis are written to an index. The selection between object tracking and appearance-based analysis depends on whether foreground activity exceeds a defined quality threshold.
14. The computer program product of claim 13 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the level of the detected foreground object activity by determining a density of the foreground relative to an entirety of the image; and wherein the threshold level of activity is a density value.
The computer program, as described above, determines the amount of foreground activity by calculating the density of the foreground objects relative to the entire image. The threshold for selecting object tracking is then a specific density value. The system switches to object tracking if the foreground density is high enough.
15. The computer program product of claim 14 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to select the object tracking-based mode in response to: determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time; determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time; determining an elapsed time between the first and the second times; comparing the elapsed time to a stability time period; and selecting the object tracking-based mode in response to determining that the elapsed time is less than the stability time period.
The computer program selects object tracking based on foreground density changes. It tracks density at two points: a first instance where it exceeds a threshold, and a second where it falls below a minimum. If the time between these instances is less than a defined stability period, object tracking is chosen. This prevents short fluctuations from triggering object tracking unnecessarily.
16. The computer program product of claim 15 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to select the object tracking-based mode by: determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity; comparing the time duration to a specified time period; selecting the object tracking-based mode in response to determining that the time duration is less than the specified time period.
In this computer program, object tracking is selected if the duration for which foreground density exceeds a certain threshold is shorter than a specified time. This prevents the system from enabling object tracking when an object has been static in the foreground for too long.
17. The computer program product of claim 16 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the density of the foreground as an average of a plurality of individual densities over the specified time period.
This computer program calculates foreground density as an average across several samples over a period of time. This average density is used to determine when to switch between object tracking and appearance-based analysis methods.
18. The computer program product of claim 17 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.
In calculating the average density within the computer program, more recent densities are given higher weight than older densities. This makes the analysis more responsive to changes in foreground activity and improves switching between tracking modes.
19. A method of providing a service for multi-mode video event indexing, the method comprising: integrating computer-readable program code into a computer system comprising a processing unit, a computer-readable memory and a computer-readable tangible storage device; wherein the computer-readable program code is embodied on the computer-readable tangible storage device and comprises instructions that, when executed by the processing unit via the computer-readable memory, cause the processing unit to: apply background subtraction to a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of video input images from the video stream input; determine a level of the detected foreground object activity with respect to the video input images from the video stream input; select from a plurality of video analytics modes and apply an object tracking-based analytic mode to the detected foreground object activity of the video input images to track a foreground object, in response to the determined level of detected quality of the detected foreground object activity meeting a threshold level of quality; select from the plurality of video analytics modes and apply a non-object tracking-based analytic mode to the detected foreground object activity of the video input images to determine object movement from extracted foreground object appearance attributes without tracking the foreground object, in response to the determined level of detected quality of the detected foreground object activity not meeting the threshold level of quality; and write results of the tracking of the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.
The invention describes a service that provides multi-mode video event indexing. This is achieved by integrating a software implementation of the techniques described in earlier claims onto a computer system. The software performs background subtraction, dynamically selects between object tracking and appearance-based analysis based on foreground activity, and outputs analysis results to an index.
20. The method of claim 19 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the level of the detected foreground object activity by determining a density of the foreground relative to an entirety of the image; and wherein the threshold level of activity is a density value.
The video event indexing service uses foreground density to measure the level of foreground object activity. Specifically, it computes the density of the foreground relative to the image, and uses this value as the quality measure with which to determine the selection of an analytic mode. The threshold is defined as a density value.
21. The method of claim 20 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to select the object tracking-based mode in response to: determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time; determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time; determining an elapsed time between the first and the second times; comparing the elapsed time to a stability time period; and selecting the object tracking-based mode in response to determining that the elapsed time is less than the stability time period.
The video event indexing service uses density changes for selecting object tracking. Object tracking is used if the density starts above a threshold, then falls below a minimum value within a specific time (stability time period). This prevents short fluctuations in foreground activity from triggering the object tracking mode.
22. The method of claim 21 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to select the object tracking-based mode by: determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity; comparing the time duration to a specified time period; selecting the object tracking-based mode in response to determining that the time duration is less than the specified time period.
In the video event indexing service, the object tracking is selected when the duration that the foreground density remains above a specific threshold is less than a specified time period. In other words, the duration that an object is present in the foreground determines the selection of the object tracking mode.
23. The method of claim 22 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the density of the foreground as an average of a plurality of individual densities over the specified time period.
In this video event indexing service, foreground density is determined from averaging densities across multiple samples over a specified time period. The resulting average density value is used as the measure of foreground activity for analytic mode selection.
24. The method of claim 23 , wherein the computer-readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.
In the video event indexing service, calculating the average density value involves giving more weight to recent densities than older ones. This makes the system more responsive to recent changes in foreground activity when dynamically selecting analytic modes.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
August 17, 2010
September 10, 2013
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.