Systems, devices, and techniques related to selecting a key frame for burst image processing are discussed. Such techniques may include generating key frame scores for at least some frames of a multi-frame burst image capture such that the key frame scores include a combination of an image quality component, a shutter lag component, and a burst image processing latency component and selecting a frame having a maximum key frame score as the key frame.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A method for performing burst image processing comprising: receiving a plurality of frames corresponding to a multi-frame burst image capture; determining a key frame score for each of at least two frames of the plurality of frames, wherein each key frame score comprises an image quality component, a shutter lag component, and a burst image processing latency component; selecting a burst image processing key frame for the plurality of frames based on the key frame scores; and combining at least portions of at least a subset of the plurality of frames using the selected key frame to generate a resultant burst image processed frame.
2. The method of claim 1 , wherein the at least two frames comprise a first frame and a second frame that is temporally subsequent to the first frame, the method further comprising: determining a maximum or expected key frame score for a third frame of the plurality of frames, wherein the third frame is temporally subsequent to the second frame; comparing the maximum or expected key frame score for the third frame to a maximum of the key frame scores of the first and second frames; and generating, in response to the maximum or expected key frame score for the third frame exceeding the maximum of the key frame scores of the first and second frames, a key frame score for the third frame or bypassing, in response to the maximum or expected key frame score for the third frame not exceeding the maximum of the key frame scores of the first and second frames, generation of a key frame score for the third frame.
3. The method of claim 2 , wherein, in response to the maximum or expected key frame score for the third frame not exceeding the maximum of the key frame scores of the first and second frames, the method further comprises: selecting the frame corresponding to the maximum of the key frame scores of the first and second frames for use as the burst image processing key frame; and bypassing key frame score generation for remaining frames of the plurality of frames, wherein said combining is performed in parallel with an image capture of at least one of the remaining frames.
4. The method of claim 2 , wherein determining the maximum or expected key frame score for the third frame comprises determining a maximum or expected image quality component for the third frame, a shutter lag component for the third frame, and a burst image processing latency component for the third frame, wherein the maximum or expected image quality component for the third frame is based on one of a maximum available image quality value or a probabilistic image quality value for the third frame generated based at least in part on image quality values of the first and second frames.
5. The method of claim 1 , further comprising: evaluating a second plurality of frames temporally prior to the plurality of frames corresponding to the multi-frame burst image capture to generate a moving object indicator; and including, in response to the moving object indicator, a moving object component in the key frame scores for each of the at least two frames.
6. The method of claim 1 , wherein the at least two frames comprise a first frame and a second frame that is temporally subsequent to the first frame, the method further comprising: determining a maximum or expected key frame score for a third frame of the plurality of frames, wherein the third frame is temporally subsequent to the second frame, the maximum or expected key frame score for the third frame comprising a maximum or expected image quality component for the third frame, a shutter lag component for the third frame, a burst image processing latency component for the third frame, and a moving object component for the third frame; comparing the maximum or expected key frame score to a maximum of the key frame scores of the first and second frames; and generating, in response to the maximum or expected key frame score exceeding the maximum of the key frame scores of the first and second frames, a key frame score for the third frame or bypassing, in response to the maximum or expected key frame score not exceeding the maximum of the key frame scores of the first and second frames, generation of a key frame score for the third frame.
7. The method of claim 1 , further comprising: evaluating at least a second subset of the plurality of frames to generate a moving object indicator; and including, in response to the moving object indicator, a moving object component in the key frame score for each of the at least two frames, wherein the moving object component is determined using a moving object function of a frame number of the at least two frames in a temporal frame number count of the plurality of frames, wherein the moving object function provides a first component value for a first frame number, a second component value for a second frame number, and a third component value for a third frame number, wherein the second frame number is temporally subsequent to the first frame number and the third frame number is temporally subsequent to the second frame number, and wherein the second component value is greater than both the first and third component values.
8. The method of claim 1 , wherein the image quality component for a first frame of the at least two frames is determined using a monotonically increasing function of an image quality value for the first frame.
9. The method of claim 8 , wherein the image quality value for the first frame comprises an autofocus blur value.
10. The method of claim 8 , further comprising: generating the image quality value for the first frame by applying a linear quadratic estimation to a sharpness value, a gyroscopic motion value, and an autofocus blur value for the first frame.
11. The method of claim 1 , wherein the image quality component for a first frame of the at least two frames is determined using a monotonically increasing function of an image quality value for the first frame, the shutter lag component is determined using a monotonically decreasing function of a shutter lag value for the first frame, and the burst image processing latency component is determined using a monotonically decreasing function of a burst image processing latency value for the first frame, wherein the shutter lag value is a time difference between a shutter actuation time and an image capture time of the first frame and the burst image processing latency value is a time difference between a sum of the selection time of the first frame and a constant burst image processing time and the shutter actuation time.
12. The method of claim 1 , wherein each key frame score comprises a sum of a product of a first weight and the image quality component, a product of a second weight and the shutter lag component, and a product of a third weight and the burst image processing latency component.
13. The method of claim 12 , further comprising: receiving selection of a particular mode from a plurality of burst optimization modes, wherein the plurality of burst optimization modes comprise at least a high image quality mode and a fast operation mode; and implementing the particular mode by modifying the first, second, and third weights, wherein, in the high image quality mode, the first weight has a first value and the third weight has a second value and, in the fast operation mode, the first weight has a third value and the third weight has a fourth value, wherein the first value is greater than the third value and the fourth value is greater than the second value.
14. A device comprising: a memory to store a plurality of frames corresponding to a multi-frame burst image capture; and one or more processors coupled to the memory, the one or more processors to: determine a key frame score for each of at least two frames of the plurality of frames, wherein each key frame score comprises an image quality component, a shutter lag component, and a burst image processing latency component; select a burst image processing key frame for the plurality of frames based on the key frame scores; and combine at least portions of at least a subset of the plurality of frames using the selected key frame to generate a resultant burst image processed frame.
15. The device of claim 14 , wherein the at least two frames comprise a first frame and a second frame that is temporally subsequent to the first frame, the one or more processors to: determine a maximum or expected key frame score for a third frame of the plurality of frames, wherein the third frame is temporally subsequent to the second frame; compare the maximum or expected key frame score for the third frame to a maximum of the key frame scores of the first and second frames; and generate, in response to the maximum or expected key frame score for the third frame exceeding the maximum of the key frame scores of the first and second frames, a key frame score for the third frame or bypass, in response to the maximum or expected key frame score for the third frame not exceeding the maximum of the key frame scores of the first and second frames, generation of a key frame score for the third frame.
16. The device of claim 15 , wherein, in response to the maximum or expected key frame score for the third frame not exceeding the maximum of the key frame scores of the first and second frames, the one or more processors are to: select the frame corresponding to the maximum of the key frame scores of the first and second frames for use as the burst image processing key frame; and bypass key frame score generation for remaining frames of the plurality of frames, wherein the processor to combine is to be performed in parallel with an image capture of at least one of the remaining frames.
17. The device of claim 14 , the one or more processors to: evaluate a second plurality of frames temporally prior to the plurality of frames corresponding to the multi-frame burst image capture to generate a moving object indicator; and include, in response to the moving object indicator, a moving object component in the key frame scores for each of the at least two frames.
18. The device of claim 14 , wherein the at least two frames comprise a first frame and a second frame that is temporally subsequent to the first frame, the one or more processors to: determine a maximum or expected key frame score for a third frame of the plurality of frames, wherein the third frame is temporally subsequent to the second frame, the maximum or expected key frame score for the third frame comprising a maximum or expected image quality component for the third frame, a shutter lag component for the third frame, a burst image processing latency component for the third frame, and a moving object component for the third frame; compare the maximum or expected key frame score to a maximum of the key frame scores of the first and second frames; and generate, in response to the maximum or expected key frame score exceeding the maximum of the key frame scores of the first and second frames, a key frame score for the third frame or bypassing, in response to the maximum or expected key frame score not exceeding the maximum of the key frame scores of the first and second frames, generation of a key frame score for the third frame.
19. The device of claim 14 , wherein the image quality component for a first frame of the at least two frames is determined using a monotonically increasing function of an image quality value for the first frame and wherein the image quality value for the first frame comprises an autofocus blur value.
20. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a device, cause the device to perform burst image processing by: receiving a plurality of frames corresponding to a multi-frame burst image capture; determining a key frame score for each of at least two frames of the plurality of frames, wherein each key frame score comprises an image quality component, a shutter lag component, and a burst image processing latency component; selecting a burst image processing key frame for the plurality of frames based on the key frame scores; and combining at least portions of at least a subset of the plurality of frames using the selected key frame to generate a resultant burst image processed frame.
21. The non-transitory machine readable medium of claim 20 , wherein the at least two frames comprise a first frame and a second frame that is temporally subsequent to the first frame, the machine readable medium comprising further instructions that, in response to being executed on the device, cause the device to perform burst image processing by: determining a maximum or expected key frame score for a third frame of the plurality of frames, wherein the third frame is temporally subsequent to the second frame; comparing the maximum or expected key frame score for the third frame to a maximum of the key frame scores of the first and second frames; and generating, in response to the maximum or expected key frame score for the third frame exceeding the maximum of the key frame scores of the first and second frames, a key frame score for the third frame or bypassing, in response to the maximum or expected key frame score for the third frame not exceeding the maximum of the key frame scores of the first and second frames, generation of a key frame score for the third frame.
22. The non-transitory machine readable medium of claim 20 , the machine readable medium comprising further instructions that, in response to being executed on the device, cause the device to perform burst image processing by, in response to the maximum or expected key frame score for the third frame not exceeding the maximum of the key frame scores of the first and second frames: selecting the frame corresponding to the maximum of the key frame scores of the first and second frames for use as the burst image processing key frame; and bypassing key frame score generation for remaining frames of the plurality of frames, wherein said combining is performed in parallel with an image capture of at least one of the remaining frames.
23. The non-transitory machine readable medium of claim 20 , the machine readable medium comprising further instructions that, in response to being executed on the device, cause the device to perform burst image processing by: evaluating a second plurality of frames temporally prior to the plurality of frames corresponding to the multi-frame burst image capture to generate a moving object indicator; and including, in response to the moving object indicator, a moving object component in the key frame scores for each of the at least two frames.
24. The non-transitory machine readable medium of claim 20 , wherein the at least two frames comprise a first frame and a second frame that is temporally subsequent to the first frame, the machine readable medium comprising further instructions that, in response to being executed on the device, cause the device to perform burst image processing by: determining a maximum or expected key frame score for a third frame of the plurality of frames, wherein the third frame is temporally subsequent to the second frame, the maximum or expected key frame score for the third frame comprising a maximum or expected image quality component for the third frame, a shutter lag component for the third frame, a burst image processing latency component for the third frame, and a moving object component for the third frame; comparing the maximum or expected key frame score to a maximum of the key frame scores of the first and second frames; and generating, in response to the maximum or expected key frame score exceeding the maximum of the key frame scores of the first and second frames, a key frame score for the third frame or bypassing, in response to the maximum or expected key frame score not exceeding the maximum of the key frame scores of the first and second frames, generation of a key frame score for the third frame.
25. The non-transitory machine readable medium of claim 20 , wherein the image quality component for a first frame of the at least two frames is determined using a monotonically increasing function of an image quality value for the first frame and wherein the image quality value for the first frame comprises an autofocus blur value.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
March 21, 2018
March 17, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.