Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A real-time stereo camera disparity estimation device comprising: an input device arranged to input measured data corresponding to rows of left and right images; a memory device arranged to buffer the input measured data; a processor for rotating data to align the rows of left and right images in a same column; a processor for data allocation to create variable window sizes to adapt a window size based on a local texture on the left and right images; a processor for metrics calculation to compute stereo matching costs for disparities between the left and right images within a block within the adapted window size within a block within the adapted window size; and a processor for adaptive disparity selection configured to select disparity values with minimum matching costs from the stereo matching costs as disparity results.
A real-time stereo vision system estimates depth (disparity) from left and right camera images. It includes: an input module to receive rows of pixel data from the left and right images; a memory buffer to store this input data; a data alignment processor to align the rows of the left and right images to be in the same column; a data allocation processor that creates variable window sizes that adapt based on the local texture of the left and right images; a matching cost calculation processor that computes the similarity between image blocks within the adapted window sizes; and a disparity selection processor that picks the disparity (depth) value with the lowest matching cost as the final disparity result.
2. The device of claim 1 , further comprising: a processor for an iterative disparity refinement configured to iteratively refine the disparity values of the disparity results of the processor for adaptive disparity selection.
The real-time stereo vision system from the previous description additionally includes a disparity refinement processor. This processor iteratively improves the initial disparity values from the disparity selection processor. It takes the initial disparity results and refines them to improve accuracy.
3. The device of claim 1 , wherein the processor for data allocation is further configured to create variable window sizes to adapt the window size to a local texture on the at least one of left and right image.
In the real-time stereo vision system, the data allocation processor (which creates variable window sizes to adapt to local texture) specifically adjusts the window size based on the texture of either the left or right image, or both. This allows the system to focus on areas with rich texture for more accurate disparity estimation.
4. The device of claim 1 , wherein the processor for metrics calculation comprises: a plurality of processing devices for multiple processed pixels in a two dimensional block to compute their stereo matching costs for candidate disparities in parallel.
In the real-time stereo vision system, the matching cost calculation processor contains multiple processing elements. These elements compute stereo matching costs in parallel for multiple pixels within a 2D block of the image for various potential disparity values. This massively parallel approach greatly speeds up the disparity estimation process.
5. The device of claim 1 , wherein the memory device comprises: dual-ports configured to write and read concurrently; a connection of read address ports to a same read address request of the processors to allow the processors to read multiple rows and a same column of the image in parallel, the memory device configured to store YCbCr or RGB data for the pixels.
In the real-time stereo vision system, the memory buffer uses dual-port memory, enabling simultaneous reading and writing. The read address ports are connected to a shared read address request from the data alignment, data allocation, matching cost calculation, and disparity selection processors. This allows these processors to read multiple rows and the same column of the image in parallel. The memory stores pixel data in YCbCr or RGB format.
6. The device of claim 5 , wherein pixels of different rows are stored in separate block RAMs of the memory device to be able to access multiple pixels in the same column in parallel.
In the real-time stereo vision system, pixels from different rows are stored in separate block RAMs (memory blocks). This separation enables parallel access to multiple pixels in the same column, speeding up the disparity estimation process. This parallel access is enabled by the dual port block ram described in claim 5 that allows for the shared read address request.
7. The device of claim 6 , wherein the data in the block RAMs are overwritten by new rows of at least one of the left and right image after they are processed by at least one of the processors.
In the real-time stereo vision system with separated block RAMs as described in claim 6, the data in the block RAMs are overwritten with new rows of left or right image data (or both) after the data has been processed by the alignment, data allocation, metrics calculation, and disparity selection processors. This allows for continuous processing of the image stream.
8. The device of claim 1 , wherein the processor for rotating data is further configured to rotate either Y, Cb or Cr, either R, G or B to make disparity estimation in any of the selected pixel data channel, and to rotate and align either left image pixels or right image pixels.
In the real-time stereo vision system, the data alignment processor can rotate and align either the Y, Cb, or Cr components of YCbCr data, or the R, G, or B components of RGB data. This enables disparity estimation to be performed on any selected color channel. Also the data alignment processor can rotate and align either the left or right image pixels.
9. The device of claim 3 , wherein the processor for data allocation to create variable window sizes comprises, a flip-flop array configured to store and shift aligned outputs of the processor for rotating data; wires connected to the flip-flops array arranged to sample the pixels while pixels are flowing inside the flip-flops array; a plurality of first sampling schemes to provide the variable window sizes; a plurality of second sampling schemes to provide constant number of contributing pixels in the variable window sizes to provide constant computational load for the variable window sizes; and a plurality of multiplexers configured to select windows to a selected window size to be used in disparity estimation process of multiple pixels in a block according to the selected window size.
In the real-time stereo vision system with variable window sizes, the data allocation processor works as follows: A flip-flop array stores and shifts the aligned pixel outputs from the data alignment processor. Wires connected to the flip-flop array sample the pixels as they flow through. Multiple pre-defined sampling schemes provide different variable window sizes. Additional sampling schemes ensure a constant number of contributing pixels within these variable window sizes to maintain a consistent computational load. Multiplexers then select the appropriate window size for disparity estimation based on the selected scheme for multiple pixels in a block.
10. The device of claim 9 , wherein the selection of window size is determined depending on a variance of neighboring pixels for variable window sizes.
In the real-time stereo vision system using variable window sizes and pixel sampling, the selection of the window size is determined by the variance (amount of change) of neighboring pixels. Higher variance (more detail) may lead to smaller windows, while lower variance may lead to larger windows.
11. The device of claim 9 , wherein a same one of the selected window size is applied to multiple searched pixels in a block.
In the real-time stereo vision system using variable window sizes, the same selected window size is applied to multiple pixels within a block being searched for disparity. This maintains consistency within the block.
12. The device of claim 9 , wherein for every searched block of pixels, window size is dynamically re-determined.
In the real-time stereo vision system using variable window sizes, for every block of pixels being searched, the window size is dynamically re-evaluated and determined. This allows for adaptive behavior throughout the entire image.
13. The device of claim 4 , wherein the plurality of processing devices are configured for a computation of metrics and comprises, a plurality of census, Hamming, SAD and BW-SAD cost computation devices for concurrent and independent disparity search of multiple pixels in the two dimensional block; and a selection device configured for a configurability through selection either of SAD or BW-SAD cost computations.
In the real-time stereo vision system with parallel matching cost calculation, the multiple processing elements are configured for different matching cost computations. These include Census transform, Hamming distance, Sum of Absolute Differences (SAD), and Block-wise SAD (BW-SAD). These computations happen concurrently and independently for multiple pixels within the 2D block to perform disparity search. Additionally, there is a selection mechanism to choose between SAD and BW-SAD cost computations.
14. The device of claim 13 , wherein the plurality of processing devices are configured to perform: SAD and BW-SAD computations for sampled pixels in a searched block to reduce an overall computational complexity; interpolation of SAD and BW-SAD values of the sampled pixels in the block to compute and estimate the SAD and BW-SAD values of all the remaining pixels in the searched block for which SAD and BW-SAD are not computed; and Hamming computations for all the pixels in the searched block.
In the real-time stereo vision system with multiple matching cost computations, the processing elements perform: SAD and BW-SAD computations for sampled pixels within the search block to reduce overall computation; interpolation of the SAD and BW-SAD values of the sampled pixels to estimate the SAD and BW-SAD values for the remaining pixels in the block; and Hamming distance computations for all pixels in the search block.
15. The device of claim 13 , wherein the processor for disparity selection comprises: a multiplier to normalize a hamming cost using adaptive penalties; and an adder for adding multiplied hamming value with SAD result to compute hybrid cost.
In the real-time stereo vision system, the disparity selection processor includes a multiplier to normalize the Hamming cost using adaptive penalties. An adder then combines the scaled Hamming cost with the SAD result to compute a hybrid matching cost. This hybrid cost is used for disparity selection.
16. The device of claim 15 , wherein the adaptive penalties are in an order of two to simplify an implementation of multipliers with shifters.
In the real-time stereo vision system, the adaptive penalty values used to normalize the Hamming cost are powers of two (e.g., 1, 2, 4, 8...). This simplifies the multiplier implementation to simple bit-shift operations, reducing hardware complexity.
17. The device of claim 15 , wherein small penalty values are used for small window size, and big penalty values are used for big window size.
In the real-time stereo vision system, small penalty values for Hamming cost normalization are used with small window sizes, and large penalty values are used with large window sizes. This balances the influence of Hamming cost relative to SAD cost according to the window size.
18. The device of claim 2 , wherein the processor for iterative disparity refinement to refine the disparity values comprises: a flip-flop array to store and shift the disparity results; and a highest frequency selection device configured to determine most frequent disparity values to replace processed disparity values with the most frequent ones to establish the disparity results.
In the real-time stereo vision system, the disparity refinement processor to refine the disparity values includes: a flip-flop array to store and shift the disparity results; and a highest frequency selection device which determines the most frequent disparity values and replaces the processed disparity values with the most frequent disparity value to establish the disparity results.
19. The device of claim 18 , wherein the highest frequency selection hardware device is configured to determine the most frequent disparity values and refine the disparities using the color similarity of neighboring pixels.
In the real-time stereo vision system with disparity refinement, the highest frequency selection processor refines disparity estimates by selecting the most frequent disparity value and also refines these disparities further using the color similarity of neighboring pixels. This combines frequency analysis with color information for improved refinement.
20. The device of claim 18 , wherein multiple rows are refined in parallel using a plurality of highest frequency selection devices to determine the most frequent disparity values.
In the real-time stereo vision system with disparity refinement, multiple rows of disparity values are refined in parallel. This is achieved using multiple highest frequency selection processors, each operating on a different row.
21. The device of claim 18 , wherein the disparity results are iteratively refined.
In the real-time stereo vision system with disparity refinement, the disparity results are iteratively refined. This means the refinement process is repeated multiple times to progressively improve the disparity estimates.
22. The device of claim 18 , wherein the disparity results are iteratively refined by processing multiple consecutive columns using a plurality of highest frequency selection devices.
In the real-time stereo vision system with disparity refinement, the refinement process iteratively processes multiple consecutive columns of disparity values using multiple highest frequency selection processors.
23. The device of claim 18 , wherein the refined disparity values are written back to the flip flop array to iteratively use refined disparity values for further refinements.
In the real-time stereo vision system with disparity refinement using a flip-flop array, the refined disparity values are written back to the flip-flop array to iteratively use the refined disparity values for further refinements. This feedback loop enhances the accuracy of the final disparity map.
24. The device of claim 18 , wherein final shifted disparity values at an end of the flip flop array are used as the output of the disparity estimation device.
In the real-time stereo vision system with disparity refinement using a flip-flop array, the final shifted disparity values at the end of the flip-flop array serve as the final output of the disparity estimation device.
Unknown
September 5, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.