Methods, apparatuses and electronic devices for identifying an object category include: determining M key point neighborhood regions from corresponding object candidate boxes according to position information of M key points in a plurality of object candidate boxes of an image to be detected, where M is less than or equal to the total number of key points of N preset object categories, and M and N are positive integers; and determining category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in the image according to the M key point neighborhood regions.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A method for identifying an object category, performed by an electronic device, comprising: determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes, wherein M is less than or equal to the total number of key points of N preset object categories, and M and N are positive integers; and determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in an image, wherein if the object category is a background category, the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes comprises: determining regions of object candidate boxes corresponding to the position information of the M key points as the M key point neighborhood regions.
2. The method according to claim 1 , wherein the convolutional neural network model comprises K convolutional layers, a pooling layer, and an output layer, wherein K is a positive integer greater than or equal to 2; and the determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in the image comprises: outputting, from a (K−1) th convolutional layer, feature maps having one-to-one correspondence to the M key points; outputting, from a K th convolutional layer, a response map of a key point obtained after conversion of each of the feature maps; respectively mapping the M key point neighborhood regions to response maps of the corresponding key points to obtain M mapping regions; outputting, from the pooling layer, M pooling results corresponding to the M mapping regions; and obtaining, from the output layer, a first confidence of each preset object category based on the M pooling results; and determining, according to the first confidence of the each preset object category, the category information of the at least one object.
3. The method according to claim 2 , wherein the obtaining, from the output layer, a first confidence of each preset object category based on the M pooling results comprises: averaging pooling results corresponding to key points of a same preset object category to obtain a score for the each preset object category; and obtaining, according to the score for the each preset object category, the first confidence of the each preset object category from the output layer.
4. The method according to claim 1 , wherein prior to the determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in an image, the method further comprises: training the convolutional neural network model; and the training the convolutional neural network model comprises: obtaining a sample image comprising position information of key points, object box marking information, and category marking information; performing convolution on the sample image to obtain a convolution result; determining, according to the object box marking information and the category marking information, whether at least one of object box position information or category information in the convolution result meets a training completion condition; in response to at least one of the object box position information or category information in the convolution result meets a training completion condition, completing training of the convolutional neural network model; and in response to at least one of the object box position information or category information in the convolution result does not meet a training completion condition, adjusting, according to at least one of the object box position information or category information in the convolution result, parameters of the convolutional neural network model, and performing, according to the adjusted parameters of the convolutional neural network model, iterative training on the convolutional neural network model, until at least one of object box position information or category information after the iterative training meets the training completion condition.
5. The method according to claim 1 , wherein prior to the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes, the method further comprises: obtaining positions of the plurality of object candidate boxes; and obtaining, according to the positions of the plurality of object candidate boxes, the position information of the M key points in the plurality of object candidate boxes.
6. The method according to claim 5 , wherein the obtaining positions of the plurality of object candidate boxes comprises: obtaining the position information of the plurality of object candidate boxes by means of a first convolutional neural network, or obtaining the position information of the plurality of object candidate boxes by means of a Selective Search method, or obtaining the position information of the plurality of object candidate boxes by an Edge Box method.
7. The method according to claim 6 , wherein prior to the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes, the method further comprises: correcting the positions of the plurality of object candidate boxes using a second convolutional neural network; and obtaining position information of the plurality of corrected object candidate boxes.
8. The method according to claim 5 , wherein the obtaining the position information of the M key points in the plurality of object candidate boxes according to the positions of the plurality of object candidate boxes further comprises: obtaining, according to the positions of the plurality of object candidate boxes, a second confidence corresponding to each of the M key points, wherein the second confidence is data indicating the possibility that a key point exists in the candidate box; and the determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in an image further comprises: marking a pooling result of a response map corresponding to a key point having a second confidence lower than a set confidence threshold as zero.
9. The method according to claim 1 , wherein if the object category is a non-background category, the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes comprises: determining, according to size information of the corresponding object candidate boxes and a preset size multiple, M rectangular regions respectively centered on positions of the M key points as the M key point neighborhood regions.
10. An electronic device for identifying an object category, comprising: a processor; and a memory for storing instructions executable by the processor; wherein execution of the instructions by the processor causes the processor to perform: determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes, wherein M is less than or equal to the total number of key points of N preset object categories, and M and N are positive integers; and determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in an image, wherein if the object category is a background category, the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes comprises: determining regions of object candidate boxes corresponding to the position information of the M key points as the M key point neighborhood regions.
11. The electronic device according to claim 10 , wherein the convolutional neural network model comprises K convolutional layers, a pooling layer, and an output layer, wherein K is a positive integer greater than or equal to 2; and the determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in the image comprises: outputting, from a (K−1)th convolutional layer, feature maps having one-to-one correspondence to the M key points; outputting, from a Kth convolutional layer, a response map of a key point obtained after conversion of each of the feature maps; respectively mapping the M key point neighborhood regions to response maps of the corresponding key points to obtain M mapping regions; outputting, from the pooling layer, M pooling results corresponding to the M mapping regions; and obtaining, from the output layer, a first confidence of each preset object category based on the M pooling results; and determining, according to the first confidence of the each preset object category, the category information of the at least one object; wherein the obtaining, from the output layer, a first confidence of each preset object category based on the M pooling results comprises: averaging pooling results corresponding to key points of a same preset object category to obtain a score for the each preset object category; and obtaining, according to the score for the each preset object category, the first confidence of the each preset object category from the output layer.
12. The electronic device according to claim 10 , wherein prior to the determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in an image, the method further comprises: training the convolutional neural network model; and the training the convolutional neural network model comprises: obtaining a sample image comprising position information of key points, object box marking information, and category marking information; performing convolution on the sample image to obtain a convolution result; determining, according to the object box marking information and the category marking information, whether at least one of object box position information or category information in the convolution result meets a training completion condition; in response to at least one of the object box position information or category information in the convolution result meets a training completion condition, completing training of the convolutional neural network model; and in response to at least one of the object box position information or category information in the convolution result does not meet a training completion condition, adjusting, according to at least one of the object box position information or category information in the convolution result, parameters of the convolutional neural network model, and performing, according to the adjusted parameters of the convolutional neural network model, iterative training on the convolutional neural network model, until at least one of object box position information or category information after the iterative training meets the training completion condition.
13. The electronic device according to claim 10 , wherein prior to the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes, the method further comprises: obtaining positions of the plurality of object candidate boxes; and obtaining, according to the positions of the plurality of object candidate boxes, the position information of the M key points in the plurality of object candidate boxes.
14. The electronic device according to claim 13 , wherein the obtaining positions of the plurality of object candidate boxes comprises: obtaining the position information of the plurality of object candidate boxes by means of a first convolutional neural network, or obtaining the position information of the plurality of object candidate boxes by means of a Selective Search method, or obtaining the position information of the plurality of object candidate boxes by an Edge Box method.
15. The electronic device according to claim 14 , wherein before determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes, the method further comprises: correcting the positions of the plurality of object candidate boxes using a second convolutional neural network; and obtaining position information of the plurality of corrected object candidate boxes.
16. The electronic device according to claim 13 , wherein the obtaining the position information of the M key points in the plurality of object candidate boxes according to the positions of the plurality of object candidate boxes further comprises: obtaining, according to the positions of the plurality of object candidate boxes, a second confidence corresponding to each of the M key points, wherein the second confidence is data indicating the possibility that a key point exists in the candidate box; and the determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in an image further comprises: marking a pooling result of a response map corresponding to a key point having a second confidence lower than a set confidence threshold as zero.
17. The electronic device according to claim 10 , wherein if the object category is a non-background category, the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes comprises: determining, according to size information of the corresponding object candidate boxes and a preset size multiple, M rectangular regions respectively centered on positions of the M key points as the M key point neighborhood regions.
18. A non-transitory computer readable storage medium with a computer program stored thereon, wherein the computer program comprises computer-readable instructions, wherein execution of the instructions by the processor causes the processor to perform: determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes, wherein M is less than or equal to the total number of key points of N preset object categories, and M and N are positive integers; and determining, according to the M key point neighborhood regions, category information of at least one object in the image to be detected using a convolutional neural network model used for identifying an object category in an image, wherein if the object category is a background category, the determining, according to position information of M key points in a plurality of object candidate boxes of an image to be detected, M key point neighborhood regions from corresponding object candidate boxes comprises: determining regions of object candidate boxes corresponding to the position information of the M key points as the M key point neighborhood regions.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
May 27, 2019
April 13, 2021
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.