Assisting Program Code Development by Processing Multimodal Data of a Meeting

1. A method for assisting program code development, the method comprising: obtaining, with a processor circuitry, multimodal data of a meeting discussing a program code development, wherein the obtaining the multimodal data of the meeting comprises: recording the meeting to obtain audio data and video streams of the meeting in real time from multi-media devices, and retrieving the multimodal data of the meeting from a cloud storage maintaining records and contents of past meetings; extracting, with the processor circuitry, a plurality of topics and a plurality of concepts from the multimodal data of the meeting, wherein a concept represents an object or an action relevant to a topic discussed during the meeting, wherein a topic is associated with a set of concepts and a concept is associated with a set of topics, and wherein the extracting the plurality of topics and the plurality of concepts comprises: converting, using a speech-to-text tool, the audio data of the meeting into textual data of the meeting, and performing natural language processing (NLP) on the textual data of the meeting to extract the plurality of topics and the plurality of concepts from the textual data of the meeting; identifying, with the processor circuitry, a plurality of meeting segments for the plurality of concepts, wherein a meeting segment represents a segment of the multimodal data of the meeting over a time period during the meeting and a concept occurring in a set of meeting segments, wherein a concept is associated with the set of meeting segments, and wherein a meeting segment being is associated with the set of concepts; determining, with the processor circuitry, a coding intent from program codes for the program code development; aligning, with the processor circuitry, the coding intent to a set of topics; for a topic of the set of topics, identifying, with the processor circuitry using a filtered topic-to-concept bipartite graph, the set of concepts associated with the topic aligned with the coding intent, wherein the filtered topic-to-concept bipartite graph is generated by: mappings using association information between the plurality of topics and the plurality of concepts, and processing the mappings between the plurality of topics and the plurality of concepts to generate the filtered topic-to-concept bipartite graph; for a concept of the set of concepts, identifying, with the processor circuitry using a concept-to-meeting-segment bipartite graph, the set of meeting segments associated with the concept, wherein the concept-to-meeting-segment bipartite graph is generated by: determining an association weight between the concept and the meeting segment associated with the concept, and creating a concept-to-meeting-segment weighted association which represents the concept-to-meeting-segment bipartite graph between the set of concepts and the set of meeting segments; for a meeting segment of the set of meeting segments, determining, with the processor circuitry, an alignment metric of the meeting segment based on an alignment metric between the coding intent and the topic, an alignment metric between the topic and the concept, and the association weight between the concept and the meeting segment associated with the concept; and outputting, with the processor circuitry via a graphical user interface (GUI), one or more meeting segments for the coding intent based on alignment metrics of the one or more meeting segments for the coding intent to a developer who is developing the program codes corresponding to the coding intent in order to facilitate the program code development.

2. The method of claim 1, wherein the method further comprises: determining the association weight between the concept and the meeting segment associated with the concept based on a time duration of the meeting segment and a number of times that the concept occurs in the meeting segment.

3. The method of claim 1, wherein the method further comprises: determining a relevance probability between the topic and the concept based on the multimodal data of the meeting; and using the relevance probability as the alignment metric between the topic and the concept.

4. The method of claim 1, wherein the method further comprises: determining a similarity between the coding intent and the topic; and using the similarity as the alignment metric between the coding intent and the topic.

5. The method of claim 1, wherein the identifying the plurality of meeting segments for the plurality of concepts comprises: identifying a first time point and a second time point when a concept occurs in the meeting, wherein the second time point is later than the first time point, and wherein a time difference between the first time point and the second time point is less than a predetermined time difference threshold; identifying a third time point when the concept occurs in the meeting, wherein the third time point is later than the second time point; and in response to a time difference between the third time point and the second time point being equal to or greater than the predetermined time difference threshold, determining a meeting segment between the first time point and the second time point to be associated with the concept.

6. The method of claim 5, wherein the identifying the plurality of meeting segments for the plurality of concepts comprises: in response to the time difference between the third time point and the second time point being less than the predetermined time difference threshold, determining a meeting segment between the first time point and the third time point to be associated with the concept.

7. The method of claim 1, wherein performing the natural language processing (NLP) on the textual data of the meeting comprises: using any one of NLP techniques including latent dirichlet allocation (LDA) and word-embedding-distance-based clustering, to extract the plurality of topics and the plurality of concepts from the textual data of the meeting.

8. The method of claim 1, wherein the method further comprises: generating coding comments for the program codes based on the one or more meeting segments.

9. The method of claim 1, wherein the outputting the one or more meeting segments for the coding intent comprises: displaying the one or more meeting segments whose alignment metrics are greater than an alignment threshold.

10. The method of claim 1, wherein the outputting the one or more meeting segments for the coding intent comprises: displaying the one or more meeting segments in descending order of the alignment metrics of the one or more meeting segments.

11. The method of claim 1, wherein the method further comprises: marking the one or more meeting segments as having been outputted for the coding intent; and in response to a request for reviewing a progress of the program code development, outputting unmarked meeting segments to assist the review.

12. A system for assisting program code development, the system comprising: a memory having stored thereon executable instructions; and a processor circuitry in communication with the memory, the processor circuitry, when executing the executable instructions, configured to: obtain multimodal data of a meeting discussing a program code development, wherein the obtaining the multimodal data of the meeting comprises: recording the meeting to obtain audio data and video streams of the meeting in real time from multi-media devices, and retrieving the multimodal data of the meeting from a cloud storage maintaining records and contents of past meetings; extract a plurality of topics and a plurality of concepts from the multimodal data of the meeting, wherein a concept represents an object or an action relevant to a topic discussed during the meeting, wherein a topic is associated with a set of concepts and a concept is associated with a set of topics, and wherein the extracting the plurality of topics and the plurality of concepts comprises: converting, using a speech-to-text tool, the audio data of the meeting into textual data of the meeting, and performing natural language processing (NLP) on the textual data of the meeting to extract the plurality of topics and the plurality of concepts from the textual data of the meeting; identify a plurality of meeting segments for the plurality of concepts, wherein a meeting segment represents a segment of the multimodal data of the meeting over a time period during the meeting and a concept occurring in a set of meeting segments, wherein a concept is associated with the set of meeting segments, and wherein a meeting segment is associated with the set of concepts; determine a coding intent from program codes for the program code development; align the coding intent to a set of topics; for a topic of the set of topics, identify, using a filtered topic-to-concept bipartite graph, the set of concepts associated with the topic aligned with the coding intent, wherein the filtered topic-to-concept bipartite graph is generated by: mappings using association information between the plurality of topics and the plurality of concepts, and processing the mappings between the plurality of topics and the plurality of concepts to generate the filtered topic-to-concept bipartite graph; for a concept of the set of concepts, identify, using a concept-to-meeting-segment bipartite graph, the set of meeting segments associated with the concept, wherein the concept-to-meeting-segment bipartite graph is generated by: determining an association weight between the concept and the meeting segment associated with the concept, and creating a concept-to-meeting-segment weighted association which represents the concept-to-meeting-segment bipartite graph between the set of concepts and the set of meeting segments; for a meeting segment of the set of meeting segments, determine an alignment metric of the meeting segment based on an alignment metric between the coding intent and the topic, an alignment metric between the topic and the concept, and the association weight between the concept and the meeting segment associated with the concept; and output, via a graphical user interface (GUI), one or more meeting segments for the coding intent based on alignment metrics of the one or more meeting segments for the coding intent to a developer who is developing the program codes corresponding to the coding intent in order to facilitate the program code development.

13. The system of claim 12, wherein the processor circuitry is further configured to: determine the association weight between the concept and the meeting segment based on a time duration of the meeting segment and a number of times that the concept occurs in the meeting segment.

14. The system of claim 12, wherein the processor circuitry is further configured to: determine a relevance probability between the topic and the concept based on the multimodal data of the meeting; and use the relevance probability as the alignment metric between the topic and the concept.

15. The system of claim 12, wherein the processor circuitry is further configured to: determine a similarity between the coding intent and the topic; and use the similarity as the alignment metric between the coding intent and the topic.

16. The system of claim 12, wherein the processor circuitry is further configured to: identify a first time point and a second time point when a concept occurs in the meeting, wherein the second time point is later than the first time point, and wherein a time difference between the first time point and the second time point is less than a predetermined time difference threshold; identify a third time point when the concept occurs in the meeting, wherein the third time point is later than the second time point; and in response to a time difference between the third time point and the second time point being equal to or greater than the predetermined time difference threshold, determine a meeting segment between the first time point and the second time point to be associated with the concept.

17. The system of claim 16, wherein the processor circuitry is further configured to: in response to the time difference between the third time point and the second time point being less than the predetermined time difference threshold, determine a meeting segment between the first time point and the third time point to be associated with the concept.

18. The system of claim 12, wherein the processor circuitry is further configured to: generate coding comments for the program codes based on the one or more meeting segments.

19. The system of claim 12, wherein the processor circuitry is further configured to: mark the one or more meeting segments as having been outputted for the coding intent; and in response to a request for reviewing a progress of the program code development, output unmarked meeting segments to assist the review.

20. A computer program product for assisting program code development, the computer program product comprising: a non-transitory machine-readable medium; and instructions stored on the non-transitory machine-readable medium, the instructions configured to, when executed by a processor circuitry, cause the processor circuitry to: obtain multimodal data of a meeting discussing a program code development, wherein the obtaining the multimodal data of the meeting comprises: recording the meeting to obtain audio data and video streams of the meeting in real time from multi-media devices, and retrieving the multimodal data of the meeting from a cloud storage maintaining records and contents of past meetings; extract a plurality of topics and a plurality of concepts from the multimodal data of the meeting, a concept representing an object or an action relevant to a topic discussed during the meeting, and a topic being associated with a set of concepts and a concept being associated with a set of topics, wherein the extracting the plurality of topics and the plurality of concepts comprises: converting, using a speech-to-text tool, the audio data of the meeting into textual data of the meeting, and performing natural language processing (NLP) on the textual data of the meeting to extract the plurality of topics and the plurality of concepts from the textual data of the meeting; identify a plurality of meeting segments for the plurality of concepts, wherein a meeting segment represents a segment of the multimodal data of the meeting over a time period during the meeting and a concept occurring in a set of meeting segments, wherein a concept is associated with the set of meeting segments, and wherein a meeting segment is associated with a set of concepts; determine a coding intent from program codes for the program code development; align the coding intent to a set of topics; for a topic of the set of topics, identify, using a filtered topic-to-concept bipartite graph, the set of concepts associated with the topic aligned with the coding intent, wherein the filtered topic-to-concept bipartite graph is generated by: mappings using association information between the plurality of topics and the plurality of concepts, and processing the mappings between the plurality of topics and the plurality of concepts to generate the filtered topic-to-concept bipartite graph; for a concept of the set of concepts, identify, using a concept-to-meeting-segment bipartite graph, the set of meeting segments associated with the concept, wherein the concept-to-meeting-segment bipartite graph is generated by: determining an association weight between the concept and the meeting segment associated with the concept, and creating a concept-to-meeting-segment weighted association which represents the concept-to-meeting-segment bipartite graph between the set of concepts and the set of meeting segments; for a meeting segment of the set of meeting segments, determine an alignment metric of the meeting segment based on an alignment metric between the coding intent and the topic, an alignment metric between the topic and the concept, and the association weight between the concept and the meeting segment associated with the concept; and output, via a graphical user interface (GUI), one or more meeting segments for the coding intent based on alignment metrics of the one or more meeting segments for the coding intent to a developer who is developing the program codes corresponding to the coding intent in order to facilitate the program code development.