Configuring 360-degree video within a virtual conferencing system

This patent application is a continuation of U.S. patent application Ser. No. 17/955,017, filed Sep. 28, 2022, which application claims the benefit of U.S. Provisional Patent Application No. 63/261,870, filed Sep. 30, 2021, entitled “CONFIGURING 360-DEGREE VIDEO WITHIN A VIRTUAL CONFERENCING SYSTEM”, which are incorporated by reference herein in their entirety.

The present disclosure relates generally to virtual conferencing systems, including configuring 360-degree video within a virtual conferencing system.

A virtual conferencing system provides for the reception and transmission of audio and video data between devices, for communication between device users in real-time.

A virtual conferencing system provides for the reception and transmission of audio and video data between devices, for communication between device users in real-time. The virtual conferencing system as described herein allows a user to design or select a virtual space with one or more rooms for real-time communication. A given room may include different elements, such as participant video elements (e.g., boxes, windows) for assigning to the video feed of a participant during virtual conferencing, and 360-degree video elements for assigning to 360-degree video sources.

The disclosed embodiments provide for a user, in designing a room for virtual conferencing, to position and/or size participant video elements (e.g., based on objects depicted in the room) and 360-degree video elements. Each of the participant video elements corresponds to a potential placeholder for a participant. On the other hand, each of the 360-degree video elements corresponds to a specified video source (e.g., live feed, pre-recorded video). During virtual conferencing within the room, the video feeds of participants are assigned to the participant video elements, and the video sources are assigned to the 360-degree video elements.

By virtue of allowing a user to customize participant video elements that are assignable to the video feeds of participants and to customize 360-degree video elements that are assignable to video sources, it is possible for the user to create different types of environments (e.g., seminars, classrooms, hangouts, and the like) for virtual conferencing. In addition, the design of the room may be more engaging for participants of the virtual conference. Without allowing the user (e.g., administrator) to configure a room in this manner, the user may otherwise be limited in configuring participant video feeds and/or 360-degree video sources for virtual conferencing. The virtual conferencing system facilitates creation of virtual conferencing environments, thereby saving time for the user, and reducing computational resources/processing power for the virtual conferencing system.

is a block diagram showing an example virtual conferencing systemfor exchanging data over a network. The virtual conferencing systemincludes multiple instances of a client device, each of which hosts a number of applications, including a virtual conference clientand other application(s). Each virtual conference clientis communicatively coupled to other instances of the virtual conference client(e.g., hosted on respective other client devices), a virtual conference server systemand third-party serversvia a network(e.g., the Internet). A virtual conference clientcan also communicate with locally-hosted applicationsusing Applications Program Interfaces (APIs).

The virtual conferencing systemprovides for the reception and transmission of audio, video, image, text and/or other signals by user devices (e.g., at different locations), for communication between users in real-time. In some cases, two users may utilize virtual conferencing to communicate with each other in one-to-one communication at their respective devices. In other cases, multiway virtual conferencing may be utilized by more than two users to participate in a real-time, group conversation. Thus, multiple client devicesmay participate in virtual conferencing, for example, with the client devicesparticipating in a group conversation in which audio-video content streams and/or message content (e.g., text, images) are transmitted between the participant devices.

A virtual conference clientis able to communicate and exchange data with other virtual conference clientsand with the virtual conference server systemvia the network. The data exchanged between virtual conference clients, and between a virtual conference clientand the virtual conference server system, includes functions (e.g., commands to invoke functions) as well as payload data (e.g., video, audio, other multimedia data, text).

The virtual conference server systemprovides server-side functionality via the networkto a particular virtual conference client. For example, with respect to transmitting audio and/or video streams, the virtual conference client(e.g., installed on a first client device) may facilitate in transmitting streaming content to the virtual conference server systemfor subsequent receipt by other participant devices (e.g., one or more second client devices) running respective instances of the virtual conference client.

The streaming content can correspond to audio and/or video content captured by sensors (e.g., microphones, video cameras) on the client devices, for example, corresponding to real-time video and/or audio capture of the users (e.g., faces) and/or other sights and sounds captured by the respective device. The streaming content may be supplemented with other audio/visual data (e.g., animations, overlays, emoticons and the like) and/or message content (e.g., text, stickers, emojis, other image/video data), for example, in conjunction with extension applications and/or widgets associated with the virtual conference client.

While certain functions of the virtual conferencing systemare described herein as being performed by either a virtual conference clientor by the virtual conference server system, the location of certain functionality either within the virtual conference clientor the virtual conference server systemmay be a design choice. For example, it may be technically preferable to initially deploy certain technology and functionality within the virtual conference server systembut to later migrate this technology and functionality to the virtual conference clientwhere a client devicehas sufficient processing capacity.

The virtual conference server systemsupports various services and operations that are provided to the virtual conference client. Such operations include transmitting data to, receiving data from, and processing data generated by the virtual conference client. This data may include the above-mentioned streaming content and/or message content, client device information, and social network information, as examples. Data exchanges within the virtual conferencing systemare invoked and controlled through functions available via user interfaces (UIs) of the virtual conference client.

Turning now specifically to the virtual conference server system, an Application Program Interface (API) serveris coupled to, and provides a programmatic interface to, application servers. The application serversare communicatively coupled to a database server, which facilitates access to a databasethat stores data associated with virtual conference content processed by the application servers. Similarly, a web serveris coupled to the application servers, and provides web-based interfaces to the application servers. To this end, the web serverprocesses incoming network requests over the Hypertext Transfer Protocol (HTTP) and several other related protocols.

The Application Program Interface (API) serverreceives and transmits virtual conference data (e.g., commands, audio/video payloads) between the client deviceand the application servers. Specifically, the Application Program Interface (API) serverprovides a set of interfaces (e.g., routines and protocols) that can be called or queried by the virtual conference clientin order to invoke functionality of the application servers. The Application Program Interface (API) serverexposes various functions supported by the application servers, including account registration, login functionality, the streaming of audio and/or video content, and/or the sending and retrieval of message content, via the application servers, from a particular virtual conference clientto another virtual conference client, the retrieval of a list of contacts of a user of a client device, the addition and deletion of users (e.g., contacts) to a user graph (e.g., a social graph), and opening an application event (e.g., relating to the virtual conference client).

The application servershost a number of server applications and subsystems, including for example a virtual conference serverand a social network server. The virtual conference serverimplements a number of virtual conference processing technologies and functions, particularly related to the aggregation and other processing of content (e.g., streaming content) included in audio-video feeds received from multiple instances of the virtual conference client. Other processor and memory intensive processing of data may also be performed server-side by the virtual conference server, in view of the hardware requirements for such processing.

The social network serversupports various social networking functions and services and makes these functions and services available to the virtual conference server. To this end, the social network servermaintains and accesses a user graph(as shown in) within the database. Examples of functions and services supported by the social network serverinclude the identification of other users of the virtual conferencing systemwith which a particular user has relationships (e.g., contacts such as friends, colleagues, teachers, students, and the like).

In one or more embodiments, a user interacting via the virtual conference clientrunning on a first client devicemay select and invite participant(s) to a virtual conference. For example, the participants may be selected from contacts maintained by the social network server. In another example, the participants may be selected from contacts included within a contact address book stored in association with the first client device(e.g., in local memory or in a cloud-based user account). In another example, the participants may be selected by the user manually entering email addresses and/or phone numbers of the participants.

The user at the first client devicemay initiate the virtual conference by selecting an appropriate user interface element provided by the virtual conference client, thereby prompting the invited participants, at their respective devices (e.g., one or more second client devices), to accept or decline participation in the virtual conference. When the participant(s) have accepted the invitation (e.g., via the prompt), the virtual conference server systemmay perform an initialization procedure in which session information is published between the participant client devices, including the user who provided the invite. Each of the participant client devicesmay provide respective session information to the virtual conference server system, which in turn publishes the session information to the other participant client devices. The session information for each client devicemay include content stream(s) and/or message content that is made available by the client device, together with respective identifiers for the content stream(s) and/or message content.

As described below with respect to, the virtual conference may correspond to a virtual space which includes one or more rooms (e.g., virtual rooms). The virtual space and its corresponding rooms may have been created at least in part by the inviting user and/or by other users. In this manner, an end user may act as an administrator, who creates their own virtual spaces with rooms, and/or designs a virtual space based on preset available rooms.

is a block diagram illustrating further details regarding the virtual conferencing system, according to some examples. Specifically, the virtual conferencing systemis shown to comprise the virtual conference clientand the application servers. The virtual conferencing systemembodies a number of subsystems, which are supported on the client-side by the virtual conference clientand on the server-side by the application servers. These subsystems include, for example, a virtual space creation systemwhich implements a virtual space design interface, and a virtual space participation systemwhich implements a virtual space navigation interface.

The virtual space creation systemprovides for a user to design one or more virtual space(s) in which participants may engage in virtual conferencing. In one or more embodiments, a virtual space corresponds to an environment with one or more rooms configured to accommodate virtual conferencing.

The virtual space may be created and/or selected (e.g., from among a set of predefined virtual spaces with rooms) by an end user who wishes to invite other users for virtual conferencing. In addition, the individual rooms of a virtual space may be newly-created and/or selected (e.g., from among a set of predefined rooms) by the end user. In one or more embodiments, the virtual space creation systemincludes a virtual space design interface, which is usable by the end user to design a virtual space, including creating and/or selecting rooms for including in the virtual space.

As discussed below with respect to, the virtual space design interfaceenables an end user (e.g., an administrator) to select and/or position multiple elements for including in a room. Examples of elements include, but are not limited to, participant video elements (e.g., for displaying the respective video feeds of participants), chat interfaces (e.g., for participants to provide text-based messages, stickers and/or reactions within a room), breakout buttons (e.g., for shuffling from a first room to one or more second rooms), and/or other user-definable elements for performing certain actions (e.g., speaking into a virtual microphone, querying an administrator via a button, and the like).

The virtual space participation systemis configured to perform virtual conferencing among participants within a virtual space. The participants may include the end user (e.g., administrator) who created the virtual space, as well as those users who were invited to participate in virtual conferencing with respect to the virtual space created/selected by the end user. The virtual space participation systemincludes a virtual space navigation interface(e.g., discussed below with respect to) that allows participants to navigate between the rooms of a virtual space, and to participate in virtual conferencing with respect to the rooms.

In one or more embodiments, the virtual space creation systemand the virtual space participation systemprovide for an end user (e.g., an administrator) to create different types of environments (e.g., virtual spaces with rooms) for virtual conferencing, and for participants to engage in virtual conferencing within such environments. Examples of such virtual conferencing include, but are not limited to: business meetings, seminars, presentations, classroom lectures, teacher office hours, concerts, reunions, virtual dinners, escape rooms, and the like.

is a schematic diagram illustrating data structures, which may be stored in the databaseof the virtual conference server system, according to certain examples. While the content of the databaseis shown to comprise a number of tables, it will be appreciated that the data could be stored in other types of data structures (e.g., as an object-oriented database).

The databaseincludes profile data, a user graphand a user tablerelating to the users (participants) of the virtual conferencing system. The user tablestores user data, and is linked (e.g., referentially) to the user graphand the profile data. Each user of the virtual conferencing systemis associated with a unique identifier (email address, telephone number, social network identifier, etc.).

The user graphstores (e.g., in conjunction with the social network server) information regarding relationships and associations between users. Such relationships may be social, professional (e.g., work at a common corporation or organization) interested-based or activity-based, merely for example. As noted above, the user graphmay be maintained and accessed at least in part by the social network server.

The profile datastores multiple types of profile data about a particular user. The profile datamay be selectively used and presented to other users of the virtual conferencing system, based on privacy settings specified by a particular user. The profile dataincludes, for example, a user name, telephone number, email address, and/or settings (e.g., notification and privacy settings), as well as a user-selected avatar representation.

The databasefurther includes a virtual spaces table. As noted above, a virtual space corresponds to an environment with one or more rooms configured to accommodate virtual conferencing. A virtual space may be newly-created by a user, or may be included within one or more sets of public virtual spaces made available (e.g., by other users, system administrators, and the like) for virtual conferencing. The virtual spaces tablestores information representing the one or more sets of public virtual spaces, as well as any private virtual space(s) created by a user (e.g., in a case where the particular user did not make such virtual space(s) public).

In one or more embodiments, the virtual spaces tablestores associations between its virtual spaces and users (e.g., within the user table) who selected those virtual spaces. In this manner, it is possible for a particular user to have one or more virtual spaces associated therewith. Moreover, the databaseincludes a rooms tablewhich may be associated with the virtual spaces within the virtual spaces table. As noted above, a room may be newly-created by a user, or may be included within one or more sets (e.g., galleries) of public rooms made available for user selection. The rooms tablestores information representing the one or more sets of rooms, as well as any private room(s) created by the user (e.g., in a case where the particular user did not make such room(s) public). The stored information is usable by the virtual conferencing systemto create the corresponding rooms for use in a virtual space. In one or more embodiments, the stored information may further include recordings (e.g., audio and/or video recordings) of a particular virtual conference, for subsequent playback by corresponding participants.

illustrates a virtual space design interfacewith interface elements for designing a virtual space, in accordance with some example embodiments. Designing the virtual space may include creation and/or selection of rooms for including in the virtual space. The virtual space design interfaceincludes a menu interface, a room elements interface, an element properties interface, a controls interface, a room list interface, a room canvas interface, and an administrator name. It is noted that elements-correspond to an example of interface elements for the virtual space design interface, and that additional, fewer and/or different interface elements may be used.

An administrator (e.g., corresponding to administrator name) may use the various interface elements to design a virtual space. In one or more embodiments, the menu interfaceincludes user-selectable categories (e.g., menu headings) relating to a virtual space (e.g., “workspace”), rooms within the virtual space, and/or elements within a room. For example, the workspace category is user-selectable for presenting options (e.g., via a drop-down list) to manage settings for the virtual space, manage invites for the virtual space, manage versions of a virtual space, publish the virtual space (e.g., for future use by users), manage virtual space publications, and/or to start/manage recordings (e.g., audio and/or video recordings) with respect to the virtual space.

The room category of the menu interfaceis user-selectable for presenting options (e.g., via a drop-down list) to manage settings for a room within the virtual space, set a room background, set an order for the rooms listed in the room list interface, create a new room, import a room from a set of available rooms, remove a room, publish a room, manage room publications, and/or to start/manage recordings with respect to a room.

In addition, the element category is user-selectable for presenting options (e.g., via a drop-down list) to insert elements into a room, insert shapes into a room, foreground/background elements, arrange/position elements, and/or group elements. Examples of elements include, but are not limited to: an action button, analog clock, audience question board, backpack item, breakout button, chat, closed caption display, closed caption input, countdown, clock, digital clock, doorbell, double-sided image, feedback, image, multiuser video chat, music, participant audio mixer, participant count, participant video element (e.g., single or multiple), picture strip, poll, random source, room preview, scheduled time, sound effect, stopwatch, take picture, text, timer, user search, video, waiting list, web media, website. Examples of shapes include, but are not limited to, a circle, rectangle and triangle.

The users category of the menu interfaceis user-selectable for presenting options (e.g., via a drop-down list) to manage users/participants of the virtual space (e.g., adding tags for participants, so as to distinguish between roles such as an administrator or an attendee/participant). In addition, the edit category is user-selectable for performing edit operations (e.g., undo, redo, cut, copy, paste), and the help category is user-selectable for performing help operations (e.g., getting started, discord, live help, submitting feedback).

In one or more embodiments, the room elements interfaceincludes user-selectable icons for inserting elements (e.g., corresponding to a subset of those available via the above-mentioned element category) into a current room. For example, the elements may be added and/or positioned within the current room by selecting the element and dragging the selected element onto the room canvas interface, which represents the layout of the current room.

In one or more embodiments, the room elements interfaceinclude icons including but not limited to: a text icon for adding text to a room; a participant video icon for adding a single participant video element (e.g., an interface element which is selectable by a single participant for displaying that participant's video feed) to a room; a multiuser video icon for adding a multiple participant video element (e.g., an interface element which is selectable by one or more participants for displaying the video feeds for those participants) to a room; a 360-degree video icon for adding a 360-degree video element (e.g., an interface element for displaying a video feed from a 360-degree video camera); a chat icon for adding a chat interface (e.g., for messaging using text, stickers, emojis, etc.) to a room; a video playback icon for adding a video playback element (e.g., screen) to a room for playback of a selected video; a background icon for selecting a background color/gradient, image or video to a room; an action icon for adding an action element (e.g., button) to a room for performing a user-defined action (e.g., speaking into a virtual microphone, querying an administrator via a button, and the like); and/or a breakout button for adding a breakout element (e.g., button) for shuffling selected participants between the current room and one or more other rooms.

In one or more embodiments, the element properties interfaceinclude various fields for setting configuration properties for above-described room elements. For example, with respect to elements in general (e.g., text, single participant video element, multi participant video element, 360-degree video element, chat interface, video element, background image, action element, breakout button), the element properties interfaceincludes fields for setting the element title/name, opacity, gradient, style, layout, borders/corners, shadows, interaction (e.g., to what extent participant(s) may delete, modify, resize the element), filtering, full screen status, conditions, accessibility and actions for the element.

For a participant video element, the element properties interfaceincludes further fields for setting the manner in which users are placed into the participant video element during virtual conferencing (e.g., automatically, manually by the participant and/or the administrator end user). For a 360-degree video element, the element properties interfaceincludes further fields for setting the manner in which 360-degree video is presented in the 360-degree video element during virtual conferencing (e.g., source URL, whether the 360-degree video element is controllable by participants). In addition, for the chat interface, the element properties interfaceincludes further properties for setting who (e.g., administrator and/or participants) can provide chat input, and/or which types of input (e.g., text, stickers, emojis, etc.) are available. For the action element, the element properties interfaceincludes further properties for setting what type of action is to be performed in response to user selection of the action element (e.g., button). Moreover, for the breakout element, the element properties interfaceincludes further properties for selecting participants and/or breakout rooms.

In one or more embodiments, the element properties interfacefurther includes fields for setting configuration properties for the room canvas interface. For example, the element properties interfaceincludes fields for selecting a number of fake participants (e.g., simulated video feeds) in order to visualize multiple users, selecting music (e.g., background music), and/or selecting reaction buttons for participants to indicate real-time reactions with respect to virtual conferencing within a room.

In one or more embodiments, the controls interfaceincludes user-selectable icons corresponding to controls (e.g., administrative controls) for the virtual space. For example, the controls interfaceinclude icons including but not limited to: a director mode icon for toggling between a director mode for designing a room and a user mode for viewing the room within the virtual space design interface(e.g., with the director mode including the room elements interfaceand the element properties interfacewhile the user mode does not); a view icon for viewing the room within the virtual space navigation interface; a share screen icon (e.g., for collaborative design with other user(s) such as co-administrators); a microphone icon for enabling or disabling the microphone; a help icon (e.g., getting started, discord, live help, submitting feedback); an invite icon (e.g., for displaying an invite link for sending to participants to visit the virtual space); a settings icon (e.g., for selecting the end user's video and audio devices for the virtual conferencing, and for selecting a user avatar); and/or an exit icon for exiting the virtual space design interface.

In one or more embodiments, the room list interfacedisplays the list of rooms for the virtual space. Each listed room is user selectable to switch to edit (e.g., in director mode) and/or view (e.g., in user mode) the selected room. As noted above, the list of rooms may be modified (e.g., by adding, importing and/or removing rooms) via the options within the room category of the menu interface.

illustrates a virtual space navigation interfacewith interface elements to navigate between the rooms of a virtual space and to participate in virtual conferencing with respect to the rooms, in accordance with some example embodiments. The virtual space navigation interfaceincludes a controls interface, a room list interface, a current room interface, a participant video elementand a participant video element. It is noted that elements-correspond to an example of interface elements for the virtual space navigation interface, and that additional, fewer and/or different interface elements may be used.

In one or more embodiments, the controls interfaceincludes user-selectable icons corresponding to controls (e.g., administrative controls) for the virtual space. For example, the controls interfaceinclude icons including but not limited to: an edit icon for redirecting to the virtual space design interfaceto edit the current room; a volume icon for adjusting a volume level for the current room; a share screen icon (e.g., for allowing others to view the room without necessarily joining the room); a microphone icon for muting and unmuting the microphone; a help icon (e.g., getting started, discord, live help, submitting feedback); an invite icon (e.g., for displaying an invite link for participants to visit the virtual space); a settings icon (e.g., for selecting the end user's video and audio devices for the virtual conferencing, and for selecting a user avatar); and/or an exit icon for exiting the virtual space design interface.

In one or more embodiments, the room list interfacedisplays the list of rooms for the virtual space. Each listed room is user selectable to switch to the selected room (e.g., for virtual conferencing). The selected room is presented as a current room within the current room interface. In this manner, a participant may navigate among the multiple rooms available within the virtual space. Alternatively or in addition, navigation between rooms is possible via a virtual space map interface (not shown) which depicts a map view of the virtual space (e.g., a floor plan) and its corresponding rooms, with each room being user selectable to navigate thereto. Alternatively or in addition, navigation between rooms is further possible by positioning a navigation button (not shown) within a room, where user selection of the button results in navigating to another room (e.g., a predefined room). As noted above, the virtual space design interfaceallows for the design of a virtual space and its corresponding rooms. As such, navigation between rooms is based at least in part on the design of the virtual space (e.g., a virtual space may include one or more of the above-mentioned room list interface, the virtual space map/floor plan interface and/or the navigation button).

With respect to the current room interface, each participant is represented as a respective participant video element. As noted above, a participant video element corresponds to an interface element (e.g., a box) which is selectable by a single participant for displaying that participant's video feed. The example ofincludes a first participant associated with the participant video elementand a second participant associated with the participant video element. In one or more embodiments, with respect to the perspective of the first participant, the participant video elementshowing the feed of the second participant may include participant button(s). For example, the participant button(s)are selectable by the first participant so as to perform a predefined action (e.g., initiate a side conversation, designate the second participant to follow the first participant when the first participant moves rooms) with respect to the second participant.

While the example ofillustrates two participants, it is possible for the current room interfaceto accommodate additional participants for virtual conferencing. The additional participants may be positioned (e.g., automatically and/or manually by dragging) based on the positioning of participant video elements (e.g., boxes) as designed by the virtual space design interface.

In one or more embodiments, the virtual space navigation interfacemay vary based on whether a given participant is an administrator or another participant (e.g., an attendee). For example, some participant video elements may be designated (e.g., via the virtual space design interface) for administrators, while other participant video elements are designated for other participants. The virtual conference server systemis configured to distinguish between these administrator or other participant roles, for example, based on the above-described tags assigned to participants via the users category of the menu interfaceprovided by the virtual space design interface.