Communication terminal, image communication system, and display control method

This patent applicationis a Reissue of U.S. Pat. No. 10,944,798, issued Mar. 9, 2021, whichis a continuation of U.S. application Ser. No. 16/355,762, filed Mar. 17, 2019, which claims priority pursuant to 35 U.S.C. § 119(a) to Japanese Patent Application No. 2018-068440, filed Mar. 30, 2018, in the Japan Patent Office, the entire disclosure of each is hereby incorporated by reference herein.

Embodiments of the present disclosure relate to a communication terminal, an image communication system, and a display control method.

Remote conference systems, such as videoconference systems, are now in widespread use, enabling users to remotely attend a conference (meeting) that is held among different sites via a communication network such as the Internet. In such videoconference systems, a communication terminal for a remote conference system is provided in a conference room (meeting room) where attendees of one party in a remote conference are attending. This communication terminal collects an image or video of the conference room including the attendees and sounds such as a speech made by an attendee, and transmits digital data converted from the collected image (video) and/or sounds to the other party's terminal provided at a different conference room. Based on the transmitted digital data, the other party's terminal displays images on a display or outputs audio from a speaker in the different conference room to establish video communication (video call). This enables the attendees to carry out the conference among remote sites, as if they are close to each other as an actual conference.

In addition, an image capturing device that is capable of capturing a spherical image in real time is connectable to such communication terminals described above to transmit the spherical image acquired by the image capturing device to each communication terminal of the other party. Each communication terminal sequentially converts the received spherical image to a planar image representing a predetermined area, which is a part of the spherical image, and displays the planar image on a display. This enables a user in each of remote sites to determine, by his or her own, a predetermined area image to be displayed, representing an image of a predetermined area that the user is interested in, from a whole image of the spherical image.

In addition, there is a known technique that superimposes a predetermined figure on an object in a video image indicated by a video image communication terminal in superimposing the figure on the video image by an image relay server that relays a video image between or among two or more video image communication terminals. This provides a video image in which the figure is combined with the object in the video image even when objects in the video image moves.

An exemplary embodiment of the present disclosure includes an image communication system including a first communication terminal and a second communication terminal. The first communication terminal includes first circuitry and a second communication terminal includes second circuitry. The first circuitry of the first communication terminal transmits, to the second communication terminal, first image data representing a first image and second image data representing a second image. The first circuitry of the first communication terminal transmits, to the second communication terminal, position information indicating a predetermined position on the first image. The second circuitry of the second communication terminal combines, based on the position information, the second image with the first image at the predetermined position on the first image to generate a combined image. The second circuitry of the second communication terminal displays, on a display, the combined image.

The accompanying drawings are intended to depict example embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted.

The terminology used herein is for describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “includes” and/or “including”, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. In describing preferred embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this patent specification is not intended to be limited to the specific terminology so selected, and it is to be understood that each specific element includes all technical equivalents that have the same function, operation in a similar manner, and achieve a similar result.

Hereinafter, a description is given of one of the embodiments of the present disclosure, with reference to the attached drawings,(to) to.

A method of generating a spherical image is described below, with reference to(to) to.

A description is now given of an external view of an image capturing device, with reference toto. The image capturing deviceis a digital camera that captures an image (captured image) representing a three dimensional spherical (360-degree) image.,, andare a left side view, a front view, and a plan view, respectively, of the image capturing device.

As illustrated in, the image capturing devicecan be held by a single hand. Referring to,, and, the image capturing deviceis provided with an imaging elementa on a front side (anterior side) and an imaging elementb on a back side (rear side) in an upper section. The imaging elements (image sensors)a andb are used in combination with optical members such as fisheye lensesa andb each of which is capable of capturing a hemispherical image having a field view of 180-degree or wider. A detailed description of the fisheye lensesa andb is deferred. As illustrated in, the image capturing deviceis also provided with an operation devicesuch as a shutter button on an opposite side of the front side.

A description is now given of an example of how the image capturing deviceis used, with reference to.is an illustration of an example of how a user uses the image capturing device. As illustrated in, the image capturing deviceis used for capturing objects surrounding the user who is holding the image capturing devicein his or her hand, for example. The imaging elementsa andb illustrated intocapture the objects surrounding the user to obtain two hemispherical images.

A description is now given of an overview of a process of generating a spherical image from the images captured by the image capturing device, with reference to(to) and(and).is an illustration of one of the two hemispherical images (front side), captured by the image capturing device.is an illustration of the other one of the two hemispherical images (back side), captured by the image capturing device.is an illustration of an image represented by Mercator projection. The image represented by Mercator projection as illustrated inis, hereinafter, referred to as a “Mercator image”.is an illustration of a Mercator image covering a sphere.is an illustration of a spherical image.

As illustrated in, an image captured by the imaging elementa is a curved hemispherical image (front side) due to the fisheye lensa, which is described later. In addition, as illustrated in, an image captured by the imaging elementb is a curved hemispherical image (back side) captured by the fisheye lensb, which is described later. The image capturing devicecombines one hemispherical image (front side) and the other hemispherical image (back side), which is reversed by 180-degree from the one hemispherical image (front side), to generate the Mercator image as illustrated in.

The Mercator image is attached so as to cover the sphere surface using Open Graphics Library for Embedded Systems (OpenGL ES) as illustrated in. This results in the spherical image as illustrated in. In other words, the spherical image is represented as an image of which the Mercator image faces toward the center of the sphere. OpenGL ES is a graphic library used for visualizing two-dimensional (2D) and three-dimensional (3D) data. The spherical image is either a still image or a video image.

Because the spherical image is an image attached to the sphere surface, a part of the image may look distorted when viewed from the user, and this may give a feeling of strangeness to the user. To cope with this, a part of the spherical image is displayed as a planar image having fewer curves. The part of the spherical image displayed as a planar image is referred to as a predetermined area. In addition, the predetermined area may also be selectable, or settable according to a user instruction. The displayed planar image corresponding the predetermined area is, hereinafter, referred to as a “predetermined area image”. A description is now given of displaying the predetermined area image, with reference toand(and).

is an illustration of relative positions of a virtual camera IC and the predetermined area T when the spherical image is represented as a three-dimensional solid sphere, according to the present embodiment. The virtual camera IC corresponds to a position of a point of view (viewpoint) of a user who is viewing the full-view spherical image CE represented as a surface area of the three-dimensional solid sphere CS.is a perspective view of.is an illustration of the predetermined area image displayed on a display. In, the spherical image CE illustrated inis represented as a surface area of the three-dimensional solid sphere CS. Assuming that the spherical image CE is a surface area of the solid sphere CS, the virtual camera IC is outside of the spherical image CE as illustrated in. The predetermined area T in the spherical image CE is an imaging area of the virtual camera IC. Specifically, the predetermined area T is identified by predetermined area information indicating an imaging direction and an angle of view of the virtual camera IC in a three-dimensional virtual space containing the spherical image CE.

The predetermined area image Q, which is an image of the predetermined area T illustrated in, is displayed on a display as an image of an imaging area of the virtual camera IC, as illustrated in. The image illustrated inis the predetermined area image represented by the predetermined area information that is set by default. In another example, a predetermined area image may be identified by an imaging area (X, Y, Z) of the virtual camera IC, i.e., the predetermined area T, rather than the predetermined area information, i.e., the position coordinate of the virtual camera IC. In the following description of the embodiment, an imaging direction (rH, rV) and an angle of view a of the virtual camera IC are used.

A description is now given of a relation between the predetermined area information and the predetermined area T, with reference to.is a diagram illustrating a relation between the predetermined area information and the predetermined area T, according to the present embodiment. As illustrated in, “rH”, “rV”, and “α” denote a horizontal radian, a vertical radian, and an angle of view, respectively, of the virtual camera IC. The position of the virtual camera IC is adjusted, such that the point of gaze of the virtual camera IC, indicated by the imaging direction (rH, rV), matches a center point CP of the predetermined area T, which is the imaging area of the virtual camera IC. The predetermined area image Q is an image of the predetermined area T, in the spherical image CE. Distance “f” denotes a distance from the virtual camera IC to the center point CP of the predetermined area T. L is a distance between the center point CP and a given vertex of the predetermined area T (2L is a diagonal line). In, a trigonometric function equation generally expressed by the following equation 1 is satisfied.L/f=tan(α/2)  (Equation 1)

is a diagram illustrating a point in a three-dimensional Euclidean space according to spherical coordinates, according to the present embodiment. A positional coordinate (r, θ, φ) is given when the center point CP is represented by a spherical polar coordinates system. The positional coordinate (r, θ, φ) represents a moving radius, a polar angle, and an azimuth angle. The moving radius r is a distance from the origin of the three-dimensional virtual space including the spherical image to the center point CP. Accordingly, the radius r is equal to Distance “f”.illustrates the relation between these items. In the following description of the embodiment, the positional coordinates (r, θ, φ) of the virtual camera IC is used.

A description is now given of an overview of a configuration of an image communication system according to the present embodiment of the disclosure, with reference to.is a schematic diagram illustrating a configuration of the image communication system according to the present embodiment.

As illustrated in, the image communication system according to the present embodiment includes an image capturing devicea, an image capturing deviceb, a videoconference terminala, a videoconference terminald, a displaya, a displayd, a communication management system, a personal computer (PC), a PC, an image capturing device, and a smartphone. The videoconference terminala, the smartphone, the PC, and the videoconference terminald communicate data with one another via a communication networksuch as the Internet. The communication networkmay be either a wireless network or a wired network.

Each of the image capturing devicea and the image capturing deviceb is a special digital camera that captures an image including an object or a view (surroundings) to obtain two hemispherical images from which a spherical image is generated, as described above. The image capturing deviceis a general-purpose digital camera that captures an image of an object or a view (surroundings) to obtain a general planar image.

Each of the videoconference terminala and the videoconference terminalb is a terminal dedicated to videoconferencing. The videoconference terminala and the videoconference terminald display, on the displaya and the displayd, respectively, a video image obtained by performing a video communication (video call) via a wired cable such as a universal serial bus (USB) cable. The videoconference terminala usually captures an image by a camera, as illustrated in, which is described later. However, when the videoconference terminala is connected to a cradlea on which the image capturing devicea is mounted, the image capturing devicea is preferentially used. Accordingly, two hemispherical images are obtained, from which a spherical image is generated. In addition, the videoconference terminala is connected to the PCand obtains a screen of the PC. When a wired cable is used for connecting the videoconference terminala and the cradlea, the cradlea supplies power to the image capturing devicea and holds the image capturing devicea in addition to establishing a communication between the image capturing devicea and the videoconference terminala. In the present embodiment, the image capturing devicea, the cradlea, the videoconference terminala, the displaya, and the PCare provided in the same site that is a site A. In addition, there are four users, a user A, a user A, a user Aand a user A, participating in the video communication in the site A. In addition, the videoconference terminald and the displayd are provided in the same site that is a site D. In the site D, there are three users, a user D, a user D, and a user D, participating in the video communication.

The communication management systemmanages and controls communication among the videoconference terminala, the videoconference terminald, the PCand the smartphone. In addition, the communication management systemmanages types of image data (a general image type and a special image type) to be transmitted or received in the communication among the videoconference terminala, the videoconference terminald, the PCand the smartphone. In other words, the communication management systemis a communication control system. In the description of the present embodiment, a spherical image is used as a special image, and a planar image is used as a general image. The communication management systemis installed in, for example, an office of a service provider that provides a video communication service. The communication management systemmay be configured as a single computer. Alternatively, the communication management systemmay be configured as a plurality of computers, and one or more units (functions, means, or storages) are arbitrarily assigned to each of the plurality of computers. That is, the communication management systemmay be implemented by a plurality of servers that operate in cooperation with one another.

The PCgenerates material image data that is image data of a material image to be displayed in the videoconference. In this disclosure, the material image is any image to be presented for participants during the videoconference. Examples of the material image include an image displayed, created, or edited by a general-purpose application being executed on the PC, and an image, which is photographed by a general-purpose digital camera, reproduced on the PC. However, these are not intended to be limiting the embodiment.

The PCcan perform a video communication by connecting with the image capturing device. In the present embodiment, the PCand the image capturing deviceare provided in the same site that is a site C. There is one user, a user C, participating in the video communication in the site C.

The smartphoneincludes a display, which is described later, and displays an image of the video communication on the display. The smartphoneincludes a complementary metal oxide semiconductor (CMOS) sensor, and usually captures an image using the CMOS sensor. In addition, the smartphoneis capable of obtaining data of two hemispherical images, which are the original image data of a spherical image, captured by the image capturing deviceb using a wireless communication such as Wireless Fidelity (Wi-Fi) or Bluetooth (registered trademark). When such a wireless communication is used, a cradleb supplies power to the image capturing deviceb and holds the image capturing deviceb, but not establish a communication. In the present embodiment, the image capturing deviceb, the cradleb, and the smartphoneare provided in the same site that is a site B. In addition, two users, a user Band a user B, are participating in the video communication in the site B.

Each of the videoconference terminala, the videoconference terminald, the PCand the smartphoneis an example of a communication terminal. OpenGL ES is installed on each of the communication terminals to enable each of the communication terminals to generate predetermined area information that indicates a partial area of the spherical image, or to generate a predetermined area image from a spherical image that is transmitted from a different one of the communication terminals.

The arrangement of the terminals (i.e., the communication terminals, the displays, the image capturing devices), the apparatuses and the users illustrated inis just an example, and any other suitable arrangement will suffice. For example, in the site C, an image capturing device that is capable of capturing a spherical image may be used in place of the image capturing device. In addition, examples of the communication terminal include a digital television, a smartwatch, and a car navigation device. In the following description, any arbitrary one of the image capturing devicea and the image capturing deviceb is referred to as a “image capturing device”. In addition, any arbitrary one of the videoconference terminala and the videoconference terminald is, hereinafter, referred to as a “videoconference terminal”. In addition, any arbitrary one of the displaya and the displayd is, hereinafter, referred to as a “display”.

A description is now given of hardware configurations of the image capturing device, the videoconference terminal, the communication management system, the PC, the PC, and the smartphoneaccording to the present embodiment, with reference toto. The image capturing deviceis a general-purpose camera, and a detailed description of the image capturing deviceis omitted here.

A description is now given of a hardware configuration of the image capturing deviceaccording to the present embodiment, with reference to.is a block diagram illustrating a hardware configuration of the image capturing deviceaccording to the present embodiment. In the following description of the embodiment, the image capturing devicethat is a spherical (omnidirectional) image capturing device having two imaging elements is used. However, the image capturing devicemay have more than two imaging elements. In addition, the image capturing deviceis not necessarily an image capturing device dedicated to omnidirectional image capturing, but may be a general-purpose digital camera or a smartphone to which an external omnidirectional image capturing unit is attachable to implement substantially the same function as the image capturing devicein the description of the present embodiment (omnidirectional image capturing device).

As illustrated in, the image capturing deviceincludes an imaging unit, an image processor, an image controller, a microphone, an audio processor, a central processing unit (CPU), a read only memory (ROM), a static random access memory (SRAM), a dynamic random access memory (DRAM), an operation device, a network interface (I/F), a communication device, and an antennaa.

The imaging unitincludes two wide-angle lenses (so-called fisheye lenses)a andb each of which has a field angle that is equal to or greater than 180 degrees to form a hemispherical image. The imaging unitfurther includes the two imaging elementsa andb corresponding to the wide-angle lensesa andb, respectively. Each of the imaging elementsa andb includes an image sensor such as a CMOS sensor and a charge-coupled device (CCD) sensor, a timing generation circuit, and a group of registers. The image sensor converts an optical image formed by the fisheye lensesa andb into electric signals to output image data. The timing generation circuit generates horizontal or vertical synchronization signals, pixel clocks and the like for the image sensor. Various commands, parameters, and the like for operations of the imaging elementsa andb are set in the group of registers.

Each of the imaging elementsa andb of the imaging unitis connected to the image processorthrough a parallel I/F bus. In addition, each of the imaging elementsa andb of the imaging unitis connected to the image controllerthrough a serial I/F bus such as an inter-integrated circuit (I2C) bus. Each of the image processorand the image controlleris connected to the CPUthrough a bus. In addition, the ROM, the SRAM, the DRAM, the operation device, the network I/F, the communication device, and an electronic compassare also connected to the bus.

The image processorobtains image data from each of the imaging elementsa andb through the parallel I/F bus and performs predetermined processing on the image data obtained from each of the imaging elementsa andb separately and combines the processed image data to generate data representing a Mercator image as illustrated in.

The image controllerusually functions as a master device while each of the imaging elementsa andb usually functions as a slave device, and the image controllersets commands in the group of registers of each of the imaging elementsa andb through the I2C bus. The image controllerreceives necessary commands from the CPU. In addition, the image controllerobtains status data of the group of registers of each of the imaging elementsa andb through the I2C bus and transmits the status data to the CPU.

The image controllerinstructs the imaging elementsa andb to output the image data at a time when the shutter button of the operation deviceis pressed. The image capturing devicecan support a preview display function (e.g., displaying a preview on a display such as a display of the videoconference terminala) or a movie display function. In case of displaying movie, the image data is continuously output from the imaging elementsa andb at a predetermined frame rate (frames per minute).

Furthermore, the image controlleroperates in conjunction with the CPUto synchronize times when the imaging elementsa andb output the image data. In the present embodiment, the image capturing devicedoes not include a display unit (display). However, in some embodiments, the image capturing devicemay include a display.

The microphoneconverts sound into audio data (signals). The audio processorobtains the audio data from the microphonethrough an I/F bus and performs predetermined processing on the audio data.

The CPUcontrols the image capturing deviceand performs necessary processing. The ROMstores various programs to be executed by the CPU. Each of the SRAMand the DRAMoperates as a work memory to store programs loaded from the ROMto be executed by the CPUor data being currently processed. More specifically, in one example, the DRAMstores image data currently processed by the image processorand data of the Mercator image on which processing has been performed.

The operation devicecollectively refers to various operation keys, a power switch, a shutter button, and a touch panel having functions of both displaying information and receiving input from a user, which may be used in combination. The user operates the operation keys to input various image capturing modes or image capturing conditions.

The network I/Fcollectively refers to an interface circuit such as a USB I/F that enables the image capturing deviceto communicate with an external media such as a secure digital (SD) card or an external personal computer. The network I/Fsupports at least one of a wired communication and a wireless communication. The data representing the Mercator image, which is stored in the DRAM, can be stored in the external media through the network I/For transmitted to the external device such as the videoconference terminala via the network I/F, as needed.

The communication devicecommunicates with an external device such as the videoconference terminala via the antennaa of the image capturing deviceby a short range wireless communication such as Wi-Fi and Near Field Communication (NFC). The communication devicemay transmit the data representing the Mercator image to a device external to the videoconference terminala.

The electronic compasscomputes an orientation and a tilt (roll angle) of the image capturing devicebased on the Earth magnetism to output orientation and tilt information. The orientation and tilt information is an example of related information, which is metadata described in compliance with Exif. In addition, the orientation and tilt information is used for performing image processing, such as image correction, on captured image data. The related information also includes data indicating a time (date) when an image is captured by the image capturing device, and data indicating a size of image data (an amount of image data), for example.

A description is now given of a hardware configuration of the videoconference terminalaccording to the present embodiment of the disclosure, with reference to.is a block diagram illustrating an example of the hardware configuration of the videoconference terminalaccording to the present embodiment. As illustrated in, the videoconference terminalincludes a CPU, a ROM, a RAM, a flash memory, a solid state drive (SSD), a medium I/F, an operation key, a power switch, a bus line, a network I/F, a camera, an imaging element I/F, a microphone, a speaker, an audio input/output (I/O) interface, a display I/F, an external device connection I/F, a short-range communication circuit, and an antennaa for the short-range communication circuit.

The CPUcontrols the entire operation of the videoconference terminal. The ROMstores a control program such as an Initial Program Loader (IPL) used for operating the CPU. The RAMis used as a work area for the CPU. The flash memorystores various data such as a communication control program, image data, and audio data. The SSDcontrols reading and/or writing of various data to and/or from the flash memoryunder control of the CPU. In alternative to the SSD, a hard disk drive (HDD) may be used. The medium I/Freads and/or writes (stores) data from and/or to a recording mediumsuch as a flash memory. The operation keyis operated according to a user input indicating an instruction in selecting a destination of a communication from the videoconference terminal, for example. The power switchis a switch that turns on or off the power of the videoconference terminal.

The network I/Fenables the videoconference terminalto establish a data communication with an external device via the communication networksuch as the Internet. The camerais an example of a built-in imaging device capable of capturing an object under control of the CPUto obtain image data. The imaging element I/Fis a circuit that controls driving of the camera. The microphoneis an example of a built-in sound collecting device capable of inputting sounds. The audio input/output interfaceis a circuit for controlling input and output of audio signals between the microphoneand the speakerunder control of the CPU. The display I/Fis a circuit for transmitting image data to an external displayunder control of the CPU. The external device connection I/Fis an interface that connects the videoconference terminalto various external devices. The short-range communication circuitis a communication circuit such as NFC standard, Bluetooth (registered trademark) or the like.

The bus line, which includes an address bus and a data bus, electrically connects to various elements, including the CPUillustrated in, one another.