Information processing apparatus and information processing method

This application is a U.S. National Phase of International Patent Application No. PCT/JP2023/002644 filed on Jan. 27, 2023, which claims priority benefit of Japanese Patent Application No. JP 2022-048905 filed in the Japan Patent Office on Mar. 24, 2022. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.

The present technology relates to an information processing apparatus, an information processing method, and a recording medium that can be applied to telepresence.

In recent years, a technology has been developed that connects multiple locations using communication to enable remote communication with a sense of realism by use of displays respectively used at the multiple locations. Such a technology may be referred to as a telepresence technology or tele-presentations. The telepresence technology may be considered an embodiment of a telecommunication technology.

In the telepresence technology, there is a need for a high-level sense of spatial co-presence and a high-level sense of reality of a person. Patent Literature 1 proposes an approach of causing persons to face each other, which is one of the purposes of the telepresence technology.

On the other hand, a stereoscopic display is known that is a display technology that makes a sense of reality of a display object greater. As such a stereoscopic display, a stereoscopic display that makes it possible to view display with naked eyes (Patent Literature 2), and a stereoscopic display that requires dedicated glasses (Patent Literature 3) are known. Note that embodiments of the stereoscopic display also include a head-mounted display (of which the disclosure is omitted) that makes an image in a real space invisible and replaces a field of view of a user with an image. (Note that Patent Literature 2 also proposes a system that enables a gesture to be input relative to a displayed stereoscopic object.)

When a stereoscopic display is applied to the telepresence technology in order to make a sense of reality of a person greater, stereoscopic displays are favorably used in all of the locations in order to share the same sense of reality. On the other hand, the diffusion rate of stereoscopic displays is not high even in recent years, compared with the diffusion rate of general non-stereoscopic displays. Thus, when a stereoscopic display is applied to the telepresence technology, communication may be expected to be performed between a stereoscopic display and a non-stereoscopic display. In this case, it may become difficult for a user of a stereoscopic display and a user of a non-stereoscopic display to interact with each other due to a difference between the users in perceived sense of co-presence or in perceived sense of reality.

Patent Literature 1: WO 2019/026598 Patent Literature 2: Japanese Patent Application Laid-open No. 2012-222386 Patent Literature 3: Japanese Patent Application Laid-open No. 2015-109111

The present disclosure proposes a technology that makes it possible to perform a proper interaction in terms of telepresence technology even when a stereoscopic display and a non-stereoscopic display communicate with each other, where there is asymmetry between respective display states of the stereoscopic display and the non-stereoscopic display.

In order to achieve the object described above, an information processing apparatus according to an embodiment of the present technology includes an image acquisition section, a depth acquisition section, and a superimposition processor. The image acquisition section acquires a captured image of a first user who uses a stereoscopic display. The depth acquisition section acquires depth information regarding a second user who uses a non-stereoscopic display that communicates with the stereoscopic display. The superimposition processor superimposes a result of interaction of the second user on the captured image of the first user on the basis of the depth information, the captured image of the first user being displayed on the non-stereoscopic display.

An information processing method according to an embodiment of the present technology is an information processing method that is performed by a computer system, the information processing method including acquiring a captured image of a first user who uses a stereoscopic display. Depth information regarding a second user who uses a non-stereoscopic display that communicates with the stereoscopic display is acquired. A result of interaction of the second user is superimposed on the captured image of the first user on the basis of the depth information, the captured image of the first user being displayed on the non-stereoscopic display.

A recording medium that records therein a program according to an embodiment of the present technology causes a computer system to execute a specified instruction. The specified instruction is used to perform a process including acquiring a captured image of a first user who uses a stereoscopic display; acquiring depth information regarding a second user who uses a non-stereoscopic display that communicates with the stereoscopic display; and superimposing a result of interaction of the second user on the captured image of the first user on the basis of the depth information, the captured image of the first user being displayed on the non-stereoscopic display.

Embodiments according to the present technology will now be described below with reference to the drawings.

[Telepresence System]

1 FIG. 1 schematically illustrates an example of a configuration of a telepresence system according to an embodiment of the present technology. Telepresence is a technology that connects multiple locations using communication to enable remote communication with a sense of realism by use of displays respectively used at the multiple locations. A telepresence systemaccording to the present technology enables users situated at places remote from each other to talk over the phone with a sense of realism while viewing their respective displays.

1 4 1 2 5 1 3 4 5 1 1 FIG. In the present embodiment, the telepresence systemis arranged for two spaces situated at places distant from each other.illustrates two spaces obtained by division being performed by a middle dashed line. A userwho uses the telepresence systemin a spaceis schematically illustrated on the left. Likewise, a userwho uses the telepresence systemin a spaceis schematically illustrated on the right. In the present embodiment, the usersandtalk over the phone using the telepresence system.

1 FIG. 2 3 2 3 1 4 5 illustrates a user in each of the spacesand. However, a plurality of users may be situated in each of the spacesand. In other words, a plurality of persons may be capable of talking over the phone using the telepresence system. The usercorresponds to an embodiment of a first user according to the present technology. The usercorresponds to an embodiment of a second user according to the present technology.

1 6 7 10 11 6 7 2 10 11 3 1 FIG. The telepresence systemincludes a stereoscopic display, a camera, a non-stereoscopic display, and a stereo camera. As illustrated in, the stereoscopic displayand the cameraare arranged in the space. The non-stereoscopic displayand the stereo cameraare arranged in the space.

6 4 6 4 4 4 6 6 6 The stereoscopic displayis a display with which a display object can be stereoscopically viewed by the user. Specifically, the stereoscopic displaydisplays thereon an image viewed by the userwith his/her right eye and an image viewed by the userwith his/her left eye. There is parallax between the respective images, and the user has a stereoscopic experience by viewing the different images with his/her right eye and with his/her left eye. In the present embodiment, an autostereoscopic display with which the usercan perform viewing with naked eyes (without wearing anything for his/her eyes) is used as the stereoscopic display. Of course, the stereoscopic displayis not limited thereto, and a stereoscopic display with which viewing is performed by, for example, dedicated glasses being worn, may be used as the stereoscopic display.

1 FIG. 14 6 14 8 5 4 8 4 5 4 14 11 3 14 schematically illustrates a captured imagedisplayed on the stereoscopic display. The captured imageincludes a user imagethat is an image of the user. The useris stereoscopically viewing the user image. This enables the userto feel as if the useris with the user. The captured imageis acquired by the stereo cameraarranged in the space. The captured imagecorresponds to an embodiment of a captured image of the second user according to the present technology.

7 6 7 2 4 15 7 The camerais arranged above the stereoscopic display. In the present embodiment, image-capturing is performed by the camerawith respect to the spacein which the useris situated. Accordingly, a captured imageis acquired. For example, a digital camera by which a moving image is captured, or an infrared camera is used as the camera. Moreover, any cameras may be used.

10 3 5 10 10 10 10 The non-stereoscopic displayarranged in the spaceis a display with which a display object can be planarly viewed by the user. In other words, the non-stereoscopic displaycan also be a commonly used display that is not used to view a display object stereoscopically. A display device using, for example, liquid crystal or electroluminescence (EL) is used as the non-stereoscopic display. Of course, the non-stereoscopic displayis not limited thereto, and any display devices may be used as the non-stereoscopic display.

1 FIG. 15 10 15 9 4 5 9 5 9 15 7 2 15 schematically illustrates the captured imagedisplayed on the non-stereoscopic display. The captured imageincludes a user imagethat is an image of the user. The useris not stereoscopically viewing the user image. Thus, the useris planarly viewing the user image. The captured imageis acquired by the cameraarranged in the space. The captured imagecorresponds to an embodiment of a captured image of the first user according to the present technology.

11 5 5 10 11 5 10 11 12 5 13 5 12 13 3 5 14 The stereo camerais arranged in front of the user. For example, when the userlooks at the non-stereoscopic displayfrom the front, the stereo camerais arranged between the userand the non-stereoscopic display. The stereo cameraincludes a left camerasituated on the left as viewed from the user, and a right camerasituated on the right as viewed from the user. Image-capturing is performed by each of the left cameraand the right camerawith respect to the spacein which the useris situated. Accordingly, two images between which there is parallax are acquired as the captured image.

[Superimposition of Avatar Image]

15 10 2 FIG. In the present embodiment, an avatar image is superimposed on the captured imagedisplayed on the non-stereoscopic display.schematically illustrates an example of superimposition of an avatar image.

2 FIG. 4 5 3 5 17 18 17 5 6 4 18 17 5 4 4 4 16 18 4 4 5 illustrates a state in which the usersandare giving pseudo-handshakes to each other. In the space, the useris extending his/her right hand. An image (a right-hand image) of the right handextended by the useris displayed on the stereoscopic display. The useris stereoscopically viewing the right-hand image. Thus, it looks like the right handof the useris protruding toward the user, as viewed from the user. The usercauses a position of the right handto coincide with an apparent position of the right-hand image. This enables the userto feel as if the useris shaking hands with user.

2 4 16 19 16 4 10 5 19 19 5 In the space, the useris extending his/her right hand. An image (a right-hand image) of the right handextended by the useris displayed on the non-stereoscopic display. The useris not stereoscopically viewing the right-hand image. Thus, the right-hand imageis planarly seen by the user.

20 10 20 17 5 20 20 15 20 18 4 2 16 18 4 5 4 20 19 10 16 18 4 5 4 20 19 10 20 18 16 4 4 19 20 10 18 16 4 20 19 Further, an avatar imageof a hand is displayed on the non-stereoscopic display. The avatar imageis an image that represents the right handof the user, and, for example, a drawing of the hand is used as the avatar image. The avatar imageis superimposed to be displayed on the captured image. The avatar imageis displayed to correspond to the apparent position of the right-hand imagestereoscopically viewed by the userin the space. For example, it is assumed that the right handand the right-hand imageare situated at positions similar to each other and it looks like the useris shaking hands with the user, as viewed from the user. In this case, the avatar imageis superimposed to be displayed at a position similar to a position of the right-hand imageon the non-stereoscopic display. It is assumed that the right handand the right-hand imageare distant from each other and it looks like the useris not shaking hands with the user, as viewed from the user. In this case, the avatar imageis superimposed to be displayed at a position distant from the right-hand imageon the non-stereoscopic display. Further, the avatar imageis superimposed to be displayed such that a positional relationship between the right-hand imageand the right handof the useras viewed from the usercorresponds to a positional relationship between the right-hand imageand the avatar imageon the non-stereoscopic display. For example, when it looks like the right-hand imageis situated on the left of the right hand, as viewed from the user, the avatar imageis superimposed to be displayed on the right of the right-hand image.

20 17 5 17 20 17 20 Further, a superimposition position for the avatar imageis changed according to a change in a position of the right hand. For example, when the usermoves the right handrightward, the superimposition position for the avatar imageis also changed rightward. Further, when the right handis extended forward in order to shake hands, the superimposition position for the avatar imageis slightly changed upward.

20 4 5 19 20 5 16 18 4 18 4 5 17 20 17 As described above, the avatar imageis superimposed to be displayed such that contents viewed by the userare consistent with contents viewed by the user. For example, there is no possibility of the right-hand imageand the avatar imagebeing seen at similar positions, as viewed from the user, despite the fact that it looks like the right handand the right-hand imageare distant from each other, as viewed from the user. Alternatively, there is also no possibility of it looking like the right-hand imageis moving forward, as viewed from the user, despite the fact that the useris extending his/her right handforward and the avatar imageis moving upward according to the extension of the right hand.

[Configuration of Telepresence System]

3 FIG. 1 11 5 11 14 5 5 11 is a block diagram illustrating an example of a configuration of the telepresence system. The stereo cameraacquires depth information regarding the user. In other words, the stereo cameraacquires the captured imageand the depth information regarding the userin the present embodiment. A method for acquiring the depth information regarding the useris not limited. For example, a ranging sensor such as a time-of-flight (ToF) sensor or light detection and ranging or laser imaging detection and ranging (LiDAR) may be provided separately from the stereo camera.

1 23 24 23 23 6 10 23 23 25 26 27 28 25 26 27 28 29 25 The telepresence systemfurther includes an information processing apparatusand a database (DB). The information processing apparatusis implemented by any computer such as a personal computer (PC). Alternatively, the information processing apparatusand the stereoscopic displayor the non-stereoscopic displaymay be integrated by the information processing apparatusbeing built in the display. The information processing apparatusincludes a controller, an operation section, a communication section, and a storage. The controller, the operation section, the communication section, and the storageare connected to each other through a bus. The respective blocks may be connected to each other using, for example, a communication network or an unstandardized unique communication approach instead of using the bus.

26 1 23 22 Examples of the operation sectioninclude a keyboard, a pointing device, a touchscreen, and other operation apparatuses. For example, an operator who manages the telepresence systemcan perform setting related to an operation of the information processing apparatususing the operation section.

27 27 23 6 23 10 6 10 23 23 7 11 23 The communication sectionis a module used to perform, for example, network communication or near field communication with another device. In the present embodiment, the communication sectioncauses the information processing apparatusto communicate with the stereoscopic display. Further, the information processing apparatusis also caused to communicate with the non-stereoscopic display. In other words, it can also be said that the stereoscopic displayand the non-stereoscopic displaycommunicate with each other through the information processing apparatus. Furthermore, the information processing apparatusis also caused to communicate with each of the cameraand the stereo camera. Moreover, the information processing apparatusmay be capable of communicating with any apparatus.

28 28 28 23 23 The storageis a storage device such as a nonvolatile memory, and, for example, an HDD or an SSD is used. Moreover, any non-transitory computer-readable storage medium may be used as the storage. The storagestores therein a control program used to control an operation of the overall information processing apparatus. A method for installing the control program on the information processing apparatusis not limited. For example, the installation may be performed through various recording media, or the installation of the program may be performed through, for example, the Internet.

25 25 The controllerincludes hardware, such as a processor including a CPU, a GPU, and a DSP; a memory including a ROM and a RAM; and a storage device including an HDD, that is necessary for a configuration of a computer. For example, an information processing method according to the present technology is performed by the CPU loading, into the RAM, a program according to the present technology that is recorded in, for example, the ROM in advance and executing the program. For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA), or another device such as an application specific integrated circuit (ASIC) may be used as the controller.

30 31 32 33 34 35 36 37 38 25 In the present embodiment, an image acquisition section, a depth acquisition section, a 3D data converter, a shortest-distance-pixel determining section, a border determination section, a 2D-space-projection section, an angle-of-view determining section, a superimposition processor, and a display controllerare implemented as functional blocks by the CPU of the controllerexecuting the program according to the present technology (such as an application program). Note that, in order to implement each functional block, dedicated hardware such as an integrated circuit (IC) may be used as appropriate.

30 4 6 30 7 15 2 4 30 14 11 The image acquisition sectionacquires a captured image of the userwho uses the stereoscopic display. Specifically, the image acquisition sectionacquires, from the camera, the captured imagecorresponding to a captured image of the spacein which the useris situated. Further, the image acquisition sectionacquires the captured imagefrom the stereo camera.

31 5 10 31 5 11 31 The depth acquisition sectionacquires depth information regarding the userwho uses the non-stereoscopic display. Specifically, the depth acquisition sectionacquires the depth information regarding the userfrom the stereo camera. Note that, for example, when a ranging sensor is provided separately from the stereo camera, the depth acquisition sectionmay acquire depth information from the ranging sensor.

32 5 3 14 5 The 3D data convertercalculates three-dimensional coordinates of the userin the spaceon the basis of the captured imageand the depth information regarding the user.

14 5 33 5 3 11 33 14 17 11 5 17 17 On the basis of the captured imageand the depth information regarding the user, the shortest-distance-pixel determining sectioncalculates three-dimensional coordinates of a portion of the userin the spacethat is situated closest to the stereo camera(a shortest-distance portion). Further, the shortest-distance-pixel determining sectiondetermines which pixels in the captured imageare pixels (shortest-distance pixels) used for image-capturing on the shortest-distance portion. For example, when the right handis situated closest to the stereo camerain a body of the user, the right handcorresponds to the shortest-distance portion, and pixels used for image-capturing on the right handare the shortest-distance pixels.

34 14 14 14 34 14 14 The border determination sectiondetermines whether the shortest-distance pixel is situated at an image-outline border of the captured image. In the present embodiment, a rectangular image is captured as the captured image. Thus, the image-outline border is a border that forms a rectangle and corresponds to an edge portion of the captured image. The border determination sectiondetermines whether the shortest-distance pixel is situated at the border. Of course, shapes of the captured imageand the image-outline border are not limited, and the captured imageand the image-outline border may be designed discretionarily.

35 5 15 5 3 7 2 15 10 2 2 15 7 15 15 2 15 7 15 15 The 2D-space-projection sectioncalculates imaginary coordinates (virtual coordinates) of the userin the captured imageon the basis of the three-dimensional coordinates of the userin the space. In the present embodiment, the cameraperforms image-capturing on the space, and the captured imageis displayed on the non-stereoscopic display. Thus, when image-capturing is performed on a target object that is situated in the space, there is a correspondence relationship between coordinates of the target object in the spaceand coordinates of the target object in the captured image. For example, when the target object is situated in an upper portion in a range of an angle of view of the camera, the target object appearing in the captured imageis also in an upper portion of the captured image. Thus, a Z coordinate of the target object in the spaceexhibits a relatively large value, and a coordinate of the target object appearing in the captured imagealso exhibits a relatively large value. On the other hand, when the target object is not situated in the range of the angle of view of the camera, the captured imagedoes not include an image of the target object. In this case, the coordinates of the target object in the captured imageare not defined.

5 2 7 5 15 5 2 7 5 5 5 15 35 5 15 24 Here, an apparent userin the spaceis considered a “target object” of which an image is captured by the camera. In other words, the following is considered: what values coordinates of the apparent userin the captured imagewill exhibit if the apparent userreally exists in the spaceand the cameracaptures an image of the user, although an image of the apparent useris actually not captured. Accordingly, the imaginary coordinates of the userin the captured imagecan be considered. The 2D-space-projection sectioncalculates the “imaginary coordinates” as the virtual coordinates of the userin the captured image. The virtual coordinates are calculated using calibration data stored in the DB.

36 2 7 36 5 15 35 36 3 33 15 5 15 3 The angle-of-view determining sectiondetermines whether an apparent position of a shortest-distance portion in the spaceis situated in the range of the angle of view of the camera. Specifically, first, the angle-of-view determining sectionacquires virtual coordinates of the userin the captured imagethat are calculated by the 2D-space-projection section. Further, the angle-of-view determining sectionacquires three-dimensional coordinates of the shortest-distance portion in the spacethat are calculated by the shortest-distance-pixel determining section. Furthermore, it is determined whether there are virtual coordinates of the shortest-distance portion in the captured image, on the basis of the virtual coordinates of the userin the captured imageand the three-dimensional coordinates of the shortest-distance portion in the space.

15 2 7 15 2 7 15 36 2 7 When there are the virtual coordinates of the shortest-distance portion in the captured image, this means that the apparent position of the shortest-distance portion in the spaceis situated in the range of the angle of view of the camera. On the other hand, when there are not the virtual coordinates of the shortest-distance portion in the captured image, this means that the apparent position of the shortest-distance portion in the spaceis not situated in the range of the angle of view of the camera. Thus, the determination of whether there are the virtual coordinates of the shortest-distance portion in the captured imageenables the angle-of-view determining sectionto determine whether the apparent position of the shortest-distance portion in the spaceis situated in the range of the angle of view of the camera.

37 20 14 20 34 14 36 2 7 20 34 36 The superimposition processorsuperimposes the avatar imageon the captured image. In the present embodiment, the avatar imageis superimposed when the border determination sectiondetermines that a shortest-distance pixel is not situated at the image-outline border of the captured imageand when the angle-of-view determining sectiondetermines that the apparent position of the shortest-distance portion in the spaceis situated in the range of the angle of view of the camera. Processing of superimposing the avatar imageand processing performed when the border determination sectionand the angle-of-view determining sectionperform other determinations will be described in detail later.

32 33 34 35 36 37 The 3D data converter, the shortest-distance-pixel determining section, the border determination section, the 2D-space-projection section, the angle-of-view determining section, and the superimposition processorcorrespond to an embodiment of a superimposition processor according to the present technology.

38 14 6 15 10 20 10 The display controllercontrols display of the captured imageon the stereoscopic display, display of the captured imageon the non-stereoscopic display, and display of the avatar imageon the non-stereoscopic display.

24 35 5 15 35 5 3 28 The DBstores therein calibration data used by the 2D-space-projection sectionto calculate virtual coordinates. Specifically, for example, a matrix is stored as the calibration data. Virtual coordinates of the userin the captured imageare calculated by the 2D-space-projection sectionperforming matrix operation on three-dimensional coordinates of the userin the space. Moreover, any information, such as a function and a table, that is used to calculate virtual coordinates may be stored. Further, information used to calculate not only virtual coordinates but also other parameters may be stored. Furthermore, these pieces of information may be stored in the storage.

[Superimposition of Interaction Result]

30 14 5 5 5 4 30 14 5 4 10 5 2 30 14 5 10 5 5 In the present embodiment, the image acquisition sectionacquires the captured imageof the userperforming interaction. The interaction refers to various motions performed by the user. Examples of the interaction of the userinclude a pseudo-handshake given to the user. In this case, the image acquisition sectionacquires the captured imageof the usergiving a pseudo-handshake to the userdisplayed on the non-stereoscopic display. The examples of the interaction also include a motion of the userperforming pseudo-pointing at a target object situated in the space. In this case, the image acquisition sectionacquires the captured imageof the userperforming pseudo-pointing at a target object displayed on the non-stereoscopic display. Furthermore, the examples of the interaction according to the present technology also include a motion, such as the usermoving his/her hand in the air, that is not performed relative to a specific target. Moreover, the examples of the interaction may include any motions performed by the user.

5 37 15 10 5 17 5 17 5 Further, a result of interaction of the useris superimposed by the superimposition processoron the captured imagedisplayed on the non-stereoscopic display, on the basis of depth information regarding the user. The interaction result refers to an object used to perform interaction. For example, when the interaction is a pseudo-handshake, the interaction result is the right handof the usergiving a pseudo-handshake. Further, when the interaction is pseudo-pointing at a target object, the interaction result is the right handof the userperforming pseudo-pointing. Moreover, any object (such as a stick) other than a hand may be the interaction result according to the interaction.

15 5 5 17 20 17 20 17 The superimposition of an interaction result refers to superimposing an image corresponding to the interaction result on the captured image. In the present embodiment, the superimposition of an interaction result includes superimposing an avatar image of a body part of the user, the body part being used by the userto perform interaction. For example, when the interaction result is the right handwith which a pseudo-handshake is given, the superimposition of the interaction result is superimposing the avatar imageof the right hand. Furthermore, when the interaction result is a hand with which pseudo-pointing is performed, the superimposition of the interaction result is also superimposing the avatar imageof the right hand. Moreover, any image (such as an image of a stick) corresponding to the interaction result may be superimposed according to the interaction result.

4 FIG. 5 5 6 6 7 7 8 8 9 FIGS.A,B,A,B,A,B,A,B,A 10 9 4 5 is a flowchart illustrating an example of processing of controlling display on the non-stereoscopic display., andB schematically illustrate the usersandgiving pseudo-handshakes to each other.

30 15 4 7 101 30 14 5 11 102 31 5 11 103 11 5 The image acquisition sectionacquires the captured imageof the userthat is captured by the camera(Step). The image acquisition sectionacquires the captured imageof the userthat is captured by the stereo camera(Step). The depth acquisition sectionacquires depth information regarding the userthat is detected by the stereo camera(Step). Specifically, a distance between the stereo cameraand each body part of the useris acquired.

32 5 3 104 32 14 30 32 5 31 5 11 14 5 5 11 14 5 The 3D data convertercalculates three-dimensional coordinates of the userin the space(Step). First, the 3D data converteracquires the captured imagefrom the image acquisition section. Further, the 3D data converteracquires the depth information regarding the userfrom the depth acquisition section. Then, three-dimensional coordinates of the userin a range of an angle of view of the stereo cameraare calculated on the basis of the acquired captured imageand depth information regarding the user. Specifically, for example, “X=10 to 50, Y=20 to 60, Z=0 to 60”, which is a coordinate range in which the useris situated in the range of the angle of view of the stereo camera, is calculated. Further, only a coordinate range for a portion that appears in the captured image(a front surface of the body of the user) may be calculated.

5 5 14 5 A method for calculating three-dimensional coordinates of the useris not limited. For example, the three-dimensional coordinates of the usercan be calculated using a known approach on the basis of the captured imageand depth information regarding the user. Further, any reference position of a coordinate system and any type of coordinate system (such as a rectangular coordinate system or a polar coordinate system) may also be adopted.

33 105 33 14 30 33 5 31 33 11 14 5 11 5 11 11 The shortest-distance-pixel determining sectiondetermines a shortest-distance pixel (Step). First, the shortest-distance-pixel determining sectionacquires the captured imagefrom the image acquisition section. Further, the shortest-distance-pixel determining sectionacquires the depth information regarding the userfrom the depth acquisition section. Furthermore, the shortest-distance-pixel determining sectioncalculates three-dimensional coordinates of a shortest-distance portion in the range of the angle of view of the stereo cameraon the basis of the captured imageand the depth information regarding the user. Note that the shortest-distance portion of which three-dimensional coordinates are calculated is a shortest-distance portion “in the angle of view of the stereo camera”. Thus, even if a portion of the userthat is situated outside of the range of the angle of view of the stereo camerais situated closest to the stereo camera, three-dimensional coordinates of such a portion will not be calculated.

5 11 5 11 5 11 In the present embodiment, three-dimensional coordinates in a specified range centered at a portion of the userthat is situated closest to the stereo cameraare calculated as the three-dimensional coordinates of the shortest-distance portion. For example, when three-dimensional coordinates of the portion of the userthat is situated closest to the stereo cameraare represented by “X=30, Y=40, Z=30”, three-dimensional coordinates in a range of “X=27 to 33, Y=37 to 43, Z=27 to 33” are calculated as the three-dimensional coordinates of the shortest-distance portion, the range being obtained by adding three to and subtracting three from each value of the original three-dimensional coordinates. For example, when the portion of the userthat is situated closest to the stereo camerais a fingertip, a specified range centered at the fingertip corresponds to the shortest-distance portion.

33 14 5 14 5 11 Further, the shortest-distance-pixel determining sectiondetermines which pixels are the shortest-distance pixels on the basis of the captured imageand the depth information regarding the user. In the present embodiment, a specified range (pixel group), in the captured image, that is centered at a pixel used for image-capturing on a portion of the userthat is closest to the stereo cameracorresponds to the shortest-distance pixels. In other words, two-dimensional coordinates such as “X=17 to 23, Y=27 to 33” are calculated as coordinate values of the shortest-distance pixels. The coordinate values of a shortest-distance pixel can be calculated on the basis of, for example, a histogram of depth information. Moreover, any approach may be used to calculate coordinate values of a shortest-distance pixel.

When coordinate values of a shortest-distance portion or coordinate values of shortest-distance pixels are calculated in the form of a specified range, this results in making the calculated coordinate values less likely to be affected by noise. In other words, coordinate values can be calculated with a high degree of accuracy. Of course, a size and the like of the specified range are not limited. Further, only coordinate values of a single point may be calculated precisely without a specified range being set.

5 FIG.A 5 FIG.A 5 FIG.A 5 FIG.A 5 FIG.A 6 FIG.A 4 2 8 6 7 41 7 8 5 5 5 6 5 41 5 11 6 schematically illustrates, by hatching, the userstanding straight in the space. Further,schematically illustrates an apparent user imageby shading. Furthermore,illustrates the stereoscopic displayand the camera. Moreover,illustrates an angle of viewof the cameraby a dashed line. Note thatillustrates the apparent user imagein the form of an image of the entirety of the user, but this merely indicates a relative position of the entirety of the userfor description. Actually, the entirety of the useris not displayed on the stereoscopic display. For example, legs of the userare situated outside of a range of the angle of view. Thus, an image of the legs of the useris not captured by the stereo cameraor the legs are not displayed on the stereoscopic display. The same applies to other figures including.

5 FIG.B 5 FIG.B 5 FIG.B 5 3 10 11 15 9 10 42 11 schematically illustrates, by shading, the userstanding straight in the space. Further,illustrates the non-stereoscopic displayand the stereo camera. The captured imageincluding the user imageis displayed on the non-stereoscopic display. Furthermore,illustrates an angle of viewof the stereo cameraby a dashed line.

6 6 FIGS.A andB 4 16 5 17 4 5 In, the useris extending his/her right hand, the useris extending his/her right hand, and the usersandare giving pseudo-handshakes to each other.

7 7 FIGS.A andB 8 8 FIGS.A andB 4 16 5 17 16 4 Inand in, the useris extending his/her right hand, and the useris extending his/her right handfurther downward than the right handof the user.

34 14 106 106 34 33 34 14 28 14 11 28 6 6 7 7 8 8 FIGS.A,B,A,B,A, andB The border determination sectiondetermines whether a shortest-distance pixel is situated at the image-outline border of the captured image(Step). The process of Stepis described below with reference to. First, the border determination sectionacquires coordinate values of a shortest-distance pixel from the shortest-distance-pixel determining section. Further, the border determination sectionacquires coordinate values of the edge portion of the captured imagefrom the storage. In the present embodiment, a rectangular image is captured as the captured image. Thus, the edge portion is rectangular. In other words, coordinate values of the edge portion are in a range of, for example, “X=10, Y=30 to 50 (on the left)”, “X=30, Y=30 to 50 (on the right)”, “X=10 to 30, Y=30 (on the lower side)”, and “X=10 to 30, Y=50 (on the upper side)”. The coordinate values of the edge portion are determined in advance according to, for example, a position of or an image-capturing angle of the stereo camera, and stored in the storage.

34 14 14 106 Further, the border determination sectioncompares the coordinate values of a shortest-distance pixel with the coordinate values of the edge portion. When the coordinate values of the shortest-distance pixel are equal to the coordinate values of the edge portion, the shortest-distance pixel is determined to be situated at the image-outline border (the edge portion) of the captured image. On the other hand, when the coordinate values of the shortest-distance pixel are not equal to the coordinate values of the edge portion, the shortest-distance pixel is determined to not be situated at the image-outline border of the captured image. As described above, the process of Stepis performed.

17 11 17 17 42 14 17 42 14 6 FIG.B 6 FIG.B For example, the right handis extended toward the stereo camerain the example illustrated in. Thus, the shortest-distance pixel is a pixel used for image-capturing on the right hand. The right handis situated in a range of the angle of view. Thus, the shortest-distance pixel is a pixel situated in the captured image(in a portion other than the edge portion). Thus, in this case, the coordinate values of the shortest-distance pixel are not equal to the coordinate values of the edge portion. The coordinate values of the shortest-distance pixel are represented by, for example, “X=20, Y=40”, and are not in the range of the coordinate values of the edge portion. In other words, when the right handis in the range of the angle of view, as illustrated in, the shortest-distance pixel is determined to not be at the image-outline border of the captured image.

7 8 FIGS.B andB 7 8 FIGS.B andB 17 11 17 42 5 11 42 5 42 5 14 17 42 14 In the examples illustrated in, the right handis extended toward the stereo camera, and the right handis situated outside of the range of the angle of view(on the lower side). The shortest-distance portion is a portion of the userthat is situated closest to the stereo camerain the range of the angle of view. Thus, in this case, the shortest-distance portion corresponds to a right wrist of the userthat is situated at a lower border in the range of the angle of view. The right wrist of the userappears in a lower edge portion of the captured image. Thus, coordinate values of the shortest-distance pixel (a pixel used for image-capturing on the right wrist) are equal to coordinate values of the edge portion. Specifically, the coordinate values of the shortest-distance pixel are values of, for example, “X=20, Y=30”, and are in a range of the coordinate values of the lower edge portion, that is, “X=10 to 30, Y=30”. In other words, when the right handis situated outside of the range of the angle of view, as in, the shortest-distance pixel is determined to be at the image-outline border of the captured image.

5 17 5 17 6 6 34 5 6 14 5 6 6 7 7 8 8 FIGS.A,B,A,B,A, andB 6 FIG.A 7 8 FIG.A orA Here, the useris trying to give a pseudo-handshake in. Thus, the right handis a result of interaction of the user. In other words, the interaction result (the right hand) is displayed on the stereoscopic displayin. On the other hand, the interaction result is not displayed on the stereoscopic displayin. Thus, it can also be said that the border determination sectiondetermines whether a result of interaction of the useris displayed on the stereoscopic display, on the basis of information regarding the edge portion of the captured imageof the userperforming interaction.

106 Note that the information regarding the edge portion that is used for determination is not limited to coordinate values, and any information may be used. Further, coordinate values of a single point are set to be coordinate values of a shortest-distance pixel when the process of Stepis described, in order to simplify the description. However, of course, the coordinate values of a shortest-distance pixel may correspond to a specified range. In this case, for example, the coordinate values of the shortest-distance pixels are determined to be equal to the coordinate values of the edge portion when coordinate values in a portion of the specified range are equal to the coordinate values of the edge portion. Alternatively, the coordinate values of the shortest-distance pixels may be determined to be equal to the coordinate values of the edge portion when coordinate values in a certain range included in the specified range are equal to the coordinate values of the edge portion.

14 106 35 5 15 107 107 6 6 FIGS.A andB When the shortest-distance pixel is determined to not be situated at the image-outline border of the captured image(No in Step), the 2D-space-projection sectioncalculates virtual coordinates of the userin the captured image(Step). In other words, the process of Stepis performed in a state illustrated in, for example,.

35 5 3 32 35 24 5 5 15 5 3 5 3 8 2 7 5 15 5 First, the 2D-space-projection sectionacquires three-dimensional coordinates of the userin the spacefrom the 3D data converter. Further, the 2D-space-projection sectionacquires calibration data from the DB. Furthermore, computing processing using calibration data is performed on the three-dimensional coordinates of the user. Accordingly, the virtual coordinates of the userin the captured imageis calculated. The following is an example of the calculation of virtual coordinates corresponding to coordinates of each point on the userin the space: virtual coordinates that correspond to values of “X=30, Y=40, Z=30” that represent coordinates of a single point on the userin the spaceare represented by “X=20, Y=40”. Note that, when an apparent position of the user imagein the spaceis not situated in the range of the angle of view of the camera, the virtual coordinates of the userin the captured imageare not defined. Thus, a point of which corresponding virtual coordinates are not calculated (do not exist) also exists on the user.

36 2 7 108 36 5 35 36 3 33 15 5 The angle-of-view determining sectiondetermines whether an apparent position of the shortest-distance portion in the spaceis situated in the range of the angle of view of the camera(Step). First, the angle-of-view determining sectionacquires the virtual coordinates of the userfrom the 2D-space-projection section. Further, the angle-of-view determining sectionacquires the three-dimensional coordinates of the shortest-distance portion in the spacefrom the shortest-distance-pixel determining section. Furthermore, it is determined whether there exist virtual coordinates of the shortest-distance portion in the captured imageon the basis of the virtual coordinates of the userand the three-dimensional coordinates of the shortest-distance portion.

For example, when values of “X=30, Y=40, Z=30” are acquired as the coordinates of the shortest-distance portion, it is determined whether virtual coordinates corresponding to “X=30, Y=40, Z=30” exist in the acquired virtual coordinates.

6 FIG.A 17 18 41 7 15 41 In the example illustrated in, an apparent position of the shortest-distance portion (the right hand), that is, the apparent position of the right-hand imageis situated in the range of the angle of view. In such a case, there exist virtual coordinates corresponding to the shortest-distance portion. In other words, if the apparent shortest-distance portion really exists, an image of the apparent shortest-distance portion will be captured by the camera, and the image of the apparent shortest-distance portion will be included in the captured image. In other words, if there exist virtual coordinates corresponding to the shortest-distance portion, the apparent position of the shortest-distance portion will be situated in the range of the angle of view.

41 41 9 FIG.A On the other hand, the apparent position of the shortest-distance portion is situated outside of the range of the angle of viewin the example illustrated in. In such a case, there exist no virtual coordinates corresponding to the shortest-distance portion. In other words, if there exist no virtual coordinates corresponding to the shortest-distance portion, the apparent position of the shortest-distance portion will be situated outside of the range of the angle of view.

36 41 As described above, the angle-of-view determining sectiondetermines whether there exist virtual coordinates corresponding to a shortest-distance portion, and determines, according to a result of the determination, whether an apparent position of the shortest-distance portion is situated in the range of the angle of view.

41 108 20 6 109 109 6 6 FIGS.A andB When it has been determined that the apparent position of the shortest-distance portion is in the range of the angle of view(Yes in Step), the avatar imageis superimposed on an image on the stereoscopic display(Step). In other words, the process of Stepis performed in a state illustrated in, for example,.

37 20 5 20 6 4 4 5 17 20 17 5 37 20 17 5 20 17 5 4 The superimposition processorsuperimposes the avatar imageof a result of interaction of the usersuch that the avatar imagecorresponds to an apparent position of the interaction result displayed on the stereoscopic display, the apparent position being an apparent position as viewed from the user. In the present embodiment, the usersandare giving pseudo-handshakes to each other. Thus, the interaction result is the right handwith which a pseudo-handshake is given. Further, the superimposition of the avatar image of the interaction result is superimposing the avatar imageof the right handof the user. Thus, the superimposition processorsuperimposes the avatar imageof the right handof the usersuch that the avatar imagecorresponds to an apparent position of the right handwith which a pseudo-handshake is given by the user, the apparent position being an apparent position as viewed from the user.

37 15 5 4 6 20 15 20 17 5 18 6 Specifically, first, the superimposition processorcalculates a position of superimposition performed with respect to the captured image, the superimposition position corresponding to the apparent position of the result of interaction of the user, the apparent position being an apparent position as viewed from the user, the interaction result being displayed on the stereoscopic display. In other words, a superimposition position for the avatar imageon the captured imageis calculated such that the avatar imagecorresponds to the apparent position of the right handof the user(the apparent position of the right-hand image) displayed on the stereoscopic display.

37 15 30 37 20 15 20 28 37 20 28 37 36 37 20 15 First, the superimposition processoracquires the captured imagefrom the image acquisition section. Further, the superimposition processoracquires the avatar imageto be superimposed on the captured image. For example, the avatar imageis stored in the storagein advance, and the superimposition processoracquires the avatar imagefrom the storage. Further, the superimposition processoracquires the virtual coordinates of the shortest-distance portion from the angle-of-view determining section. Then, the superimposition processorsuperimposes the avatar imageon the captured imageusing the acquired virtual coordinates of the shortest-distance portion as a superimposition position.

6 FIG.B 17 3 15 20 15 20 18 2 16 4 18 5 4 5 4 19 4 20 5 15 4 5 5 4 5 For example, in the example illustrated in, the shortest-distance portion is the right handin the space, and coordinates of a lower portion of the captured imageare acquired as virtual coordinates of the shortest-distance portion. Thus, the avatar imageis superimposed on the lower portion of the captured image. The superimposition processing is performed such that the avatar imagecorresponds to the apparent position of the right-hand imagein the space. For example, when the right handof the useroverlaps an apparent right-hand imageof the userand it looks like the usersandare giving pseudo-handshakes to each other, as viewed from the user, the right-hand imageof the useroverlaps the avatar imageof the hand of the userin the captured imageand it also looks like the usersandare giving pseudo-handshakes to each other, as viewed from the user. In other words, contents viewed by the userare consistent with contents viewed by the user.

38 15 10 20 15 38 37 15 20 15 10 20 15 15 20 10 6 FIG.B The display controllercontrols display of the captured imageon the non-stereoscopic display, where the avatar imageis superimposed on the captured image. Specifically, the display controlleracquires, from the superimposition processor, the captured imageon which the avatar imageis superimposed. Then, display of the captured imageon the non-stereoscopic displayis controlled, where the avatar imageis superimposed on the captured image. This results in the captured imageon which the avatar imageis superimposed being displayed on the non-stereoscopic display, as illustrated in.

38 14 6 Further, the display controllercontrols display of the captured imageon the stereoscopic display.

38 14 30 14 6 14 6 Specifically, the display controlleracquires the captured imagefrom the image acquisition section. Then, display of the captured imageon the stereoscopic displayis controlled. This results in the captured imagebeing displayed on the stereoscopic display.

14 106 6 110 110 7 7 8 8 FIGS.A andB orA andB When it has been determined that the shortest-distance pixel is situated at the image-outline border of the captured image(Yes in Step), a marker indicating a direction of the shortest-distance portion is presented on the stereoscopic display(Step). In other words, the process of Stepis performed in a state illustrated in, for example,.

46 37 17 42 42 46 7 8 FIGS.B andB In the present embodiment, an arrowis superimposed to be displayed by the superimposition display sectionas the marker indicating the direction of the shortest-distance portion. For example, in the examples illustrated in, the right handis situated outside of the range of the angle of view(on the lower side), and the shortest-distance portion (a wrist) is situated in a lower edge portion in the range of the angle of view. In this case, the shortest-distance portion is situated in a lower direction. In other words, the arroworiented downward is superimposed to be displayed.

6 111 37 15 5 6 17 42 18 6 15 47 15 7 8 FIGS.B andB An alert indicating “beyond the image outline” is presented on the stereoscopic display(Step). In the present embodiment, the superimposition processorsuperimposes an alert image on the captured imagewhen a result of interaction of the useris not displayed on the stereoscopic display. In the examples illustrated in, the right handis situated outside of the range of the angle of view. Thus, the right-hand imageis not displayed on the stereoscopic display. In other words, the interaction result is not displayed. In such a case, the alert image is superimposed on the captured image. In the present embodiment, a wordindicating “Warning” is superimposed on the captured imageas the alert image. Of course, another word or drawing may be superimposed as the alert image.

41 108 6 112 112 9 9 FIGS.A andB When it has been determined that the apparent position of the shortest-distance portion is not situated in the range of the angle of view(No in Step), a marker indicating an apparent direction of the shortest-distance portion is presented on the stereoscopic display(Step). In other words, the process of Stepis performed in a state illustrated in, for example,.

15 37 15 18 41 17 15 17 15 15 18 6 41 4 17 15 9 9 FIGS.A andB In the present embodiment, when the calculated superimposition position is not in a displayed-image range for the captured image, the superimposition processorsuperimposes the marker on the captured image. In the example illustrated in, the apparent position of the right-hand imageis situated outside of the range of the angle of view. Thus, there exist no virtual coordinates of the right handin the captured image. However, the virtual coordinates of the right handcan also be defined in such a case. In other words, the virtual coordinates are not in a coordinate range for the captured image, but can be defined as coordinates situated outside of the coordinate range for the captured image. Specifically, the apparent position of the right-hand imageis situated in a forward direction (on the side of the stereoscopic display) in the range of the angle of view, as viewed from the user. Thus, the virtual coordinates of the right handare situated outside on the lower side of the coordinate range for the captured image.

37 15 15 10 20 10 If the superimposition processorcalculates a superimposition position on the basis of the virtual coordinates defined as described above, the superimposition position not being situated in the captured imagewill be calculated. Thus, the superimposition does not exist in the displayed-image range for the captured imageon the non-stereoscopic display. In other words, the avatar imageis not displayed on the non-stereoscopic display.

15 45 15 45 17 15 17 15 45 18 41 4 17 15 45 15 45 20 46 17 42 45 46 15 9 FIG.B 6 FIG.A 6 FIG.B 7 8 FIGS.B andB A marker is superimposed on the captured imagein such a case. In the present embodiment, an arrowis superimposed on the captured imageas the marker. The arrowis oriented toward a direction of the virtual coordinates of the right handwith respect to the coordinate range for the captured image. In the example illustrated in, for example,, the virtual coordinates of the right handare situated outside on the lower side of the coordinate range for the captured image. Thus, the arrowis also oriented downward. Likewise, when, for example, the apparent position of the right-hand imageis situated on the right of the range of the angle of view, as viewed from the user(on a front side in), the virtual coordinates of the right handare situated outside on the left of the coordinate range for the captured image. Thus, the arroworiented leftward is superimposed to be displayed on the left in the captured image. Note that, in order to distinguish the arrowdisplayed when the avatar imageis not superimposed to be displayed (the example illustrated in) from the arrowdisplayed when the right handis situated outside of the range of the angle of view(the examples illustrated in), the arrowsandmay be displayed in different colors or shapes. Further, the marker superimposed on the captured imageis not limited to an arrow, and any marker may be superimposed.

1 5 10 4 6 6 10 6 10 In the telepresence systemaccording to the present embodiment, a result of interaction of the userwho uses the non-stereoscopic displayis superimposed on an image of the userwho uses the stereoscopic display, as described above. This makes it possible to perform a proper interaction in terms of telepresence technology even when the stereoscopic displayand the non-stereoscopic displaycommunicate with each other, where there is asymmetry between respective display states of the stereoscopic displayand the non-stereoscopic display.

A sense of spatial co-presence and a sense of reality of a person are important factors for telepresence. Eye contact made with a communication partner is one of factors for attaining a sense of spatial co-presence. A sense of co-presence is attained by talking to a communication partner in a state in which eye contact has been made with the communication partner. Various approaches have been proposed in order to make eye contact with a communication partner in telepresence.

Further, stereoscopic viewing of a video is one of factors for attaining a sense of reality of a person. When a user views a video of a communication partner stereoscopically, this makes it possible to attain a sense of reality of a person, the sense of reality causing the user to feel as if the person is with the user. Various approaches have been proposed in order to provide stereoscopic viewing of a video in telepresence. On the other hand, if both users use their stereoscopic displays, there will be a need for a special system. This results in more strict constraints on what can be done. Thus, any telepresence system including stereoscopic displays used by respective users is not yet available as a realistic solution.

1 Thus, a telepresence system including a non-stereoscopic display as a display used by one of users has been proposed. In other words, a telepresence system having an asymmetric configuration in which one of users uses a stereoscopic display to stereoscopically views another of the users, and the other of the users uses a non-stereoscopic display to planarly views the one of the users, has been proposed. The telepresence system having such a configuration imposes less strict constraints on technologies and costs, and is actually used in various scenes. The telepresence systemaccording to the present technology is also an example of such a telepresence system including a stereoscopic display and a non-stereoscopic display.

In a telepresence system having such a configuration, a higher quality of communication is ensured for a user who views a stereoscopic display than for a user who views a non-stereoscopic display. Thus, such a telepresence system is expected to be used when there is a superior-and-inferior relationship between users.

For example, the telepresence system is expected to be used in a scene in which an insurance product or a financial product is explained. In this case, a customer who receives an explanation views a stereoscopic display, and an explainer views a non-stereoscopic display. When the customer stereoscopically views an image of the explainer, this enables the customer to receive an explanation about a product while feeling as if the explainer is with the customer. Further, the telepresence system is expected to be used in a scene of a handshake session of, for example, an idol. When a fan stereoscopically views an image of an idol, this enables the fan to have experience in feeling as if the fan is actually shaking hands with the idol.

10 10 11 11 FIGS.A,B,A, andB 10 10 11 11 FIGS.A,B,A, andB 20 15 However, in such a telepresence system, a user who views a non-stereoscopic display is not allowed to check a relative position of his/her hand.schematically illustrate a telepresence system according to a comparative example.illustrate, as a comparative example, a telepresence system in which the avatar imageor a marker is not superimposed to be displayed on the captured image.

4 5 5 4 4 5 4 5 5 4 4 5 4 5 5 4 4 10 10 FIGS.A andB In the telepresence system according to the comparative example, the usercan also cause a position of his/her hand to coincide with an apparent position of a hand of the userwhen, for example, the userextends his/her hand to give a pseudo-handshake to the user, since the userstereoscopically views a video of the user. Thus, the usersandcan give pseudo-handshakes to each other. On the other hand, the userdoes not know where to extend his/her hand to give a pseudo-handshake to the userwhen the userextends his/her hand, as illustrated in, since the userplanarly views a video of the user. In other words, the userdoes not know where to extend his/her hand so that it looks like the hand of the useroverlaps the hand of the user, as viewed from the user.

5 4 5 5 5 4 5 4 It is also necessary that the userrespond to a handshake suggestion given by the userin such a case. Thus, the userextends, from necessity, his/her hand toward a position estimated by the userhimself/herself while estimating that the usercould shake hands with the userby extending his/her hand around a certain region. Consequently, an apparent position of the hand of the useris shifted from a position of the hand of the userin most cases. This results in being unable to shake hands.

1 20 15 20 5 4 5 5 4 20 19 4 5 20 4 5 5 4 4 In the telepresence systemaccording to the present technology, the avatar imageis superimposed to be displayed on the captured imagesuch that the avatar imagecorresponds to an apparent position of a hand of the user, the apparent position being an apparent position as viewed from the user. Consequently, the userunderstands a relative position of his/her hand. For example, when the apparent position of the hand of the useris situated lower than the position of the hand of the user, the avatar imageis accordingly superimposed to be displayed on a lower portion of the right-hand imageof the user. The userchecks a position of the avatar imageto understand that the apparent position of his/her hand is situated lower than the position of the hand of the user. Then, the usermoves his/her hand upward to correct the apparent position of his/her hand. This enables the userto cause the apparent position of his/her hand to coincide with the position of the hand of the user(to give a pseudo-handshake with the user).

20 20 5 20 5 20 Further, the avatar imageof a body part used to perform interaction is superimposed as an interaction result. When, for example, a pseudo-handshake is given as interaction, the avatar imageof a hand is superimposed to be displayed. This enables the userto intuitively understand that the superimposed and displayed avatar imagerepresents the hand of the userand that the avatar imagemoves according to movement of his/her hand.

15 20 37 15 Furthermore, a marker is superimposed on the captured imagewhen a superimposition position for the avatar imagethat is calculated by the superimposition processoris not in the displayed-image range for the captured image.

5 10 5 5 4 Consequently, the userunderstands that a virtual position of his/her hand is situated outside of a displayed-image range for the non-stereoscopic display. Further, this enables the userto correct the virtual position of his/her hand by moving his/her hand, such that the virtual position of his/her hand is in the displayed-image range. In other words, this enables the userto cause the apparent position of his/her hand to get close to the position of the hand of the user.

45 45 5 5 4 In the present embodiment, the arrowis superimposed to be displayed as a marker, and the arrownotifies, using its direction, a direction in which the virtual position of the hand of the useris situated outside of a screen. Consequently, the userunderstands in which direction his/her hand is to be moved, and can cause the apparent position of his/her hand to get close to the position of the hand of the usermore smoothly.

15 37 5 6 5 6 4 5 4 5 5 4 Further, an alert image is superimposed to be displayed on the captured imageby the superimposition processorwhen the hand of the useris not displayed on the stereoscopic display. Consequently, the userunderstands that an image of his/her hand is not displayed on the stereoscopic displayviewed by the user. In other words, the userno longer believes that the useris seeing the hand of the userdespite the fact that the hand of the useris not seen by the user. This makes it possible to perform communication more smoothly.

37 5 6 14 5 5 6 Furthermore, the superimposition processordetermines whether the hand of the useris displayed on the stereoscopic display, on the basis of information regarding the edge portion of the captured image. This enables the userto accurately determine whether the hand of the useris displayed on the stereoscopic display.

30 14 11 31 5 11 11 14 1 Moreover, the image acquisition sectionacquires the captured imagecaptured by the stereo camera, and the depth acquisition sectionacquires depth information regarding the userthat is detected by the stereo camera. In other words, the stereo cameraacquires both the captured imageand depth information. This makes it possible to obtain the telepresence systemhaving a simpler configuration, compared to, for example, when a ranging sensor used to acquire depth information is separately provided.

6 4 1 Further, an autostereoscopic display is used as the stereoscopic displayin the present embodiment. This enables the userto use the telepresence systemwith less effort without wearing, for example, dedicated glasses.

38 15 10 14 6 20 15 4 5 5 4 20 Furthermore, the display controllercontrols each of display of the captured imageon the non-stereoscopic displayand display of the captured imageon the stereoscopic display, where the avatar imageis superimposed on the captured image. This enables the userto check an image of the user. Further, this enables the userto check an image of the userand the avatar image.

The present technology is not limited to the embodiments described above, and can achieve various other embodiments.

12 12 FIGS.A andB 12 12 FIGS.A andB 12 12 FIGS.A andB 5 20 5 37 20 17 5 20 17 5 4 49 2 50 49 10 5 49 50 schematically illustrates the userperforming another interaction. The avatar imagemay be superimposed to be displayed not only when the userextends his/her hand to give a pseudo-handshake but also in response to any other interaction. In the example illustrated in, the superimposition processorsuperimposes the avatar imageof the right handof the usersuch that the avatar imagecorresponds to an apparent position of the right handwith which the userperforms pseudo-pointing at a target object, the apparent position being an apparent position as viewed from the user. In the example illustrated in, a documentis situated in the spaceas a target object, and a document imagethat is an image of the documentis displayed on the non-stereoscopic display. The userperforms pseudo-pointing at the documentby pointing at the document image.

20 17 5 15 37 17 20 1 4 5 5 4 5 12 FIG.B 6 6 FIGS.A andB Further, the avatar imageof the right handof the useris superimposed to be displayed on the captured image. The superimposition processorcalculates virtual coordinates of the right hand, and the avatar imageis superimposed on the calculated virtual coordinates to perform the superimposition and displaying as illustrated in, as in the case of the example illustrated in, for example,. For example, this makes it possible to smoothly explain about an insurance product or a financial product using a document when the telepresence systemis used for the explanation of the insurance product or the financial product. Specifically, for example, this results in the usersandproperly understanding which line in the document the useris pointing at, and thus in the usersandcommunicating with each other with consistency.

13 FIG. 13 FIG. 10 4 6 10 30 53 15 4 6 6 14 5 5 53 37 20 53 10 schematically illustrates a third-person-viewpoint image being displayed on the non-stereoscopic display. Images of the userand the stereoscopic displaythat are captured at a viewpoint of a third person may be displayed on the non-stereoscopic display. In the example illustrated in, the image acquisition sectionacquires a third-person-viewpoint imagethat corresponds to the captured imagein which the userand the stereoscopic displayappear, the stereoscopic displaydisplaying thereon the captured imageof the user. Further, with respect to the userappearing in the third-person-viewpoint image, the superimposition processorsuperimposes the avatar imageon the third-person-viewpoint imagedisplayed on the non-stereoscopic display.

7 6 4 4 6 7 7 53 4 6 7 4 6 The camerais arranged on the right of the stereoscopic display, as viewed from the user, such that the userand the stereoscopic displayare in a range of an angle of view of the camera. This results in the cameracapturing the third-person-viewpoint imagein which both the userand the stereoscopic displayappear. Of course, the cameramay be arranged at any position or in any orientation that makes it possible to perform image-capturing on both the userand the stereoscopic display.

53 54 6 53 55 9 6 The third-person-viewpoint imageincludes a stereoscopic-display imagethat is an image of the stereoscopic display. Further, the third-person-viewpoint imageincludes a user imagethat is an image of the user imagedisplayed on the stereoscopic display.

37 20 53 20 55 53 17 53 20 56 20 55 56 55 56 55 20 56 6 6 FIGS.A andB Further, the superimposition processorsuperimposes the avatar imageon the third-person-viewpoint image. The avatar imageis superimposed with respect to the user imageincluded in the third-person-viewpoint image. Specifically, virtual coordinates of the right handin the third-person-viewpoint imageare calculated, and the avatar imageis superimposed to be displayed on the calculated virtual coordinates, as in the case of the example illustrated in, for example,. Further, an avatar imageof an arm is superimposed to be displayed such that the avatar imageis connected to the user image. For example, a drawing of the arm is displayed as the avatar imageof the arm. In other words, with respect to a position of the user image, the avatar imageof the arm is superimposed to be displayed such that the arm is extended from the user image, and the avatar imageof the hand is superimposed to be displayed on a tip of the avatar imageof the arm.

5 5 4 20 20 6 6 FIGS.A andB This enables the userto check how the userlooks to the user. Further, this makes it easy to grasp a height of the avatar image, compared to when the avatar imageis displayed at a normal viewpoint, as in the case of, for example,.

6 10 6 4 10 5 1 4 5 The present technology can be applied to not only stationary displays such as the stereoscopic displayand the non-stereoscopic display, but also head-mounted displays (HMDs). For example, an HMD that enables stereoscopic viewing is provided instead of the stereoscopic display, and the userwears the HMD enabling stereoscopic viewing. Further, a commonly used HMD (an HMD that makes it possible to view an image planarly) is provided instead of the non-stereoscopic display, and the userwears the commonly used HMD. The telepresence systemhaving such a configuration can also be provided. This makes it possible to provide the userwith, for example, a greater sense of realism or a greater sense of reality of the user.

17 5 20 17 5 20 32 17 5 20 17 A real image of the right handof the usermay be displayed as the avatar image. Further, point cloud or the like of the right handof the usermay be displayed as the avatar image. In this case, for example, the 3D data convertercalculates three-dimensional coordinates of the right hand, and the point cloud is generated on the basis of the calculated three-dimensional coordinates. Consequently, the userunderstands more intuitively that the avatar imagecorresponds to his/her right hand.

17 5 1 17 20 A dedicated recognition engine may determine whether a shortest-distance portion is the right handof the user. For example, the telepresence systemincludes a recognition engine used to recognize a hand, and whether a shortest-distance portion is a hand is determined. This makes it possible to calculate virtual coordinates of the right handmore accurately, and to superimpose the avatar imagemore accurately.

9 FIG.B 9 FIG.A 45 18 18 41 45 15 18 41 45 15 18 41 45 15 45 17 5 17 In the example illustrated in, a superimposition position for the arrowmay be calculated according to the apparent position of the right-hand image. For example, when the apparent position of the right-hand imageis situated outside on the lower side of the range of the angle of view, as in the case of, the arrowis superimposed to be displayed near a lower border of the captured image. In this case, for example, when the apparent position of the right-hand imageis situated outside in a rightward region on the lower side of the range of the angle of view, the arrowmay be displayed near a rightward region at the lower border of the captured image. Likewise, when the apparent position of the right-hand imageis situated outside in a leftward region on the lower side of the range of the angle of view, the arrowmay be displayed near a leftward region at the lower border of the captured image. In this case, the superimposition position for the arrowis calculated on the basis of, for example, virtual coordinates of the right hand. Consequently, the userunderstands a virtual position of his/her right handmore precisely.

6 10 7 11 23 1 1 FIG. A portion of or all of the functions of the stereoscopic display, non-stereoscopic display, camera, and stereo cameraillustrated inmay be included in the information processing apparatus. Further, the telepresence systemmay be implemented by a plurality of computers or a single computer.

14 FIG. 500 23 500 501 502 503 505 504 506 507 508 509 510 505 is a block diagram illustrating an example of a hardware configuration of a computerby which the information processing apparatuscan be implemented. The computerincludes a CPU, a ROM, a RAM, an input/output interface, and a busthrough which these components are connected to each other. A display section, an input section, a storage, a communication section, a drive, and the like are connected to the input/output interface.

506 507 507 506 508 508 510 511 509 509 509 500 The display sectionis a display device using, for example, liquid crystal or EL. Examples of the input sectioninclude a keyboard, a pointing device, a touchscreen, and other operation apparatuses. When the input sectionincludes a touchscreen, the touchscreen may be integrated with the display section. The storageis a nonvolatile storage device, and examples of the storageinclude an HDD, a flash memory, and other solid-state memories. The driveis a device that can drive a removable recording mediumsuch as an optical recording medium or a magnetic recording tape. The communication sectionis a modem, a router, or another communication apparatus that can be connected to, for example, a LAN or a WAN and is used to communicate with another device. The communication sectionmay perform communication wirelessly or by wire. The communication sectionis often used in a state of being separate from the computer.

500 508 502 500 503 502 Information processing performed by the computerhaving the hardware configuration described above is performed by software stored in, for example, the storageor the ROM, and hardware resources of the computerworking cooperatively. Specifically, the information processing method according to the present technology is performed by loading, into the RAM, a program included in the software and stored in the ROMor the like and executing the program.

500 511 500 500 For example, the program is installed on the computerthrough the removable recording medium. Alternatively, the program may be installed on the computerthrough, for example, a global network. Moreover, any non-transitory storage medium that is readable by the computermay be used.

The information processing method according to the present technology may be executed and the information processing system and the information processing apparatus according to the present technology may be implemented by a plurality of computers working cooperatively, the plurality of computers being a plurality of computers connected through, for example, a network to be capable of communicating with each other. In other words, the information processing method according to the present technology can be executed not only in a computer system that includes a single computer, but also in a computer system in which a plurality of computers operates cooperatively.

Note that, in the present disclosure, the system refers to a set of components (such as apparatuses and modules (parts)) and it does not matter whether all of the components are in a single housing. Thus, a plurality of apparatuses accommodated in separate housings and connected to each other through a network, and a single apparatus in which a plurality of modules is accommodated in a single housing are both the system.

The execution of the information processing method according to the present technology by the computer system includes, for example, both the case in which the acquisition of a captured image, the acquisition of depth information, the superimposition of an interaction result, the superimposition of a marker, the superimposition of an alert image, the calculation of superimposition position, the display control, and the like are executed by a single computer; and the case in which the respective processes are executed by different computers. Further, the execution of the respective processes by a specified computer includes causing another computer to execute a portion of or all of the processes and acquiring a result of it. In other words, the information processing method according to the present technology is also applicable to a configuration of cloud computing in which a single function is shared and cooperatively processed by a plurality of apparatuses through a network.

10 The telepresence system, contents displayed on the stereoscopic display and the non-stereoscopic display, the information processing apparatus, the respective processing flows, and the like described with reference to the respective figures are merely embodiments, and any modifications may be made thereto without departing from the spirit of the present technology. In other words, for example, any other configurations or algorithms for purpose of practicing the present technology may be adopted.

At least two of the features of the present technology described above can also be combined. In other words, the various features described in the respective embodiments may be combined discretionarily regardless of the embodiments. Further, the various effects described above are not limitative but are merely illustrative, and other effects may be provided.

Note that the present technology may also take the following configurations.

(1) An information processing apparatus, including:

an image acquisition section that acquires a captured image of a first user who uses a stereoscopic display; a depth acquisition section that acquires depth information regarding a second user who uses a non-stereoscopic display that communicates with the stereoscopic display; and a superimposition processor that superimposes a result of interaction of the second user on the captured image of the first user on the basis of the depth information, the captured image of the first user being displayed on the non-stereoscopic display.(2) The information processing apparatus according to (1), in which the image acquisition section acquires a captured image of the second user performing the interaction, and on the basis of the depth information, the superimposition processor superimposes an avatar image of the result of the interaction of the second user on the captured image of the first user such that the avatar image corresponds to an apparent position of the interaction result displayed on the stereoscopic display, the apparent position being an apparent position as viewed from the first user.(3) The information processing apparatus according to (2), in which the superimposition of the interaction result includes superimposing an avatar image of a body part of the second user on the captured image of the first user, the body part being used by the second user to perform the interaction.(4) The information processing apparatus according to (3), in which the image acquisition section acquires the captured image of the second user giving a pseudo-handshake to the first user displayed on the non-stereoscopic display, and the superimposition processor superimposes an avatar image of a hand of the second user on the captured image of the first user such that the avatar image corresponds to an apparent position of the hand of the second user, the apparent position being an apparent position as viewed from the first user, the pseudo-handshake being given with the hand of the second user.(5) The information processing apparatus according to (3) or (4), in which the image acquisition section acquires the captured image of the second user performing pseudo-pointing at a target object displayed on the non-stereoscopic display, and the superimposition processor superimposes an avatar image of a hand of the second user on the captured image of the first user such that the avatar image corresponds to an apparent position of the hand of the second user, the apparent position being an apparent position as viewed from the first user, the pseudo-pointing being performed with the hand of the second user.(6) The information processing apparatus according to any one of (2) to (5), in which on the basis of the depth information, the superimposition processor calculates a position of superimposition performed with respect to the captured image of the first user, the superimposition position corresponding to the apparent position of the result of the interaction of the second user, the apparent position being an apparent position as viewed from the first user, the interaction result being displayed on the stereoscopic display.(7) The information processing apparatus according to (6), in which the superimposition processor superimposes a marker on the captured image of the first user when the calculated superimposition position is not in a displayed-image range for the captured image of the first user.(8) The information processing apparatus according to any one of (1) to (7), in which the superimposition processor superimposes an alert image on the captured image of the first user when the result of the interaction of the second user is not displayed on the stereoscopic display.(9) The information processing apparatus according to (8), in which on the basis of information regarding an edge portion of a captured image of the second user performing the interaction, the superimposition processor determines whether the result of the interaction of the second user is displayed on the stereoscopic display.(10) The information processing apparatus according to any one of (1) to (9), in which the image acquisition section acquires a third-person-viewpoint image that is a captured image in which the first user and the stereoscopic display appear, the stereoscopic display displaying thereon a captured image of the second user, and with respect to the second user displayed on the stereoscopic display appearing in the third-person-viewpoint image, the superimposition processor superimposes the result of the interaction of the second user on the third-person-viewpoint image displayed on the non-stereoscopic display.(11) The information processing apparatus according to any one of (1) to (10), in which the image acquisition section acquires a captured image of the second user that is captured by a stereo camera, and the depth acquisition section acquires the depth information regarding the second user, the depth information being detected by the stereo camera.(12) The information processing apparatus according to any one of (1) to (11), in which the stereoscopic display is an autostereoscopic display.(13) The information processing apparatus according to any one of (1) to (11), in which the image acquisition section acquires a captured image of the second user, and the information processing apparatus further includes a display controller that controls each of display of the captured image of the first user on the non-stereoscopic display and display of the captured image of the second user on the stereoscopic display, the captured image of the first user being a captured image on which the result of the interaction of the second user is superimposed.(14) An information processing method, including: acquiring, by a computer system, a captured image of a first user who uses a stereoscopic display; acquiring, by the computer system, depth information regarding a second user who uses a non-stereoscopic display that communicates with the stereoscopic display; and superimposing, by the computer system, a result of interaction of the second user on the captured image of the first user on the basis of the depth information, the captured image of the first user being displayed on the non-stereoscopic display.(15) A recording medium that records therein a program that causes a computer system to execute an instruction used to perform a process including: acquiring a captured image of a first user who uses a stereoscopic display; acquiring depth information regarding a second user who uses a non-stereoscopic display that communicates with the stereoscopic display; and superimposing a result of interaction of the second user on the captured image of the first user on the basis of the depth information, the captured image of the first user being displayed on the non-stereoscopic display.

1 telepresence system 4 user 5 user 6 stereoscopic display 7 camera 8 user image 9 user image 10 non-stereoscopic display 11 stereo camera 14 captured image 15 captured image 16 right hand 17 right hand 18 right-hand image 19 right-hand image 20 avatar image 23 information processing apparatus 30 image acquisition section 31 depth acquisition section 32 3D data converter 33 shortest-distance-pixel determining section 34 border determination section 35 2D-space-projection section 36 angle-of-view determining section 37 superimposition processor 38 display controller 41 angle of view 42 angle of view 45 arrow 46 arrow 47 word 49 document 53 third-person-viewpoint image