Display Control Device and Display Control Method

The present invention relates to a display control apparatus and a display control method for controlling display on a head-mounted display.

There is widespread use of image display systems allowing a user wearing a head-mounted display to view a target space from a free viewpoint. For example, there is known electronic content of which the target space is a virtual three-dimensional space and which realizes virtual reality (VR) with images displayed on the head-mounted display in a manner reflecting the user's line-of-sight direction. Utilizing the head-mounted display can enhance the sense of immersion in the visual world and improve the operability of applications such as games. In addition, walk-through systems which allow the user wearing the head-mounted display and physically moving around to walk around virtually in a space displayed as imagery have also been developed.

The deeper the sense of immersion in the visual world, the harder it becomes for the user to focus on real-world situations. It might happen that the user has unwittingly moved to an unexpected position or remains unaware of the changing surroundings such an approaching person. In view of this, there have been proposed techniques which enable the head-mounted display to have a function of letting the real-world situation be viewed temporarily during display of content imagery, with a view to avoiding hazards and enhancing convenience (e.g., see PTL 1 and PTL 2).

PCT Patent Publication No. WO2017/022769

Japanese Patent Laid-open No. 2018-10657

In recent years, VR technologies have been applied to diverse scenes of the real world, as testified by widespread use of the term “metaverse.” If such situations accelerate from now on, it might happen that the state of stay in a VR space becomes commonplace. The longer the period of staying in the VR space, the larger the number of scenes that require returning from the VR space to the real world will become, in addition to specific situations requiring a short-term avoidance of hazards during an electronic game. Thus, there is a need for technologies that permit a seamless alternation between the VR space and the real world without compromising the world view of the VR space.

The present invention has been made in view of the above circumstances. An object of the invention is therefore to provide a technology that implements the seamless alternation between the VR space and the real world without compromising the world view of the VR space.

In solving the problem above and according to one embodiment of the present invention, there is provided a display control apparatus. This display control apparatus is characterized to include an output part configured to output data of an image representing a virtual space to a head-mounted display, a virtual event detection part configured to detect that a predetermined event related to a virtual object in the virtual space has occurred, and a mode switchover part configured to switch, upon occurrence of the event, an output target to be output by the output part from the data of the image representing the virtual space to data of a see-through image representing a real world.

According to another embodiment of the present invention, there is provided a display control method. This display control method is characterized to include a step of outputting data of an image representing a virtual space to a head-mounted display, a step of detecting that a predetermined event related to a virtual object in the virtual space has occurred, and a step of switching, upon occurrence of the event, an output target to be output by an output part from the data of the image representing the virtual space to data of a see-through image representing a real world.

It is to be noted that suitable combinations of the above constituent elements and the expressions of the present invention, when converted between a system, a computer program, a recording medium that records the computer program in a readable manner, and a data structure, among others, are also effective as embodiments of the present invention.

According to the present invention, it is possible to implement a seamless alternation between the VR space and the real world without compromising the world view of VR space.

1 FIG. 100 100 102 104 104 106 102 108 100 108 100 is an exemplary external view of a head-mounted display. In this example, the head-mounted displayincludes an output mechanism partand a wearing mechanism part. The wearing mechanism partincludes a wearing bandworn by a user to secure an apparatus around the head. The output mechanism partincludes a housingshaped to cover the left and right eyes of the user when the head-mounted displayis worn by the user. Inside the housingis a display panel directly facing the eyes when the head-mounted displayis worn.

108 100 100 100 100 100 Also inside the housingare eyepieces for magnifying images, the lenses being interposed between the display panel and the user's eyes when the head-mounted displayis worn. In addition, the head-mounted displaymay further have speakers or earphones mounted at positions corresponding to the user's ears when the head-mounted displayis worn. Moreover, the head-mounted displaymay incorporate motion sensors that detect translational and rotational motions of the user's head wearing the head-mounted displayand, by extension, the positions and postures of the head at different points in time.

100 110 108 110 100 The head-mounted displayis also equipped with a stereo cameraat the front of the housing. The present embodiment provides a mode in which moving images captured by the stereo cameraare displayed with a small delay to let the user view the unmodified real space in the user's facing direction. In the description that follows, this mode will be referred to as a “see-through mode.” For example, the head-mounted displayautomatically enters see-through mode during a period in which content images are not displayed.

100 110 108 110 This allows the user to check the status of the surroundings before, after, and during interruption of content execution without taking off the head-mounted display. It is to be noted that, although the illustrated example indicates the stereo camerabeing mounted at the lower front of the housing, the camera position is not limited to any specific arrangement. Also, cameras other than the stereo cameramay be mounted alternatively.

110 The images captured by the stereo cameramay also be used as content images. For example, a virtual object may be combined with captured images and displayed in a manner reflecting the position, posture, and motion of a real object in the field of view of the camera, thereby realizing augmented reality (AR) or mixed reality (MR). In addition, regardless of whether or not to include captured images into what is being displayed, it is also possible to analyze the captured images and determine the position, posture, and motion of the object to be rendered, through the use of the result of the analysis.

100 For example, captured images may be subjected to stereo matching to extract corresponding points of a subject from the images. The distance to the subject may then be obtained by use of the principle of triangulation. Alternatively, known techniques such as visual simultaneous location and mapping (SLAM) may be used to acquire the position and posture of the head-mounted displayrelative to the surrounding space and, by extension, the position and posture of the user's head. Such processes permit rendering and display of a virtual world in a field of view reflecting the viewpoint position and line-of-sight direction of the user.

2 FIG. 1 100 200 100 10 200 8 20 8 is a view depicting an exemplary configuration of an image display system in the present embodiment. An image display systemincludes the head-mounted displayand a content processing apparatus. The head-mounted displayis connected to the content processing apparatusin a wired or wireless manner. The content processing apparatusis connected by wire or wirelessly with a network, and is connected to a content providing servervia the network.

200 100 200 100 200 The content processing apparatusis an information processing apparatus that basically generates display images by processing content and transmits the generated images to the head-mounted displayfor display thereon. Typically, the content processing apparatusidentifies the viewpoint position and line-of-sight direction of the user on the basis of the position and posture of the user's head wearing the head-mounted display, and generates display images in a field of view reflecting the identified viewpoint direction and line-of-sight direction. For example, while advancing an electronic game, the content processing apparatusgenerates and displays images representing a virtual world that constitutes a game stage.

200 200 20 200 100 200 20 1 200 8 20 It is to be noted that the details and display image viewpoints of the content processed by the content processing apparatusare not limited to anything specific. Also, the content processing apparatusmay generate display images on the basis of the image data streamed from the content providing server. It is to be noted that the content processing apparatusmay be incorporated in the head-mounted display. Further, the manner of sharing functions between the content processing apparatusand the content providing serveris not limited to anything specific. Moreover, in the image display system, the number of content processing apparatusesconnected to the networkand the number of network-connected content providing serversconnected thereto are not limited to any number.

20 200 20 200 200 100 For example, the content providing servermay acquire the movements and operations of multiple users from each content processing apparatusand have what is acquired reflected in the movements of what are generally called avatars representing the users in a virtual world. In addition, the content providing servermay distribute three-dimensional data of the virtual world to each content processing apparatus. In this case, each content processing apparatusgenerates display images from the transmitted three-dimensional data at a predetermined rate and causes the head-mounted displayto display the generated images. In this manner, multiple users located remotely from each other can get together, communicate with one another, and participate in one game by means of their avatars.

3 FIG. 100 300 100 20 302 302 a b depicts an exemplary image displayed on the head-mounted display. In this example, a display imagerepresents a virtual space including multiple avatars from a panoramic viewpoint (third-person perspective). In the real world, each user wearing the head-mounted displaylaunches an application and performs operations to participate in the virtual space provided by the content providing server. This causes the user's own avatar (e.g., an avatar) to appear in the virtual space, allowing the user to communicate with the avatars of other participating users (e.g., an avatar).

302 302 304 a a The avatarof the user may move in a manner reflecting the user's movement in the real world, or may be moved by the user operating a controller, not depicted. For example, the user moves the own avatarin the virtual space, talks to another avatar, or grabs an item (e.g., a glass) existing in the virtual space. The conversation with other avatars may be performed in real voice or in text. The processes related to the control and display of such a virtual space have been conceived of and already implemented diversely depending on the details of content and thus will not be discussed further in detail.

100 In any case, there are no particular limitations on the types and purposes of the virtual space to be displayed, the types of viewpoints, the presence or absence of avatars, the participation or non-participation of other users, and the like in the present embodiment. In the description that follows, the virtual space that includes user-manipulable objects exemplified by avatars will be referred to as a “VR space.” Also, the state where images representing the VR space are displayed on the head-mounted displayand where the user's own object or the user's viewpoint is manipulated in the VR space will be referred to as “staying in the VR space.”

In recent years, diverse places such as the workplace, meeting rooms, banquet halls, eateries, streets, and event venues have been created as a VR space such as the illustrated one in which people can stay and buy things as if in the real world. That is, the scenes for utilizing the VR space are not just limited to temporary entertainment such as electronic games but continue to expand in the real world. This trend is expected to intensify from now on.

100 Under such circumstances, the longer the period in which the user stays in the VR space constituting a virtual workplace or a virtual banquet hall, the higher the possibility of experiencing matters to be dealt with in the real world such as consumption of food and drink, elimination, receiving a visitor, or taking calls during that period. Every time such a necessity comes up, the user needs to remove and again wear the head-mounted display, which can be bothersome and reduce the quality of the user experience in the VR space.

100 Further, as illustrated, with regard to the VR space which provides a scene for social interaction between the users, it is not unusual to have a parallel state in which the users staying in the VR world prepare and enjoy food and drink and have conversations with each other also in the real world, for example, thus exhibiting similar behaviors in both the VR world and the real world. In this case, however, the user wearing the head-mounted displayand having to grope for food and drink in the real world may feel stressed unlike in the real party.

100 100 If the above-mentioned video see-through function is incorporated in the head-mounted displayto let the display target be switched at a desired timing, the user can at least verify the real-world situation without taking off the head-mounted display. However, if VR imagery is cut off by a controller, for example, the timeline in the VR space is abruptly interrupted, which can compromise its world view. Particularly in a VR space where the users interact with each other, they may have feelings of discomfort upon noticing a sudden disappearance of a participating member.

100 In view of this, in the present embodiment, when the action of an object in the VR space meets a predetermined condition, which is a trigger, the display on the head-mounted displayis switched to a real-world image. That is, a link from the VR space to the real world is set on a given object in the VR space. This extends considerably means for expressing a temporary exit of a user from the VR space where the user is staying. As a result, even when the user stays in the VR space over an extended time period, the user can deal seamlessly with matters to attend to in the real world as needed while the world view of the VR space is maintained. Further, the other users do not have a sense of strangeness when noticing an exit of the relevant user from the shared VR space.

4 FIG. 200 200 222 224 226 230 230 228 228 232 234 236 238 240 depicts an internal circuit configuration of the content processing apparatus. The content processing apparatusincludes a central processing unit (CPU), a graphics processing unit (GPU), and a main memory. These components are interconnected via a bus. The busis further connected with an input/output interface. The input/output interfaceis connected with a communication part, a storage part, an output part, an input part, and a recording medium driving part.

232 234 236 100 238 100 240 The communication partincludes a peripheral device interface such as a universal serial bus (USB) interface or an Institute of Electrical and Electronics Engineers (IEEE) 1394 interface and a network interface for a wired or wireless local area network (LAN), for example. The storage partincludes a hard disk drive and a nonvolatile memory. The output partoutputs data to the head-mounted display. The input partreceives input of data from the head-mounted display. The recording medium driving partdrives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

222 200 234 222 234 226 232 224 224 222 236 226 The CPUcontrols the content processing apparatusas a whole by executing an operating system stored in the storage part. Also, the CPUexecutes various programs (e.g., a VR game application and the like) read from the storage partor the removable recording medium and loaded into the main memoryor downloaded via the communication part. The GPUhas a function of a geometry engine and a function of a rendering processor. The GPUperforms rendering processes in accordance with rendering instructions from the CPU, and outputs the result of the rendering to the output part. The main memoryincludes a random access memory (RAM) and stores the programs and data necessary for processing.

5 FIG. 100 100 120 122 124 126 128 128 130 130 132 134 110 136 138 depicts an internal circuit configuration of the head-mounted display. The head-mounted displayincludes a CPU, a main memory, a display part, and an audio output part. These components are interconnected via a bus. The busis further connected with an input/output interface. The input/output interfaceis connected with a communication partincluding a wireless communication interface, a motion sensor, the stereo camera, a gaze point detector, and a proximity sensor.

120 100 128 124 126 200 122 120 The CPUprocesses the information acquired from the components of the head-mounted displayvia the bus, and supplies the display partand the audio output partwith display image data and audio data obtained from the content processing apparatus. The main memorystores the programs and data required by the CPUfor processing.

124 100 124 124 100 The display partincludes a display panel such as a liquid display panel or an organic electroluminescent (EL) panel, and displays images in front of the eyes of the user wearing the head-mounted display. The display partmay realize a stereoscopic view by displaying a pair of stereo images in regions corresponding to the right and left eyes. The display partmay further include a pair of lenses for expanding the user's viewing angle, the lenses being interposed between the display panel and the user's eyes when the user wears the head-mounted display.

126 100 132 200 132 134 100 The audio output partincludes speakers or earphones mounted at positions corresponding to the ears of the user wearing the head-mounted display, the speakers or earphones allowing the user to hear sound. The communication partis an interface that transmits and receives data to and from the content processing apparatus. The communication partimplements communication by use of a known wireless communication technology such as Bluetooth (registered trademark). The motion sensorincludes a gyro sensor and an acceleration sensor, and acquires the angular velocity and acceleration rate of the head-mounted display.

1 FIG. 110 110 As depicted in, the stereo camerais constituted by a pair of video cameras for capturing images of the surrounding real space in a field of view corresponding to the user's viewpoint from left and right viewpoints. Items in the line-of-sight direction of the user (typically, directly in front of the user) are caught in the frames of moving images captured by the stereo camera.

136 124 136 136 The gaze point detectortracks the spot gazed at by the user on the display panel of the display part. The gaze point detectorincludes a mechanism that emits reference light such as infrared rays to the user's eyeballs and cameras that capture the light reflected from the eyeballs. The gaze point detectorthus derives the gaze point from the orientation of the pupils. It is to be noted that, in the present embodiment, as will be discussed later, the images captured by these cameras are also used to acquire the state of the eyes and eyelids.

138 100 134 110 136 138 100 200 132 The proximity sensordetects the state in which the user wears the head-mounted display. Measurement by the motion sensor, images captured by the stereo camera, information regarding the eye images and the gaze point acquired by the gaze point detector, and information acquired by the proximity sensorregarding the head-mounted displaybeing worn or taken off are transmitted as needed to the content processing apparatusvia the communication part.

6 FIG. 6 FIG. 200 200 20 100 200 100 20 depicts functional blocks of the content processing apparatus. It is to be noted that, as discussed above, the content processing apparatushas the functions of processing, on its own, various kinds of content such as electronic games and of establishing communication with the content providing serverand head-mounted display.mainly indicates the blocks of the functions related to switchover between a mode for displaying VR space images (referred to as a “VR space mode”) and the see-through mode. Of the functions of the content processing apparatus, those illustrated in the figure may be implemented as the display control apparatus. Further, at least some of the functions illustrated in the figure may be incorporated in the head-mounted displayor in the content providing server.

6 FIG. 4 FIG. 222 224 226 234 The multiple functional blocks incan be implemented in a hardware configuration that includes, for example, the CPU, the GPU, the main memory, and the storage partdepicted in, or can be implemented by software using computer programs that implement the functions of the multiple functional blocks. It will thus be understood by those skilled in the art that these functional blocks can be implemented by hardware alone, by software alone, or by a combination of both in diverse forms and that the implementation is not limited to a particular form.

200 50 52 54 56 58 60 62 64 The content processing apparatusincludes a VR space image acquisition partthat acquires images representing the VR space, a see-through image generation partthat generates images representing the real world, a VR event detection partthat detects an event in the VR space from among the events triggering mode switchover, a real event detection partthat detects an event in the real world, an event setting information storage partthat stores setting information regarding an event triggering the mode switchover, an event setting processing partthat receives the settings of the event triggering the mode switchover from the user, a mode switchover partthat controls the mode switchover, and an output partthat outputs display images.

50 50 20 50 20 3 FIG. In the VR space mode, the VR space image acquisition partacquires at a predetermined rate the display image representing the VR space, such as one illustrated in. The VR space image acquisition partmay process content on its own and generate the display image of the VR space as a result of the process, or generate the display image on the basis of information supplied from the content providing serverwith regard to the VR space. As another alternative, the VR space image acquisition partmay decode or modify as needed the display image supplied from the content providing server, to generate the display image.

100 3 FIG. In the VR space, there exists at least a virtual object that gazes at the image representing the VR space, i.e., a virtual object manipulated by the user wearing the head-mounted displayto which the image is output. This virtual object is not limited to anything specific; it may be an avatar such as one of those depicted in, or a character or a vehicle manipulated by the user in an electronic game. In the description that follows, such a virtual object will be referred to as a “self-object.”

52 52 110 100 The display image of the VR space may be expressed from the viewpoint of the self-object (from a first-person perspective), or from some other suitable viewpoint. In the see-through mode, the see-through image generation partgenerates at a predetermined rate the display image representing what the real world is like in real time. Specifically, the see-through image generation partacquires instantaneously the moving image captured by the stereo cameraof the head-mounted display, and modifies the acquired image into the display image.

54 54 In the VR space mode, the VR event detection partdetects the occurrence of a predetermined event related to an object in the VR space. This event has been set as the trigger for switchover from the VR space mode to the see-through mode. For example, the VR event detection partdetects the self-object executing a predetermined action such as exiting from a door that exists in the VR space or grabbing an item found in the VR space.

54 It is to be noted that the events detected by the VR event detection partare not limited to the actions belonging strictly to the self-object. The events may also be such occurrences as the self-object moving into a predetermined region in the VR space, the self-object coming into contact with a predetermined object, or the self-object doing an act that causes another object to produce a predetermined change.

54 54 50 As another alternative, the VR event detection partmay detect events caused by motions of multiple virtual objects including those of other users. The VR event detection partmay detect the occurrence of a predetermined event by referencing images acquired by the VR space image acquisition part, or by identifying changes of the self-object in the VR space on the basis of the operations or actions of the user.

56 56 100 136 56 The real event detection partdetects the occurrence of a predetermined event in the real world. For example, the real event detection partdetects that a predetermined state related to the user's eyes has continued for at least a predetermined time period, such as the eyelids being closed, the eyes blinking repeatedly, or the eyes gazing in a predetermined direction relative to the display screen. The movement of the user's eyelids or the position of the gaze point relative to the display screen is obtained by acquiring from the head-mounted displaythe information regarding the captured image of the eyes or the gaze point obtained by the gaze point detector. Besides detecting the eye-related events, the real event detection partmay detect, as an event, that a part of the user's body observable by sensors, not depicted, has entered a state that meets a predetermined condition.

56 100 134 100 110 56 Further, the real event detection partmay detect predetermined changes occurring in the position and posture of the user in the real world. The position and posture of the user wearing the head-mounted displayare obtained by use of measurement taken by the motion sensorof the head-mounted displayor the result of analyzing the images captured by the stereo camera. Such functions may be utilized by the real event detection partto detect that the user has performed a predetermined gesture.

56 56 138 100 56 As another alternative, the real event detection partmay use a timer, not depicted, to detect the elapse of a predetermined time period following mode switchover. The real event detection partmay also detect the occurrence of an event upon obtaining the point in time, detected by the proximity sensor, at which the head-mounted displayis worn. Further, the real event detection partmay detect a predetermined operation performed by the user on an input apparatus, not depicted. Such events in the real world may be used, in combination with an event in the VR space, as the trigger for switching to the see-through mode. Alternatively, an event in the real world may be used singly as the trigger for switchover from the see-through mode to the VR space mode.

58 54 56 The event setting information storage partstores the setting information regarding the events to be detected by the VR event detection partor the real event detection partas the trigger for switchover between the VR space mode and the see-through mode. The event setting information may include, for example, information set in common for all pieces of content beforehand, information set by content creators for individual pieces of content in a manner reflecting their details, and information set by individual users.

60 54 56 58 The event setting processing partreceives from the user the settings of the events to be detected by the VR event detection partor by the real event detection partas the trigger for switchover between the VR space mode and the see-through mode, and stores the received event settings as the setting information into the event setting information storage part. For example, the user can discretionally make the settings of the action desired to be performed by the self-object and the object related to the action as the trigger for switchover from the VR space mode to the see-through mode. This allows the user to better memorize the trigger for mode switchover and to create dramatic effects accommodating the user's intentions and preferences in the VR space.

54 62 56 62 62 When the VR event detection partdetects the occurrence of an event in the VR space, the mode switchover partswitches display from the VR space mode to the see-through mode. Also, when the real event detection partdetects the occurrence of an event in the real world, the mode switchover partswitches display from the see-through mode to the VR space mode. Alternatively, the mode switchover partmay use the occurrence of an event in the real world for switchover from the VR space mode to the see-through mode.

62 50 64 62 52 64 62 64 In VR space mode, the mode switchover partacquires the data of the display image representing the VR space from the VR space image acquisition part, and outputs the acquired data to the output part. Further, in the see-through mode, the mode switchover partacquires the data of the display image representing the real world generated by the see-through image generation part, and outputs the acquired data to the output part. It is to be noted that, in the see-through mode, the mode switchover partmay output both the data of the display image representing the VR space and the data of the display image representing the real world to the output part.

64 62 64 60 The output partoutputs the display image data output from the mode switchover part, to the head-mounted display for display thereon. Also, the output partoutputs the data of a setting screen through which the event setting processing partreceives event-related settings from the user, to the head-mounted display for display thereon.

64 64 64 In the see-through mode, the output partmay integrate the display image representing the real world with the display image representing the VR space to generate the final display image. For example, the output partmay put a real-world display image in a partial region of the display image of the VR space to generate the final display image. Alternatively, the output partmay overlay a translucent real-world display image onto a VR space display image background to generate the final display image.

64 100 64 64 The output partmay change the audio data output to the head-mounted display, upon display mode switchover. For example, the output partmay output the audio of the VR space in the VR space mode, and output the voice and environmental sound of the real world in the see-through mode. As another alternative, in the see-through mode, the output partmay lower the volume of the sound in the VR space to let the voice and environmental sound of the real world reach directly the user's ears.

64 100 64 100 100 Further, in the see-through mode, the output partmay output to the head-mounted displaythe audio from such devices as a mobile phone used by the user in the real world. For example, in a case where the user selects the see-through mode to speak on the phone, the output partmay communicably connect to the phone to acquire audio data from the communicating party in the call, and output the acquired audio data to the head-mounted display. This enables the user still wearing the head-mounted displayto have a call with a party unrelated to the VR space.

54 In order to implement the embodiment described above, preferably, the event to be detected by the VR event detection partis set in consideration of the purpose of switching from the VR space mode to the see-through mode. For example, in a case where the action of answering an incoming call or placing a call in the real world is purposed for switching to the see-through mode, the action of the self-object grabbing a virtual phone as an object in the VR space is set as the event to be detected.

200 50 50 In this manner, associating the action of the self-object in the VR space with the user's own action in the real world permits intuitive operations and reduces the possibility of compromising the world view of the VR space due to a sudden incoming call. It is to be noted that, in this case, when the content processing apparatusacquires from the phone a notification of an incoming call, the VR space image acquisition partmay change the display image of the VR space in a manner indicating the incoming call in the VR space. For example, the VR space image acquisition partmay flash a virtual phone indicator in the VR space to represent the incoming call in the real world.

62 64 64 When the self-object grabs the object of the virtual phone in the VR space, the mode switchover partswitches to the see-through mode, and notifies the output partof the details of the event that has occurred. This allows the output partin the see-through mode to recognize that the audio output is switched to the voice of the communicating part, on the basis of the setting information. When the event to be detected in the VR space is associated beforehand with the handling of audio data upon switching to the see-through mode, it is possible to control not only the voice of the communicating partner but also the above-described audio output.

7 FIG. 310 312 314 312 316 is a view depicting a perceptive image of a manner of switchover from the VR space mode to the see-through mode, the switchover being triggered by an event in the VR space. The user staying in a VR spacemanipulates a self-objectto communicate with another objector to participate in an electronic game. In the process, it may occur to the user that there is a matter to attend to in the real world. In that case, the user moves the self-objectto the proximity of a doorset as the trigger for mode switchover.

316 312 54 54 312 312 316 312 316 312 316 312 316 When the relation between the doorand the self-objectmeets a predetermined condition, the VR event detection partdetermines that an event has occurred. For example, the VR event detection partdetermines that the predetermined condition is satisfied when the self-objectenters any one of states including a state in which the self-objectcomes into a predetermined range around the door, a state in which the self-objectcomes into contact with the door, a state in which the self-objectperforms the action of opening the door, and a state in which the self-objectgoes out of the door.

312 316 312 316 312 20 200 Whether a contact has occurred between the self-objectand some other object such as the doorcan be determined by application of what is generally called a collision detection technique in which calculations using a bounding box, for example, are performed to determine whether collision has taken place between objects in a common electronic game. The display of the VR space depicts the self-objectexiting from the door. In the case of a VR space where multiple users are staying, an action like this of the self-objectis reflected by the content providing serverin three-dimensional information regarding the VR space, the information being distributed to the content processing apparatusof each user for visual recognition.

54 316 200 312 62 100 110 320 318 7 FIG. At the time when the VR event detection partdetermines the occurrence of the event related to the door, the content processing apparatusof the user manipulating the self-objectcauses the mode switchover partto switch to the see-through mode. This allows the head-mounted displayof the user to display a real-time image of the real world captured by the stereo camera.depicts what it is like in the own room(of a user) of the real world.

310 320 316 310 312 Such display changes enable the user to have a sensation of moving from the VR spaceto the own roomthrough the door. This makes it possible for the user to feel that the world of the VR spacecontinues without interruption while attending to matters in the real world in the see-through mode. The extent of compromising the world view of the VR space or spoiling the fun for the user is reduced considerably. In addition, the other users staying in the VR space can recognize the exit of the user of the objectin an intuitive manner.

316 312 312 318 312 7 FIG. It is to be noted that the dooras the trigger for mode switchover may be replaced by, for example, a bathroom door or an emergency exit in the VR space. For example, if the user feels having to go to the bathroom in the real world, the user may cause the self-objectto go to the bathroom in the VR space so as to switch to the see-through mode. In the user's consciousness, there is thus a seamless connection between the VR space and the real world. It is to be noted that, although the self-objectand the userhave the same appearance in, needless to say, the self-objectmay alternatively be a distorted character in the VR space.

62 56 100 56 100 Further, the mode switchover partmay switch from the see-through mode back to the VR space mode when a predetermined condition is met regarding any one of the above-described items detectable by the real event detection part; such items include the user's body part such as the eyes, the position and posture of the user, an elapsed time, the state of the head-mounted displaybeing worn, and operations on the input apparatus. For example, when the real event detection partdetects, as an event, that the user goes to the bathroom while wearing the head-mounted displayand returns to the own room, the user's movement in the real world connects seamlessly to the return to the VR space.

8 FIG. 330 332 334 334 330 a b depicts another example of a manner of switchover from the VR space mode to the see-through mode, the switchover being triggered by an event in the VR space. In this example, a case of representing a VR space where the user is having a party with other users as viewed from a first-person perspective is assumed. A display imagethus excludes the face and the like of the self-object and indicates only a handof the self-object along with objectsandas the other users in this example. Obviously, the display images for the other users exhibit the object of the relevant user looking at the display image.

The self-object may be caused to behave as if consuming food and drink placed on a virtual table while communicating by real voice and the like with the other participating members. In this manner, a state similar to that of the real party is created in the virtual world. Meanwhile, each user may prepare foods and drinks within their reach in the real world when taking part in the party in the VR space. This allows the users to actually enjoy eating and drinking, thereby boosting the festive mood of the party.

336 336 332 336 54 It is assumed here that a glassin the VR space is set as the trigger for mode switchover. When the relation between the glassand the self-object (the hand) meets the predetermined condition involving the glassgetting grabbed by the self-object, the VR event detection partdetermines that an event has occurred. The above-mentioned collision detection technique, for example, can also be applied to this case.

336 200 330 62 100 340 110 342 344 At the time when the event related to the glassis determined to have occurred, the content processing apparatusof the user looking at the display imagecauses the mode switchover partto switch to the see-through mode. This allows the head-mounted displayof the user to display a real-world imagecaptured in real time by the stereo camera. The user can view, grab by hand, and consume foodand drinkprepared in the real world, so that there is no stress or hazard associated with fumbling.

7 FIG. 330 The manner described above has the purpose of permitting eating and drinking in the real world without burdening or jeopardizing the user. That often means a few seconds or so is enough for the stay in the see-through mode. Thus, even if the self-object is not made to exit from the VR space, unlike the case in, the other users are not very likely to have a sense of strangeness. As a result, the convenience of eating and drinking in the real world can be improved for the user viewing the display imageas well as for the other users while the world view of the VR space is maintained.

62 62 With the above characteristics taken into account, the mode switchover partmay count the time following the switchover to the see-through mode and, upon elapse of a predetermined time period, return automatically to the VR space mode. In this case, given that there are differences in the time required to verify and consume different kinds of foods and drinks, the objects set as the trigger for mode switchover may be differentiated by the type of actual food and drink. The mode switchover partmay change the time period in which the see-through mode is maintained, depending on the type of the object as the trigger.

9 FIG. 58 350 352 352 252 352 352 a b c d a depicts an exemplary data structure of the setting information stored in the event setting information storage part. Event setting informationhas a structure in which each assumed purposeis associated with an eventas the trigger for switchover from the VR space mode to the see-through mode, with an eventas the trigger for switchover from the see-through mode back to the VR space mode, and with detailsof audio control in the see-through mode. It is to be noted that the assumed purposesare used as the preconditions for setting the other items.

100 A setting example in the second row assumes that the user goes to the bathroom. The exit of the self-object from the door in the VR space is set as the trigger for switching to the see-through mode. The return of the head-mounted displayto its initial position is set as the trigger to return to the VR space mode. In this case, the volume of the audio in the VR space is lowered in the see-through mode such that the sound in the real world will be easier to listen to.

A setting example in the third row assumes that the user consumes food and drink while staying in the VR space. The self-object grabbing a glass in the VR space is set as the trigger for switching to the see-through mode. The elapse of a predetermined time period is set as the trigger to return to the VR space mode. In this case, in the see-through mode, the volume of the audio in the VR space is left unchanged such that conversations in the virtual party or the like will continue.

200 100 A setting example in the fourth row assumes that the user makes a call on the phone in the real world. The self-object grabbing a mobile phone in the VR space is set as the trigger for switching to the see-through mode. Ending the call is set as the trigger for returning to the VR space mode. The end of the call may be detected either by the content processing apparatusreceiving a notification to that effect from the phone or upon occurrence of a predetermined event such as a gesture made by the user. In this case, the voice of the communicating party in the call in the see-through mode is acquired from the phone and output such that the user can speak while wearing the head-mounted display.

350 9 FIG. It is to be noted that the setting details and the data structure of the event setting informationinare only examples and may be replaced by specific values, subdivided into more detailed items, or omitted in some items. Also, although only the events occurring in the VR space are set as the trigger for switchover from the VR space mode to the see-through mode in the illustrated example, this is not limitative of how the present invention is embodied.

That is, events in the real world, such as the user's body part, the position and posture of the user, and operations on the input apparatus, may be used also as the conditions for switchover from the VR space mode to the see-through mode. As another alternative, conditions may be set for the events in both the VR space and the real world such that, when the conditions for both worlds are met simultaneously, switchover from the VR space mode to the see-through mode is performed.

350 As discussed above, at least part of the event setting informationmay be set discretionally by the user. For example, the time having to go to the bathroom to leave the virtual space or the time needed to eat and drink varies depending on the individual users. Thus, in particular, in a case where the elapsed time after the switchover to the see-through mode is used as the condition for switching to the VR space mode, the user may discretionally set that time. This reduces the stress stemming from an excessively prolonged or unduly short see-through mode.

10 FIG. 10 FIG. 3 FIG. 60 60 380 depicts an exemplary setting screen displayed by the event setting processing partfor receiving event-related settings from the user. Because the assumed purpose of exiting a mode and the object that can be used as the trigger for mode switchover vary depending on the type and details of VR space, the event setting processing partmay preferably receive the user settings specific to each VR space actually created. A setting screeninassumes the presence of the VR space illustrated in.

60 380 60 380 60 382 384 The event setting processing partdisplays an image that represents the structure of a given VR space and the objects therein as the setting screenfor the user who participates in this VR space for the first time or for the user who requests changes in the setting details. The event setting processing partmay change as needed the viewpoint of the VR space displayed on the setting screen, in accordance with the user's operations to change the viewpoint. Further, the event setting processing partdisplays a message“Select object” and a cursoroperable by the user in an overlaid manner.

382 60 384 380 60 58 Using the message, the event setting processing partprompts the user to select from the screen the object to be set as the trigger for switchover from the VR space mode to the see-through mode. Using the cursor, the user selectively designates the object to be set as the trigger from among various objects displayed on the setting screen. The event setting processing partreceives the designated object and adds it to the setting information stored in the event setting information storage part.

60 9 FIG. In practice, the event setting processing partmay receive user settings with respect to various items such as those indicated in. When the user discretionally sets the event constituting the trigger for mode switchover, it is possible to implement the switchover more easily. This also makes it easier to memorize the event required for the switchover. Depending on how the settings are made, it is also possible to execute the switchover to the see-through mode in an entertaining manner.

200 100 50 20 11 FIG. Explained below are the operation of the content processing apparatusimplemented in the above-described configuration.is a flowchart of processing steps performed by the content processing apparatus for controlling switchover between the VR space mode and the see-through mode. For example, this flowchart is started when the user wears the head-mounted displayand performs operations to request participation in the VR space. In response, the VR space image acquisition partacquires VR space-related data typically from the content providing serverto generate display images consecutively.

54 56 62 64 58 10 62 50 64 64 100 12 54 14 Meanwhile, the VR event detection part, the real event detection part, the mode switchover part, and the output partread the setting information related to the event from the event setting information storage part(S). The mode switchover partacquires the image data representing the VR space from the VR space image acquisition part, and supplies the acquired data to the output part. This causes the output partto start outputting the data to the head-mounted display(S). The VR event detection partmonitors the occurrence of the event that has been set as the trigger for switchover from the VR space mode to the see-through mode (S).

54 16 54 18 56 14 For example, the VR event detection partdetermines, at a predetermined rate, whether the event has occurred or not, by calculating the position of the object in contact with other objects or within the VR space and matching what is calculated against the condition set as the occurrence of the event. During a period in which the display is not terminated by the user ending the application, for example (Y in S), the VR event detection partkeeps monitoring the occurrence of the event (N in S). Alternatively, as discussed above, the real event detection partmay also monitor the occurrence of the event in the real world in Sand use the occurrence of the event for the basis of mode switchover.

54 18 62 20 52 62 64 64 100 64 100 When the VR event detection partdetects the occurrence of the event (Y in S), the mode switchover partswitches the display to the see-through mode (S). Specifically, the see-through image generation partgenerates a display image representing the real world. The mode switchover partacquires the data thus generated and supplies the output parttherewith. This causes the output partto start outputting the data to the head-mounted display. At this point, on the basis of the event setting information, the output partmay modify the audio being output to the head-mounted display.

56 22 56 100 In the see-through mode, the real event detection partmonitors the occurrence of the event that has been set as the trigger for switchover from the see-through mode to the VR space mode (S). For example, the real event detection partdetermines, at a predetermined rate, whether or not a predetermined condition is met regarding at least one of such items as the state of the user's body part, the position and posture of the user, an elapsed time, the user's operations on the input apparatus, and the state of the head-mounted displaybeing worn.

24 56 26 56 26 62 12 16 24 During a period in which the display is not terminated by the user ending the application, for example (Y in S), the real event detection partkeeps monitoring the occurrence of the event (N in S). When the real event detection partdetects the occurrence of the event (Y in S), the mode switchover partswitches the display to the VR space mode by resuming the output of VR space image data (S). Thereafter, the above steps are similarly repeated. When it becomes necessary to stop the display in the VR space mode or in the see-through mode because the user has terminated the application, for example, the processing is brought to an end (N in S, N in S).

In the above-described embodiment involving the techniques for implementing VR using a head-mounted display, the event to be detected is set for an object or for its movement in the VR space. When that event has occurred, display is switched to an image that represents the real world. This enables the user while wearing the head-mounted display to deal with matters to attend to in the real world, such as eating and drinking, elimination, or receiving a visitor. As a result, the annoyance of wearing and taking off the head-mounted display is reduced. This leads to lowering psychological barriers to wearing the head-mounted display and, by extension, to the participation in the VR space.

Further, the entrance to the real world is created on the side of the VR space. This allows the user to have the sensation of continuously experiencing the virtual world in the VR space while still in the real world, which reduces the possibility of compromising the world view of the VR space. Moreover, the event as the trigger for mode switchover, the condition for switching the display from the real world back to the VR space, and the details of audio control is changed depending on the purpose of switching the display. This makes it possible for the user to attend to matters in the real world without stress and to alternate between the VR space and the real world more naturally and seamlessly in the user's consciousness. Furthermore, any user's exit from a VR space participated in by multiple users can be recognized by the remaining users from the occurrence of the exit in the VR space, so that they can continue their stay in the VR space without having a sense of strangeness.

As described above, the present invention can be applied to various information processing apparatuses including game machines, head-mounted displays, and personal computers as well as to image display systems incorporating any one of these apparatuses.

The present invention relates to a display control apparatus and a display control method for controlling display on a head-mounted display.

1 : Image display system 50 : VR space image acquisition part 52 : See-through image generation part 54 : VR event detection part 56 : Real event detection part 58 : Event setting information storage part 60 : Event setting processing part 62 : Mode switchover part 64 : Output part 100 : Head-mounted display 200 : Content processing apparatus