Detection device and program

This is a Continuation of application Ser. No. 17/482,621 filed Sep. 23, 2021, which is a Continuation of application Ser. No. 15/123,901 filed Sep. 6, 2016, which is a National Stage Application of PCT/JP2014/084715 filed Dec. 26, 2014. The entire disclosures of the prior applications are hereby incorporated by reference herein in their entireties.

The present invention relates to a detection device and the program.

There is an operation input device known in the related art that detects, via a three-dimensional image sensor, a pointing operation performed within space surrounded by a display screen and reference points and thereby enables execution of processing assigned to an object present at a position on the display screen that corresponds to the position at which the pointing operation has been detected within the space. At this operation input device, a pointing operation performed in space with regard to a mark on a calibration image brought up on display at the display screen is detected with the 3-D image sensor and the pointing space is calibrated based upon a plane in alignment the positions where the pointing operation has been detected, the display screen and a reference point (see PTL 1).

PTL 1: Japanese Laid Open Patent Publication No. 2005-321869

There is an issue in that the pointing space cannot be calibrated unless a pointing operation correctly performed on the calibration image is detected.

According to the 1st aspect of the present invention, a detection device comprises: a detection unit that detects a predetermined non-contact operation by a detection reference; and a control unit that changes the detection reference when the predetermined non-contact operation is not detected by the detection reference.

According to the 2nd aspect of the present invention, a detection device comprises: a detection unit that includes a detection reference that detects a non-contact operation; and a control unit that changes the detection reference based on the non-contact operation detected outside the detection reference.

According to the 3rd aspect of the present invention, a program enabling a computer to execute comprises: processing for detecting a predetermined non-contact operation at a detection reference; and processing for changing the detection reference when the predetermined non-contact operation is not detected by the detection reference.

According to the 4th aspect of the present invention, a program enabling a computer to execute comprises: processing for detecting a non-contact operation by a detection reference; and processing for changing the detection reference based on the non-contact operation detected outside the detection reference.

In reference to drawings, the display device achieved in the first embodiment will be described. The first embodiment will be described in reference to an example in which the display device in the embodiment is mounted in a portable telephone. It is to be noted that the display device in the embodiment may be mounted in an electronic apparatus other than a portable telephone such as a portable information terminal device such as a tablet terminal, a wristwatch-type terminal and so on, a personal computer, a music player, a land-line telephone unit or a wearable device.

is a disassembled oblique view of a display deviceandshows part of the display devicein an enlarged side elevation. It is to be noted that for purposes of facilitating the explanation, a coordinate system assuming an X axis, a Y axis and a Z axis is set relative to the display deviceas indicated in the figure. It is to be also noted that the coordinate system set for these purposes does not need to be an orthogonal coordinate system assuming the X axis, the Y axis and the Z axis, and it may instead be a polar coordinate system or a cylindrical coordinate system. In other words, any of these coordinate systems may be set relative to the display device, as long as the X axis is set to extend along the shorter sides of the rectangular display area of the display device, the Y axis is set to extend along the longer sides of the rectangular display area of the display deviceand the Z axis is set to extend along a direction perpendicular to the display area.

The display deviceincludes a bodyhaving installed therein a control unit, a display unit, an image forming optical systemand an operation detector. The display unit, the image forming optical systemand the operation detectorare disposed inside the body. The display unit, configured with, for instance, a liquid crystal display or an organic EL display, includes a plurality of display pixel arrays arranged in a two-dimensional pattern. Under control executed by the control unit, the display unitdisplays an image corresponding to display image data. The image forming optical systemis disposed further upward relative to the display unit(on the + side along the Z direction) over a predetermined distance from the display unit. The image forming optical systemmay be configured by, for instance, layering two micro-lens arrays, each having convex micro-lensesarranged in a two-dimensional array, along the Z direction as clearly indicated in.

The image forming optical systemforms a midair imageof a display image displayed at the display unitin the space above the display device. Namely, an image displayed by the display unitat the display devicecan be viewed by the user of the display deviceas the midair image, floating above the display device. The midair imageincludes a plurality of iconsA (operation buttons) corresponding to operation buttons via which various settings may be selected for the display deviceand instructions for executing various functions of the display devicemay be issued. The iconsA in the embodiment may be set over, for instance, three rows by five columns. It is to be noted that instead of the micro-lens arrays, pinhole arrays or slit arrays may be used to configure the image forming optical system.

The operation detector, disposed further upward (on the + side along the Z direction) relative to the image forming optical system, may be configured with, for instance, a transparent capacitive panel (hereafter will be referred to as a capacitive panel) of the known art. The operation detectorconfigured with a capacitive panel forms an electric field with electrodes constituted of a substantially transparent material. When the user moves his finger or a stylus toward the midair imagein order to operate manipulate adjust the display position of the midair image, the operation detectordetects the position of the finger or the stylus as an electrostatic capacitance value. For instance, it may compare the electrostatic capacitance values detected at the four corners of the capacitive panel so as to detect the position of the user's finger along the X axis and the Y axis based on the electrostatic capacitance values detected at the four corners. The operation detectormay take an electrostatic capacitance detection range, which is a predetermined range set upward relative to itself, as will be described in detail later, and detect the distance (i.e., the position on the Z axis) between the operation detectorand the finger or the stylus within the predetermined detection range based on the electrostatic capacitance values detected at the four corners of the transparent capacitive panel by, for instance, comparing the electrostatic capacitance values detected at the four corners. The midair imagemust be formed via the image forming optical systemso that it occupies a position within the predetermined detection range of the operation detector, and preferably a position around the midway point of the predetermined detection range along the up/down direction. As described above, the operation detectordetects an operation by a user at the display position of the midair imagewith his finger or with a stylus, enabling the user to perform operations on the midair imagewithout having to directly touch the operation detector. It is to be noted that while the following description will be provided by assuming that the user uses his finger to perform an operation at the display position of the midair image, the similar manner will apply to an operation performed via a stylus or the like.

is a block diagram showing the control unit, and the display unitand the operation detectorcontrolled by the control unit, among the components configuring the display device. The control unit, comprising a CPU, a ROM, a RAM and the like, includes an arithmetic operation circuit that controls various structural elements of the display device, including the display unitand the operation detector, based on a control program and executes various types of data processing. The control unitincludes an image generation unit, a display control unit, a calibration unit, a detection reference control unitand a storage unit. The storage unitincludes a nonvolatile memory where the control program is stored, a storage medium where image data to be displayed at the display unitand the like are stored, and the like. It is to be noted that the correspondence between the distance from the surface of the operation detectorto the fingertip and the electrostatic capacitance manifesting when the operation detectordetects the fingertip is stored in advance in the storage unit. Accordingly, as the fingertip is positioned within the predetermined detection range of the operation detector, the operation detectoris able to detect the electrostatic capacitance at the fingertip, and thus detect the position of the fingertip along the Z direction based on the detected electrostatic capacitance and the correspondence stored in the storage unit, as explained earlier.

Based upon image data stored in the storage medium, the image generation unitgenerates display image data corresponding to a display image to be displayed at the display unit. The display control unitbrings up the image corresponding to the display image data generated by the image generation unitat the display unit. As the user performs an operation at the display position of an iconA in the midair image, the display control unitexecutes display image switchover control for the display unitin correspondence to the type of iconA that has been operated. It is to be noted that in response to a user operation performed at the display position of the iconA in the midair image, the display control unitmay execute control other than the display image switchover control for the display unit. For instance, assuming that the user performs an operation at the display position of the iconA in the midair imagewhile a movie image is on display as the display image at the display unit, the display control unitmay execute control for playing the movie image currently displayed at the display unitor for stopping the movie image playback.

The calibration unitexecutes calibration processing in first and second calibration processing modes, as will be described in detail later. The detection reference control unitsets a detection plane, i.e., a detection reference, in the space above the display device, and specifically, at the position of the midair image(or over a specific range around the midair image) within the predetermined detection range of the operation detector. The detection reference control unitalso decides that the user's finger has reached the detection reference based on an electrostatic capacitance value detected by the operation detector. Namely, the detection reference control unitdecides that the user has performed an operation at the display position of the iconA when the finger position (the position on the X axis, the Y axis and the Z axis) corresponding to the value of the electrostatic capacitance detected by the operation detection unitmatches the position of the detection reference having been set. The detection reference control unitmay set the detection reference at a predetermined specific initial position and changes or adjusts the detection reference position based on the results of the calibration processing to be described later. The initial position for the detection reference is stored in advance in the storage unit. It is to be noted that the initial position for the detection reference may be a universal position shared by all users, or a different initial position may be set for each user based on the history of use of the display deviceby the particular user. The initial position for the detection reference and a newly changed position for the detection reference may be set over the entire plane (the X axis and the Y axis) of the operation detectoror may be set over part of the plane. The detection reference having been set when the display devicewas last used, stored in the storage unit, may also be read out and set as the initial position for the detection reference. It is to be noted that the detection reference control unitmay decide that the user has performed an operation at the display position of the iconA when the finger position corresponding to the electrostatic capacitance value detected by the operation detectorroughly matches the position for the detection reference, as well as when the finger position exactly matches the position for the detection reference. A specific range over which the finger position is considered to roughly match the position for the detection reference may be set in advance.

presents an example of a midair imagethat may be displayed by the display deviceandschematically illustrates the positional relationship among the bodyor the operation detector, the midair imageand a detection reference. The midair imageinincludes 15 iconsA set over three rows by five columns, as explained earlier. In, the detection referenceis set near the position of the midair image, and specifically, slightly above the midair image, as shown in the figure, by the detection reference control unit. In, icons in the midair imageare indicated asA marked with bold dotted lines. It is to be noted that while the iconsA all assume a height wise position matching that of the midair imagesince they are part of the midair image, the bold dotted lines indicating the iconsA are offset from the position of the solid line indicating the midair imageinso as to distinguish them from the midair image.

In, the midair imageis formed above the operation detectorin the display device, at a position set apart from the operation detectorby a distance H, whereas the detection referenceis set at a position further upward relative to the operation detector, set apart from the operation detectorby a distance H(H<H). As explained earlier, the operation detectorassumes an electrostatic capacitance detection rangeA set above its surface. In, the electrostatic capacitance detection limit above the operation detectoris indicated with a dotted line, and the interval between the electrostatic capacitance detection limitand the operation detectoris indicated as an electrostatic capacitance detection rangeA. The midair imageand the detection referenceare set so as to take positions within the electrostatic capacitance detection rangeA. It is to be noted that while the detection referenceinis set above the midair image, it may instead be set further downward relative to the midair imageor may be set so as to match the position of the midair image, as long as it takes a position within the electrostatic capacitance detection rangeA of the operation detector. A range outside the area set for the detection referencewithin the detection rangeA will hereafter be referred to as outside a detection reference.

It is to be noted that while the midair imageand the detection referenceare shown inas flat planes ranging parallel to the XY plane, they do not need to be flat planes but instead may be curved planes. The detection referencemay include stages, each corresponding to one of the iconsA, as indicated in, instead of being formed as a flat plane. In other words, the distance between a given iconA and the part of the detection referencecorresponding to the particular icon may be different from the distance between another iconA and the part of the detection referencecorresponding to the other icon. Assuming stages in the detection reference, as described above, is particularly effective when the midair imageis a 3-D image and the positions of the plurality of iconsA are offset relative to one another along the Z direction, i.e., along the up/down direction. By offsetting the positions of the detection reference, each corresponding to one of the iconsA in correspondence to the offset with which the plurality of iconsA in the 3-D midair imageare shifted along the up/down direction, for instance, it is also possible that the distances between the iconsA and the corresponding positions in the detection referenceremain equal.

When the user's fingertip has moved to a point set apart from the operation detectorby the distance H, the operation detectoroutputs a detection output corresponding to the distance H. Based upon the detection output provided by the operation detector, the detection reference control unitdecides that the user's fingertip position has matched the detection referenceand accordingly decides that the user has performed an operation with his fingertip at the display position of the iconA. Through this process, the display devicedetects an operation by the user at the display position of the particular iconA in the midair imageand executes a function corresponding to the iconA having been operated. For instance, it executes display image switchover at the display unit.

While the iconsA take positions set apart from the operation detectorby the distance H, they are displayed as part of the midair imageand for this reason, the visual perception of the display positions of the iconsA in the midair image, i.e., their height H, of one user may be different from that of another user. The visual perception of the display positions of the iconsA of a given user may change depending upon the environment in which he operates the display device. For instance, when the detection referenceis set so as to align with the position of the midair image, a user, having moved his finger toward an iconA in the midair imagein order to perform an operation at the display position of the particular iconA, may feel that there is still some distance between his finger and the iconA although the finger has, in fact, reached the icon, i.e., the detection reference. Under such circumstances, an icon operation will be executed against the user's wishes. Another user, having moved his finger toward an iconA in the midair image in order to perform an icon operation, may feel that his finger has reached the iconA and is performing an operation at the display position of the iconA while his finger is actually still set apart from the iconA, i.e., the detection reference. In this case, no icon operation will be executed against the user's wishes. In either scenario, the user is bound to feel that the icon operation is uncomfortable.

Accordingly, in addition to a midair operation mode that may be set for the midair image, as described earlier, a calibration processing mode can be set in the display devicein the embodiment, so as to customize the response to icon operations to suit the user. In the calibration processing mode, the positional relationship between the midair imageand the detection referenceis set so as to establish a suitable relation best suited to the operational perception or operation characteristics of a specific user, the environment in which the display device is used and the like. As explained earlier, first and second calibration processing modes are available in the display devicein the embodiment. In the first calibration processing mode, calibration processing is executed while the midair image operation mode is not in effect, i.e., while midair image operation mode execution is not underway, whereas in the second calibration processing mode, calibration processing is executed while the midair image operation mode execution, following startup of the display device, is underway. The processing in the first and second calibration processing modes is executed by the calibration unitshown in. It is to be noted that the first or second calibration processing mode may be selected by operating a calibration processing mode selector operation button (not shown) located at the display device, and the control unitmay select and execute the midair image operation mode when neither the first calibration processing mode nor the second calibration processing mode is selected via the calibration processing mode selector operation button. If the display device does not have a calibration processing mode selector operation button, the second calibration processing mode may be a default mode. The first calibration processing mode and the second calibration processing mode will now be described in that order. It is to be noted that the first or second calibration processing mode may be selected via an icon in the midair image instead of via an operation button.

The first calibration processing mode will be described first. As the display deviceis started up and the user operates the calibration processing mode selector operation button to select the first calibration processing mode, the calibration unitinstarts the first calibration processing mode. The image generation unitgenerates display image data, and the display unitbrings up a display image to be used in calibration processing based on the display image data.andshow a midair imageof the display image generated for the calibration processing. The midair imageincludes an iconA for calibration, and a message “execute calibration. Please touch this icon” is superimposed on the calibration iconA. The detection reference control unitmay select an initial setting for the detection referenceso as to set it at a position near the midair image, e.g., slightly above the midair image, as indicated in. The initial position for the detection referencemay instead be set so as to align with the midair imageor may be set slightly below the midair image. It is to be noted that it is not always necessary to display the message “execute calibration. Please touch this icon” when calibration processing is to be executed. For instance, the user, having selected the calibration processing mode may already know a specific operation he needs to perform in the calibration processing mode and in such a case, the message does not need to be displayed.

As the user, following the instructions in the message superimposed on the iconA in the midair image, moves his fingertip F down toward the iconA, as shown in, until the fingertip F reaches the electrostatic capacitance detection rangeA of the operation detectorshown in, the operation detectordetects movement of the user's fingertip F toward the iconA, i.e., the downward movement, as a change in the electrostatic capacitance.

As the fingertip F moves further down and reaches a position indicated by a dotted lineslightly above the detection referencein, the user feels that his fingertip F, having reached the display position of the iconA, has performed a press down operation on the iconA, and moves the fingertip F upward by a predetermined distance. The operation detectordetects the downward movement of the fingertip F described above, i.e., the fingertip F pressing down the iconA, and the subsequent upward movement by the predetermined distance as changes in the electrostatic capacitance. Once the operation detectordetects the fingertip F pressing down the iconA and the subsequent upward movement of the fingertip F by the predetermined distance described above, the detection reference control unitdecides that an operation has been performed at the display position of the iconA. It is to be noted that the reach position to which the user's fingertip F moves downward in order to press down the iconA for an operation at the display position of the iconA before the fingertip F moves upward by the predetermined distance will be referred to as a reach position. Namely, position indicated by the dotted linewill be referred to as the reach position.

Upon deciding that the fingertip F, having reached the reach position, has performed an operation at the display position of the iconA, the detection reference control unitmoves, i.e., changes, the detection referenceto the reach positionas indicated inand also stores the position data indicating the position of the changed detection referenceinto the storage unitin. It is to be noted that the detection reference control unitmay instead move, i.e., change, the detection referenceto a position further upward relative to the reach position, set apart from the reach positionby a predetermined distance d, as indicated inand store position data indicating the position of the changed detection referenceinto the storage unit. This predetermined distance dmay be set to, for instance, approximately 1 mm through 10 mm. The predetermined distance dmay be changed in correspondence to each user who operates the device, and in such a case, the predetermined distance dmay be set based on the distance measured from the tip of the user's finger to the first joint. For instance, the predetermined distance dmay be set to a specific value falling into a range between ¼ and ½ of the distance from the user's fingertip to the first joint.

While the reach positionto which the finger reaches is further upward relative to the detection referencein the example described above, the reach positionmay instead be further downward relative to the detection reference, and in such a case, too, decision-making with regard to the reach positionshould be made as described above and the detection referenceshould be changed based on the reach positionthus determined. While decision-making with regard to the reach positionaligned with the detection referenceis also executed as described above, the detection referencemay not be changed in this case, since the reach positionis aligned with the detection reference.

It is to be noted that when the reach positionis further downward relative to the detection reference, the finger passes through the detection referenceand the detection reference control unitthus decides that the finger has reached the detection referencebased on the detection output provided by the operation detectorbefore the finger reaches the reach position. In this situation, no display switchover takes place at the display unitin the first calibration processing mode. Likewise, when the reach positionis aligned with the detection reference, no display switchover takes place at the display unit. However, when the finger has moved to reach the detection reference, the user may be informed that the finger has reached the detection reference with, for instance, a highlight display of the iconA or by flashing the iconA.

It is to be noted that while the user presses down the iconA as an operation performed at the display position of the iconA in the example described above, the present invention is not limited to this example. Namely, the operation detectormay detect a predetermined non-contact operation performed by the user in relation to the iconA and, in such a case, the detection referencemay be changed based on the location where the specific non-contact operation is performed. The predetermined non-contact operation may be performed by the user by making a gesture operation to touch the iconA. In this case, the detection referencemay be changed based on the position at which the user has made the gesture of touching the iconA. The operation performed by the user making a gesture of touching the iconA may be, for instance, a gesture of swiping the iconA with the user's hand. The position at which the user has performed the operation by making a gesture of touching the iconA may be determined based on the position at which the user's hand, having made the swiping gesture, is determined to have stopped or based on the position at which the user has started making the swiping gesture.

As described above, it is decided in the first calibration processing mode that the user perceives that his finger has performed an operation at the display position of the iconA in the midair imageby detecting a shift from a downward movement to an upward movement by the predetermined distance, and the positional relationship between the detection referenceand the midair imageis changed by changing the detection referencefor the particular user to a position set apart from the finger reach positionby the predetermined distance dalong the upward direction. Namely, based on the operation by the user, which is one type of information on the user operating the device, the positional relationship between the detection referenceand the display position of the midair imageis changed. It is to be noted that when changing the detection referencefor a given user, it is not always necessary to detect the identity of the user operating the device and that the positional relationship between the detection referencebut the display position of the midair imagemay be changed based on the detection output provided by the operation detectorhaving detected the operation described above.

It is to be noted that the detection reference control unit, changing the position of the detection reference, may move the entire detection referenceor may move only the part of the detection referencecorresponding to the iconA operated by the user's finger.

It is to be also noted that in the description provided above, the detection referenceis changed to the position further upward set apart from the reach positionby the predetermined distance din the example presented infor the following reason. Namely, a user operating a touch panel normally touches the touch panel with his finger and gently presses the finger down on the touch panel. When the user performs an operation at the display position of the iconA in the midair image, he tends to simulate this process, i.e., when performing the operation at the display position of the iconA in the midair imagewith his fingertip, the user is likely to move his finger slightly downward before moving it upward by a predetermined distance, instead of immediately moving his finger upward by the predetermined distance after the operation at the display position. Accordingly, the finger reach positiontends to be slightly lower than the position at which the user perceives that the operation has been performed at the display position of the iconA with the finger and the detection referenceis then changed to the position set apart from the reach positionby the predetermined distance dalong the upward direction. However, the extent to which the finger is moved down to press the iconA may be relatively small or the finger may hardly move downward to touch the iconA, for some users, and accordingly, the position of the detection referencemay be changed so as to substantially match the reach position.

In the calibration processing mode, which is selected in order to set the positional relationship between the calibration midair imageand the detection referenceto a suitable relationship suited to the operation characteristics of the user, as explained earlier, the detection referencemay be changed so as to be substantially aligned with the reach position, as described above, or to a position further downward relative to the reach position, in correspondence to the operation characteristics by the user or the display device. For instance, when the reach positionof a given user is above the midair imageand the distance between the reach positionand the upper limitof the electrostatic capacitance detection rangeA is less than the distance between the reach positionand the midair image, the detection referencemay be changed so as to be in alignment with the reach positionor to a position further downward relative to the reach position, since the detection referencechanged to take a position further upward relative to the reach positionwould be too close to the upper limitof the electrostatic capacitance detection rangeA.

It is to be noted that the method through which the reach positionis determined is not limited to that described above, through which the reach positionis determined based on the shift from the downward movement to the upward movement by the predetermined distance, and it may be determined through any of the various other methods to be described below. For instance, the user, perceiving that his finger, having reached the display position of the iconA, has performed the press down operation on the iconA, stops moving his finger downward, i.e., stops pressing down the icon. The detection reference control unitmay decide that the finger has stopped pressing down when there is not almost any change in the value of the electrostatic capacitance detected by the operation detector, and may determine the position at which the finger has stopped pressing down as the reach position. It is to be noted that it may decide that the downward movement has stopped when the value of the electrostatic capacitance detected by the operation detectorhas remained unchanged for a short span of time of, for instance, 0.1 sec through 1 sec. In another method, the reach position may be determined by detecting the velocity vector of the movement of the user's finger, i.e., the finger movement velocity and the finger movement direction, based on a change in the electrostatic capacitance, by detecting that the direction of the finger velocity vector has changed from the downward direction to the opposite direction and that the velocity vector along the opposite direction has reached a predetermined level, and by designating the position taken by the finger when the velocity vector achieving the predetermined level along the opposite direction is detected as the reach position. When the predetermined level for the velocity vector is set substantially equal to 0 in this method, the position taken by the finger when the direction of the velocity vector shifts from downward to the opposite direction, i.e., the lowermost position, will be determined to be the reach position, whereas when the predetermined level is set to a value other than 0 in this method, a position taken by the finger, set apart from the lowermost position by a predetermined distance along the upward direction is determined as the reach position. As explained above, the reach position is set at the lowermost position taken by the finger as the finger is judged by the detection reference control unitto have performed an operation at the display position of the icon or at a position near the lowermost position.

While the reach position is determined in the example presented above in reference to the part of the finger or the stylus coming in contact with the iconA in the midair image, i.e., the position of the fingertip or the position of the lowermost part of the stylus, the reach position may instead be determined in reference to the position of the fingernail tip of the user's finger or in reference to the position of the first joint of the finger. The icon may be also operated with the user's foot or elbow instead of the user's finger, and in such a case, the reach position may be determined in reference to the foot or the elbow. When the icon operation is performed via a stylus, a specific position on the stylus may be marked and the reach position may be determined in reference to the marked position. It is desirable that when the reach position is determined in reference to the position of the first joint of the finger, the position of the stylus mark or the like, the operation detectorbe configured with an image-capturing device or the like, such as that to be described in reference to variation 8, instead of the capacitive panel.

The relationship between the first calibration processing mode described above and the midair image operation mode will be described in reference to the flowchart presented in. After the display deviceis started up, the processing shown in the flowchart presented inis executed based on a program by the control unit. The program is stored in the storage unit. In step S, the first calibration processing mode, selected by the user via the calibration mode selector operation button, is recognized, and then the operation proceeds to step S. In step S, the calibration unitshown instarts the first calibration processing mode before the operation proceeds to step S. In step S, the image generation unitgenerates display image data for the calibration display image, the display control unitbrings up on display at the display unitthe calibration image based on the display image data and the detection reference control unitsets the detection reference at the initial position. The display image at the display unitis the calibration midair imageingenerated by the image forming optical system. The midair imageincludes the iconA with the message “execute calibration. Please touch this icon”. In step S, the operation detectordetects a downward movement of the user's fingertip F, and the operation proceeds to step S.

In step S, the detection reference control unitshown indetermines based on the detection output provided by the operation detector, whether or not the finger has reached the reach position. When an affirmative decision is made in step S, i.e., when it is decided that the finger has reached the reach position, the operation proceeds to step S. When a negative decision is made in step S, i.e., when it is decided that the finger has not become still, the operation waits in standby until an affirmative decision is made. In step S, the detection reference control unitchanges the position of the detection referencebased on the reach positionand stores the position data indicating the position of the changed detection referenceinto the storage unitshown inbefore the operation proceeds to step S. In step S, the operation exits the first calibration processing mode and proceeds to step S. In step S, the midair image operation mode is started, and the operation then proceeds to step S. In step S, the midair imagefor the midair image operation mode shown in, which includes iconsA, is displayed, the position data indicating the position of the detection referencehaving been changed in the first calibration processing mode through step Sare read out from the storage unitand the detection referenceis set at a position near the midair imagebased on the position data, as indicated in. The detection reference, having been set in the first calibration processing mode to suit the operation characteristics of the user is thus used in the midair image operation mode.

As the user moves his finger down toward the midair imagein order to perform an operation at the display position of an iconA, the operation detectorshown indetects the downward movement of the user's finger in step S, and then the operation proceeds to step S. In step S, the detection reference control unitdetermines based on the detection output divided by the operation detectorwhether or not the finger has reached the detection reference. When an affirmative decision is made in step S, i.e., when it is decided that the finger has reached the detection reference, the operation proceeds to step S. When a negative decision is made in step S, i.e., when it is decided that the finger has not reached the detection reference, the operation waits in standby until an affirmative decision is made. In step S, the display control unitswitches the display image at the display unitto a display image corresponding to the iconA having been operated, and then the operation proceeds to step S. In step S, a decision is made as to whether or not an operation has been performed to stop the display device. When an affirmative decision is made in step S, i.e., when an operation has been performed to stop the display device, the display devicestops. When a negative decision is made in step S, however, the operation returns to step S.

As explained above, the detection reference is changed based on an operation by the user so as to change the positional relationship between the midair image and the detection reference in the first calibration mode. Since the detection reference for the midair image operation mode is set at the position of the detection reference having been changed through the first calibration mode, the midair image operation mode can be executed based on the detection reference suitable for the operation characteristics of the user or the operating environment in which the display deviceis operated.

While the first calibration processing mode is executed immediately after the display deviceis started up so as to precede the midair image operation mode in the example described above, the first calibration processing mode may instead be executed following the midair image operation mode. For instance, the user, having experienced frustration while performing an operation at the display position of an iconA in the midair image operation mode, may operate the calibration processing mode selector operation button at the display devicein order to select the first calibration processing mode. In this case, the first calibration processing mode is executed by interrupting the midair image operation mode that is underway and the midair image operation mode is then resumed after the first calibration processing ends. It is to be noted that while the display deviceselects the first calibration mode in response to an operation of the operation button by the user in the example described above, the first calibration processing mode may instead be executed upon detecting signs of annoyance experienced by the user, which is likely attributable to difficulty in performing an operation at the display position of the iconA. The display devicemay, for instance, read the pulse rate of the user (biometrics) and a pulse rate exceeding a predetermined value may be detected as a sign of user discomfort.

Next, the second calibration processing mode will be described in reference toand. It is to be noted that the processing described earlier in reference to the first calibration mode may also be executed, as appropriate, in the second calibration processing mode described below.illustrates the midair imageon display for the midair image operation mode, the detection referenceat the initial position and the finger reach position, whereaspresents a flowchart of the operation executed in the second calibration processing mode. The processing in the flowchart presented inis executed following startup of the display device, by the control unitbased on a program.

In step S, the second calibration processing mode is recognized as the selected mode, and then the operation proceeds to step S. In step S, the midair image operation mode and the second calibration processing mode start concurrently, before the operation proceeds to step S. In step S, the midair imageshown in, which includes the iconsA, is displayed and the detection reference control unitinsets the detection referenceat a predetermined initial position, e.g., at the position taken by the midair imageor at a position slightly above the position taken by the midair image, before the operation proceeds to step S. At this time, a message “calibration executed during icon operation” is briefly displayed in the midair image. However, it is not necessary that this message be displayed.

As the user moves his finger downward in order to perform an operation at the display position of an iconA, the operation detectorstarts detection of finger movement in step S, and then the operation proceeds to step S. In step S, the detection reference control unitdetermines based on the detection output provided by the operation detectorwhether or not the finger moving downward has passed through the detection reference. When an affirmative decision is made in step S, i.e., when the finger moving downward has passed through the detection referenceand has moved further down, the operation proceeds to step S. Finindicates the finger having passed through the detection referenceduring its downward movement. In step S, the detection reference control unit, having decided that the finger Fhas reached the detection reference, i.e., it has passed through the detection reference, executes icon display switchover so as to switch the midair imagein correspondence to the iconA having been operated. The operation then proceeds to step S. In step S, the detection reference control unitdetermines whether or not the finger Fhas reached the reach position, and when an affirmative decision is made, the operation proceeds to step S, whereas when a negative decision is made, the operation is held until an affirmative decision is made. In step S, the detection reference control unitchanges the position of the detection referencebased on the reach position. It is to be noted that the position of the detection referencehaving been changed may be exactly at the user's fingertip or may be above the position of the user's fingertip. In such a case, icon display switchover does not occur since icon display switchover has already been executed once in step S.

When a negative decision is made in step S, i.e., when the finger moving downward has not yet passed through the detection reference, the operation proceeds to step S. In step S, the detection reference control unitdetermines based on the detection output provided by the operation detectorwhether or not the finger has reached the reach position, and when an affirmative decision is made, the operation proceeds to step S. When, on the other hand, a negative decision is made, the operation is held until an affirmative decision is made. A finger Finindicates that the reach positionis in alignment with the detection reference. In step S, the detection reference control unitdetermines based on the detection output from the operation detectorwhether or not the reach positionis in alignment with the detection reference, and when an affirmative decision is made, the operation proceeds to step S, whereas when a negative decision is made, the operation proceeds to step S. In step S, icon display switchover is executed since the reach positionis in alignment with the detection referencebut the detection referenceis not changed.

In step S, with the reach positionlocated above the detection referenceas indicated by a finger Fin, the detection reference control unitchanges the position of the detection referencebased on the reach position, i.e., changes the position of the detection referenceto a point near the reach position, and then the operation proceeds to step S. In this situation, when the position of the detection referencehaving been changed is exactly at the user's fingertip or is further upward relative to the position of the user's fingertip, icon display switchover is executed. When, on the other hand, the detection referencehaving been changed still does not reach the position of the user's fingertip, icon display switchover is not executed.

In step S, a decision is made as to whether or not an end operation has been performed in order to exit the second calibration processing mode, and when an affirmative decision is made, the operation proceeds to step S, whereas when a negative decision is made, the operation returns to step S.

In the second calibration processing mode described above, which is executed concurrently while the midair image operation mode is underway, the user is able to perform an operation at the display position of the midair image by using the detection reference suitable for the user without having to be aware that calibration processing is also in progress. It is to be noted that the first/second calibration processing mode selection does not necessarily need to be made by the user and instead, the display devicemay automatically select either the first calibration processing mode or the second calibration processing mode. It is not necessary that both the first calibration processing mode and the second calibration processing mode be available at the display deviceand the display devicemay use one of these calibration modes.

The display deviceachieved in the first embodiment as described above allows for the following variations.

The display device in variation 1 calculates the velocity or the acceleration of the user's fingertip based on the detection output provided by the operation detector, predicts a finger reach position based on the velocity or the acceleration having been calculated and changes the detection reference based on the predicted reach position. FIG.is a block diagram showing a control unitand a display unitand an operation detectorcontrolled by the control unit, among the structural components in the display deviceachieved in variation 1.