One-handed operation of a device user interface

The present invention relates to technology that enables a user to operate a handheld device having a display, and more particularly to technology that enables a user to use one hand to simultaneously hold and operate a handheld device having a display.

Today's smartphones have touchscreens that not only display information to a user, but also enable the user to supply input to the device, such as selections and data. The user interacts with touchscreens primarily by touching the display with one or more fingers. However in the general case, touch interaction can have many types of gestural inputs (micro-interactions) such as tap, double-tap, slide, tap-slide, tap-hold, swipe, flip, pinch, and the like.

Inputting information in this way is very simple and accurate when the user holds the phone with one hand while interacting with the touchscreen with the other. Quite often, however, the user is holding the smartphone with one hand while the other hand is busy doing other things, for example, carrying a bag or similar. Relatively long ago, when phones were small and had physical buttons only on parts of the front surface, it was relatively easy for most people to use just one hand to both operate the phone and hold it (i.e., one-handed operation). However, with today's large phones, this is very difficult with the touch-based User Interface (UI) and it is consequently quite common that people drop the phone while trying to do so. For this reason, there have been various attempts to solve this problem.

For example, in some of today's smartphones, it is possible to activate one-handed operation through the settings-menu, whereby the complete display content is scaled down to a sub-area of the display which can then be reached by, for example, the thumb of the hand holding the phone. In that solution, the content becomes smaller as the same content fit into only a subset of the display.

In a different approach, US 2014/0380209 A1 describes technology in which the complete display content is shifted so that each displayed item retains its original size but with only parts of the content being shown.

In still another approach, U.S. Pat. No. 10,162,520 B2 describes technology in which a keyboard on the touchscreen is re-sized into a limited area of the display screen that is reachable by the thumb of the one hand holding the smartphone.

Currently, there are different sensors that can detect movements above or at the side of a handheld device (e.g., a smartphone). For example, US 2014/0267142 A1 describes touch or multi-touch actions being continued or extended off-screen via integrating touch sensor data with touchless gesture data. Sensors providing such functionality include radar, cameras on the side of the smartphone, infrared, and the like. As described in an article accessible at the URL en.wikipedia.org/wiki/Google_ATAP, project Soli involves development of a radar-based gesture recognition technology. There are different technologies (e.g., radar, ultra-sound, capacitive, light etc.) for detecting proximity and distance between the phone and an object above the phone.

The sensor-based solutions to the problem of one-handed device operation, such as those mentioned above, do not explicitly address the problem of one-handed operation. For example, the technology described in US 2014/0267142 is intended to sense activities aside the phone, and hence is suitable for two-handed operation. Many of the gesture-based technologies such as employed by project Soli are similar in that they detect gestures by hands that are separated from the mobile device.

Technologies that re-scale the display content are problematic in that they make the content more difficult to read, and this might limit the user experience when a person needs to employ the technology (e.g., when only able to use one hand due to being in transit).

Furthermore, technologies such as that which is described in US 2014/0380209 A1, in which only a subset of the display content is visible, might be problematic.

And technologies such as that which is described in U.S. Pat. No. 10,162,520 B2 are limited to certain use cases and are limiting as to which applications are adapted. This is then more limiting and disruptive to the user experience.

All of above-mentioned technologies are less flexible in that different positions of the hand lead to different reachability of the thumb.

There is therefore a need for technology that addresses the above and/or related problems.

It should be emphasized that the terms “comprises” and “comprising”, when used in this specification, are taken to specify the presence of stated features, integers, steps or components; but the use of these terms does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

Moreover, reference letters may be provided in some instances (e.g., in the claims and summary) to facilitate identification of various steps and/or elements. However, the use of reference letters is not intended to impute or suggest that the so-referenced steps and/or elements are to be performed or operated in any particular order.

In accordance with one aspect of the present invention, the foregoing and other objects are achieved in technology (e.g., methods, apparatuses, nontransitory computer readable storage media, program means) in which a user interface of a device is operated, wherein the user interface comprises a hover and touch sensitive display device. The operation comprises receiving user information from the hover and touch sensitive display device and detecting that the received information corresponds to a hover control gesture, wherein the hover control gesture comprises a swipe gesture followed by hover information. In response to the detecting, the device is operated in a hover control mode that comprises using continuously supplied hover information to control placement of a cursor display on the hover and touch sensitive display device.

In another aspect of some but not necessarily all embodiments consistent with the invention, an initial placement of the cursor display following the detecting is a detected location at which a first object performing the swipe gesture lifted off of the hover and touch sensitive display device.

In yet another aspect of some but not necessarily all embodiments consistent with the invention, using the continuously supplied hover information to control placement of the cursor display on the hover and touch sensitive display device comprises moving the cursor display in one of two directions along a line of movement in correspondence with a trajectory of the detected swipe gesture, wherein a placement of the cursor display along the line of movement is proportional to a detected height of the first object from the hover and touch sensitive display device.

In still another aspect of some but not necessarily all embodiments consistent with the invention, the placement of the cursor display along the line of movement is continuously adjusted in correspondence with changes in detected height of the first object from the hover and touch sensitive display device. In some but not necessarily all such embodiments, the placement of the cursor display along the line of movement is continuously adjusted further in correspondence with a speed at which detected height of the first object from the hover and touch sensitive display device changes.

In another aspect of some but not necessarily all embodiments consistent with the invention, operation includes detecting that the hover information indicates a movement of the object parallel to a plane of the hover and touch sensitive display device, and in response thereto adjusting the placement of the cursor display in a direction that is orthogonal to the line of movement. In some but not necessarily all such embodiments, adjusting the placement of the cursor display in the direction that is orthogonal to the line of movement comprises adjusting the placement of the cursor display in the direction that is orthogonal to the line of movement by an amount that is proportional to an amount of movement of the object that is parallel to the plane of the hover and touch sensitive display device.

In yet another aspect of some but not necessarily all embodiments consistent with the invention, operation comprises one of:

In still another aspect of some but not necessarily all embodiments consistent with the invention, operation comprises using radar information to detect the height of the first object from the hover and touch sensitive display device.

In another aspect of some but not necessarily all embodiments consistent with the invention, operation comprises, while in hover control mode, detecting that the cursor display is pointing to an executable function of the device when a first predefined number of taps on the device by a second object is detected, and in response thereto causing the device to perform the executable function.

In yet another aspect of some but not necessarily all embodiments consistent with the invention, operating the device in the hover control mode is enabled in response to a detection of a predefined enabling user input to the device.

In still another aspect of some but not necessarily all embodiments consistent with the invention, the predefined enabling user input to the device comprises any one or more of:

In another aspect of some but not necessarily all embodiments consistent with the invention, operation of the device comprises causing operation of the device to leave the hover control mode in response to detecting that the first object is touching the hover and touch sensitive display device.

The various features of the invention will now be described in connection with a number of exemplary embodiments with reference to the figures, in which like parts are identified with the same reference characters.

To facilitate an understanding of the invention, many aspects of the invention are described in terms of sequences of actions to be performed by elements of a computer system or other hardware capable of executing programmed instructions. It will be recognized that in each of the embodiments, the various actions could be performed by specialized circuits (e.g., analog and/or discrete logic gates interconnected to perform a specialized function), by one or more processors programmed with a suitable set of instructions, or by a combination of both. The term “circuitry configured to” perform one or more described actions is used herein to refer to any such embodiment (i.e., one or more specialized circuits alone, one or more programmed processors, or any combination of these). Moreover, the invention can additionally be considered to be embodied entirely within any form of non-transitory computer readable carrier, such as solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein. Thus, the various aspects of the invention may be embodied in many different forms, and all such forms are contemplated to be within the scope of the invention. For each of the various aspects of the invention, any such form of embodiments as described above may be referred to herein as “logic configured to” perform a described action, or alternatively as “logic that” performs a described action.

In one aspect of embodiments consistent with the invention, the technology involves a device having a user interface comprising a hover and touch sensitive display device. The hover and touch sensitive display device can comprise, for example, one or multiple sensors (e.g., capacitive proximity, ultra-sound, radar, and the like) capable of detecting the distance of an object (e.g., a finger) above the display surface. In some but not necessarily all inventive embodiments, the sensor may also be capable of detecting a gesture (e.g., that the object is moving to the left or the right when it is above the hover and touch sensitive device). In some but not necessarily all inventive embodiments, the device further comprises an Inertial Motion Unit (IMU) capable of detecting rapid movements of the device itself.

In another aspect of embodiments consistent with the invention, when the device is held with one hand, having for example the thumb above the touchscreen for one-handed operation with touch control, the device is capable of detecting that a swipe movement performed on the display surface was followed by the lifting of the thumb. The swipe movement forms a trajectory on the screen. When the thumb lifts from the display, a cursor indicates the place where it was at the point of liftoff, and as the thumb lifts more and more from the display, the cursor continues along the trajectory proportionally to the thumb's distance from the screen. If the thumb lowers again, the cursor moves back accordingly.

In yet another aspect of some but not necessarily all embodiments consistent with the invention, the activation of a function that the cursor is pointing to is triggered by a tap or a double tap on the phone by any of the other fingers holding the phone (e.g., detected by the IMU).

In still another aspect of some but not necessarily all embodiments consistent with the invention, the cursor can be controlled left/right from the trajectory by detecting and responding to thumb movements made to the left/right as the thumb is held above the screen.

In yet other aspects of some but not necessarily all embodiments consistent with the invention, the one-handed mode of operation can be activated and/or deactivated in a number of different ways. These and other aspects are discussed in greater detail in the following.

In one exemplary, non-limiting example, a system consistent with the invention comprises at least one device (e.g., a smartphone) having at least some but not necessarily all of the following characteristics:

In embodiments consistent with the invention, the user interface (UI) of a device is controlled by detected interactions between a hover and touch sensitive display of the device and an object. In most circumstances, the object will be a finger (a term used herein to include any of the four fingers and opposable thumb of a hand) of the user, and in most of those circumstances, the thumb will be used because, for most people, the thumb is the most natural digit/object for performing the described gestures and movements. Accordingly, in the following descriptions, the thumb is described as the finger/object controlling the UI. However, this is done merely for purposes of illustration. Those of ordinary skill in the art will readily appreciate that any finger and even some objects (e.g., stylus) can be used as the object in place of the thumb.

depicts a hover and touch sensitive display deviceof a user device (e.g., smartphone—not shown in order to avoid cluttering the figure). An object (e.g., user's thumb)starts touching the screen at a point indicated by “X”, and performs a hover control gesture. The hover control gesturecomprises the objectmaking a swipe movement from the starting point “X” to a locationat which the objectlifts off from the device surface, and then continuing to rise to a heightabove the hover and touch sensitive display device. The device is configured to respond to the hover control gestureby causing a cursor to appear on the screen of the hover and touch sensitive display deviceat the positionat which liftoff occurred, and to continue moving along the trajectory that the objecthad before it was lifted.

In another aspect, the device is configured to cause the displayed cursor to remain still in response to the objectbecoming still while hovering above the hover and touch sensitive display device.

In still another aspect, as the objectis raised or lowered, the cursor moves accordingly forward or backward along the trajectory and by an amount that is proportional to the distance between the objectand the hover and touch sensitive display device.

illustrates some of the same features and the same activity but from the side, clearly showing that the objectinitially makes a hover control gesturethat comprises a movement on the devicefollowed by a lifting above the device.

To further illustrate aspects of embodiments consistent with the invention,illustrates a devicehaving a hover and touch sensitive display deviceon front surface of the device. The perspective adopted inis of the deviceas seen from above the hover and touch sensitive display device. Components of the hover control gestureare illustrated. In particular, an object (e.g., thumb)makes a swipe gesturestarting at a first touch pointon the hover and touch sensitive display deviceand extending to an endpoint. From the endpointthe object liftsinto the air.

The deviceis configured to detect that the hover control gesturehas been performed, and to respond to the detection by determining a trajectoryof the swipeand also by displaying a cursorinitially at the point of liftoff. The cursordoes not remain at its positionat the point of liftoff, however, but instead moves along the trajectoryof the swipeby an amount that is proportional to the object's heightabove the hover and touch sensitive display device. The cursor accordingly moves forward or backwards along the trajectoryin correspondence with the object moving higher or lower above the hover and touch sensitive display device.

In another aspect of embodiments consistent with the invention, by moving the object up or down above the hover and touch sensitive display device, the user can cause the cursorto move to an indicated executable functionthat is displayed on the hover and touch sensitive display device. At this point, if any of the other fingers currently on the devicemakes a tap (e.g., detected by the device's IMU), the executable functionpointed to by the cursoris activated. Certain functions might require a double tap before activation is initiated, depending on the UI, app, or context.

In a further aspect of some but not necessarily all embodiments, sensors (for example radar) of the devicecan detect not only the distance between the objectand the hover and touch sensitive display device, but also movements of the objectin the air that are parallel to the plane of the hover and touch sensitive display device(e.g., movements to the right or left as seen from above the device). The deviceis further configured to move the cursornot only along the trajectoryaccording to the height, but also to the left or the right in a directionthat is orthogonal to the trajectoryin dependence on the object's movement. Consequently, the objectcan control the exact position of the cursoralong two orthogonal axes when it is in the air.

Further aspects of inventive embodiments will now be described with reference to, which is in one respect a flowchart of actions taken by the deviceto enter and operate in a hover control mode that enables one-handed operation of the device. In other respects, the blocks depicted incan also be considered to represent means(e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

At step, one-handed operation is activated. Preferably, this is in response to a pre-defined trigger so that the devicewill not behave in an unpredictable manner during normal two-handed operation. The trigger can be a predefined input pattern from the user, such as but not limited to an initial swipe movement of the thumb from the bottom to the center of the screen followed by a double tap of any of the other fingers. In alternative embodiments, the predefined triggering input can be other gestures or combination of gestures including but not limited to shaking or tilting the device back and forth while holding the thumb on the screen, or voice control.

At step, it is detected that an object (e.g., finger or thumb of the user) is touching the hover and touch sensitive display device. This is an ordinary touch-based user interface in the area reachable by, for example, the user's thumb. As long as the thumb is still touching the screen, the device operates in accordance with the principles of the ordinary touch-screen user interface.

In some but not necessarily all embodiments, part of the movement on the screen is recorded for a subsequent trajectory estimation. Alternatively, the current trajectory is estimated as the thumb moves on the screen in performance of the swipe gesture so that it is readily available.

As the objectis in contact with the screen, the devicechecks to determine whether the object has been lifted (decision block). If not (“No” path out of decision block), processing reverts to stepand operation continues as described above.

But if it is detected that the objecthas been lifted (“Yes” path out of decision block), this may indicate completion of the hover control gestureand in response to the hover control gesture, the cursorand function activation are controlled and operated as described above with reference toto continue moving the cursorto a point on the hover and touch sensitive display devicethat is not reachable by a touch movement. However, lifting of the objectmay alternatively have occurred because the user is in the process of tapping the screen at a present position of the thumb (e.g., to select or activate an indicated function).

To distinguish between the two possibilities, in the illustrated exemplary embodiment it is determined whether the objecthas lifted above the screen and lowered again immediately (e.g., as determined by a certain amount of max time, e.g. 0.5 seconds, between liftoff and a second contact with the hover and touch sensitive display device) (decision block). If so (“Yes” path out of decision block), this is interpreted as a tap on the hover and touch sensitive display device, and operation follows conventional procedure in the case of a tap, for example by activating an executable function indicated at the point of contact (step). Processing then reverts back to stepand operation continues as discussed above.