Head up display apparatus and display control method thereof

The present invention relates to a technique for a display apparatus such as a head up display (HUD: Head Up Display) apparatus, and relates to a technique for providing image information of augmented reality (AR: Augmented Reality) to a driver.

As AR display apparatuses each having an AR function and systems of the AR display apparatuses, a HUD apparatus mounted on a car or others is cited. In the HUD apparatus (called AR-HUD in some cases), by projection of image light, a virtual image is superimposed on a transparent real image in a visual recognition area (screen) of a windshield, a combiner or others in a field of view of a user such as a driver. The virtual image will be described as an AR image below in some cases. As the virtual image, for example, image information for driving support, etc., related to the real image including a road and a car ahead of the subject car, etc., is exemplified. As a specific example, an image of vehicle information such as a car speed, an arrow image such as navigation system indicating a car-running direction, a frame image for use in attention seeking that indicates approach of an oncoming car, a pedestrian, a bicycle or others and for use in emphasis of a road sign or others are exemplified. As source data for use in creation of the virtual image, information of on-vehicle sensor, engine controller, navigation system and others are usable.

The HUD apparatus includes, for example, a display element, an optical source, and an optical system such as a lens or a mirror for use in guiding image light to the visual recognition area. The image light from the display element is projected on an area (referred to as an AR display area, a HUD display range, etc., below in some cases) inside the visual recognition area of the windshield, the combiner or others through the mirror or others. The light reflected on the area inside the screen enters the driver's eyes, and forms an image on a retina, so that the driver visually recognizes the image as the virtual image.

As related-art examples related to the on-vehicle HUD apparatus, Japanese Patent Application Laid-Open Publication No. 2010-70066 (Patent Document 1) is cited. The Patent Document 1 describes that shift of a projection position of the image from a line of sight of either one of eyes due to vibration of the vehicle or change of a driver's posture is reduced in the head up display to enhance visual recognition of the display information.

Patent Document 1: Japanese Patent Application Laid-Open Publication No. 2010-70066

In an on-vehicle system including a conventional HUD apparatus, a virtual image is superimposed on a transparent real image in a visual recognition area of a windshield or others by using an AR function. At this time, when the driver is watching the visual recognition area, display shift between an object of the real image and the virtual image may occur depending on a driver's eye position (referred to as point-of-view position in some cases). In other words, a display position of the virtual image that is designed by the HUD apparatus and a display position of the virtual image that is practically visually recognized by the driver do not match each other in some cases.

As causes of the display shift between the real image and the virtual image, a case of the eye position change from a basic-setting eye position due to motion such as posture change caused when the driver moves, a case of a person having a different human body size or others who is watching the visual recognition area of the HUD apparatus as the same basic setting, etc., are cited.

In the HUD technique, depending on an eye position of a person (such as the driver) who is watching the visual recognition area, such as a position in height in a vertical direction, difference in a sight such as shift of a relative positional relation between the virtual image and the real image occurs. By a relation between the position of the virtual image formed by the HUD apparatus and a position of a watcher's eyes, a state of the image formed on the retina of the watcher is determined. When a predetermined relation between the position of the virtual image formed by the HUD apparatus and the position of the watcher's eyes is not satisfied, the watcher cannot visually recognize the virtual image. A range in which the eye position should be put for allowing the watcher to visually recognize the image is called eye box.

For example, when the point-of-view position of the driver changes in the vertical direction in the eye box range including a basic setting point-of-view position so as to be out of the eye box range, the driver cannot watch the virtual image or an entire display range of the virtual image. The driver adjusts the optical system such as an angle of the mirror so that the driver can watch the virtual image. However, as a result of the adjustment, the display shift between the real image and the virtual image occurs. In the visual recognition area of the field of view of the driver, the virtual image is displayed at a position that largely shifts from the object of the real image, and therefore, the driver is difficult to or cannot visually recognize the virtual image. The larger the display shift between the real image and the virtual image is, the more difficult the recognition of the virtual image by the driver is, and the more difficult a linking recognition between the real image and the virtual image is. That is, for the driver, such a virtual image is unsuitable.

A purpose of the present invention is related to a technique of a HUD apparatus having an AR function to provide a technique capable of achieving favorable AR display by reducing display shift between a real image and a virtual image in.

A typical embodiment of the present invention is a head up display apparatus, and a feature of the embodiment is to have the following configuration.

A head up display apparatus of one embodiment is a head up display apparatus displaying, to a driver, a virtual image superimposed on scenery ahead of a vehicle by projecting an image onto a windshield or a combiner of the vehicle, the head up display apparatus including: an image input unit configured to receive an image captured by a camera as input and extract a predetermined object from the image; an information acquiring unit configured to acquire object information containing a position of the object inside the image and distance information containing a distance from the object in a space; an image creating unit configured to create an image of the virtual image to be superimposed on the object; a converter configured to, by using the acquired information, perform a conversion processing for correcting a position of a display area that is a range in which the image is displayable in a visual recognition area of the windshield or the combiner and a display position of the image inside the display area at least in a vertical up and down direction; a display controller configured to perform control for superimposing the image on the visual recognition area by using data caused after the correction; and a display configured to superimpose the image on the visual recognition area in accordance with the control, and the converter performs the conversion processing so that a display position of the image inside the display area matches a position of the object caused when the object is watched from a point-of-view position of the driver as a basic setting through the visual recognition area.

According to a typical embodiment of the present invention, in a technique of to HUD apparatus having an AR function, display shift between a real image and a virtual image is reduced, so that favorable AR display is achieved.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the same components are denoted by the same reference symbols throughout all the drawings for describing the embodiments, and the repetitive description thereof will be omitted.

With reference to, a head up display apparatus (HUD apparatus) of a first embodiment of the present invention and others will be described. The HUS apparatus of the first embodiment is an AR display apparatus having an AR function, and a case of mounting of the HUD apparatus as an on-vehicle AR-HUD will be described. A display control method of the first embodiment is a method having steps executed by the HUD apparatus of the first embodiment. The HUD apparatus of the first embodiment has a function of performing a conversion processing that automatically corrects the display position shift between the real image and the virtual image at the time of AR display by performing a program processing of a controller. In the first embodiment, even if the driver does not adjust the mirror of the optical system or others, an AR display content is corrected in a setting state without the adjustment.

As a comparative example of the first embodiment, the following problem of an on-vehicle system including a conventional HUD apparatus will be described. The HUD apparatus of the comparative example does not have the function of correcting the display shift between the real image and the virtual image as described in the first embodiment. Conventionally, in a usage example, the driver takes a driver's seat of a car, and performs a basic setting by adjusting the seat so as to match the driver's human body size and posture, and then, adjusting an AR display area (a HUD display range in which an AR image is displayable) of a visual recognition area of the HUD apparatus. For example, the driver manually adjusts an angle of a mirror of an optical system of the HUD apparatus so that the virtual image of the AR display area can be watched at a preferable position inside the visual recognition area ahead of a line of sight of the driver himself/herself. Note that such adjustment is referred to as basic setting in some cases.

When the driver is a person having a standard human body size, in a completion state of this basic setting, the driver can favorably visually recognize the virtual image displayed in the AR display area of the visual recognition area, and can use the AR function during the driving. Even if the point-of-view position of the driver slightly changes, the AR display area has room as long as the point of view is within the eye box range set as the basic setting, and therefore, the driver can visually recognize the virtual image. In this case, the driver does not particularly need readjustment of the angle of the mirror or others.

However, in some cases, the HUD apparatus at the time of usage of the AR function in the comparative example causes the shift between the display position of the virtual image that is practically watched from the point of view of the driver and the display position of the virtual image inside the AR display area that is designed by the HUD apparatus. For example, when the point-of-view position of the driver changes in a vertical direction from a point-of-view position set of basic setting, the virtual image is displayed at a position that is shift from the object of the real image in the visual recognition area of the field of view of the driver, and therefore, the driver is difficult to visually recognize the virtual image. For example, when the change of the point-of-view position of the driver is large so that the point-of-view position is out of the eye box range including the point-of-view position of the basic setting, the virtual image is not visually recognized or visually recognized in a state with the large display shift between the real image and the virtual image. The display shift appears in a distance between the real image and the virtual image inside the visual recognition area or others. When the display shift is smaller than a certain degree, the driver can recognize the object and the virtual image so as to link the object with the virtual image. However, when the display shift is equal or larger than the certain degree, the image is unsuitable as the virtual image of the AR.

In the HUD apparatus of the comparative example, the point-of-view position of the driver and a position of a camera for use in the AR function are different from each other. Further, in the HUD apparatus of the comparative example, an image-capturing direction and an angle of view of the camera are different from a direction and an angle of view of the AR display area watched from the point of view of the driver. Therefore, in some cases, the HUD apparatus of the comparative example causes the shift between the designed display position of the AR image and the position of the AR image watched from the driver.

[HUD Apparatus and On-Vehicle System]

shows a configuration of an on-vehicle systemincluding a HUD apparatusof the first embodiment. The on-vehicle systemis a system mounted on a car. The driver who is a user operates and uses the on-vehicle systemand the HUD apparatus. This HUD apparatusis particularly an AR-HUS apparatus having the AR function.

The on-vehicle systemincludes an ECU (Engine controller), the HUD apparatus, an outside-image capturing unitincluding a camera, an image data storage, a communication unit, a GPS (Global Positioning System) receiver, a car navigation unit, a sensor, a DB unitand others, each of which is connected to an on-vehicle bus and CAN (Car Area Network). The on-vehicle systemincludes an audio output unit, an operation unit, a power supply unit and others, which are not illustrated.

The HUD apparatusincludes a controllerand a display. The HUD apparatushas the AR function. The HUD apparatusdisplays an AR image in an AR display areaof a visual recognition areaof a windshieldby using the AR function, so that various types of information can be given to the driver. The HUD apparatushas a function of automatically correcting a position of the AR display areathat is a HUD display range in which the virtual image is displayable in the visual recognition area. By this function, the display shift between the virtual image and the real image in the watching of the visual recognition areafrom the point of view of the driver is reduced. In combination use with the audio output unit, to the driver, the HUD apparatuscan give audio output such as guidance, an alarm, etc., using the car navigation function or the AR function. The HUD apparatusalso includes an operational panel, an operational button, etc., so that manual operational input by the driver such as turning ON/OFF of the AR function, user setting of the AR function, the adjustment of the angle of the mirror of the optical system, etc., is also achieved.

The controllercontrols the entire HUD apparatus. The controllerincludes hardware such as a CPU, a ROM and a RAM, and corresponding software. The controllerand other units may be implemented by using hardware such as a microcomputer or FPGA. The controllerachieves each unit such as an image input unitby, for example, causing the CPU to read a program from the ROM and execute a processing in accordance with the program. The controllerstores various types of data and information in an internal memory or an external memory if needed, and reads the data and information or others. The controllerretains setting information for use in the AR function and others in a nonvolatile memory of memories. The setting information includes setting information for use in a conversion processing of the display position converterand user setting information. The user setting information includes basic setting information for the angle of the mirror of the optical systemand others.

The controllerreceives an image captured by the cameraas input, creates image data that is the AR data for use in displaying the virtual image onto the AR display areaof the visual recognition areawhile using each types of the acquired information, and feeds the data to the display. The controllerhas a function of correcting the position of the AR display areaof the visual recognition areaand the display position of the AR image inside the AR display areaat the time of creation of the AR data. The controllercan adjust a state of the angle of the mirror of the optical systemby controlling the display, so that the position of the AR display areaof the visual recognition areacan be adjusted for the basic setting.

The displayis configured of a projection-type image display apparatus (projector) or others. The displayincludes a display driving circuit, a display element, an optical source, an optical systemand a driving unit, which are connected to one another. The displaygenerates image light for use in displaying the virtual image onto the visual recognition areaon the basis of the control by the controllerand the image data, and projects the image light onto the visual recognition area.

The display driving circuitgenerates a display driving signal for use in the AR display in accordance with the image data output from the AR display, and feeds the signal to the display elementand the optical sourcefor driving control.

On the basis of the display driving signal, the optical sourcegenerates illumination light to be supplied to the display element. The optical sourceis configured of, for example, a high-pressure mercury lamp, a xenon lamp, an LED element, a laser element, or others. The light from the optical sourceis caused to enter the display elementthrough an illumination optical system that is not illustrated. The illumination optical system collects and equalizes the illumination light, and is emitted to the display element.

The display elementgenerates the image light on the basis of the display driving signal and the illumination light from the optical source, and emits the image light to the optical system. The display elementis configured of, for example, an SLM (Spatial Light Modulator), a DMD (Digital Micromirror Device, registered trademark), a MEMS device, an LCD (a transparent-type liquid crystal panel or a reflection-type liquid crystal panel), or others.

The optical systemincludes elements such as a lens and a mirror for use in guiding the image light from the display elementto the visual recognition areaof the windshield(). To the optical system, the driving unitis connected. For example, the image light from the display elementis expanded by the lens of the optical system, is reflected by the mirror, and is projected onto the visual recognition area. That is, in a part of the visual recognition area, the AR display areathat is the HUD display range in which the image light is projected is generated. The image light is reflected on the AR display areain the visual recognition area, enters the driver's eyes, and forms the image on the retina. In this manner, in the AR display areaof the visual recognition areaof the field of view of the driver, the virtual image is superimposed on the transparent real image.

The driving unitis an optical system driving unit for use in driving the optical system, and includes a component such as a motor for use in driving the lens, the mirror and others. The driving unitdrives the optical systemso that, for example, the angle of the mirror changes in accordance with the manual operational input by the driver or the control from the controller. The driving unitincludes an operational button for use in, for example, adjusting the angle of the mirror. The driver can change the angle of the mirror in positive and negative directions with reference to a standard-set angle by performing up-and-down manual operations of the operational button. For example, the angle of the mirror changes in the positive direction (that is a direction of increasing the angle) while a first portion of the operational button is pushed, and the angle of the mirror changes in the negative direction (that is a direction of decreasing the angle) while a second portion thereof is pushed.

The ECUperforms vehicle control including engine control and entire control of the on-vehicle system. The ECUis, in other words, a vehicle controller. The ECUmay have a sophisticated function for use in driving support and driving automatic control. In this case, the ECUmay output information related to the function to the HUD apparatusand control the HUD apparatusso as to allow the HUD apparatusto display the AR image related to the function. On the basis of the detection information from the sensor, the ECUrecognizes the vehicle information containing a car-running state such as a car speed, and uses the information for the control. The HUD apparatuscan acquire the vehicle information from the ECUand use the information for the AR display.

The outside-image capturing unitincludes the camera, and captures external-state images of the subject car by using one or more camerasduring the stop or the running of the subject car or others to acquire image data (containing time-series image frames) and around-vehicle information. The outside-image capturing unitstores the image data or others in the image data storage, or outputs the image data or others to the ECUor the HUD apparatus.

The camerais an outside camera that is placed at a predetermined position of the vehicle so as to have predetermined orientation and angle of view (). The position of the camerais, for example, vicinity of a front bumper of the vehicle, vicinity of a side of the windshield, vicinity of a back mirror on a lateral side of the vehicle or others. The cameracaptures images in a predetermined direction (image-capturing direction) within a range of a predetermined angle of view so as to include the vehicle and a forward side of the driver, and outputs the image data.

The outside-image capturing unitmay include a signal processor that processes the images of the cameraor not include it. The signal processor may process the images of one or more camerasto compute and acquire the around-vehicle information or others. The signal processor may be included in the ECUor the HUD apparatus. On the basis of the analysis of the images of the camera, the outside-image capturing unitmay determine states of other cars, people, buildings, road surface, terrain, weather and others around the subject car.

The outside-image capturing unitmay have a function of measuring a distance between the subject car and the object of the real image. When two or more cameras such as a stereo camera are included as the camerain the outside-image capturing unit, the distance from the object can be computed by using two images captured by the two cameras on right and left sides in a publicly-known distance measuring method based on binocular disparity. Even when only one camera is included in the outside-image capturing unit, the distance from the object can be computed on the basis of a position of the object inside the image of the camera. Alternatively, the distance from the object may be computed by combination use of the images of the cameraand the detection information of a different sensor.

The image data storagestores the image data output from the cameraor others. The image data storagemay be set inside the outside-image capturing unitor the HUD apparatus.

The communication unitis a unit including a communication interface device that communicates with an external mobile network, the Internet or others. On the basis of the control by the ECU, the HUD apparatusor others, the communication unitcan communicate with, for example, a server on the Internet or others. For example, the HUD apparatuscan refer to and acquire source data, related information and others for use in the AR display from the server through the communication unit.

The communication unitmay include an inter-car communication wireless receiver, a road-to-car/car-to-road communication wireless receiver, a VICS (Vehicle Information and Communication System: road traffic information communication system, registered trademark) receiver or others. The inter-car communication is communication between the subject car and a surrounding different car. The road-to-car/car-to-road communication is communication between the subject car and surrounding road or apparatus such as a traffic signal.

A GPS receiveracquires current-position information (such as latitude, longitude, and altitude) of the subject car on the basis of signals from a GPS satellite. The ECU, the HUD apparatusand the car navigation unitcan acquire the current-position information of the subject car from the GPS receiver, and use the information for the control.

The car navigation unitis an existing car navigation system unit mounted on a car, and performs a publicly-known navigation processing to store the map information, the position information or others acquired from the GPS receiverand use the information. The ECUand the HUD apparatuscan acquire the information from the car navigation unitand perform the control. The HUD apparatusmay refer to the map information or others from the car navigation unitand use the information as the source data for the AR display. On the basis of the source data, the HUD apparatusmay create, for example, an arrow image as an example of the AR image for use in car navigation toward a destination in the car-running direction.

The sensorhas a publicly-known sensor group mounted on a car, and outputs the detection information. The ECUand the HUD apparatusacquires the detection information, and perform the control. Examples of a sensor device in the sensorinclude a car speedometer, an acceleration sensor, a gyrosensor, a geomagnetic sensor (an electronic compass), an engine start sensor, a shift position sensor, a handle steering angle sensor, a headlight sensor, an external-light sensor (a chromaticity sensor and an illuminance sensor), an infrared sensor (an approaching-object sensor), a temperature sensor and others. The acceleration sensor and the gyrosensor detect an acceleration rate, an angular rate, an angle or others as the state of the subject car.

The sensormay include a distance sensor that measures the distance between the subject car and the object. The distance sensor is achieved by, for example, an optical sensor, and the distance can be computed by using time that is taken while the emitted light impacts against the object and returns to the sensor. In the case with the distance sensor, the information acquiring unitof the HUD apparatuscan acquire the distance information from the distance sensor.

The DB unitis configured of a storage or others, and the source data or information to be used for the AR display is stored in a DB. The source data is, for example, basic image data for use in creating the AR image (example: frame image) or others. The information is, for example, basic information or related information of the object (example: oncoming car), a reference image or others. Note that the DB unitmay be set inside the HUD apparatusor a data center or others on a communication network out of the on-vehicle system. In the DB of the DB unit, the information externally acquired through the communication unitmay be stored. The DB of the DB unitmay be used in combination with the DB of the car navigation unit.

The windshieldis a part of the car, and is made of a glass having transparency and stiffness, a film having predetermined optical characteristics or others (). In the visual recognition areaof the windshield, the AR display areais formed at the time of the usage of the AR function so that the virtual image is superimposed on the real image in the AR display area. Ahead of the windshield, note that an AR-dedicated display plate (such as a combiner) may be arranged. The visual recognition areacorresponds to a range in which the AR display areacan be arranged by the adjustment.

The image input unitinputs the image captured by the camera, and extracts the predetermined object for use in the AR from the image. The information acquiring unitacquires the object information, the distance information, the vehicle information and others as the information that is necessary for the AR display, the display positional conversion and others. The object information is the information containing the position of the object inside the image. The distance information is the information containing the distance between the subject car and the object in a space. The vehicle information is the information containing the car speed or others. The AR image creating unitcreates a basic AR image to be superimposed on the object.

The display position converteris, in other words, a correcting unit. The display position converterperforms a conversion processing that corrects the position of the AR display areathat is the HUD display range in which the AR image is displayable in the visual recognition areaand corrects the display position of the AR image in the AR display areaby using the information acquired by the information acquiring unit. The AR displaycontrols the displayso as to superimpose the AR image on the visual recognition areaby using the corrected data.

[Driver's Seat]

schematically shows arrangement examples of units in the first embodiment on a plane on which vicinity of the driver's seat of the car is watched in a lateral direction (X direction). The windshieldis put on a front side of the driver's seat, the visual recognition areais set as a part of the windshield, and the AR display areais set as a part of the visual recognition area. The displayof the HUD apparatus, the car navigation unitnot illustrated and others are placed at a position of a part of a dashboard on the front side of the driver's seat, such as a position of a console. In, the display elementand a mirrorA of the optical systemare shown. The diagram shows a case in which the image light is emitted from the display elementto, for example, the front side (a Z direction) and is reflected toward the windshieldin an upper direction (a Y direction) by the mirrorA. The reflection light from the mirrorA is reflected on the visual recognition areainside the windshieldtoward the driver's eyes. Note that the configuration of the optical systemis not limited to this, and any configuration is applicable as long as its position and its angle at which the image light (the reflection light of the mirrorA) is projected onto the visual recognition areaare variable.

The driver's eye position and point-of-view position are shown as a point “p”. A line of sight “L” extending from the point pthat is the eye position is shown with a chain line. Note that the point-of-view position may be different from the eye position. For example, the point-of-view position may be computed as an intermediate point between both eyes on right and left sides, or be computed as a center point of a head, a face, or others.