System and Method of Operating System

This application claims priority to Japanese Patent Application No. 2024-203512, filed on November 21, 2024, the entire contents of which are incorporated herein by reference.

The present disclosure relates to a system and a method of operating the system.

Technology for controlling operations of vehicles according to the preferences of users of the vehicles has been proposed. For example, Patent Literature (PTL) 1 discloses an example of systems that associate information on the settings of vehicle interior environment with identification information on users, and make the settings of vehicle interior environment that each user prefers.

When in-vehicle apparatuses to be mounted in vehicles differ from vehicle type to vehicle type, there are variations in convenience that users enjoy from operations of the vehicles controlled by the in-vehicle apparatuses. Thus, there is room to enhance and further improve the level of user convenience.

Hereinafter, a system and the like that can improve user convenience will be disclosed.

A system according to the present disclosure includes:

a first in-vehicle apparatus mounted in a first vehicle, the first in-vehicle apparatus having a suggestion model generated by machine-learning a combination of a first input by a first user of the first vehicle and a first operation of the first vehicle in response to the first input; and

a second in-vehicle apparatus mounted in a second vehicle of a different vehicle type from the first vehicle, the second in-vehicle apparatus being configured to determine, using the suggestion model acquired from the first in-vehicle apparatus, a second operation of the second vehicle in response to a second input by a second user of the second vehicle.

Another aspect of the present disclosure is a method of operating a system including first and second in-vehicle apparatuses mounted in first and second vehicles of different vehicle types, respectively, the method including:

having, by the first in-vehicle apparatus, a suggestion model generated by machine-learning a combination of a first input by a first user of the first vehicle and a first operation of the first vehicle in response to the first input; and

determining, by the second in-vehicle apparatus using the suggestion model acquired from the first in-vehicle apparatus, a second operation of the second vehicle in response to a second input by a second user of the second vehicle.

According to the system and the like in the present disclosure, it is possible to improve user convenience.

An embodiment will be described below with reference to the drawings.

1 FIG. 1 10 12 1 12 2 13 1 13 2 11 10 13 1 13 2 12 1 12 2 13 1 13 2 11 is a diagram illustrating an example configuration of a vehicle control system according to the embodiment. A vehicle control systemincludes at least one server apparatusand in-vehicle apparatuses-and-mounted in vehicles-and-, respectively, which are communicably connected to each other via a network. The server apparatusis, for example, one or more server computers that belong to a cloud computing system or another computing system and that function as a server that implements various functions. The vehicles-and-are different types of passenger cars, commercial vehicles, or the like, and include internal combustion engine vehicles, Hybrid Electric Vehicles (HEVs), Plug-in Hybrid Electric Vehicles (PHEVs), or the like. The in-vehicle apparatuses-and-are computers that have communication functions and information processing functions, and control operations of the vehicles-and-, respectively. The networkmay include, for example, a mobile communication network, the Internet, an ad hoc network, a local area network (LAN), a metropolitan area network (MAN), other networks, or any combination thereof.

13 1 13 2 13 1 13 2 13 1 13 2 12 1 12 2 13 1 13 2 13 2 13 1 13 2 In the present embodiment, the vehicle-is a higher-class model than the vehicle-. For example, the vehicle-belongs to a so-called high-end vehicle category, while the vehicle-belongs to a so-called low-end vehicle category that is less expensive than high-end vehicles. The vehicle-is provided with equipment to provide more sophisticated user experiences than the vehicle-, and the in-vehicle apparatus-is configured to perform control that offers more user convenience than the in-vehicle apparatus-. In the present embodiment, reducing the imbalance in user convenience between the vehicles-and-and enhancing the level of user convenience in the vehicle-improve the user convenience of the vehicles-and-as a whole.

12 1 13 1 13 1 12 1 12 2 12 1 13 2 13 2 1 12 2 12 1 In the present embodiment, the in-vehicle apparatus-has a suggestion model that has machine-learned combinations of inputs from users of the vehicle-(one or more users including drivers and passengers, hereinafter referred to as high-end vehicle users) and operations of the vehicle-(hereinafter referred to as high-end vehicle operations) in response to the inputs. The in-vehicle apparatus-also updates the suggestion model, using combinations of the high-end vehicle operations and the reactions of the high-end vehicle users to those operations. The in-vehicle apparatus-determines, using the suggestion model acquired from the in-vehicle apparatus-, an operation of the vehicle-(hereinafter referred to as a low-end vehicle operation) in response to an input from a use of the vehicle-(at least one user including a driver or a passenger, hereinafter referred to as a low-end vehicle user). Here, the inputs from the high-end vehicle users or low-end vehicle user are intentional inputs made by each user, and spoken content corresponding to wishes and preferences of each user. Such inputs are hereinafter referred to as intentional inputs. The reactions of the high-end vehicle users to the high-end vehicle operations are various actions taken by the high-end vehicle users, including physical conditions such as body temperature and pulse, signs of drowsiness or discomfort in captured images, and operations on air conditioning or audio. According to the vehicle control system, it is possible to realize the more sophisticated low-end vehicle operation in response to the input from the low-end vehicle user by using, in the in-vehicle apparatus-, the suggestion model that has been updated in the in-vehicle apparatus-through reinforcement learning using the reactions of the high-end vehicle users. Therefore, user convenience can be improved as a whole.

2 FIG. 12 1 2 12 1 12 2 12 1 12 2 121 122 123 124 125 126 127 15 12 1 12 2 illustrates an example configuration of the in-vehicle apparatuses-and 12-. The in-vehicle apparatuses-and-are configured equivalently in the following respects. That is, the in-vehicle apparatus-,-has a communication interface, a memory, a controller, a positioner, an input interface, an output interface, and a detector. These components may be configured as a single control apparatus, as two or more control apparatuses, or with another apparatus such as a control apparatus and a communication device. The control apparatus includes, for example, an electronic control unit (ECU) or the like. The communication device includes, for example, a data communication module (DCM) or the like. The components are communicably connected to each other or to equipment in the vehicle, by an in-vehicle network compliant with a standard such as a controller area network (CAN). The in-vehicle apparatus-,-may be configured to include an information processing apparatus such as a smartphone or tablet terminal.

121 12 121 11 13 1 13 2 h The communication interfacehas a module compliant with a mobile communication standard such as Long Term Evolution (LTE), 4th Generation (4G), or 5tGeneration (5G), a module compliant with in-vehicle LAN such as CAN, or the like. The in-vehicle apparatusperforms, via the communication interface, information communication with other apparatuses via the networkconnected through a nearby router apparatus or a mobile communication base station, and information communication with each component of the vehicle-,-via the in-vehicle LAN.

122 122 122 123 123 122 21 22 The memoryincludes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these types. The semiconductor memories are, for example, random access memory (RAM) or read only memory (ROM). The RAM is, for example, static RAM (SRAM) or dynamic RAM (DRAM). The ROM is, for example, electrically erasable programmable ROM (EEPROM). The memoryfunctions as, for example, a main memory, an auxiliary memory, or a cache memory. The memorystores information to be used for operations of the controllerand information obtained by the operations of the controller. In the present embodiment, the memorystores a suggestion modeland a language model.

21 12 1 13 1 21 13 1 12 1 21 21 12 1 12 2 10 12 2 13 2 21 The suggestion modelhas been generated in advance through machine learning using combinations of intentional inputs from high-end vehicle users and high-end vehicle operations as training data, and has been stored in the in-vehicle apparatus-. In the vehicle-, the suggestion modelsuggests a high-end vehicle operation in response to an intentional input from a high-end vehicle user. Here, the suggestion includes an instruction for executing the high-end vehicle operation for the vehicle-. The in-vehicle apparatus-updates the suggestion modelthrough reinforcement learning using reactions of the high-end vehicle users, as described in the following procedure. The updated suggestion modelis transmitted from the in-vehicle apparatus-to the in-vehicle apparatus-via the server apparatus, and is stored in the in-vehicle apparatus-. In the vehicle-, the suggestion modelsuggests a low-end vehicle operation in response to an intentional input from a low-end vehicle user.

22 22 12 1 12 2 22 The language modelis a model that has been generated by machine-learning patterns, structures, and meanings of natural language from a large amount of text data, to generate, summarize, or analyze sentences by natural language. For example, the language modelis a relatively small language model based on a transformer architecture, having several millions to several hundreds of millions of parameters. In the in-vehicle apparatuses-and-, the format, scale, or the like of the language modelmay differ.

123 123 12 12 The controllerincludes one or more processors, one or more dedicated circuits, or a combination thereof. The processors are general purpose processors such as central processing units (CPUs), or dedicated processors such as graphics processing units (GPUs) specialized for particular processing. The dedicated circuits are, for example, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or the like. The controllerexecutes information processing related to operations of the in-vehicle apparatuswhile controlling components of the in-vehicle apparatus.

123 123 123 123 123 123 The functions of the controllerare realized by execution of a control/processing program by a processor included in the controller. The control/processing program is a program for causing a computer to execute processing of steps included in the operations of the controller, thereby enabling the computer to realize the functions corresponding to the processing of the steps. That is, the control/processing program is a program for causing a computer to function as the controller. Some or all of the functions of the controllermay be realized by a dedicated circuit included in the controller.

124 124 123 123 12 The positionerincludes one or more global navigation satellite system (GNSS) receivers. The GNSS includes, for example, global positioning system (GPS), quasi-zenith satellite system (QZSS), BeiDou, global navigation satellite system (GLONASS), and/or Galileo. The positionertransmits a positioning result to the controller, and the controllercalculates positional information on the in-vehicle apparatus.

125 125 123 123 The input interfaceincludes one or more interfaces for input. The interfaces for input include, for example, a microphone that accepts audio input, physical keys, capacitive keys, a pointing device, a touch screen integrally provided with a display, or the like. The interfaces for input also include a camera that captures images of the interior of the vehicle. The input interfaceaccepts operations for inputting various information, including spoken voice of a user, and transmits the input information to the controlleror transmits the captured images to the controller.

126 126 123 The output interfaceincludes one or more interfaces for output. The interfaces for output include, for example, a speaker or a display. The display is, for example, a liquid crystal display (LCD) or an organic electro-luminescence (EL) display. The output interfaceoutputs information obtained by operations of the controller.

127 13 1 13 2 127 12 1 The detectorhas interfaces with one or more sensors that detect the states of various parts of the vehicles-and-, or has the one or more sensors. The sensors include, for example, sensors for vehicle speed, acceleration, and the like, and sensors for indoor and outdoor temperature, humidity, and the like. The sensors of the detectorin the in-vehicle apparatus-further include sensors that measure and detect the physical conditions such as body temperature and pulse of high-end vehicle users using infrared rays or other means, sensors that detect operation amounts for in-vehicle equipment such as air conditioning and audio by the high-end vehicle users, and other sensors.

3 FIG. 3 FIG. 12 1 21 123 12 1 is a flowchart illustrating an operation procedure of the in-vehicle apparatus-for reinforcement learning of the suggestion model. Each step inis a step of information processing executed by the controllerof the in-vehicle apparatus-.

31 123 123 125 123 22 22 In S, the controlleracquires an intentional input. The controlleracquires spoken voice from a high-end vehicle user through the input interface. The controllerconverts spoken content into text through, for example, voice recognition processing, and inputs the text to the language model, to acquire an intentional input corresponding to the spoken content from the language model. The intentional input corresponds to the spoken content that indicates the condition of the high-end vehicle user, including anxiety, tension, drowsiness, boredom, concentration, or discomfort, such as "I want to go to XX," "I want to listen to the song XX," "I want to focus on driving," or "I want to refresh myself."

32 123 123 10 10 13 1 10 123 22 10 In S, the controlleracquires attribute information on the user. The controlleracquires attribute information on the high-end vehicle user from, for example, the server apparatus. The attribute information includes, for example, information such as the age, gender, residence, and occupation of the high-end vehicle user. The attribute information may be, in advance, transmitted to and stored in the server apparatusfrom any information processing apparatus at the time of, for example, purchasing the vehicle-. The attribute information may be stored in the server apparatusin any text format or the like. The attribute information may be acquired from posted content or the like of the high-end vehicle user on Social Network Service (SNS). The controllercan acquire the attribute information by collecting various types of text and posted content from various servers, and analyzing the text and posted content using the language modelor a large-scale language model held by the server apparatus.

33 123 13 1 123 21 21 13 1 123 13 1 123 122 123 122 In S, the controllerdetermines an operation of the vehicle-. The controllerinputs the intentional input from the high-end vehicle user to the suggestion model, and determines, using the suggestion model, a corresponding operation of the vehicle-, that is, a high-end vehicle operation. The determined high-end vehicle operation includes, for example, an adjustment of air conditioning temperature or humidity corresponding to the intentional input of the high-end vehicle user, an adjustment of audio (song selection, volume adjustment), a recommendation (voice output) for a parking or stopping location, or the like. Furthermore, the controllermay determine the operation of the vehicle-taking into account the attribute information on the high-end vehicle user. For example, the controllerperforms settings for air conditioning or song selections according to the age and gender of the high-end vehicle user. Setting information for air conditioning, information on songs, and the like suitable for each age group and gender are stored in advance in the memory. The controllerselects a parking or stopping location close to the residence or workplace of the high-end vehicle user based on map information. The map information is stored in advance in the memory. For example, when the high-end vehicle user is a male corporate employee in his 50s and the time is in the afternoon, the high-end vehicle user is likely to be wearing a suit, so the air conditioning can be set to a lower temperature and a higher airflow. When the high-end vehicle user is a woman in her 60s and the season is summer, the high-end vehicle user is likely to be wearing lightly, so the temperature should not be set too low and the airflow should be reduced.

34 123 13 1 123 13 1 126 In S, the controllerinstructs the operation of the vehicle-. The controllertransmits an instruction to execute the high-end vehicle operation, to equipment, an actuator, or the like of the vehicle-to realize the high-end vehicle operation. The instruction includes, for example, an instruction for an adjustment of temperature or humidity for an air conditioning system, an instruction for song selection, volume adjustment, or the like for an audio system, an instruction for voice output by the output interfacefor a parking or stopping location, or the like.

35 123 123 13 1 127 125 123 122 123 In S, the controlleracquires and analyzes a reaction of the user. The controlleracquires a reaction of the high-end vehicle user, from the physical condition of the high-end vehicle user and an operation on the vehicle-acquired by the detector, captured images acquired by the input interface, or the like. The controlleranalyzes the reaction of the high-end vehicle user and determines whether the reaction is a positive reaction or a negative reaction to the most recent high-end vehicle operation. For example, in a case in which the temperature or humidity has been changed, the reaction can be classified as a positive reaction when a proper body temperature and a proper pulse rate corresponding to the changed temperature or humidity are obtained. Otherwise, the reaction is classified as a negative reaction. Information on the proper body temperature and the proper pulse rate for each temperature or humidity is set freely in advance and stored in the memory. In the captured images, when the high-end vehicle user who has been indicating signs of drowsiness or discomfort indicates signs of awakening or comfort, the reaction is determined as a positive reaction. Otherwise, the reaction is determined as a negative reaction. The controllerderives a pattern of the actions, expressions, and gestures of the high-end vehicle user through image processing on sequential frames, and determines whether the user indicates signs of awakening or comfort by matching the derived pattern with a pre-set pattern. Alternatively, when a cancellation or opposing operation by the high-end vehicle user is detected in response to a change in temperature or humidity or a change in audio, the reaction is determined as a negative reaction.

36 123 21 123 31 34 123 21 36 36 13 1 In S, the controllerperforms reinforcement learning of the suggestion model. The controllerexecutes steps Sto Sa predetermined number of times at any intervals, for example, every few seconds, to acquire combinations of intentional inputs and high-end vehicle operations in response thereto, and combinations of the high-end vehicle operations and reactions thereto. The controllerexecutes reinforcement learning of the suggestion modelusing positive reactions as rewards through any algorithm. Step Scan be executed at any frequency. Step Smay be executed, for example, for each single travel of the vehicle-, or at any intervals, such as every few days.

37 123 21 21 122 In S, the controllerupdates the suggestion modelwith one after the reinforcement learning, and stores the updated suggestion modelin the memory.

38 123 21 21 12 2 12 2 21 10 21 122 21 12 1 12 2 38 21 37 38 21 In S, the controllerduplicates the suggestion model, and transmits information on the duplicated suggestion modelto the in-vehicle apparatus-. The in-vehicle apparatus-receives the information on the duplicated suggestion modelvia the server apparatusand stores the duplicated suggestion modelin the memory, so the updated suggestion modelis shared among the in-vehicle apparatuses-and-. Step Smay be executed every time the suggestion modelis updated in step S, or step Smay be executed when the suggestion modelis updated any multiple times.

4 FIG. 4 FIG. 12 2 21 123 12 2 is a flowchart illustrating an operation procedure of the in-vehicle apparatus-after acquiring the suggestion model. Each step inis a step of information processing executed by the controllerof the in-vehicle apparatus-.

41 123 123 125 123 22 22 In S, the controlleracquires an intentional input. The controlleracquires spoken voice from a low-end vehicle user through the input interface. The controllerconverts spoken content into text through, for example, voice recognition processing, and inputs the text to the language model, to acquire an intentional input corresponding to the spoken content from the language model. The intentional input corresponds to the spoken content that indicates the condition of the low-end vehicle user, including anxiety, tension, drowsiness, boredom, concentration, or discomfort, such as "I want to go to XX," "I want to listen to the song XX," "I want to focus on driving," or "I want to refresh myself."

42 123 123 10 10 13 2 10 123 22 10 In S, the controlleracquires attribute information on the user. The controlleracquires attribute information on the low-end vehicle user from, for example, the server apparatus. The attribute information includes, for example, information such as the age, gender, residence, and occupation of the low-end vehicle user. The attribute information is, in advance, transmitted to and stored in the server apparatusfrom any information processing apparatus at the time of, for example, purchasing the vehicle-. The attribute information may be stored in the server apparatusin any text format or the like. The attribute information may be acquired from posted content or the like of the low-end vehicle user on SNS. The controllercan acquire the attribute information by collecting various types of text and posted content from various servers, and analyzing the text and posted content using the language modelor a large-scale language model held by the server apparatus.

43 123 13 2 123 21 21 13 2 123 13 2 123 122 123 122 In S, the controllerdetermines an operation of the vehicle-. The controllerinputs the intentional input from the low-end vehicle user to the suggestion model, and determines, using the suggestion model, a corresponding operation of the vehicle-, that is, a low-end vehicle operation. The determined low-end vehicle operation includes, for example, an adjustment of air conditioning temperature or humidity corresponding to the intentional input of the low-end vehicle user, an adjustment of audio (song selection, volume adjustment), a recommendation (voice output) for a parking or stopping location, or the like. Furthermore, the controllermay determine the operation of the vehicle-taking into account the attribute information on the low-end vehicle user. For example, the controllerperforms settings for air conditioning or song selections according to the age and gender of the low-end vehicle user. Setting information for air conditioning, information on songs, and the like suitable for each age group and gender are stored in advance in the memory. The controllerselects a parking or stopping location close to the residence or workplace of the low-end vehicle user based on map information. The map information is stored in advance in the memory.

44 123 13 2 123 13 2 126 In S, the controllerinstructs the operation of the vehicle-. The controllertransmits an instruction to execute the low-end vehicle operation, to equipment, an actuator, or the like of the vehicle-to realize the low-end vehicle operation. The instruction includes, for example, an instruction for an adjustment of temperature or humidity for an air conditioning system, an instruction for song selection, volume adjustment, or the like for an audio system, an instruction for voice output by the output interfacefor a parking or stopping location, or the like.

13 2 21 13 1 As described above, the low-end vehicle operation based on the intentional input of the low-end vehicle user is executed in the vehicle-, using the suggestion modelacquired from the vehicle-. Therefore, according to the present embodiment, user convenience can be improved.

While the embodiment has been described with reference to the drawings and examples, it should be noted that various modifications and revisions may be implemented by those skilled in the art based on the present disclosure. Accordingly, such modifications and revisions are included within the scope of the present disclosure. For example, functions or the like included in each means, each step, or the like can be rearranged without logical inconsistency, and a plurality of means, steps, or the like can be combined into one or divided.