Electronic Apparatus, Agent System, Program Thereof, Smart Glasses, Order Terminal, and Program Thereof

The present invention relates to an electronic apparatus, an agent system, a program thereof, smart glasses, an order terminal, and a program thereof.

Patent Literature 1 discloses a technique for determining an appropriate action of a robot with respect to a state of a user. In the related art of Patent Literature 1, a reaction of a user in a case in which the robot executes a specific action is recognized, and in a case in which an action of the robot with respect to the recognized reaction of the user cannot be determined, the action of the robot is updated by receiving information regarding an action suitable for the recognized state of the user from a server.

Patent Literature 1: Japanese Patent No. 6053847

An object of the present disclosure is to provide an electronic apparatus, an agent system, a program thereof, smart glasses, an order terminal, and a program thereof capable of using an agent system that understands emotions of a user and the agent system and interacts with the user.

An agent system related to a first aspect includes a reception unit that receives a budget of settlement means available for obtaining a product, a history acquisition unit that acquires history information that is information regarding an acquisition history in which a product acquired by a user in the past is recorded, a product determination unit that determines a product to be acquired on the basis of the budget received by the reception unit and the history information obtained from the history acquisition unit, and an operation execution unit that executes an operation of acquiring the product determined by the product determination unit at a predetermined timing.

In the agent system related to the first aspect, an agent system related to a second aspect further includes a user state recognition unit capable of acquiring information regarding a state of the user, and an emotion determination unit that determines an emotion of the user on the basis of the information regarding the state of the user acquired by the user state recognition unit, in which the history information includes an acquisition history in which a product acquired by the operation execution unit and an emotion of the user determined by the emotion determination unit for the product are recorded in association with each other.

In the agent system related to the second aspect, according to an agent system related to a third aspect, the emotion determination unit further determines an emotion of an agent on the basis of the state of the user acquired via the user state recognition unit, and the product determination unit determines a product to be acquired on the basis of the emotion of the agent determined by the emotion determination unit.

In the agent system related to the second aspect, according to an agent system related to a fourth aspect, the user state recognition unit further acquires information regarding a surrounding situation of the user, and the product determination unit determines a product to be acquired on the basis of the surrounding situation acquired by the user state recognition unit.

In the agent system related to the fourth aspect, according to an agent system related to a fifth aspect, the information regarding the surrounding situation is information regarding a stock for each product estimated from image information indicating the surrounding situation of the user, and the product determination unit determines a product to be acquired on the basis of the information regarding the stock.

In the agent system related to the fifth aspect, according to an agent system related to a sixth aspect, the operation execution unit executes an operation of acquiring the product at a timing at which the stock falls below a predetermined threshold continuously for a predetermined period.

In the agent system related to the second aspect, according to an agent system related to a seventh aspect, the reception unit further receives setting information of at least one of information regarding a category of a product or information regarding a preference of the user, and the product determination unit determines a product to be acquired on the basis of the setting information received by the reception unit.

A program related to an eighth aspect causes a computer to operate as the agent system related to any one of the first to seventh aspects.

An agent system related to a ninth aspect includes a reception unit that receives a keyword indicating a target desired by a user and a concept word indicating directionality of the keyword, a proposal determination unit that determines proposal information which is information to be proposed to the user on the basis of the keyword and the concept word received by the reception unit, a specifying unit that specifies a thing required by the user from the proposal information determined by the proposal determination unit, a user state recognition unit that acquires image information regarding a surrounding situation of the user, and a position output unit that outputs position information indicating a position of information regarding the thing in a case in which the image information acquired by the user state recognition unit includes the information regarding the thing specified by the specifying unit.

In the agent system related to the first aspect, according to an agent system related to a tenth aspect, the user state recognition unit is capable of acquiring information regarding a state of the user, the agent system further includes an emotion determination unit that determines an emotion of the user on the basis of the information regarding the state of the user acquired by the user state recognition unit, and a history acquisition unit that acquires history information that is information in which a past action of the user and an emotion of the user determined by the emotion determination unit when the action is performed are stored in association with each other, and the proposal determination unit determines the proposal information on the basis of the emotion of the user determined by the emotion determination unit and the history information acquired by the history acquisition unit.

A program related to an eleventh aspect causes a computer to operate as the agent system related to the first aspect or the second aspect.

An agent system related to a twelfth aspect includes a face information acquisition unit that acquires face information that is information regarding a face of a conversation partner of a user, a state recognition unit capable of acquiring information regarding a state of the conversation partner, an emotion determination unit that determines an emotion of the conversation partner on the basis of the information regarding the state of the conversation partner acquired by the state recognition unit, a published information acquisition unit that acquires published information regarding the conversation partner from information on the Internet including social media on the basis of the face information and the emotion of the conversation partner, a person information acquisition unit that acquires person information specifying the conversation partner from the published information, and an information output unit that outputs at least one of the emotion of the conversation partner, the published information, or the person information.

In the agent system related to the twelfth aspect, according to a thirteenth aspect, the person information acquisition unit acquires the person information from a database in which face information of a plurality of persons and person information for specifying each of the plurality of persons are stored in association with each other on the basis of the face information acquired by the face information acquisition unit.

In the agent system related to the twelfth aspect, according to an agent system related to a fourteenth aspect, the information output unit outputs a summary generated from a combination of at least two of the emotion of the conversation partner, the published information, and the person information.

In the agent system related to the twelfth aspect, according to an agent system related to a fifteenth aspect, the information output unit outputs conversation information that is generated on the basis of at least one of the emotion of the conversation partner, the published information, or the person information and is information for conversation with the conversation partner.

A program related to a sixteenth aspect causes a computer to operate as the above agent system.

Smart glasses related to a seventeenth aspect can use the above agent system.

According to an eighteenth aspect of the disclosure, an electronic apparatus is provided. The electronic apparatus is capable of using an agent system including an emotion determination unit that determines an emotion of a user or an emotion of an agent, and an interaction processing unit that determines speech content of the agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, the electronic apparatus including an acquisition unit capable of acquiring information regarding a state of the user, an execution unit that causes the agent system to create a summary sentence for a letter string in a specific region that the user intends to read by the acquisition unit acquiring a preset prompt, and an output unit that outputs the summary sentence generated in the agent system to the user as the speech content. In the disclosure, the electronic apparatus is a wearable terminal worn by the user, an image of the specific region on a line of sight of the user is captured by a camera mounted on the wearable terminal, the letter string in the image of the specific region captured by the camera is converted into text data from which letters are recognizable, and the text data is sent to the acquisition unit. In the disclosure, the specific region is a double spread region of a book gripped by the user, and the summary sentence is created and spoken with the double spread region as a unit. In the disclosure, a summary sentence of a plurality of units for each of double spread pages is created by determining whether or not page turning of the book is performed, and the summary sentence is output in synchronization with the page turning.

According to a nineteenth aspect of the disclosure, an electronic apparatus is provided. The electronic apparatus is capable of using an agent system including an emotion determination unit that determines an emotion of a user or an emotion of an agent, and an interaction processing unit that determines speech content of the agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, the electronic apparatus including an acquisition unit capable of acquiring information regarding a state of the user, an execution unit that causes the agent system to create a sentence that changes an emotion of the user related to a specific event to positive by the acquisition unit acquiring a predefined prompt when the emotion determination unit analyzes that the emotion of the user related to the specific event is negative, and an output unit that outputs the sentence generated in the agent system to the user as the speech content. In this case, the determination of the emotion of the user may be repeated until the emotion of the user changes from negative to positive. The electronic apparatus may be applied to determination of the emotion of the user on the basis of sensor information from which an action range of the user is detected in synchronization with an action of the user. The electronic apparatus may be a wearable terminal worn by the user, and may aggregate the sensor information in the action range along a line of sight of the user on the basis of a sensor mounted on the wearable terminal. The sensor may be a camera mounted on the wearable terminal, and the sensor information may be an image in the action range of the user captured by the camera. The sensor may be a microphone mounted on the wearable terminal, and the sensor information may be environmental sound in the action range of the user recorded by the microphone.

According to a twentieth aspect of the disclosure, an electronic apparatus is provided. The electronic apparatus is capable of using an agent system including an emotion determination unit that determines an emotion of a user or an emotion of an agent, and an interaction processing unit that determines speech content of the agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, the electronic apparatus including an acquisition unit capable of acquiring information regarding a state of the user, a determination unit that determines whether or not a specific action of an action history target has been achieved on the basis of an action history of the action history target, an execution unit that causes the agent system to generate a sentence regarding the specific action on the basis of a determination result of the determination unit regarding the specific action in a case in which there is an instruction to determine whether or not the specific action has been achieved from the user, and an output unit that outputs the sentence generated in the agent system to the user as the speech content. In this case, the electronic apparatus may further include a storage unit that stores action history images captured in time series following an action of the action history target, and the determination unit may determine whether or not the specific action has been achieved on the basis of the action history images stored in the storage unit. A relationship in which the user is an administrator or a guardian and the action history target is a managed person or a protected person may be established between the user and the action history target. The relationship between the user and the action history target may be a relationship between a parent and a child under protection of the parent, and the specific action may be selected from actions registered in advance as a ToDo list of the child.

According to a twenty-first aspect of the disclosure, an agent system is provided. The agent system includes an emotion determination unit that determines an emotion of a user or an emotion of an agent, an interaction processing unit that determines speech content of the agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, a generation unit that generates a sentence for giving an instruction for an action corresponding to a sudden change in an emotion in a case in which the emotion of the user or the emotion of the agent related to a specific event shows the sudden change exceeding a predefined change degree, and an execution unit that executes the action corresponding to the sudden change on the user between an action before the emotion change and an action after the emotion change on the basis of the generated sentence. In this case, the action corresponding to the sudden change in the emotion may be executed by combining at least one sense of hearing, vision, touch, smell, or taste, or two or more senses. The execution unit may adjust a degree of emphasis of the action corresponding to the sudden change in the emotion in proportion to a degree of the sudden change in the emotion. The emotion may be classified into happiness, anger, sadness, and pleasure, and the execution unit may vary an action according to emotions before and after a change in the emotion. In the disclosure, in a case in which the user feels intimidated or scared, the execution unit may execute an action corresponding to a change in the emotion that alleviates intimidation or scare. In a case in which the user is delighted, the execution unit may execute an action corresponding to a sudden change in the emotion that promotes and excites delight.

According to a twenty-second aspect of the disclosure, an agent system is provided. The agent system includes plurality of agents each including an acquisition unit that acquires information regarding a state of a user, an emotion determination unit that determines an emotion of the user on the basis of the information regarding the state of the user acquired by the acquisition unit, and an interaction processing unit that determines speech content of an agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, and an arbitration unit that is mounted on at least one of the agents and that determines one piece of speech content in a case in which the speech content differs among the plurality of agents when the speech content is requested for the emotion of the user. In this case, the acquisition unit may be capable of acquiring vocal sound information output by the user or text information created by the user, and the speech content may be generated by using a sentence generation model that generates a sentence for answering to the vocal sound information or the text information input via the acquisition unit. The arbitration unit may determine any one of a plurality of pieces of new speech content on the basis of a history of which agent's speech content has been determined in the past. Each of the plurality of agents may be mounted on a wearable device that can be worn by the user.

An order terminal related to a twenty-third aspect includes a reception unit that receives order information indicating an order of a product from a user, an input unit that inputs the order information received by the reception unit to a sentence generation model, a presentation unit that presents check information for checking the order information output from the sentence generation model to the user, and an order placement unit that places an order for a product in accordance with an answer of the user to the check information.

In the order terminal related to the twenty-third aspect, an order terminal related to a twenty-fourth aspect further includes an acquisition unit capable of acquiring information regarding a state of the user, an emotion determination unit that determines an emotion of the user on the basis of the information regarding the state of the user acquired by the acquisition unit, and a conversation processing unit that determines speech content of an agent who has a conversation with the user on the basis of the emotion determined by the emotion determination unit, in which the conversation processing unit determines the speech content on the basis of the check information, and the presentation unit presents the product indicated by the order information to the user according to the speech content determined on the basis of the check information.

In the order terminal related to the twenty-fourth aspect, according to an order terminal related to a twenty-fifth aspect, the conversation processing unit determines, as the speech content, a product proposal for the user based on the emotion of the user determined by the emotion determination unit.

In the order terminal related to the twenty-fourth aspect, according to an order terminal related to a twenty-sixth aspect, the acquisition unit acquires consumption information regarding consumption of a product ordered by the user, and the conversation processing unit determines a proposal of a product for the user based on the consumption information acquired by the acquisition unit as the speech content.

In the order terminal related to the twenty-fourth aspect, according to an order terminal related to a twenty-seventh aspect, the reception unit receives the order information from a plurality of the users at a time, and the presentation unit presents the check information in which the order information from a plurality of the users received by the reception unit is aggregated to the user.

In the order terminal related to the twenty-fourth aspect, according to an order terminal related to a twenty-eighth aspect, the acquisition unit acquires configuration information regarding configurations of a plurality of the users, and the conversation processing unit determines, as the speech content, a proposal of a product for the plurality of the users based on the configuration information acquired by the acquisition unit.

A program related to a twenty-ninth aspect causes a computer to operate as the order terminal according to any one of the first to sixth aspects.

According to a thirtieth aspect of the disclosure, an electronic apparatus is provided. The electronic apparatus is capable of using an agent system including an emotion determination unit that determines an emotion of a user or an emotion of an agent, and an interaction processing unit that determines speech content of the agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, the electronic apparatus including an acquisition unit that acquires at least one of information regarding a state of the user or input of vocal sound including instruction information of the user, and an output unit that outputs the speech content generated in the agent system to the user, in which the emotion determination unit determines the emotion of the user on the basis of the state of the user obtained from the acquisition unit, and the interaction processing unit performs setting on the basis of the instruction information of the user.

According to a thirty-first aspect of the disclosure, an electronic apparatus is provided. The electronic apparatus is capable of using an agent system including an emotion determination unit that determines an emotion of a user or an emotion of an agent, and an interaction processing unit that determines speech content of the agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, the electronic apparatus including an acquisition unit capable of acquiring information regarding a state of the user, and an output unit capable of outputting the speech content generated in the agent system to the user, in which the output unit outputs speech information to another electronic apparatus capable of communicating with a computer network in a case in which communication with the computer network is disabled, and the acquisition unit acquires response information corresponding to the speech information from the another electronic apparatus.

According to a thirty-second aspect of the disclosure, an electronic apparatus is provided. The electronic apparatus is capable of using an agent system including an emotion determination unit that determines an emotion of a user or an emotion of an agent, and an interaction processing unit that determines speech content of the agent for interacting with the user on the basis of the emotion determined by the emotion determination unit, the electronic apparatus including an acquisition unit capable of acquiring information regarding a state of the user, and an output unit capable of outputting the speech content generated in the agent system to the user. The acquisition unit acquires a state of another user from another agent system, and the output unit outputs an emotion of the another user determined by the emotion determination unit on the basis of the state of the another user as the speech content of the agent.

Hereinafter, the disclosure will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Not all combinations of features described in the embodiments are essential to the solution of the invention.

1 FIG. 5 5 100 101 102 300 10 10 10 10 100 11 11 11 101 12 12 102 10 10 10 10 10 11 11 11 11 12 12 12 101 102 100 5 100 a b c d a b c a b a b c d a b c a b schematically illustrates an example of a systemaccording to a first embodiment. The systemincludes a robot, a robot, a robot, and a server. A user, a user, a user, and a userare users of the robot. The user, the user, and the userare users of the robot. A userand a userare users of the robot. In the description of the present embodiment, the user, the user, the user, and the usermay be collectively referred to as a user. The user, the user, and the usermay be collectively referred to as a user. The userand the usermay be collectively referred to as a user. The robotand the robothave substantially the same functions as those of the robot. Therefore, the systemwill be described focusing on the functions of the robot.

100 10 10 100 10 10 300 20 100 10 300 100 300 10 300 10 The robothas a conversation with the userand provides a video to the user. In this case, the robothas a conversation with the user, provides a video to the user, and the like in cooperation with the serverand the like that can communicate via a communication network. For example, the robotnot only learns an appropriate conversation by itself, but also performs learning so that a conversation with the usercan be advanced more appropriately in cooperation with the server. The robotcauses the serverto record captured video data and the like of the user, requests the serverto transmit the video data and the like as necessary, and provides the video data and the like to the user.

100 100 100 10 100 The robothas an emotion value indicating the type of its own emotion. For example, the robothas emotion values indicating the intensities of respective emotions such as “happy”, “angry”, “sad”, “pleasant”, “pleased”, “displeased”, “safe”, “anxious”, “gloomy”, “excited”, “worried”, “relieved”, “sense of fulfilment”, “sense of emptiness”, and “normal”. For example, in a case in which the robothas a conversation with the userin a state in which the emotion value of excitement is great, the robot emits vocal sound at a fast speed. As described above, the robotcan express its own emotion by action.

100 100 10 100 10 10 100 The robotmay be configured to determine an action of the robotcorresponding to the emotion of the userby matching a sentence generation model with an emotion engine. Specifically, the robotmay be configured to recognize an action of the user, determine an emotion of the userfor the action of the user, and determine an action of the robotcorresponding to the determined emotion.

100 10 100 10 More specifically, in a case in which the robotrecognizes the action of the user, the robotautomatically generates the action content to be taken by the robot with respect to the action of the userby using the preset sentence generation model. The sentence generation model may be interpreted as an algorithm and an operation for automatic interaction processing using letters. Since the sentence generation model is known as disclosed in, for example, Japanese Patent Application Laid-Open (JP-A) No. 2018 081444 and chatGPT (Internet search <URL: https://openai.com/blog/chatgpt>), a detailed description thereof will be omitted. Such a sentence generation model is configured by a large language model (LLM).

10 100 100 As described above, in the present embodiment, it is possible to reflect the emotions of the userand the robotand various types of linguistic information in the action of the robotby combining the large language model and the emotion engine. That is, according to the present embodiment, a synergistic effect can be obtained by combining the sentence generation model and the emotion engine.

100 10 100 10 10 10 100 100 10 The robothas a function of recognizing an action of the user. The robotrecognizes the action of the userby analyzing a face image of the useracquired by a camera function and the vocal sound of the useracquired by a microphone function. The robotdetermines an action to be executed by the roboton the basis of the recognized action of the useror the like.

100 100 10 100 10 The robotstores a rule defining an action to be executed by the roboton the basis of the emotion of the user, the emotion of the robot, and the action of the user, and performs various actions according to the rule.

100 100 10 100 10 10 100 100 10 100 10 100 10 Specifically, the robothas a reaction rule for determining an action of the roboton the basis of the emotion of the user, the emotion of the robot, and the action of the user. In the reaction rule, for example, in a case in which the action of the useris “smiling”, an action of “smiling” is set as the action of the robot. In the reaction rule, an action of “apologizing” is defined as an action of the robot, with respect to a case in which the action of the useris “angry”. In the reaction rule, an action of “answering” is defined as an action of the robotwith respect to a case in which the action of the useris “inquiry”. In the reaction rule, an action of “talking” is defined as an action of the robotwith respect to a case in which the action of the useris “gloomy”.

100 10 100 100 In a case in which the robotrecognizes that the action of the useris “angry” on the basis of the reaction rule, the robot selects an action of “apologizing” defined in the reaction rule as the action to be executed by the robot. For example, in the case of selecting the action of “apologizing”, the robotperforms an action of “apologizing” and outputs vocal sound expressing the word “apologizing”.

100 10 100 In a case in which a condition that the emotion of the robotis “normal” (that is, “happy”=0, “angry”=0, “sad”=0, and “pleasant”=0) and the state of the useris “one person, lonely” is satisfied, it is defined that the change content of the emotion that the emotion of the robotis “worried” and the action of “talking” can be executed.

100 100 10 100 100 10 100 In a case in which the robotrecognizes that the current emotion of the robotis “normal” and the useris alone in a lonely state on the basis of the reaction rule, the emotion value of “sad” of the robotis increased. The robotselects the action of “talking” defined in the reaction rule as an action to be executed on the user. For example, in a case in which the action of “talking” is selected, the robotconverts a word “What's wrong?” indicating that the robot is concerned into vocal sound indicating that the robot is concerned, and outputs the vocal sound.

100 300 10 100 10 10 The robottransmits, to the server, user reaction information indicating that a positive reaction has been obtained from the userthrough this action. The user reaction information includes, for example, the user action of “angry”, the action of the robotof “apologizing”, the positive reaction of the user, and an attribute of the user.

300 100 300 100 101 102 300 100 101 102 The serverstores the user reaction information received from the robot. The serverreceives and stores the user reaction information not only from the robotbut also from each of the robotand the robot. The serveranalyzes the user reaction information from the robot, the robot, and the robot, and updates the reaction rule.

100 300 300 100 100 100 101 102 The robotreceives the updated reaction rule from the serverby inquiring the serverabout the updated reaction rule. The robotincorporates the updated reaction rule into the reaction rule stored in the robot. As a result, the robotcan incorporate the reaction rule acquired by the robot, the robot, or the like into its own reaction rule.

2 FIG. 100 100 200 210 220 230 232 234 236 238 250 252 280 schematically illustrates a functional configuration of the robot. The robotincludes a sensor unit, a sensor module unit, a storage unit, a user state recognition unit, an emotion determination unit, an action recognition unit, an action determination unit, a storage control unit, an action control unit, a control target, and a communication processing unit.

252 100 100 100 100 100 100 Examples of the control targetinclude a display device, a speaker, an LED of an eye portion, and motors that drive an arm, a hand, a foot, and the like. The posture and the action of the robotare controlled by controlling motors for arms, hands, and feet. Some of the emotions of the robotcan be expressed by controlling these motors. The expression of the robotcan be expressed by controlling a light emission state of the LED of the eye portion of the robot. The posture, the gesture, and the expression of the robotare examples of the attitude of the robot.

200 201 202 203 204 201 201 100 202 203 203 204 200 The sensor unitincludes a microphone, a 3D depth sensor, a 2D camera, and a distance sensor. The microphonecontinuously detects vocal sound and outputs vocal sound data. The microphonemay be provided on the head of the robotand may have a function of performing binaural recording. The 3D depth sensoranalyzes infrared patterns from infrared images continuously captured by an infrared camera by continuously applying the infrared patterns, and detects an outline of an object. The 2D camerais an example of an image sensor. The 2D cameracaptures an image with visible light and generates image information regarding visible light. The distance sensordetects a distance to an object by emitting, for example, laser light or ultrasonic waves. The sensor unitmay further include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.

100 252 200 100 100 252 2 FIG. Among the constituents of the robotillustrated in, the constituents other than the control targetand the sensor unitare examples of the constituents included in the action control system included in the robot. The action control system of the robotcontrols the control target.

220 221 222 222 10 10 10 220 10 10 100 252 200 220 2 FIG. The storage unitincludes a reaction ruleand history data. The history dataincludes past emotion values and a history of actions of the user. The emotion values and the history of actions are recorded for each userby being associated with identification information regarding the user, for example. At least a part of the storage unitis implemented by a storage medium such as a memory. A person DB that stores a face image of the user, attribute information regarding the user, and the like may be included. Among the constituents of the robotillustrated in, the functions of the constituents other than the control target, the sensor unit, and the storage unitcan be realized by a CPU operating on the basis of a program. For example, the functions of these constituents can be implemented as the operation of the CPU by basic software (OS) and a program operating on the OS.

210 211 212 213 214 200 210 210 200 230 The sensor module unitincludes a vocal sound emotion recognition unit, a speech understanding unit, an expression recognition unit, and a face recognition unit. Information detected by the sensor unitis input to the sensor module unit. The sensor module unitanalyzes information detected by the sensor unitand outputs an analysis result to the user state recognition unit.

211 210 10 201 10 211 10 212 10 201 10 The vocal sound emotion recognition unitof the sensor module unitanalyzes the vocal sound of the userdetected by the microphoneto recognize the emotion of the user. For example, the vocal sound emotion recognition unitextracts a feature amount such as a frequency component of the vocal sound and recognizes the emotion of the useron the basis of the extracted feature amount. The speech understanding unitanalyzes the vocal sound of the userdetected by the microphoneand outputs letter information indicating the speech content of the user.

213 10 10 10 203 213 10 The expression recognition unitrecognizes the expression of the userand the emotion of the userfrom the image of the usercaptured by the 2D camera. For example, the expression recognition unitrecognizes the expression and the emotion of the useron the basis of the shapes, positional relationships, and the like of the eyes and the mouth.

214 10 214 10 10 203 The face recognition unitrecognizes the face of the user. The face recognition unitrecognizes the userby matching a face image stored in a person DB (not illustrated) with a face image of the usercaptured by the 2D camera.

230 10 210 210 The user state recognition unitrecognizes a state of the useron the basis of the information analyzed by the sensor module unit. For example, processing mainly related to perception is performed by using the analysis result from the sensor module unit. For example, perception information such as “There is one father.” and “The probability that the father does not smile is 90%.” is generated. Processing of understanding the meaning of the generated perception information is performed. For example, semantic information such as “One father looks lonely.” is generated.

232 10 210 10 230 210 10 10 The emotion determination unitdetermines an emotion value indicating the emotion of the useron the basis of the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit. For example, the information analyzed by the sensor module unitand the recognized state of the userare input to a neural network trained in advance, and an emotion value indicating the emotion of the useris acquired.

10 Here, the emotion value indicating the emotion of the useris a value indicating whether the emotion of the user is positive or negative. For example, in a case in which the emotion of the user is a bright emotion accompanied with pleasure or comfort, such as “happy”, “pleasant”, “pleased”, “safe”, “excited”, “relieved”, and “sense of fulfillment”, a positive value is indicated, and the value becomes larger as the emotion becomes brighter. In a case in which the emotion of the user is an emotion that makes the user feel unpleasant, such as “angry”, “sad”, “displeased”, “anxious”, “gloomy”, “worried”, and “sense of emptiness”, a negative value is indicated, and an absolute value of the negative value increases as the user feels unpleasant. In a case in which the emotion of the user is not included in any of the above emotions (“normal”), a value of 0 is indicated.

232 100 210 10 230 100 The emotion determination unitdetermines an emotion value indicating the emotion of the roboton the basis of the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit. The emotion value of the robotincludes an emotion value for each of the plurality of emotion classifications, and is, for example, a value (0 to 5) indicating the intensity of each of “happy”, “angry”, “sad”, and “pleasant”.

232 100 100 210 10 230 Specifically, the emotion determination unitdetermines an emotion value indicating the emotion of the robotaccording to a rule for updating the emotion value of the robot, the rule being defined in association with the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit.

230 10 232 100 230 10 100 For example, in a case in which the user state recognition unitrecognizes that the userlooks lonely, the emotion determination unitincreases the emotion value of “sad” of the robot. In a case in which the user state recognition unitrecognizes that the userhas a smiling face, the emotion value of “happy” of the robotis increased.

232 100 100 100 100 100 10 The emotion determination unitmay determine the emotion value indicating the emotion of the robotin further consideration of the state of the robot. For example, in a case in which a remaining battery level of the robotis low or in a case in which the surrounding environment of the robotis dark, the emotion value of “sad” of the robotmay be increased. In the case of the usercontinuously talking to the robot even though the remaining battery level is low, the emotion value of “angry” may be increased.

234 10 210 10 230 210 10 10 The action recognition unitrecognizes an action of the useron the basis of the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit. For example, the information analyzed by the sensor module unitand the recognized state of the userare input to a neural network trained in advance, probabilities of a plurality of predetermined action classifications (for example, “smile”, “angry”, “ask a question”, and “gloomy”) are acquired, and an action classification having the highest probability is recognized as the action of the user.

100 10 10 100 10 10 As described above, in the present embodiment, the robotacquires the speech content of the userafter identifying the user, but in acquiring and using the speech content, the action control system of the robotaccording to the present embodiment considers protection of personal information and privacy of the userin addition to acquiring necessary consent according to laws and regulations from the user.

236 10 234 10 232 222 232 10 100 236 222 10 236 10 236 10 100 100 236 100 100 The action determination unitdetermines an action corresponding to the action of the userrecognized by the action recognition uniton the basis of the current emotion value of the userdetermined by the emotion determination unit, the history dataof the past emotion values determined by the emotion determination unitbefore the current emotion value of the useris determined, and the emotion value of the robot. In the present embodiment, a case in which the action determination unituses one most recent emotion value included in the history dataas the past emotion value of the userwill be described, but the disclosed technology is not limited to this aspect. For example, the action determination unitmay use a plurality of most recent emotion values as the past emotion values of the user, or may use emotion values that are earlier by a unit period such as one day before. The action determination unitmay determine an action corresponding to the action of the userin further consideration of the history of the past emotion values of the robotin addition to the current emotion value of the robot. The action determined by the action determination unitincludes a gesture performed by the robotor speech content of the robot.

236 100 10 100 10 221 10 10 236 10 10 The action determination unitaccording to the present embodiment determines the action of the roboton the basis of a combination of the past emotion value and the current emotion value of the user, the emotion value of the robot, the action of the user, and the reaction rule, as an action corresponding to the action of the user. For example, in a case in which the past emotion value of the useris a positive value and the current emotion value is a negative value, the action determination unitdetermines an action for positively changing the emotion value of the useras an action corresponding to the action of the user.

221 100 10 100 10 10 10 10 100 The reaction ruledefines an action of the robotaccording to the combination of the past emotion value and the current emotion value of the user, the emotion value of the robot, and the action of the user. For example, in a case in which the past emotion value of the useris a positive value, the current emotion value is a negative value, and the action of the useris gloomy, a combination of a gesture and speech content at the time of making an inquiry to encourage the userincluding a gesture is defined as the action of the robot.

221 100 100 1296 10 10 100 100 10 10 236 100 222 10 For example, the reaction ruledefines an action of the robotfor all combinations of patterns of emotion values of the robot(patterns that is the fourth power of six values of the values “0” to “5” of “happy”, “angry”, “sad”, and “pleasant”), a pattern of the combination of the past emotion value and the current emotion value of the user, and an action pattern of the user. That is, for each pattern of the emotion value of the robot, the action of the robotaccording to the action pattern of the useris defined for each of a plurality of combinations such as combinations of the past emotion value and the current emotion value of the userbeing a negative value and a negative value, a negative value and a positive value, a positive value and a negative value, a positive value and a positive value, a negative value and normal, and normal and normal. The action determination unitmay transition to an operation mode of determining the action of the robotby using the history data, for example, in a case in which the userhas made speech that intends a conversation continued from a past topic such as “I want to talk about the topic I discussed earlier”.

221 100 1296 100 221 100 100 In the reaction rule, at least one of a gesture and statement content may be defined as an action of the robotfor each pattern (patterns) of the emotion values of the robotat the maximum. Alternatively, in the reaction rule, at least one of a gesture and statement content may be defined as an action of the robotfor each of groups of the patterns of the emotion values of the robot.

100 221 100 221 The intensity of each gesture included in the action of the robotdefined in the reaction ruleis defined in advance. In each piece of speech content included in the action of the robotdefined in the reaction rule, the intensity of the speech content is defined in advance.

238 10 222 236 100 232 100 236 236 10 222 The storage control unitdetermines whether or not to store data including the action of the userin the history dataon the basis of the intensity of the action predefined for the action determined by the action determination unitand the emotion value of the robotdetermined by the emotion determination unit. Specifically, in a case in which a total value of the intensities that is a sum of a sum total of the emotion values for the plurality of respective emotion classifications of the robot, the intensity predefined for the gesture included in the action determined by the action determination unit, and the intensity predefined for the speech content included in the action determined by the action determination unitis a threshold or more, it is determined to store data including the action of the userin the history data.

238 10 222 236 210 10 10 230 222 In a case in which the storage control unitdetermines to store the data including the action of the userin the history data, the action determined by the action determination unit, information (for example, any surrounding information including data such as vocal sound, an image, and a smell at the place) analyzed by the sensor module unitfrom the current time point to a certain period before, and a state (for example, the expression and the emotion of the user) of the userrecognized by the user state recognition unitare stored in the history data.

250 252 236 236 250 252 250 100 250 100 250 236 232 The action control unitcontrols the control targeton the basis of the action determined by the action determination unit. For example, in a case in which the action determination unitdetermines an action including speech, the action control unitcauses a speaker included in the control targetto output vocal sound. In this case, the action control unitmay determine an emission speed of the vocal sound on the basis of the emotion value of the robot. For example, the action control unitdetermines a higher vocal sound emission speed as the emotion value of the robotbecomes greater. As described above, the action control unitdetermines an execution form of the action determined by the action determination uniton the basis of the emotion value determined by the emotion determination unit.

250 10 236 10 10 200 200 10 10 200 10 10 280 The action control unitmay recognize a change in emotion of the userwith respect to execution of the action determined by the action determination unit. For example, the change in emotion may be recognized on the basis of the vocal sound or expression of the user. A change in the emotion of the usermay be recognized on the basis of detection of an impact by the touch sensor included in the sensor unit. In a case in which an impact is detected by the touch sensor included in the sensor unit, it may be recognized that the emotion of the useris worsened, or in a case in which it is determined that the reaction of the useris smiling or happy from the detection result in the touch sensor included in the sensor unit, it may be recognized that the emotion of the useris improved. Information indicating the reaction of the useris output to the communication processing unit.

250 236 100 232 100 232 100 236 250 232 100 236 250 After the action control unitexecutes the action determined by the action determination unitin the execution form determined according to the emotion of the robot, the emotion determination unitfurther changes the emotion value of the roboton the basis of the user's reaction to the execution of the action. Specifically, the emotion determination unitincreases the emotion value of “happy” of the robotin a case in which the user's reaction to the action determined by the action determination unitperformed on the user in the execution form determined by the action control unitis not bad, and the emotion determination unitincreases the emotion value of “sad” of the robotin a case in which the user's reaction to the action determined by the action determination unitperformed on the user in the execution form determined by the action control unitis poor.

250 100 100 100 250 252 100 100 250 252 100 The action control unitexpresses the emotion of the roboton the basis of the determined emotion value of the robot. For example, in a case in which the emotion value of “happy” of the robotis increased, the action control unitcontrols the control targetto cause the robotto perform a gesture of being happy. In a case in which the emotion value of “sad” of the robotis increased, the action control unitcontrols the control targetso that the posture of the robotbecomes a head-drooping posture.

280 300 280 300 280 300 300 280 221 The communication processing unitperforms communication with the server. As described above, the communication processing unittransmits the user reaction information to the server. The communication processing unitreceives the updated reaction rule from the server. Upon receiving the updated reaction rule from the server, the communication processing unitupdates the reaction rule.

300 100 101 102 300 100 The serverperforms communication between the robot, the robot, and the robotand the server, receives the user reaction information transmitted from the robot, and updates the reaction rule on the basis of the reaction rule including the action for which a positive reaction has been obtained.

3 FIG. 3 FIG. 100 210 schematically illustrates an example of an operation flow related to an operation of determining an action of the robot. The operation flow illustrated inis repeatedly executed. In this case, it is assumed that information analyzed by the sensor module unitis input. “S” in the operation flow represents a step to be executed.

100 230 10 210 First, in step S, the user state recognition unitrecognizes a state of the useron the basis of the information analyzed by the sensor module unit.

102 232 10 210 10 230 In step S, the emotion determination unitdetermines an emotion value indicating the emotion of the useron the basis of the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit.

103 232 100 210 10 230 232 10 222 In step S, the emotion determination unitdetermines an emotion value indicating the emotion of the roboton the basis of the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit. The emotion determination unitadds the determined emotion value of the userto the history data.

104 234 10 210 10 230 In step S, the action recognition unitrecognizes an action classification of the useron the basis of the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit.

106 236 100 10 102 222 100 10 234 221 In step S, the action determination unitdetermines an action of the roboton the basis of a combination of the current emotion value of the userdetermined in step Sand the past emotion value included in the history data, the emotion value of the robot, the action of the userrecognized by the action recognition unit, and the reaction rule.

108 250 252 236 In step S, the action control unitcontrols the control targeton the basis of the action determined by the action determination unit.

110 238 236 100 232 In step S, the storage control unitcalculates a total value of the intensities on the basis of the intensity of the action predefined for the action determined by the action determination unitand the emotion value of the robotdetermined by the emotion determination unit.

112 238 10 222 114 In step S, the storage control unitdetermines whether or not the total value of the intensities is a threshold or more. In a case in which the total value of the intensities is less than the threshold, the data including the action of the useris not stored in the history data, and the processing is ended. On the other hand, in a case in which the total value of the intensities is the threshold or more, the processing proceeds to step S.

114 236 210 10 230 222 In step S, the action determined by the action determination unit, the information analyzed by the sensor module unitfrom the current time point to a certain period before, and the state of the userrecognized by the user state recognition unitare stored in the history data.

100 100 10 222 100 222 10 100 100 222 10 10 10 As described above, the robotdetermines the emotion value indicating the emotion of the roboton the basis of the user state, and determines whether or not to store data including the action of the userin the history dataon the basis of the emotion value of the robot. As a result, the capacity of the history datathat stores data including an action of the usercan be suppressed. For example, in a case in which the robotdetermines that the user state of 10 years ago is the same as the user state after 10 years, the robotreads the history dataof 10 years ago, and thus, can present the state of the userof 10 years ago (for example, an expression and an emotion of the user), and any surrounding information including data such as vocal sound, an image, and a smell at the place to the user.

100 100 10 100 10 10 According to the robot, it is possible to cause the robotto execute an appropriate action with respect to the action of the user. Conventionally, an action of a user is classified to determine an action including an expression or an appearance of a robot. On the other hand, the robotdetermines the current emotion value of the user, and executes an action on the useron the basis of the past emotion value and the current emotion value.

10 100 100 10 100 10 100 100 10 Therefore, for example, in a case in which the userwho was fine yesterday is depressed today, the robotmay say, “You were fine yesterday. What's wrong with you today?”. The robotmay also make speech with a gesture. For example, in a case in which the userwho was depressed yesterday is fine today, the robotmay say, “You were depressed yesterday, but you look fine today?”. For example, in a case in which the userwho was fine yesterday is better today than yesterday, the robotmay say, “You are better today than yesterday. What's better than yesterday?”. For example, the robotmay make speech such as “Recently, the mood is stable, which is good.” to the userwhose emotion value is 0 or more and whose state in which the fluctuation range of the emotion value is within a certain range continues.

100 10 10 10 100 100 10 10 100 For example, in a case in which the robotasks a question of “Did you finish the homework that you said yesterday?” to the userand an answer of “I did it” is obtained from the user, the robot may make affirmative speech such as “Good!” and make an affirmative gesture such as applause or thumbs-up. For example, in a case in which the usersays, “The presentation we discussed the day before yesterday was successful”, the robotcan make affirmative speech such as “Good job!” and also make the above affirmative gesture. As described above, the robotperforms an action based on the history of the state of the user, whereby the usercan be expected to feel a sense of closeness to the robot.

100 10 10 100 10 10 10 10 10 In the above embodiment, the case in which the robotrecognizes the userby using the face image of the userhas been described, but the disclosed technology is not limited to this aspect. For example, the robotmay recognize the userby using vocal sound uttered by the user, a mail address of the user, an ID of an SNS of the user, an ID card in which a wireless IC tag is built and which is possessed by the user, or the like.

100 100 300 300 300 The robotis an example of an electronic apparatus including an action control system. An application target of the action control system is not limited to the robot, and the action control system can be applied to various electronic apparatuses. The functions of the servermay be implemented by one or more computers. At least some functions of the servermay be implemented by a virtual machine. At least some of the functions of the servermay be implemented in a cloud.

4 FIG. 1200 100 300 1200 1200 1200 1200 1212 1200 schematically illustrates an example of a hardware configuration of a computerthat functions as the robotand the server. A program installed in the computercan cause the computerto function as one or more “units” of the device according to the present embodiment, or cause the computerto execute an operation associated with the device according to the present embodiment or one or more “units” thereof, and/or cause the computerto execute a process according to the present embodiment or a stage of the process. Such a program may be executed by a CPUto cause the computerto execute certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

1200 1212 1214 1216 1210 1200 1222 1224 1226 1210 1220 1226 1224 1200 1230 1220 1240 The computeraccording to the present embodiment includes a CPU, a RAM, and a graphic controller, which are connected to each other via a host controller. The computeralso includes input/output units such as a communication interface, a storage device, a DVD drive, and an IC card drive, which are connected to the host controllervia an input/output controller. The DVD drivemay be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage devicemay be a hard disk drive, a solid state drive, or the like. The computeralso includes a ROMand legacy input/output units such as a keyboard, which are connected to the input/output controllervia an input/output chip.

1212 1230 1214 1216 1212 1214 1218 The CPUoperates according to programs stored in the ROMand the RAM, thereby controlling each unit. The graphic controlleracquires image data generated by the CPUin a frame buffer or the like provided in the RAMor itself, and causes the image data to be displayed on the display device.

1222 1224 1212 1200 1226 1227 1224 The communication interfacecommunicates with other electronic devices via a network. The storage devicestores programs and data used by the CPUin the computer. The DVD drivereads a program or data from the DVD-ROMor the like and provides the program or data to the storage device. The IC card drive reads a program and data from the IC card and/or writes the program and data to the IC card.

1230 1200 1200 1240 1220 The ROMstores therein a boot program or the like executed by the computerat the time of activation and/or a program depending on hardware of the computer. The input/output chipmay also connect various input/output units to the input/output controllervia a USB port, a parallel port, a serial port, a keyboard port, a mouse port, or the like.

1227 1224 1214 1230 1212 1200 1200 The program is provided by a computer readable storage medium such as the DVD-ROMor an IC card. The program is read from a computer readable storage medium, installed in the storage device, the RAM, or the ROM, which is also an example of a computer readable storage medium, and executed by the CPU. The information processing described in such a program is read by the computerand provides cooperation between the program and the various types of hardware resources. A device or a method may be configured by implementing an operation or processing of information according to use of the computer.

1200 1212 1214 1222 1212 1222 1214 1224 1227 For example, in a case in which communication is executed between the computerand an external device, the CPUmay execute a communication program loaded in the RAMand instruct the communication interfaceto perform communication processing on the basis of processing described in the communication program. Under the control of the CPU, the communication interfacereads transmission data stored in a transmission buffer area provided in a recording medium such as the RAM, the storage device, the DVD-ROM, or the IC card, transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

1212 1214 1224 1226 1227 1214 1212 The CPUmay cause the RAMto read all or a necessary portion of a file or database stored in an external recording medium such as the storage device, the DVD drive(DVD-ROM), or the IC card, and may execute various types of processing on data in the RAM. Next, the CPUmay write back the processed data to the external recording medium.

1212 1214 1214 1212 1212 Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium and subjected to information processing. The CPUmay execute various types of processing on the data read from the RAM, including various types of operations, information processing, condition determination, conditional branching, unconditional branching, information retrieval/replacement, and the like, which are described throughout the disclosure and specified by a command sequence of a program, and writes back the results to the RAM. The CPUmay search for information in a file, a database, or the like in the recording medium. For example, in a case in which a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute is stored in the recording medium, the CPUmay search for an entry in which the attribute value of the first attribute matches a specified condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and thus acquire the attribute value of the second attribute associated with the first attribute satisfying a predetermined condition.

1200 1200 1200 The programs or software modules described above may be stored in the computeror a computer readable storage medium near the computer. A recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a computer readable storage medium, and provides a program to the computervia the network.

Blocks in the flowcharts and the block diagrams in the present embodiments may represent stages of a process in which an operation is performed or “units” of a device having a role of performing an operation. Specific stages and “units” may be implemented by a dedicated circuit, a programmable circuit supplied along with computer readable instructions stored on a computer readable storage medium, and/or a processor supplied along with computer readable instructions stored on a computer readable storage medium. Dedicated circuits may include digital and/or analog hardware circuits, and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuits may include reconfigurable hardware circuits including, for example, AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as field programmable gate arrays (FPGA) and programmable logic arrays (PLA).

Computer readable storage media may include any tangible device capable of storing instructions executed by a suitable device, and, as a result, a computer readable storage medium having instructions stored thereon has a product including instructions that may be executed to create means for executing operations designated in the flowcharts or block diagrams. Examples of the computer readable storage medium may include an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, and a semiconductor storage medium. More specific examples of the computer readable storage medium may include a Floppy (registered trademark) disk, a diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (an EPROM or a flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a Blu-Ray (registered trademark) disk, a memory stick, and an integrated circuit card.

The computer readable instructions may include either source codes or object codes written in any combination of one or more programming languages, including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or an object oriented programming language such as Smalltalk (registered trademark), or JAVA (registered trademark), C++, and conventional procedural programming languages such as the “C” programming language or similar programming languages.

The computer readable instructions may be provided to a processor of a general purpose computer, a special purpose computer, or another programmable data processing device, or a programmable circuit locally or via a local area network (LAN) or a wide area network (WAN) such as the Internet, in order to cause the processor of the general purpose computer, the special purpose computer, or another programmable data processing device or the programmable circuit to execute the computer readable instructions to generate means for executing operations designated in the flowcharts or the block diagrams. Examples of the processor include a computer processor, a processing unit, a microprocessor, a digital signal processor, a controller, and a microcontroller.

Although the technology of the disclosure has been described by using each embodiment, the technology of the disclosure is not limited to the scope disclosed in each embodiment. It is apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It is apparent from the description of the claims that modes to which such modifications or improvements are added can also be included in the technical scope of the disclosure.

The order of execution of each piece of processing of operations, procedures, steps, stages, and the like in the devices, the systems, the programs, and the methods illustrated in the claims, the specification, and the drawings can be realized in any order unless “before”, “prior to”, or the like is explicitly stated, and unless the output of the previous processing is used in the later processing. Even in a case in which the operation flow in the claims, the specification, and the drawings is described by using “first,”, “next,”, and the like for convenience, this does not mean that it is essential to perform in this order.

232 232 5 FIG. The emotion determination unitmay determine an emotion of the user according to a specific mapping. Specifically, the emotion determination unitmay determine an emotion of the user on the basis of an emotion map (see) that is a specific mapping.

5 FIG. 300 300 300 is a diagram illustrating an emotion mapon which a plurality of emotions are mapped. In the emotion map, emotions are disposed concentrically radially from the center. The closer to the center of the concentric circle, the more the emotion of the primitive state is disposed. Emotions indicating states and actions generated from the state of mind are disposed outside the concentric circle. The emotion is a concept including an affection and a mental state. On the left side of the concentric circle, emotions generated from reactions generally occurring in the brain are disposed. On the right side of the concentric circle, emotions induced by situation determination are generally disposed. In the up and down directions of the concentric circles, emotions generated from reactions generally occurring in the brain and induced by situational judgment are disposed. The emotion of “pleased” is disposed on the upper side of the concentric circle, and the emotion of “displeased” is disposed on the lower side. As described above, in the emotion map, a plurality of emotions are mapped on the basis of a structure in which emotions occur, and emotions that are likely to occur at the same time are mapped close to each other.

232 100 100 (1) For example, in a case in which the emotion engine, which is the emotion determination unitof the robot, detects an emotion at about 100 msec, determination of the reaction operation (for example, a quick-response) of the robotmay be performed at a timing at which a detection frequency is set to be similar to at least the detection frequency (100 msec) of the emotion engine, or may be performed at a timing at which the detection frequency is set to be higher than the detection frequency. The detection frequency of the emotion engine may be interpreted as a sampling rate.

100 300 The emotion is detected in about 100 msec, and the reaction operation (for example, a quick-response) is performed immediately in conjunction with the detection, whereby an unnatural quick-response is eliminated, and an interaction that naturally reads the atmosphere can be realized. The robotperforms a reaction operation (quick-response or the like) according to the directionality and the degree (intensity) of the mandala of the emotion map. The detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to a situation (for example, in the case of playing sports), the age of the user, or the like.

300 100 100 100 100 (2) In comparison with the emotion map, the directionality of the emotion and the intensity of the degree thereof may be set in advance, and the quick-response motion and the strength of the quick-response may be set. For example, in a case in which the robotfeels a sense of stability, safety, or the like, the robotcontinues listening to the speech while nodding. In a case in which the robotfeels anxious, hesitant, or suspicious, the robotmay tilt its head or stop swinging its head.

300 300 These emotions are distributed in the 3:00 direction of the emotion map, and usually come and go between safety and anxiety. In the right half of the emotion map, situation recognition is superior to internal sensation, and thus gives a calm impression.

100 100 300 (3) In a case in which the robotfeels good in the case of receiving the compliment, a filler “Oh” may come in front of the line. In a case in which the robot feels pain in the case of receiving harsh words, a filler “Ugh!” may come in front of the line. A physical reaction such as a gesture of the robotcrouching while saying “Ohh!” may be included. These emotions are distributed around the 9:00 o'clock position on the emotion map.

300 (4) In the left half of the emotion map, internal sensation (reaction) is superior to situation recognition. Therefore, an impression of unintentional reaction may be given.

100 100 100 300 In a case in which the robothas a favorable emotion in situation recognition while remembering internal sensation (reaction) of satisfaction, the robotmay nod deeply while looking at the other party, or may say “yeah, yeah”. As described above, the robotmay generate a balanced favorable emotion to the other party, that is, an action such as allowance or tolerance to the other party. Such emotions are distributed around 12:00 in the emotion map.

100 100 300 In contrast, in a case in which the robotfeels antipathy in the situation recognition while remembering internal sensation (reaction) of displeasure, the robotmay shake its head sideways, and may turn red the LED of the eye and look at the other party in the case of emotion hatred. Such emotions are distributed around 6:00 in the emotion map.

300 300 300 (5) Since the inside of the emotion maprepresents the inside of the mind and the outside of the emotion maprepresents an action, the emotion is more visible (appears in the action) toward the outside of the emotion map.

100 300 (6) In a case in which the robotlistens to a person's speech while remembering the safety distributed around 3:00 on the emotion map, the robot slightly shakes its head vertically and says “uh-huh”. However, in the direction of love around 12:00, the robot may perform strong nodding such as shaking its head deeply vertically.

232 210 10 300 10 210 10 300 900 6 FIG. 6 FIG. The emotion determination unitinputs the information analyzed by the sensor module unitand the recognized state of the userto a neural network trained in advance, acquires an emotion value indicating each emotion indicated in the emotion map, and determines an emotion of the user. This neural network is trained in advance on the basis of a plurality of pieces of learning data that is a combination of the information analyzed by the sensor module unitand the recognized state of the userand the emotion value indicating each emotion indicated in the emotion map. This neural network is trained such that, as in an emotion mapillustrated in, emotions disposed close to each other have close values.illustrates an example in which a plurality of emotions such as “safe”, “calm”, and “reassuring” have close emotion values.

232 100 232 210 10 230 100 300 100 210 10 100 300 100 10 100 10 900 6 FIG. The emotion determination unitmay determine an emotion of the robotaccording to a specific mapping. Specifically, the emotion determination unitinputs the information analyzed by the sensor module unit, the state of the userrecognized by the user state recognition unit, and the state of the robotto a neural network trained in advance, acquires an emotion value indicating each emotion indicated in the emotion map, and determines an emotion of the robot. This neural network is trained in advance on the basis of a plurality of pieces of learning data that is a combination of the information analyzed by the sensor module unit, the recognized state of the user, the state of the robot, and the emotion value indicating each emotion indicated in the emotion map. For example, the neural network is trained on the basis of learning data indicating that the emotion value “3” of “happy” is obtained in a case in which the robotis recognized as being petted by the userfrom the output of the touch sensor (not illustrated), and learning data indicating that the emotion value “3” of “angry” is obtained in a case in which the robotis recognized as being hit by the userfrom the output of the acceleration sensor (not illustrated). This neural network is trained such that, as in the emotion mapillustrated in, emotions disposed close to each other have close values.

236 The action determination unitadds a fixed sentence for inquiry about the action content of the robot corresponding to the action of the user to text representing the action of the user, the emotion of the user, and the emotion of the robot, and inputs the text to the interaction function, thereby generating the action content of the robot.

236 100 100 232 100 For example, the action determination unitacquires text indicating the state of the robotfrom the emotion of the robotdetermined by the emotion determination unitby using an emotion table as illustrated in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and a text indicating the state of the robotis stored for each index number.

100 232 100 100 In a case in which the emotion of the robotdetermined by the emotion determination unitcorresponds to the index number “2”, text “very pleasant state” is obtained. In a case in which the emotion of the robotcorresponds to a plurality of index numbers, a plurality of pieces of text indicating the state of the robotare obtained.

10 100 10 236 An emotion table as illustrated in Table 2 is prepared for an emotion of the user. Here, in a case in which the action of the user is to say “***”, the emotion of the robotis the index number “2”, and the emotion of the useris the index number “3”, a sentence such as “The robot is in a very pleasant state. The user is in a normally pleasant state. The user says,” ***”. How do you answer as a robot?” is input to the sentence generation model to acquire the action content of the robot. The action determination unitdetermines an action of the robot from the action content.

TABLE 1 Index Type of Emotion State of Number Emotion Value Robot 1 Pleasant 5 Very Pleasant State 2 Pleasant 4 Extremely Pleasant State 3 Pleasant 3 Normally Pleasant State 4 Pleasant 2 Slightly Pleasant State 5 Pleasant 1 Only a Little Pleasant State . . . . . . . . . . . .

TABLE 2 Index Type of Emotion State of Number Emotion Value User 1 Pleasant 5 Very Pleasant State 2 Pleasant 4 Extremely Pleasant State 3 Pleasant 3 Normally Pleasant State 4 Pleasant 2 Slightly Pleasant State 5 Pleasant 1 Only a Little Pleasant State . . . . . . . . . . . .

100 That is, since the robotcan change the action of the robot according to the index number corresponding to the emotion of the robot, the user has an impression that the robot has a mind, and is promoted to take an action such as talking to the robot.

236 222 100 The action determination unitmay generate the action content of the robot by adding a fixed sentence for inquiry about the action content of the robot corresponding to the action of the user and inputting the fixed sentence to the sentence generation model having the interaction function after adding not only the text indicating the action of the user, the emotion of the user, and the emotion of the robot but also the text indicating the content of the history data. As a result, the robotcan change the action of the robot according to the history data indicating the emotion or the action of the user, and thus, the user has an impression that the robot has personality, and is promoted to take an action such as talking to the robot. The history data may further include an emotion or an action of the robot.

100 The robotmay be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.

100 50 50 10 10 10 7 FIG. In the above description, the case in which the action control system is applied to the robothas been exemplified, but the disclosure is not limited to this aspect.is a functional block diagram of an agent systemconfigured by using some or all of the functions of the action control system. The agent systemis a computer system that performs a series of actions according to the intention of the userthrough an interaction with the user. The interaction with the usercan be performed by vocal sound or text.

50 50 The agent systemcan be mounted on, for example, a robot, a doll, a stuffed toy, a pendant, a smart watch, a smartphone, a smart speaker, an earphone, a camera, and a personal computer. The agent systemmay be implemented in a web server and used via a web browser operating on a communication terminal such as a smartphone possessed by the user.

50 10 50 10 50 70 The agent systemserves as, for example, a butler, a secretary, a teacher, a partner, a friend, a lover, or a teacher acting for the user. The agent systemnot only interacts with the userbut also provides advice, guidance to a destination, recommendation according to user's preference, or the like. The agent systemperforms reservation, order, payment, or the like on a service provider.

50 53 10 10 60 50 10 50 10 10 50 10 50 10 The agent systemincludes an emotion engineto determine an emotion of the userand an emotion of the agent itself, and acts for the userwhile taking into account the emotions of the userand the agent. Here, the agent is an agent that operates on software. In other words, the agent systemunderstands the emotion of the userand reads the mood to realize heartfelt support, assistance, advice, and service provision. The agent systemcomforts, encourages, and energizes the userby attending to the concern of the user. The agent systemplays with the userand draws a picture diary to remind the user of the past. The agent systemperforms an action that increases the sense of happiness of the user.

50 51 52 53 54 55 56 57 The agent systemincludes an interaction processing unit, a character setting unit, an emotion engine, an interaction history storage unit, a sentence generation model, a command acquisition unit, and a Robotic Process Automation (RPA).

51 236 250 51 10 The interaction processing unitincludes an action determination unitand an action control unitthat perform processing related to speech. The interaction processing unitgenerates speech content of an agent for interacting with the user, and outputs the speech content by at least one of vocal sound or text.

52 52 50 10 10 51 52 10 10 51 52 10 9 FIG. The character setting unitsets a character of an agent (corresponding to an agentA illustrated in) in a case in which the agent systeminteracts with the useron the basis of designation from the user. That is, the speech content output from the interaction processing unitis output through the agent having the set character. As the character, for example, a real celebrity or famous person such as an actor, an entertainer, an idol, or an athlete can be set. It is also possible to set a fictitious character appearing in a cartoon, a movie, or an animation. For example, it is possible to set, as the character of the agent, “Queen Anne” performed by “Audrey Hepburn” appearing in the movie “Roman Holiday”. In a case in which the agent's character is known, since the vocal sound, the wording, the tone, and the personality of the character are known, the prompt setting in the character setting unitis automatically performed only by the userdesignating his/her favorite character. The vocal sound, the wording, the tone, and the personality of the set character are reflected in the interaction with the user. In other words, the interaction processing unitsynthesizes vocal sound corresponding to the character set by the character setting unit, and outputs the speech content of the agent by the synthesized vocal sound. As a result, the usercan feel as if he/she is interacting with his/her favorite character (for example, a favorite actor).

50 52 50 10 10 50 10 In a case in which the agent systemis mounted on a device having a display such as a smartphone, for example, an icon, a still image, or a moving image of an agent having a character set by the character setting unitmay be displayed on the display. The image of the agent is generated by using, for example, an image synthesis technology such as 3D rendering. In the agent system, an interaction with the usermay be performed while the image of the agent makes a gesture according to the emotion of the user, the emotion of the agent, and the speech content of the agent. The agent systemmay output only vocal sound without outputting an image in the case of interacting with the user.

53 232 10 50 10 10 51 53 The emotion engineincludes an emotion determination unit, and determines an emotion value indicating the emotion of the userand an emotion value of the agent itself. The emotion value of the agent itself is reflected in the emotion of the set character. In a case in which the agent systeminteracts with the user, not only the emotion of the userbut also the emotion of the character is reflected in the interaction. That is, the interaction processing unitoutputs the speech content in an aspect according to the emotion determined by the emotion engine.

50 10 10 50 50 10 10 The emotion of the agent is also reflected in a case in which the agent systemperforms an action toward the user. For example, in a case in which the userrequests the agent systemto take a photograph, whether or not the agent systemtakes a photograph in response to the request of the user is determined according to the degree of the emotion of “gloomy” of the agent. In a case in which the agent has a positive emotion, the agent performs a favorable interaction or action on the user, and in a case in which the agent has a negative emotion, the agent performs a defiant interaction or action on the user.

54 10 50 54 222 220 10 10 50 54 50 10 54 50 10 51 54 54 10 10 10 54 10 The interaction history storage unitis a storage medium for storing a history of interactions performed between the userand the agent system. The interaction history storage unitmay be realized by the history dataof the storage unitor may be realized by an external cloud storage. In the case of interacting with the useror performing an action toward the user, the agent systemdetermines the interaction content or the action content in consideration of the content of the interaction history stored in the interaction history storage unit. For example, the agent systemascertains the hobby and the preference of the useron the basis of the interaction history stored in the interaction history storage unit. The agent systemgenerates interaction content matching the hobby and the preference of the userand provides a recommendation. The interaction processing unitgenerates speech content on the basis of the interaction history stored in the interaction history storage unit. The interaction history storage unitstores personal information such as a name, an address, a telephone number, and a credit card number of the useracquired through an interaction with the user. Here, the agent may voluntarily make speech such as “Is credit card number to be registered?”, to the userto ask a question about whether or not to register the personal information, and may store the personal information in the interaction history storage unitaccording to an answer of the user.

55 55 The sentence generation modelgenerates a sentence that is a response to the input text or vocal sound. The sentence generation modelmay be interpreted as an algorithm and an operation for automatic interaction processing using letters. Since the sentence generation model is known as disclosed in, for example, JP-A No. 2018-081444 and chatGPT (Internet search <URL: https://openai.com/blog/chatgpt>), a detailed description thereof will be omitted. Such a sentence generation model is configured by a large language model (LLM).

51 55 51 10 10 53 54 55 51 52 55 50 55 10 50 The interaction processing unitgenerates speech content on the basis of the sentence generated by the sentence generation model. Specifically, the interaction processing unitinputs the text or vocal sound input by the user, the emotions of both the userand the agent specified by the emotion engine, and the conversation history stored in the interaction history storage unitto the sentence generation model, and generates the speech content of the agent. In this case, the interaction processing unitmay further input the character's personality set by the character setting unitto the sentence generation modelto generate the speech content of the agent. In the agent system, the sentence generation modelis not located on the front-end side serving as a touch point with the user, but is used as a tool of the agent system.

56 212 10 10 50 The command acquisition unitincludes the speech understanding unit, and acquires an agent command from vocal sound output from the useror text through an interaction with the user. The command includes, for example, content of actions to be executed by the agent system, such as information search, store reservation, ticket arrangement, purchase of products/services, payment, route guidance to a destination, and recommendation provision.

57 56 57 70 57 The RPAperforms an action according to the command acquired by the command acquisition unit. The RPAperforms actions related to use of the service provider, such as information search, store reservation, ticket arrangement, purchase of products/services, and payment. The RPAis an example of an “operation execution unit” in the disclosure.

57 10 70 54 10 50 10 54 10 50 10 10 10 The RPAreads the personal information regarding the usernecessary for executing the action related to the use of the service providerfrom the interaction history storage unitand uses the personal information. For example, in the case of purchasing a product in response to a request from the user, the agent systemreads and uses personal information such as a name, an address, a telephone number, and a credit card number of the userstored in the interaction history storage unit. It is unkind to request the userto input personal information in the initial setting, and it is also unpleasant for the user. In the agent systemaccording to the present embodiment, instead of requesting the userto input personal information in the initial setting, the personal information acquired through the interaction with the useris stored, and read and used as necessary. As a result, it is possible to avoid making the user feel unpleasant, and convenience of the useris improved.

50 1 6 The agent systemexecutes the interaction processing according to the following Stepsto.

1 50 52 50 10 10 (step) The agent systemsets a character of the agent. Specifically, the character setting unitsets a character of the agent in a case in which the agent systeminteracts with the useron the basis of the designation from the user.

2 50 10 10 10 100 222 100 103 10 10 10 100 222 (step) The agent systemacquires the state of the userincluding the vocal sound or text input from the user, the emotion value of the user, the emotion value of the robot, and the history data. Specifically, the processing similar to steps Sto Sis performed to acquire the state of the userincluding the vocal sound or text input from the user, the emotion value of the user, the emotion value of the robot, and the history data.

3 50 51 10 10 53 54 55 10 60 53 54 55 10 10 (step) The agent systemdetermines speech content of the agent. Specifically, the interaction processing unitinputs the text or vocal sound input by the user, the emotions of both the userand the character specified by the emotion engine, and the history of conversations stored in the interaction history storage unitto the sentence generation model, and generates the speech content of the agent. For example, a fixed sentence “In this case, how do you answer as an agent?” is added to the text or vocal sound input by the userand the text representing the emotions of both the userand the character specified by the emotion engineand the interaction history stored in the conversation history storage unit, and the text is input to the sentence generation modelto acquire the speech content of the agent. As an example, in a case in which the text or the vocal sound input by the useris “I need a reservation at 7 μm at a good Chinese restaurant nearby.”, “Understood.”, and “These are recommended restaurants. 1.AAAA. 2.BBBB. 3.CCCC. and 4.DDDD” are acquired as the speech content of the agent. In a case in which the text or the vocal sound input by the useris “Fourth DDDD is good”, “Yes. I will make a reservation. How many seats do you need?” are acquired as the speech content of the agent.

4 50 51 52 (step) The agent systemoutputs the speech content of the agent. Specifically, the interaction processing unitsynthesizes vocal sound corresponding to the character set by the character setting unit, and outputs the speech content of the agent by the synthesized vocal sound.

5 50 51 55 6 2 (step) The agent systemdetermines whether or not a timing to execute a command of the agent has arrived. Specifically, the interaction processing unitdetermines whether or not the timing to execute the command of the agent has arrived on the basis of the output of the sentence generation model. For example, in a case in which the output of the sentence generation model includes that the agent executes the command, it is determined that the timing to execute the command of the agent has arrived, and the processing proceeds to Step. On the other hand, in a case in which it is determined that it is not the timing to execute the command of the agent, the processing returns to Stepdescribed above.

6 50 56 10 10 57 56 10 51 55 51 52 10 51 51 55 51 52 2 (step) The agent systemexecutes the command of the agent. Specifically, the command acquisition unitacquires the command of the agent from the vocal sound output from the useror the text through an interaction with the user. The RPAperforms an action corresponding to the command acquired by the command acquisition unit. For example, in a case in which the command is “information search”, information search is performed by a search site by using a search query obtained through an interaction with the userand an application programming interface (API). The interaction processing unitinputs the search result to the sentence generation modeland generates the speech content of the agent. The interaction processing unitsynthesizes vocal sound corresponding to the character set by the character setting unit, and outputs the speech content of the agent by the synthesized vocal sound. In a case in which the command is “store reservation”, the reservation is made by making a phone call to the reservation destination store by using the reservation information obtained through the interaction with the user, the reservation destination store information, and the API by using phone software. In this case, the interaction processing unitacquires the speech content of the agent with respect to the vocal sound input from the other party by using the sentence generation model having the interaction function. The interaction processing unitinputs the result of the store reservation (whether or not the reservation is correct) to the sentence generation model, and generates the speech content of the agent. The interaction processing unitsynthesizes vocal sound corresponding to the character set by the character setting unit, and outputs the speech content of the agent by the synthesized vocal sound. The processing returns to Stepdescribed above.

6 54 54 50 10 10 In Step, the result of the action (for example, store reservation) executed by the agent is also stored in the interaction history storage unit. The result of the action executed by the agent stored in the interaction history storage unitis used by the agent systemto ascertain a hobby or a preference of the user. For example, in a case in which the same store is reserved a plurality of times, it is recognized that the userlikes the store, or the reserved time period, or the reservation content such as the content of the course or the fee is used as a criterion for choosing the store at the time of the next reservation.

50 70 50 50 As described above, the agent systemcan execute interaction processing and perform an action related to use of the service provideras necessary. While the user is sleeping, the agent systemmay search for news or the like that the user is interested in on the basis of the taste of the user acquired in advance, and in a case in which the user wakes up, the agent systemmay deliver the news from itself. For example, the agent says “The Yankees seems to have won the game yesterday. Let's go to the baseball park to cheer on this evening.” In a case in which the user says “I need to consult father”, the father says “I doubt whether a ticket can be obtained”. The agent says “I will check whether a ticket can be obtained”, and the request for a seat or the like is interrogated from an interaction with the user to secure a ticket.

8 9 FIGS.and 8 FIG. 50 50 10 50 10 10 10 are diagrams illustrating an example of the operation of the agent system.illustrates an aspect in which the agent systemmakes a restaurant reservation through an interaction with the user. The agent systemcan ascertain a preference of the userbased on an interaction history with the user, provide a recommendation list of restaurants that match the preference of the user, and perform a reservation of a selected restaurant.

9 FIG. 50 10 50 10 10 10 50 10 50 10 10 On the other hand,illustrates an aspect in which the agent systemaccesses a mail order site through an interaction with the userto purchase a product. The agent systemcan estimate a remaining amount of the beverage stocked by the useron the basis of the interaction history with the user, and can propose and execute the purchase of the beverage to the user. The agent systemcan ascertain the taste of the user on the basis of the past interaction history with the user, and recommend a snack that the user likes. As described above, the agent systemsupports the daily life of the userby performing various actions such as restaurant reservation or product purchase payment while communicating with the useras an agent such as a butler.

50 72 72 72 10 72 74 76 78 80 82 84 86 88 72 10 FIG. The agent systemdescribed above can be applied to smart glasses.is a block diagram illustrating an example of a hardware configuration of the smart glasses. The smart glassesare glasses-type smart devices, and are worn by the usersimilarly to general glasses. The smart glassesinclude a control device, a display, a touch panel, a speaker, a sensor group, a microphone, a camera, and a communication I/F. The smart glassesare examples of an “electronic apparatus” and a “wearable terminal” in the disclosure.

74 72 76 10 76 76 72 10 78 10 80 10 The control deviceis a device that controls the entire smart glasses, and includes, for example, a processor, a memory, and a storage (not illustrated). The displaydisplays various types of information to the user. The displayis, for example, a liquid crystal display. The displayis provided, for example, in a lens portion of the smart glasses, and allows the display content to be visually recognized by the user. The touch panelreceives an input from the user. The speakeroutputs vocal sound indicating various types of information to the user.

82 10 82 82 82 82 82 10 The sensor groupdetects a state of the user. The sensor groupincludes a plurality of sensors. For example, the sensor groupincludes an acceleration sensorA, a temperature sensorB, and a heart rate sensorC. These sensors are merely examples, and other sensors may be mounted in order to detect the state of the user.

84 10 72 86 72 86 10 86 88 72 74 76 78 80 82 84 86 88 90 The microphoneacquires vocal sound output by the useror an environmental sound around the smart glasses. The cameracan image the surroundings of the smart glasses. The cameracan also image the face of the user. The camerais, for example, a CCD camera. The communication I/Fmanages communication between the smart glassesand the outside. The control device, the display, the touch panel, the speaker, the sensor group, the microphone, the camera, and the communication I/Fare connected to a bus.

11 FIG. 50 72 72 10 50 72 10 72 50 50 74 50 72 88 50 72 is a diagram illustrating an example of a usage aspect of the agent systemby the smart glasses. The smart glassesrealize provision of various services to the userusing the agent system. For example, in a case in which the smart glassesare operated (for example, sound is input to a microphone, or a touch panel is tapped with a finger) by the user, the smart glassesstart to use the agent system. Here, using the agent systemincludes that the control devicehas a function similar to that of the agent systemand uses the function, and also includes an aspect in which the smart glassescommunicate with the outside (for example, a server) via the communication I/Fto use the agent systemstored outside the smart glasses.

10 72 50 10 50 50 52 52 In a case in which the useroperates the smart glasses, a touch point is generated between the agent systemand the user. That is, a service is started to be provided by the agent system. As described above, in the agent system, the character (for example, the character of Audrey Hepburn) of the agentA is set by the character setting unit.

53 232 10 10 82 72 10 82 72 82 As described above, the emotion engineincludes the emotion determination unit, and determines an emotion value indicating the emotion of the userand an emotion value of the agent itself. Here, the emotion value indicating the emotion of the usermay be estimated from the various sensors included in the sensor groupmounted on the smart glasses. For example, in a case in which the heart rate of the userdetected by the heart rate sensorC mounted on the smart glassesincreases, emotion values such as “anxiety” and “fear” are estimated to be great. The heart rate sensorC is an example of an “acquisition unit” in the disclosure.

82 82 10 As a result of measuring the body temperature of the user with the temperature sensorB, for example, in a case in which the body temperature exceeds the average body temperature, the emotion value such as “pain” or “bitter” is estimated to be great. For example, in a case in which it is detected by the acceleration sensorA that the userperforms some sport, an emotion value such as “pleasant” is estimated.

10 10 86 72 86 For example, the emotion value (for example, the emotion value of “gloomy”) of the usermay be estimated on the basis of the expression (for example, the gloomy expression) of the usercaptured by the cameramounted on the smart glasses. In this case, the camerais an example of an “acquisition unit” in the disclosure.

10 10 84 72 10 84 For example, the emotion value of the usermay be estimated from the vocal sound or speech content of the useracquired by the microphonemounted on the smart glasses. For example, in a case in which the useris raising his/her vocal sound, an emotion value such as “angry” is estimated. In this case, the microphoneis an example of an “acquisition unit” in the disclosure.

53 50 72 72 10 72 50 72 54 54 50 54 54 222 220 72 54 10 In a case in which the emotion value estimated by the emotion engineis greater than a predefined value, the agent systemcauses the smart glassesto acquire information regarding a surrounding situation. Specifically, for example, the camera mounted on the smart glassesis caused to capture an image or a moving image indicating a surrounding situation of the user(for example, a surrounding person or object). The microphone mounted on the smart glassesis caused to record surrounding environmental sound. Examples of other information regarding the surrounding situation include date, time, and position information, and information indicating weather. The agent systemacquires information regarding the surrounding situation in a case in which the emotion value becomes great from the smart glasses. The information regarding the surrounding situation is stored in the surrounding situation storage unitA together with the emotion value. The surrounding situation storage unitA is a storage medium for storing information regarding the surrounding situation acquired by the agent system. Similar to the interaction history storage unit, the surrounding situation storage unitA may be realized by the history dataof the storage unit, or may be realized by an external cloud storage. As described above, the surrounding situation obtained by the smart glassesis stored in the surrounding situation storage unitA as a so-called life log in a state of being associated with the emotion value of the userat that time.

50 54 50 10 10 50 54 In the agent system, information indicating the surrounding situation is stored in the surrounding situation storage unitA in association with the emotion value. As a result, the agent systemascertains personal information such as a hobby, a preference, or a personality of the user. For example, in a case in which an image indicating a state of baseball watching is associated with an emotion value such as “happy” or “pleasant”, the hobby of the useris baseball watching, and a favorite team or player is ascertained by the agent systemfrom the information stored in the surrounding situation storage unitA.

10 10 50 54 54 In the case of interacting with the useror performing an action toward the user, the agent systemdetermines the interaction content or the action content in consideration of the content of the surrounding situation stored in the surrounding situation storage unitA. The interaction content or the action content may be determined in consideration of the interaction history stored in the interaction history storage unitas described above in addition to the surrounding situation.

51 55 51 10 10 53 54 55 51 54 55 As described above, the interaction processing unitgenerates the speech content on the basis of the sentence generated by the sentence generation model. Specifically, the interaction processing unitinputs the text or the vocal sound input by the user, the emotions of both the userand the agent specified by the emotion engine, the conversation history stored in the interaction history storage unit, the personality of the agent, and the like to the sentence generation model, and generates the speech content of the agent. The interaction processing unitinputs the surrounding situation stored in the surrounding situation storage unitA to the sentence generation model, and generates the speech content of the agent.

80 72 10 51 76 80 76 The generated speech content is output by vocal sound from, for example, the speakermounted on the smart glassesto the user. In this case, a synthesized vocal sound corresponding to the character of the agent is used as the vocal sound. The interaction processing unitreproduces the vocal sound quality of the character of the agent (for example, Audrey Hepburn) to generate a synthesized vocal sound, or generates a synthesized vocal sound (for example, in the case of an emotion of “angry”, vocal sound in which tone is enhanced) according to the emotion of the character. The speech content may be displayed on the displayinstead of the vocal sound output or together with the vocal sound output. The speakerand the displayare examples of an “output unit” in the disclosure.

57 10 10 57 70 The RPAexecutes an operation according to the command (for example, a command of the agent acquired from vocal sound output by the useror text through interaction with the user). The RPAperforms actions related to use of the service provider, such as information search, store reservation, ticket arrangement, product/service purchase, payment, route guidance, and translation.

57 10 As another example, the RPAexecutes an operation of transmitting content input by vocal sound of the user(for example, a child) through interaction with the agent to the other party (for example, the parent). Examples of transmission means include message application software, chat application software, and mail application software.

57 80 72 10 10 In a case in which the operation of the RPAis executed, for example, vocal sound indicating that the execution of the operation is finished is output from the speakermounted on the smart glasses. For example, vocal sound such as “Reservation of the store has been completed” is output to the user. For example, in a case in which the reservation of the store is full, vocal sound such as “Reservation could not be made. What would you like to do?” is output to the user.

72 10 50 72 10 50 As described above, in the smart glasses, various services are provided to the userby using the agent system. Since the smart glassesare worn by the user, the agent systemcan be used in various scenes such as a home, a workplace, and a place outside the house.

72 10 10 10 72 86 10 50 10 The smart glassesare worn by the user, and are thus suitable for collecting a so-called life log of the user. Specifically, the emotion value of the useris estimated on the basis of detection results from various sensors or the like mounted on the smart glassesor recording results from the cameraor the like. Thus, the emotion value of the usercan be collected in various scenes, and the agent systemcan provide a service or speech content suitable for the emotion of the user.

72 10 86 84 10 10 50 10 50 10 50 10 In the smart glasses, the surrounding situation of the usercan be obtained by the camera, the microphone, and the like. The surrounding situation and the emotion value of the userare associated with each other. As a result, it is possible to estimate what kind of emotion the userhas in what kind of situation. As a result, the accuracy in a case in which the agent systemascertains the hobby/preference of the usercan be improved. In the agent system, the hobby/preference of the useris accurately ascertained, so that the agent systemcan provide a service or speech content suitable for the hobby/preference of the user.

50 92 92 92 10 92 93 94 95 96 92 12 FIG. The agent systemcan also be applied to the smart pendant.is a block diagram illustrating an example of a hardware configuration of the smart pendant. The smart pendantis a pendant type smart device, and is worn by the usersimilarly to a general pendant. The smart pendantincludes a control device, a speaker, a microphone, and a communication I/F. The smart pendantis an example of an “electronic apparatus” and a “wearable terminal” in the disclosure.

93 92 94 10 94 94 10 10 94 The control deviceis a device that controls the entire smart pendant, and includes, for example, a processor, a memory, and a storage (not illustrated). The speakeroutputs vocal sound indicating various types of information to the user. The speakeris, for example, a speaker (that is, the directional speaker) capable of outputting vocal sound having directivity. Speakeris set to have directivity toward the ear of user. As a result, the vocal sound is suppressed from reaching a person other than the user. The speakeris an example of an “output unit” in the disclosure.

95 10 92 92 10 92 10 10 95 96 92 93 94 95 96 97 The microphoneacquires vocal sound output by the useror an environmental sound around the smart pendant. The smart pendantis worn to be carried from the neck of the user. Thus, the smart pendantis located relatively close to the mouth of the userwhile being worn. This facilitates acquisition of vocal sound output by the user. The microphoneis an example of an “acquisition unit” in the disclosure. The communication I/Fmanages communication between the smart pendantand the outside. The control device, the speaker, the microphone, and the communication I/Fare connected to a bus.

50 92 50 72 92 10 10 10 11 FIG. The agent systemcan also be used in the smart pendant. That is, the service (see) provided by using the agent systemin the smart glassesdescribed above is also provided in the smart pendant. For example, emotion values of the userand the agent are estimated through an interaction with the agent in which the character is set, and speech content of the agent is changed or content of a service to be provided is changed according to the emotion values. For example, the surrounding situation of the userand the emotion value at that time are recorded as a life log, and are reflected in the estimation of the emotion value of the useror the content of the provided service.

72 92 82 72 10 50 Similarly to the smart glasses, the smart pendantmay also be provided with a sensor group (corresponding to the sensor groupin the smart glasses). As a result, information indicating the physical condition or the surrounding situation of the usercan be detected, and an emotion value can be estimated or the surrounding situation can be recorded in the agent system.

92 72 92 92 50 92 10 10 50 10 As described above, also in the smart pendant, effects similar to those of the smart glassesdescribed above can be obtained. The smart pendantis lightweight and easy to carry compared with a stuffed toy, a smart speaker, a smartphone, smart glasses, or the like. Compared with other smart terminals such as a smartphone and smart glasses, the smart pendantis lightweight, is worn usually, and has high portability. Therefore, it is realized that the agent systemis used in various scenes such as a home, a workplace, and a place outside the house. The smart pendantis more suitable by collecting the life log of the user, and can thus collect the emotion value of the userin various scenes, and the agent systemcan provide a service or speech content suitable for the emotion of the user.

50 60 60 Next, a second embodiment will be described while omitting or simplifying an overlapping portion with the above embodiment. The second embodiment is characterized in that an agent systempurchases a product on the basis of a budget received from a userand a history of products acquired by the user.

Here, the “budget” is a budget for cash, electronic money, virtual currency, points, or the like as a settlement means. The “budget” may be a budget of any payment method such as a pre-payment method (prepaid type), a post-payment method (post-payment type), or an immediate payment method (debit type). The term “acquisition” includes purchase, rental, and contract, and “products” include both articles and services.

50 In the second embodiment, a case in which the agent systempurchases an article with prepaid electronic money will be described.

13 FIG. 2 FIG. 50 50 is a diagram schematically illustrating an example of a functional configuration of the agent systemaccording to the second embodiment. As described above, the agent systemis configured by using some or all of the functions of the action control system illustrated in.

13 FIG. 50 502 504 506 As illustrated in, the agent systemof the present embodiment includes a reception unit, an acquisition unit, and a product determination unitin addition to the functional configuration of the above-described embodiment.

502 502 60 502 60 The reception unitof the present embodiment has a function of receiving an amount of electronic money that can be used to purchase an article. Specifically, the reception unitreceives the amount of electronic money charged by the user. As an example, the reception unitreceives an amount of “10,000 yen” charged by the useras the available electronic money. The amount of electronic money is an example of a “budget” in the disclosure.

504 60 504 222 220 504 60 60 2 FIG. The history acquisition unitof the present embodiment has a function of acquiring a purchase history in which articles purchased by the userin the past are recorded. Specifically, the acquisition unitacquires the purchase history from the history data(see) in the storage unit. As an example, the history acquisition unitacquires a list of an item of an article purchased by the user, a purchase date, a purchase quantity, an emotion value of the user, and the like. The purchase history is an example of “history information” in the disclosure.

506 506 60 506 60 506 The product determination unitof the present embodiment has a function of determining an article to be purchased on the basis of the amount of electronic money and the purchase history. As an example, the product determination unitdetermines articles to be purchased in descending order of purchase frequency so that the total amount falls within the range of 10,000 yen. In a case in which an emotion value of the userfor a purchased article is recorded in the purchase history, the product determination unitmay determine an article to be purchased on the basis of the emotion value of the user. The product determination unitmay determine the quantity of articles to be purchased.

506 506 53 506 221 220 2 FIG. The product determination unithas a function of determining an article to be purchased on the basis of the emotion value of the agent. Specifically, the product determination unitdetermines an article to be purchased on the basis of the emotion value of the agent determined by the emotion engine. As an example, the product determination unitdetermines an article to be purchased by using the reaction rule(see) of the storage unit.

221 60 60 221 100 60 60 60 60 60 60 60 In the reaction rulein the present embodiment, an article to be purchased is defined according to a combination of the past emotion value and the current emotion value of the user, the emotion value of the agent, and the action of the user. That is, the reaction rulein the present embodiment is characterized in that an article to be purchased is defined instead of the action of the robotdefined in the first embodiment. For example, in a case in which the past emotion value of the useris a positive value, the current emotion value is a negative value, and the action of the useris gloomy, an article for encouraging the useris defined as the article to be purchased by the agent. Examples of the article for encouraging the userinclude a favorite food of the user, a doll of a favorite character of the user, and a product of a favorite artist of the user.

221 (1) An article for congratulating a user (2) Article for apology to a user (3) An article for expressing gratitude to a user (4) An article for making a user laugh (5) An article for distracting a user (6) An article for comforting a user (7) An article for solving a problem of a user (8) An article for entertaining a user (9) An article for annoying a user (10) An article for reassuring a user For example, the articles to be purchased defined in the reaction ruleinclude the following (1) to (10).

222 54 221 For example, a specific article name determined by analyzing the history dataand the interaction history storage unitis registered in the article defined in the reaction rule. For example, “strawberry shortcake” is registered as “(1) an article for congratulating the user”, and “box of sweets” is registered as “(2) an article for apology to the user”. The registered article name may be updated every certain period.

221 506 506 60 506 60 506 In a case in which the article to be purchased is not registered in the reaction ruleor in a case in which the emotion value of the agent is a negative value, the product determination unitneed not determine an article to be purchased. As an example, in a case in which the emotion of the agent is “dislike”, “anger”, “not forgivable”, “hated”, or the like, the product determination unitdoes not determine an article to be purchased. In a case in which the emotion of the agent becomes “happy” because the agent is complimented by the user, the product determination unitdetermines “an article for expressing gratitude to the user” as an article to be purchased. In a case in which the emotion of the agent becomes “gloomy” because the agent is reprimanded by the user, the product determination unitdetermines “an article for apology to the user” as an article to be purchased.

57 57 221 57 The RPAof the present embodiment has a function of performing an action related to purchase of the determined article. Specifically, the RPApurchases an article from an online shopping site by using the amount of received electronic money, the article name registered in the reaction ruleof the article to be purchased, and the API. The RPAdoes not need to purchase all the articles determined by one purchase.

57 57 57 The RPAof the present embodiment has a function of performing an action related to purchase of an article at a predetermined timing. Specifically, the RPAperforms the action in a case in which the emotion value of the agent becomes a predetermined numerical value or more. As an example, in a case in which the emotion value of “pleasant” of the agent becomes 4 or more, the RPAperforms an action related to purchase of an article for entertaining the user. The case in which the emotion value of the agent is a predetermined numerical value or more is an example of “predetermined timing”. Examples of the predetermined timing include a timing at which the emotion value of the user becomes a predetermined numerical value or more, and a timing at which a purchase instruction is given from the user.

57 60 57 55 506 60 60 60 506 57 60 506 57 60 The RPAof the present embodiment has a function of transmitting a reason for determining an article to be purchased to the user. Specifically, the RPAinputs, to the sentence generation model, on what basis the product determination unithas determined an article to be purchased, and transmits the reason to the userby using text generated as a message for the user. As an example, a case will be described in which since the useris happy to have a birthday, the emotion of the agent is “happy”, and the product determination unitdetermines to purchase “an article for congratulating the user”. In this case, the RPAinputs text “Happy Birthday!” to a gift message field of an online shopping site. As another example, a case will be described in which the emotion of the agent becomes “gloomy” because the agent is reprimanded by the user, and the product determination unitdetermines to purchase “an article for apology to the user”. In this case, the RPAdescribes a message “Sorry. This is an article for apology.” in the text of an email and transmits the message, or transmits the message to the userthrough an interaction.

14 FIG. 50 is a flowchart illustrating an example of a flow of product determination processing in which the agent systemdetermines an article to be purchased according to the second embodiment.

200 50 60 50 60 14 FIG. In step Sin, the agent systemreceives an amount of money that can be used by the user. As an example, the agent systemreceives the amount of electronic money charged by the user.

201 50 60 50 60 222 In step S, the agent systemacquires a purchase history of the user. Specifically, the agent systemacquires the purchase history of the userfrom the history data.

202 50 50 232 60 230 In step S, the agent systemacquires an emotion of the agent. In the agent system, specifically, the emotion determination unitdetermines the emotion of the agent on the basis of the state of the useracquired by the user state recognition unit.

203 50 50 50 15 FIG. In step S, the agent systemdetermines an article to be purchased on the basis of the received amount of money, the acquired purchase history, and the emotion of the agent. The agent systemends the product determination processing. The agent systemmay proceed to product acquisition processing illustrated inthat will be described later after the product determination processing.

15 FIG. 50 50 is a flowchart illustrating an example of a flow of product acquisition processing in which the agent systempurchases an article according to the second embodiment. After the product determination processing, the agent systemexecutes the product acquisition processing at a predetermined timing.

300 50 50 203 50 57 15 FIG. 14 FIG. In step Sin, the agent systemexecutes an operation of purchasing the determined article. Specifically, the agent systempurchases the article determined in step Sin. Specifically, the agent systemcauses the RPAto purchase an article from an online shopping site by using the amount of electronic money, the determined article, and the API.

301 50 50 238 222 15 FIG. In step Sin, the agent systemadds the information regarding the purchased article to the purchase history. Specifically, the agent systemcauses the storage control unitto add information regarding the purchased article to the purchase history of the history data.

302 50 60 50 230 60 In step S, the agent systemacquires the state of the userat the time of recognizing the purchased article. Specifically, the agent systemcauses the user state recognition unitto acquire the state of the user.

303 50 60 60 50 232 60 In step S, the agent systemdetermines an emotion of the userfor the purchased article from the acquired state of the user. Specifically, the agent systemcauses the emotion determination unitto determine the emotion of the user.

304 50 60 50 238 60 222 50 In step S, the agent systemadds the emotion of the userto the information regarding the purchased article stored in the purchase history. Specifically, the agent systemcauses the storage control unitto add the information regarding the purchased article and the emotion value of the userto the purchase history of the history datain association with each other. The agent systemends the product acquisition processing.

50 60 60 50 60 50 60 60 60 The agent systemof the second embodiment purchases an article determined on the basis of the electronic money received from the userand the purchase history of the user. That is, the agent systemcan purchase an article purchased in the past by the user. That is, since the agent systempurchases, on behalf of the user, an article that the userregularly purchases, it is possible to save the trouble of the userpurchasing the article.

50 60 50 60 60 60 Meanwhile, the agent systemcan purchase an article that the userhas not purchased in the past. That is, the agent systemcan provide the userwith a new experience by purchasing, on behalf of the user, an article that has not been purchased by the user.

50 60 50 60 50 60 60 The agent systemof the second embodiment determines an article to be purchased on the basis of a purchase history in which a purchased article and an emotion of the userfor the article are recorded. Therefore, the agent systemcan reflect the past emotion of the userfor the purchased article in the determination of an article to be purchased. That is, the agent systemcan purchase an article that the userhas felt favorable and does not need to purchase an article that the useris indifferent to.

50 50 50 50 60 The agent systemof the second embodiment determines an article to be purchased on the basis of an emotion of the agent. Thus, the agent systemcan determine whether or not to purchase an article according to the emotion of the agent, or can change an article to be purchased. The agent systemcan express the emotion of the agent by using the article to be purchased. That is, the agent systemcan communicate with the userthrough the purchased article.

50 60 60 50 60 60 The agent systemof the second embodiment transmits a message related to the purchased article to the user. Therefore, the usercan ascertain the emotion of the agent through the message. The agent systemcan notify the userthat an article has been purchased in accordance with the emotion of the user.

60 In the second embodiment, an emotion of the agent is used to determine an article to be purchased. In Modification 1, a surrounding situation of the useris used to determine an article to be purchased. In the second embodiment, an article is purchased by using prepaid electronic money, but in Modification 1 of the second embodiment, a case of purchasing an article by using a credit card will be described. In Modification 1, the portion described as the “amount of electronic money” in the above description except for a special description is replaced with a “limit amount of credit card settlement”. Hereinafter, content unique to Modification 1 will be described.

230 60 230 203 200 230 60 2 FIG. In Modification 1, the user state recognition unitillustrated inhas a function of acquiring an image of a surrounding situation of the userand recognizing the surrounding situation. Specifically, the user state recognition unitacquires the image via the 2D cameraof the sensor unit. The user state recognition unitapplies a known image recognition technology to the image to estimate the quantity of each article captured in the image. The image of the surrounding situation of the useris an example of “information regarding the surrounding situation of the user” in the disclosure, and the quantity of each article is an example of “information regarding a stock of each product” in the disclosure.

230 60 230 As an example, in a case in which the user state recognition unitrecognizes a place where articles that are used daily by the usersuch as a refrigerator or a pantry are stored, the user state recognition unitacquires an image of the place and estimates the quantity of each article stored in the place. The image of the refrigerator or the pantry is an example of “image information” in the disclosure.

502 502 60 502 13 FIG. In Modification 1, the reception unitillustrated inhas a function of receiving a limit amount of credit card settlement that can be used for purchase of an article. Specifically, the reception unitreceives the limit amount input by the user. As an example, the reception unitreceives an amount of money for which signature-free settlement is possible. The limit amount of the credit card settlement is an example of a “budget” in the disclosure.

506 60 506 230 60 506 The product determination unitin Modification 1 has a function of determining an article to be purchased on the basis of an image of a surrounding situation of the user. Specifically, the product determination unithas a function of determining an article to be purchased on the basis of the quantity of each article estimated by the user state recognition unit. As an example, in a case in which the estimated quantity of articles is less than the minimum quantity of articles set by the user, the product determination unitdetermines the article as an article to be purchased.

60 57 57 230 57 24 60 57 In a case in which the estimated quantity of articles is less than the minimum quantity set by the usercontinuously for a predetermined period, the RPAin Modification 1 performs an action related to purchase of the article. Specifically, the RPAdetermines whether or not to purchase the article by using the quantity of each article estimated by the user state recognition unit, the estimation date, and the minimum quantity set for each article. As an example, in a case in which the number of PET bottles of 500 mL-sized carbonated water falls below twelve continuously for seven days, the RPApurchasesPET bottles of the carbonated water (for one box of cardboard). Twelve that is the minimum quantity set by the useris an example of a “predetermined threshold” in the disclosure, and seven days is an example of a “predetermined period” in the disclosure. After performing the action related to purchase of an article, the RPAneed not purchase the article until the purchased article arrives.

16 FIG. 50 is a flowchart illustrating an example of a flow of acquisition determination processing in which the agent systemaccording to Modification 1 of the second embodiment determines whether or not to purchase an article.

400 50 60 50 60 16 FIG. In step Sin, the agent systemreceives an amount of money that can be used by the user. As an example, the agent systemreceives a credit card limit amount set by the user.

401 50 60 50 60 222 In step S, the agent systemacquires a purchase history of the user. Specifically, the agent systemacquires the purchase history of the userfrom the history data.

402 50 60 50 200 In step S, the agent systemacquires an image indicating a surrounding situation of the user. Specifically, the agent systemacquires the image from the sensor unit.

403 50 50 230 In step S, the agent systemestimates information regarding the quantity of each article from the acquired image. Specifically, the agent systemcauses the user state recognition unitto estimate the quantity of each article captured in the image.

404 50 In step S, the agent systemdetermines an article to be purchased on the basis of the received amount of money, the purchase history of the user, and the information regarding the quantity for each article.

405 50 50 405 406 50 405 In step S, the agent systemdetermines whether or not the quantity of articles is less than the minimum quantity set by the user continuously for a predetermined period. In a case in which the agent systemdetermines that the quantity of articles is less than the minimum quantity set by the user continuously for the predetermined period (step S: YES), the processing proceeds to step S. In a case in which the agent systemdetermines that the quantity of articles is not less than the minimum quantity set by the user continuously for the predetermined period (step S: NO), the acquisition determination processing is ended.

406 50 50 15 FIG. In step S, the agent systemexecutes the product acquisition processing illustrated in. The agent systemends the acquisition determination processing.

50 60 50 60 60 50 50 60 The agent systemof Modification 1 determines an article to be purchased on the basis of a surrounding situation of the user. Therefore, the agent systemcan change an article to be purchased according to a change in the surrounding situation of the user. Since an instruction of the useris unnecessary in a case in which the agent systempurchases the article, the agent systemcan purchase the article according to the change even in a case in which the userdoes not notice the change in the surrounding situation.

50 60 50 60 50 The agent systemof Modification 1 determines an article to be purchased on the basis of a quantity of articles estimated from an image obtained by imaging a surrounding situation of the user. Therefore, the agent systemcan change an article to be purchased according to the quantity of articles owned by the user. That is, the agent systemdoes not need to purchase an article having a sufficient quantity.

50 60 50 50 60 The agent systemof Modification 1 purchases the article at a timing at which the quantity of articles estimated from a video obtained by imaging the surrounding situation of the userfalls below a predetermined threshold continuously for a predetermined period. That is, the agent systemcan purchase the article so that the stock of the article does not run out. That is, the agent systemcan save the trouble of the userwho checks the quantity of articles and purchases an article that is likely to be insufficient.

60 60 In the second embodiment, an emotion of the agent is used to determine an article to be purchased. In Modification 2, information received from the useris used to determine a service to be contracted. In the second embodiment, an article is purchased by using prepaid electronic money, but in Modification 2 of the second embodiment, a case in which a service is to be contracted with a point balance given to the userwill be described. In Modification 2, the portion described as the “amount of electronic money” and the “limit amount of credit card settlement” in the above description except for the case of special description is replaced with the “balance of points”. In Modification 2, except for the case of special description, the portion described as the “article” in the above description is replaced with a “service”, and the portion described as “purchase” is replaced with “contract”. Hereinafter, contents unique to Modification 2 will be described.

502 502 60 60 502 60 13 FIG. In Modification 2, the reception unitillustrated inhas a function of receiving a balance of points that can be used for a service contract. Specifically, the reception unitreceives the balance input by the userfrom the total amount of points owned by the user. As an example, the reception unitreceives a balance of “10,000 pt” out of the points owned by the user. The service contract is an example of “product acquisition” in the disclosure, and the balance of points is an example of a “budget” in the disclosure.

502 60 502 60 60 The reception unitof Modification 2 has a function of receiving at least one of information regarding a service category or information regarding a preference of the user. Specifically, the reception unitof Modification 2 presents a plurality of service categories and a plurality of preferences to the useras options, and receives a service category and a preference selected by the user.

502 (1) Moving image (2) Music (3) Book/electronic book (4) Game (5) Home appliance/digital device (6) Sports (7) Beauty (8) Healthcare/dieting (9) Fashion (10) Housing related category (11) Eating and drinking related category (12) Leisure The plurality of service categories presented by the reception unitinclude, for example, the following categories.

502 502 502 The plurality of preferences presented by the reception unitare items obtained by subdividing each category. An item obtained by subdividing each category will be described by using the case of “(1) moving image” as an example. Items obtained by subdividing the category of “(1) moving image” are, for example, a “movie”, a “drama”, an “animation”, and a “variety”. The reception unitmay present a “comedy”, an “action”, a “horror”, a “romance”, a “fantasy”, an “SF”, a “documentary”, and the like, which are items obtained by further subdividing a “movie”. The reception unitmay present a “movie director”, a “release year”, “cast”, a “target age”, and the like, which are items obtained by subdividing a “movie” from another viewpoint.

504 60 504 222 220 504 60 60 The history acquisition unitaccording to Modification 2 has a function of acquiring a contract history in which services contracted by the userin the past are recorded. Specifically, the acquisition unitacquires the contract history from the history dataof the storage unit. As an example, the history acquisition unitacquires a list of items of services contracted by the user, contract dates, contracted quantities, emotion values of the user, and the like. The contract history is an example of “history information” in the disclosure.

506 502 506 60 60 506 60 60 The product determination unitin Modification 2 has a function of determining a service to be contracted on the basis of the information received by the reception unit. Specifically, the product determination unitdetermines a service suitable for the preference of userfrom the category selected by user. As an example, the product determination unitdetermines a viewing service of a movie of a favorite movie director of the userfrom the category of the moving image. The moving image is an example of “information regarding a category of a product” in the disclosure, and the director whom the userlikes is an example of “information regarding a preference of a user” in the disclosure.

506 506 60 506 60 The product determination unithas a function of determining an article to be purchased on the basis of the balance of points and the contract history. As an example, the product determination unitdetermines services to be contracted in descending order of contract frequency so that the total points fall within the range of 10,000 pt. In a case in which an emotion value of the userfor a contracted service is recorded in the contract history, the product determination unitmay determine a service to be contracted on the basis of the emotion value of the user.

57 57 60 57 60 60 57 60 The RPAof Modification 2 performs an action related to the determined service contract. As an example, the RPAmakes a service contract at a providing site that provides the service by using the point balance input by the user, the determined service, and the API. As an example, the RPAmakes a contract to allow the userto view a movie of a favorite movie director at a site that provides a movie viewing service. In a case in which the agreement of the useris required in the contract, the RPAperforms an action related to the contract except for the procedure related to the agreement of the user.

17 FIG. 50 is a flowchart illustrating an example of a flow of product determination processing in which the agent systemaccording to Modification 2 of the second embodiment determines a service to be contracted.

500 50 60 50 50 60 17 FIG. In step Sin, the agent systemreceives an amount of money that can be used by the user. As an example, the agent systemreceives a balance of points that can be used by the agent systemfrom the total amount of points owned by the user.

501 50 60 50 60 222 In step S, the agent systemacquires the contract history of the user. Specifically, the agent systemacquires the contract history of the userfrom the history data.

502 50 60 50 502 60 In step S, the agent systemreceives information regarding the category of the service from the user. Specifically, the agent systemreceives the information when the category presented by the reception unitis selected by the user.

503 50 60 60 50 60 502 In step S, the agent systemreceives information regarding the preference of the userfrom the user. Specifically, the agent systemreceives the information by the userselecting an item obtained by subdividing each category presented by the reception unit.

504 50 60 60 50 50 15 FIG. In step S, the agent systemdetermines a service to be contracted on the basis of the received amount of money, the contract history of the user, and the information received from the user. The agent systemends the product determination processing. The agent systemmay proceed to the product acquisition processing illustrated inafter the product determination processing.

50 60 50 60 50 60 The agent systemof Modification 2 determines a service to be contracted on the basis of at least one of information regarding a category of the service and information regarding a preference of the user. That is, the agent systemcan contract the service on the basis of the setting of the user. That is, the agent systemcan contract a service suitable for the preference of the user.

232 60 210 60 230 60 232 60 60 54 222 In the first and second embodiments, the emotion determination unitdetermines the emotion value indicating the emotion of the userand the emotion value of the agent itself on the basis of the information analyzed by the sensor module unitand the state of the userrecognized by the user state recognition unit. However, the state of the useris not necessarily acquired. Therefore, the emotion determination unitmay determine an emotion value indicating the emotion of the userand an emotion value of the agent itself on the basis of the interaction history between the userand the agent. The interaction history may be stored in either the interaction history storage unitor the history data.

502 60 502 In the second embodiment, as an example, the reception unitreceives the amount of electronic money charged by the user, but a method of receiving a budget is not limited thereto. For example, the reception unitmay automatically receive points acquired by the user through shopping or the like, or automatically receive an amount of money set by the user every month.

506 221 506 506 In the second embodiment, the product determination unitdetermines an article to be purchased on the basis of the reaction rule, but a method of determining an article is not limited thereto. For example, the product determination unitdetermines an article according to the level of the emotion value of the agent. As an example, the product determination unitdetermines an article to be purchased as an expensive article as the emotion value of the agent increases.

50 60 60 50 60 60 In the second embodiment, the agent systemcauses the userto perform a procedure regarding the procedure that the userneeds to perform. For example, the agent systemchecks whether or not the usergives agreement or input regarding settlement, contract, or the like that requires agreement or input of the user.

50 200 50 72 86 10 FIG. In Modification 1 of the second embodiment, the agent systemacquires the image of the surrounding situation of the user from the sensor unit, but acquisition means of the image is not limited thereto. In a case in which the agent systemis mounted on the smart glassesillustrated in, the image may be acquired via the camera.

50 60 50 In Modification 1 of the second embodiment, the agent systempurchases the article in a case in which the quantity of the articles is less than the minimum quantity set by the usercontinuously for a predetermined period, but a purchase timing of the article is not limited thereto. The agent systemmay immediately purchase the article in a case in which the quantity of the article falls below the minimum quantity.

50 50 In Modification 1 of the second embodiment, as an example, the agent systempurchases the article in a case where the quantity of the article falls below the predetermined threshold continuously for seven days, but it is not always necessary to recognize the quantity of the articles continuously for seven days. For example, in a case in which the agent systemrecognizes that the quantity of the articles is insufficient on the first day and then recognizes that the quantity of the articles is insufficient on the eighth day, it can be regarded that the quantity is below the predetermined threshold continuously for seven days.

50 60 Next, a third embodiment will be described while omitting or simplifying an overlapping portion with the above embodiments. The third embodiment is characterized in that the agent systemoutputs information regarding a position of an object necessary for realizing a proposal to the user.

50 72 10 FIG. Hereinafter, a case in which the agent systemof the present embodiment outputs a name and position of a necessary ingredient to the smart glasses(see) will be described.

18 FIG. 2 FIG. 50 50 is a diagram schematically illustrating an example of a functional configuration of the agent systemaccording to the third embodiment. As described above, the agent systemis configured by using some or all of the functions of the action control system illustrated in.

230 60 230 60 203 200 230 60 60 2 FIG. The user state recognition unitof the present embodiment illustrated inhas a function of acquiring a video of a surrounding situation of the userand recognizing the surrounding situation. Specifically, the user state recognition unitacquires a video of a surrounding situation of the uservia the 2D cameraof the sensor unit. The user state recognition unitapplies a known image recognition technology to the video of the surrounding situation of the userto estimate information regarding an object captured in the video. The video of the surrounding situation of the useris an example of “image information regarding the surrounding situation of the user”.

18 FIG. 50 501 502 503 504 505 505 As illustrated in, the agent systemof the present embodiment includes a reception unit, a proposal determination unit, a specifying unit, a position output unit, and a history acquisition unitin addition to the functional configuration of the first embodiment described above, similarly to the second embodiment. The history acquisition unitwill be described in a modification of the third embodiment that will be described later.

501 501 60 501 60 84 72 10 FIG. The reception unitof the present embodiment has a function of receiving words/phrases indicating names and seasonings of ingredients. Specifically, the reception unitreceives a word/phrase input from the user. As an example, the reception unitreceives words/phrases of “pork” and “salty-sweet” included in the vocal sound of the userinput via the microphone(see) of the smart glasses. The name of the ingredient is an example of a “keyword indicating a target desired by the user”. The word/phrase indicating the seasoning is an example of a “concept word indicating the directionality of the keyword”.

502 60 502 501 55 502 55 60 502 55 502 55 60 The proposal determination unitof the present embodiment has a function of determining content to be proposed to the user. Specifically, the proposal determination unitinputs a word/phrase indicating the name and the seasoning of the ingredient received by the reception unitto the sentence generation model. The proposal determination unitdetermines a recipe output from the sentence generation modelas content to be proposed to the user. As an example, the proposal determination unitinputs words/phrases of “pork” and “salty-sweet” to the sentence generation model. The proposal determination unitdetermines the recipe of “stir-fried pork and kimchi” generated by the sentence generation modelas content to be proposed to the user. The content to be proposed is an example of “proposal information”.

503 60 502 503 55 503 55 55 The specifying unitof the present embodiment has a function of specifying a necessary object that is an object required by the userin order to realize the proposal determined by the proposal determination unit. Specifically, the specifying unitspecifies a necessary object by using the sentence generation model. As an example, the specifying unitinputs text “Extract necessary ingredients from the recipe” to the sentence generation modeland acquires a list of names of necessary ingredients from the sentence generation model. The necessary object is an example of a “thing”.

504 504 202 203 204 200 504 60 2 FIG. 18 FIG. The position output unitof the present embodiment has a function of acquiring and outputting information regarding a position of an object captured in a video. Specifically, the position output unitacquires information regarding the position measured by using at least one of the 3D depth sensor, the 2D camera, or the distance sensorof the sensor unitillustrated in. As an example, the position output unitinacquires a distance and a direction from the position of the userto the position of “pork” captured in the video.

503 60 230 504 504 76 72 2 FIG. 10 FIG. In a case in which the necessary object specified by the specifying unitis captured in the video of the surrounding situation of the useracquired by the user state recognition unit(see), the position output unitoutputs information regarding the position of the necessary object. As an example, the position output unitoutputs the distance and the direction to the position of “pork” to the display(see) of the smart glasses. Whether or not the object captured in the video is a necessary object may be determined by using a known image recognition technology.

19 FIG. 50 is a flowchart illustrating an example of a flow of output processing in which the agent systemaccording to the third embodiment outputs information regarding a position of a necessary object.

200 50 50 60 50 60 19 FIG. In step SA in, the agent systemreceives information regarding one ingredient and seasoning thereof. Specifically, the agent systemreceives an input from the user. As an example, the agent systemreceives, as an input, vocal sound of the usersuch as “I want to make a salty-sweet dish using pork”.

201 50 50 In step SA, the agent systemexecutes proposal generation processing that will be described later. The agent systemacquires a recipe based on one ingredient and seasoning thereof by executing the proposal generation processing.

202 50 60 50 76 72 10 FIG. In step SA, the agent systemproposes the acquired recipe to the user. As an example, the agent systemoutputs a recipe of “stir-fried pork and kimchi” as letter information to the display(see) of the smart glasses.

203 50 50 55 50 In step SA, the agent systemspecifies a plurality of ingredients necessary for the proposed recipe. Specifically, the agent systemspecifies a plurality of ingredients by using the sentence generation model. As an example, the agent systemspecifies “pork”, “Kimchi”, “garlic”, “ginger”, “sesame”, “green onion”, and the like as necessary ingredients.

204 50 60 50 60 203 2 FIG. In step SA, the agent systemacquires a video of a surrounding situation of the user. As an example, the agent systemacquires a video of the surroundings of the userfrom the 2D camera(see).

205 50 60 205 50 209 205 50 206 50 In step SA, the agent systemdetermines whether or not there is the specified ingredient around the user. In a case in which it is determined that there is no specified ingredient (step SA: NO), the agent systemproceeds to step SA. On the other hand, in a case in which it is determined that the specified ingredient is present (step SA: YES), the agent systemproceeds to step SA. A condition that a predetermined time elapses before the agent systemdetermines that there is no specified ingredient may be used.

206 50 50 202 203 50 In step SA, the agent systemacquires information regarding the position of the specified ingredient. Specifically, the agent systemacquires information regarding the position of the specified ingredient on the basis of the information acquired from the 3D depth sensorand the 2D camera. As an example, the agent systemacquires a distance and a direction to a place where there is pork.

207 50 50 76 72 50 10 FIG. In step SA, the agent systemoutputs information regarding the name and the position of the specified ingredient. Specifically, the agent systemoutputs the information regarding the name and the position of the specified ingredient on the display(see) of the smart glasses. As an example, the agent systemoutputs text such as “pork” and “10 m in the diagonal right direction”.

208 50 208 50 208 50 204 In step SA, the agent systemdetermines whether or not information regarding positions of all the specified ingredients has been output. In a case in which it is determined that the information regarding the positions of all the specified ingredients has been output (step SA: YES), the agent systemends the output processing. On the other hand, in a case in which it is determined that the information regarding the positions of all the specified ingredients has not been output (step SA: NO), the agent systemreturns to step SA.

209 50 50 60 50 55 60 In step SA, the agent systemproposes an alternative. The agent systemspecifically proposes an alternative to the specific ingredient determined not to be around the user. As an example, the agent systeminputs a text “What is an alternative to pork?” to the sentence generation model, acquires the output name of the alternative, and proposes the alternative to the user.

210 50 60 50 60 209 60 210 50 211 60 210 50 In step SA, the agent systemdetermines whether or not the userhas accepted the proposal. Specifically, the agent systemdetermines whether or not the userhas accepted the proposal in step S. In a case in which it is determined that the userhas accepted the proposal (step S: YES), the agent systemproceeds to step S. On the other hand, in a case in which it is determined that the userhas not accepted the proposal (step SA: NO), the agent systemends the output processing.

211 50 205 50 210 60 50 204 In step SA, the agent systemdetermines the alternative as the specified ingredient. Specifically, in step SA, the agent systemdetermines, as the specific ingredient, the alternative accepted in step SA instead of the ingredient determined not to be around the user. The agent systemreturns to step S.

20 FIG. 55 is a flowchart illustrating an example of a flow of proposal determination processing of determining a proposal by using the sentence generation modelaccording to the third embodiment.

300 50 55 50 55 20 FIG. In step SA in, the agent systeminputs the received information regarding one ingredient and seasoning thereof to the sentence generation model. As an example, the agent systeminputs text such as “Please memorize the word/phrase to be input from now. “Pork” and “salty-sweet“ ” to the sentence generation model.

301 50 55 50 55 In step SA, the agent systeminputs generation of a recipe based on the input information to the sentence generation model. As an example, the agent systeminputs, to the sentence generation model, text indicating an output instruction such as “Output a recipe of a dish on the basis of the input ingredient and seasoning”.

302 50 55 50 In step SA, the agent systemacquires the recipe output from the sentence generation model. The agent systemends the proposal generation processing.

50 60 50 60 50 60 60 The agent systemof the third embodiment displays a proposal on the basis of information input from the user. That is, the agent systemcan embody and display the desire of the userestimated from the input information. In other words, the agent systemenables the userto recognize what the userdesires.

50 60 50 60 50 60 50 60 50 60 The agent systemof the third embodiment displays the position of the necessary object required by the user, proposed by the agent systemin a case in which there is the necessary object around the user. That is, the agent systemallows the userto recognize that there is a necessary object nearby. That is, the agent systemcan reduce the possibility that the useroverlooks the necessary object. The agent systemenables the userto save time and effort to search for a necessary object.

50 55 60 60 55 60 60 60 60 In the third embodiment, the agent systemcauses the sentence generation modelto generate a proposal for the useron the basis of a word/phrase received from the user. In the modification, the sentence generation modelis caused to generate a proposal for the useron the basis of the action history of the userand the emotion of the userin addition to the word/phrase received from the user. Hereinafter, contents unique to the modification will be described.

505 60 505 60 222 220 505 60 18 FIG. 2 FIG. The history acquisition unitof the present embodiment illustrated inhas a function of acquiring an action history in which an action performed by the userin the past is recorded. Specifically, the history acquisition unitacquires an action history of the userfrom the history data(see) in the storage unit. As an example, the history acquisition unitacquires past emotion values and a history of actions of the user. The action history is an example of “history information”.

21 FIG. 55 is a flowchart illustrating an example of a flow of a proposal determination processing of determining a proposal by using the sentence generation modelaccording to the modification of the third embodiment.

400 50 60 50 230 60 21 FIG. 2 FIG. In step SA in, the agent systemrecognizes a state of the user. Specifically, the agent systemcauses the user state recognition unit(see) to recognize the state of the user.

401 50 60 60 50 53 60 50 60 18 FIG. In step SA, the agent systemdetermines an emotion of the useron the basis of the recognized state of the user. Specifically, the agent systemcauses the emotion engine(see) to determine the emotion of the user. As an example, the agent systemdetermines the emotion of the useras “gloomy”.

402 50 60 In step SA, the agent systemacquires the action history of the user.

50 60 222 220 Specifically, the agent systemacquires the action history of the userfrom the history dataincluded in the storage unit.

403 50 60 55 50 60 55 50 In step SA, the agent systemprovides the acquired action history of the userto the sentence generation model. Specifically, the agent systemuploads the data of the action history of the userto the sentence generation model. The agent systemcan generate an answer with reference to the data of the action history when the data of the action history is uploaded.

404 50 60 55 50 200 60 401 55 50 55 19 FIG. In step SA, the agent systeminputs the received information and the determined emotion of the userto the sentence generation model. Specifically, the agent systeminputs the information received in step SA (see) and the emotion of the userdetermined in step SA to the sentence generation model. As an example, the agent systeminputs text such as “Please memorize the word/phrase to be input from now. “pork”, “salty-sweet”, and “user's emotion is sad“ ” to the sentence generation model.

405 50 55 60 50 55 50 302 In step SA, the agent systeminputs, to the sentence generation model, generation of a recipe based on the provided action history and the input emotion of the user. As an example, the agent systeminputs, to the sentence generation model, text indicating an instruction such as “Output a recipe of a dish on the basis of the uploaded action history and the current emotion of the user”. The agent systemproceeds to step SA described above, and ends the proposal generation processing.

50 60 60 60 50 60 60 50 50 60 50 60 The agent systemof the modification determines a proposal for the useron the basis of the action history of the userand the emotion of the user. That is, the agent systemcan make a proposal reflecting the past action of the userand the current emotion of the user. For example, the agent systemascertains a tendency of wanting to eat a spicy dish when the user is emotionally depressed, and proposes a recipe of a spicy dish. The agent systemproposes a recipe of a dish that is good for digestion when it is determined that the userreflected in the mirror is in poor physical condition due to a hangover or the like and looks distressed. In other words, the agent systemcan make a proposal in accordance with the action history and the emotion of the user.

501 502 501 In the third embodiment, a case in which the reception unitreceives a word/phrase indicating the name and seasoning of an ingredient and the proposal determination unitproposes a recipe has been described, but a word/phrase received by the reception unitis not limited thereto. Table 3 below illustrates examples of combinations of keywords, concept words, and proposal information.

TABLE 3 Keyword Concept Word Proposal Information Shirt Casual Brand Information Lunch Healthy Menu Information Event Pleasant Festival Information Plant Easily Grown Ornamental Plant Information Harajuku Delicious Sweet Information in Harajuku . . . . . . . . .

501 60 60 501 60 In the third embodiment, the reception unitreceives an input based on the vocal sound of the useras an example, but input means of the useris not limited thereto. The reception unitmay receive an input based on text, an image, a video, or the like by the user.

502 502 502 In the third embodiment, the specifying unithas a function of specifying a necessary object, but a specified target is not limited to an object. The specifying unitmay specify information required by the user. As an example, the specifying unitspecifies a route to a festival venue, the names of train stations where passengers get on and off, and the like. The information required by the user is an example of a “thing”.

50 In the third embodiment, the agent systemuses the list of the names of the ingredients as the information regarding the necessary object, but the present invention is not limited thereto. The information regarding the necessary object may be information regarding the necessary object represented by letters, images, videos, sounds, and the like.

504 60 504 504 504 In the third embodiment, as an example, the position output unitacquires the distance and direction from the position of the userto the position of “pork”, but acquired information regarding a position is not limited thereto. The position output unitmay acquire a distance and a direction to the position at which the related information appears. As an example, the position output unitmay acquire a distance and a direction to a signboard indicating that there is “pork”. A plurality of pieces of information regarding a position may be simultaneously displayed. The position output unitmay determine information to be displayed and information not to be displayed according to the degree of relevance between the necessary object and the object captured in the video.

504 504 504 In the third embodiment, the position output unitoutputs the information regarding the name and position of the specified ingredient, but output information is not limited thereto. The position output unitmay output information regarding specified ingredients such as nutrients of ingredients, compatible ingredients, and cooking methods. That is, the position output unitcan output information related to a necessary object when outputting information regarding a position.

504 76 72 504 60 504 In the third embodiment, the position output unitoutputs the information regarding the position of the necessary object to the displayof the smart glasses, but an output method is not limited thereto. The position output unitmay output the information regarding the position by vocal sound through, for example, a speaker provided in the electronic apparatus or an earphone worn by the user. The position output unitmay output information regarding a position by vibration.

504 202 203 204 504 86 72 In the third embodiment, the position output unitacquires the information regarding the position of the necessary object by using at least one of the 3D depth sensor, the 2D camera, or the distance sensor, but acquisition means is not limited thereto. The position output unitmay acquire the information regarding the position by using the cameraof the smart glasses.

50 55 55 55 220 In the third embodiment, the agent systemprovides the sentence generation modelwith the action history data, but a method in which the sentence generation modeluses the action history data is not limited thereto. The sentence generation modelmay use the action history data by directly referring to the action history data in the storage unit.

50 Next, a fourth embodiment will be described while omitting or simplifying an overlapping portion with the above embodiments. The fourth embodiment is characterized in that the agent systemoutputs information regarding a conversation partner on the basis of information regarding a face of the conversation partner.

50 72 72 10 FIG. Hereinafter, a case in which the agent systemof the present embodiment outputs the information regarding an interaction partner acquired on the basis of information regarding a face of the interaction partner to the smart glasses(see) will be described. The interaction partner is an example of a “conversation partner”. The smart glassesare an example of “smart glasses”.

22 FIG. 2 FIG. 50 50 is a diagram schematically illustrating an example of a functional configuration of an agent systemaccording to the fourth embodiment. As described above, the agent systemis configured by using some or all of the functions of the action control system illustrated in.

200 60 2 FIG. The sensor unitof the present embodiment illustrated indetects information regarding an interaction partner of the user.

230 60 210 230 The user state recognition unitof the present embodiment recognizes a state of the interaction partner of the useron the basis of the information analyzed by the sensor module unit. The user state recognition unitis an example of a “state recognition unit”.

232 210 230 210 The emotion determination unitof the present embodiment determines an emotion value indicating the emotion of the interaction partner on the basis of the information analyzed by the sensor module unitand the state of the interaction partner recognized by the user state recognition unit. For example, the information analyzed by the sensor module unitand the recognized state of the interaction partner are input to a neural network trained in advance, and an emotion value indicating the emotion of the interaction partner is acquired.

22 FIG. 50 502 504 506 508 As illustrated in, the agent systemof the present embodiment includes a face information acquisition unit, a published information acquisition unit, a person information acquisition unit, and an information output unitin addition to the functional configuration of the above-described embodiment.

502 502 60 502 200 2 FIG. The face information acquisition unithas a function of acquiring face information. Specifically, the face information acquisition unitacquires feature amount data indicating the feature amount of the face of the interaction partner of the user. As an example, the face information acquisition unitacquires feature amount data of the face of the interaction partner measured by the sensor unit(see). The feature amount data of the face is an example of “face information”.

504 504 502 232 The published information acquisition unithas a function of acquiring information regarding an interaction partner from information on the Internet. Specifically, the published information acquisition unitspecifies a post of the interaction partner from a post on a social networking service (SNS) on the basis of the feature amount data of the face acquired by the face information acquisition unitand the emotion value determined by the emotion determination unit, and acquires post content of the specified post. The acquired post content is not limited to text, and may be vocal sound, an image, a moving image, or the like. The post on the SNS is an example of “information on the Internet including social media”. The post content is an example of “published information that is published”.

504 502 504 232 504 The published information acquisition unitof the present embodiment specifies a poster whose feature amount data of the face acquired by the face information acquisition unitis similar to feature amount data of the face included in the image indicating the poster. Here, “being similar” indicates being similar at a predetermined ratio, and faces are not necessarily similar to an extent of being authenticable in a known face authentication technology. The published information acquisition unitspecifies a post in which an emotion value estimated from post content of a post within a predetermined period (for example, the last week) of the specified poster matches an emotion value of the interaction partner acquired by the emotion determination unit. As an example, in a case in which the emotion value of the interaction partner corresponds to “gloomy”, the published information acquisition unitspecifies posts including a “photo of a broken cup” and a “gloomy emoji” among the posts of the specified poster. A plurality of posts to be specified may be present as long as the posters are the same.

506 506 504 506 504 The person information acquisition unithas a function of acquiring a profile. Specifically, the person information acquisition unitacquires the profile of the poster who made the post specified by the published information acquisition unit. As an example, the person information acquisition unitspecifies text in which a profile is described from a home screen capable of displaying a list of posts of the specified poster, and acquires information such as the name, age, gender, and occupation of the poster included in the specified profile. The profile of the poster may be acquired from the post content acquired by the published information acquisition unit. The profile is an example of “person information”.

508 508 232 504 506 76 72 508 60 508 508 10 FIG. The information output unithas a function of outputting acquired information. Specifically, the information output unitoutputs the emotion value acquired by the emotion determination unit, the post content acquired by the published information acquisition unit, and the profile acquired by the person information acquisition unitto the display(see) of the smart glasses. The information output unitmay output the acquired information in a format that is easy for the userto understand. As an example, in the case of acquiring that the emotion of the interaction partner is “gloomy”, the information output unitoutputs an objective expression such as “being gloomy”. The information output unitdoes not need to output all of the emotion value, the post content, and the profile, and may output information including at least one thereof.

23 FIG. 10 FIG. 50 60 50 60 60 76 72 is a flowchart illustrating an example of a flow of partner information output processing of outputting information regarding an interaction partner. The partner information output processing of the present embodiment is processing executed in a case in which the agent systemrecognizes that an interaction partner of the useris present and in a case in which the agent systemrecognizes that the interaction partner of the userhas been changed. The partner information output processing may be executed at any timing of the user. In the present embodiment, the information output through the partner information output processing is output to the display(see) of the smart glasses.

200 50 60 50 200 23 FIG. 2 FIG. In step SB in, the agent systemacquires feature amount data of the face of the interaction partner of the user. Specifically, the agent systemacquires feature amount data of the face of the interaction partner measured by the sensor unit(see).

201 50 50 60 50 In step SB, the agent systemexecutes emotion acquisition processing that will be described later. The agent systemacquires an emotion value of the interaction partner of the userthrough the emotion acquisition processing. As an example, the agent systemacquires that the emotion of the interaction partner is “gloomy”.

202 50 50 203 50 208 In step SB, the agent systemdetermines whether or not a post of the interaction partner has been specified from posts on the SNS on the basis of the acquired feature amount data of the face and emotion value. In a case of determining that the post of the interaction partner has been specified, the agent systemproceeds to step SB. On the other hand, in a case in which it is determined that a post of the interaction partner has not been specified, the agent systemproceeds to step SB.

203 50 50 In step SB, the agent systemacquires post content of the specified post. As an example, the agent systemacquires post content such as “I broke my favorite cup yesterday.”.

204 50 50 205 50 207 In step SB, the agent systemdetermines whether or not a profile of the interaction partner has been specified on the basis of the specified post. In a case in which it is determined that the profile has been specified, the agent systemproceeds to step SB. On the other hand, in a case in which it is determined that the profile has not been specified, the agent systemproceeds to step SB.

205 50 50 204 50 In step SB, the agent systemacquires the specified profile. Specifically, the agent systemacquires the profile specified in step SB. As an example, the agent systemacquires the name “A”.

206 50 50 201 203 205 50 50 In step SB, the agent systemoutputs the acquired emotion value, post content, and profile. Specifically, the agent systemoutputs the emotion value acquired in step SB, the post content acquired in step SB, and the profile acquired in step SB. As an example, the agent systemoutputs information such as “being gloomy”, “recently, my favorite cup has been broken”, and “Mr. A”. The agent systemends the partner information output processing.

207 50 50 201 203 50 50 In step SB, the agent systemoutputs the acquired emotion value and post content. Specifically, the agent systemoutputs the emotion value acquired in step SB and the post content acquired in step SB. As an example, the agent systemoutputs information such as “being gloomy” and “recently, my favorite cup has been broken”. The agent systemends the partner information output processing.

208 50 50 201 50 50 In step SB, the agent systemoutputs the acquired emotion value. Specifically, the agent systemoutputs the emotion value acquired in step SB. As an example, the agent systemoutputs information such as “being gloomy”. The agent systemends the partner information output processing.

24 FIG. is a flowchart illustrating an example of a flow of emotion acquisition processing of acquiring an emotion value of an interaction partner.

300 50 60 50 230 2 FIG. In step SB, the agent systemrecognizes a state of the interaction partner of the user. Specifically, the agent systemcauses the user state recognition unit(see) to recognize the state of the interaction partner.

301 50 50 232 53 2 FIG. 22 FIG. In step SB, the agent systemdetermines an emotion value of the interaction partner. Specifically, the agent systemcauses the emotion determination unit(see) to determine the emotion value of the interaction partner. The emotion engine(see) may determine the emotion value of the interaction partner.

302 50 50 50 60 In step SB, the agent systemacquires the determined emotion value. The agent systemends the emotion acquisition processing. The agent systemmay repeatedly execute the emotion acquisition processing as long as the usercontinues the interaction.

50 504 502 232 506 50 508 50 50 60 60 In the agent systemaccording to the fourth embodiment, the published information acquisition unitacquires post content on an SNS on the basis of the feature amount of the face of the interaction partner acquired by the face information acquisition unitand the emotion value of the interaction partner acquired by the emotion determination unit, and the person information acquisition unitacquires the profile of the interaction partner. The agent systemoutputs the information acquired by the information output unit. Media such as an SNS may not fully specify an individual. However, according to the agent systemof the present embodiment, since the profile of the interaction partner is acquired by using the feature amount of the face and the emotion value of the interaction partner in addition to the post information on the SNS, the information regarding the interaction partner can be obtained with high accuracy. According to the agent systemof the present embodiment, since the information regarding the interaction partner can be complemented with respect to the user, the usercan smoothly have a conversation with the interaction partner.

50 50 The agent systemaccording to the fourth embodiment acquires the profile of the interaction partner on the basis of the specified post. The agent systemaccording to Modification 1 acquires the profile of the interaction partner from a database. Hereinafter, contents unique to Modification 1 will be described.

506 223 506 502 22 FIG. A person information acquisition unitaccording to Modification 1 illustrated inhas a function of acquiring a profile from a person information DBthat will be described later. Specifically, the person information acquisition unitacquires a profile of a person having a high degree of similarity between the feature amount data of the face of the interaction partner acquired by the face information acquisition unitand the registered feature amount data of the face. Here, “high similarity” indicates that faces are similar to an extent of being authenticable in a known face authentication technology.

25 FIG. 220 223 221 222 As illustrated in, the storage unitof Modification 1 further includes a person information DBin addition to the reaction ruleand the history data.

223 2231 2232 The person information DBis a database in which feature amount data of faces of a plurality of persons and profiles are registered. The person information DB includes a person information tableand a face feature amount table.

2231 2231 The person information tableis an information table in which the profile of each person is registered. The person information tableincludes a face information ID and a profile stored in association with the face information ID.

The face information ID is a combination of predetermined letters associated with the feature amount of the face and the profile.

The profile is information for specifying an individual. The profile includes a name, an affiliation/position, and other information. The profile may include an age, a gender, an occupation, and other information specifying an individual.

2232 2232 The face feature amount tableis a data table in which face feature amounts are registered. The face feature amount tableincludes the above-described face information ID and face feature amount data stored in association with the face information ID.

200 2 FIG. In the face feature amount data, face feature amount data measured by a camera, an infrared sensor, a laser radar, or the like is stored. The stored face feature amount data is, for example, face feature amount data measured by the sensor unit(see).

26 26 FIGS.(A) and(B) 223 are diagrams illustrating a schematic example of a data table included in the person information DB.

26 FIG.(A) 2231 As illustrated in, the person information tablestores a face information ID and person information in association with each other. As person information corresponding to the face information ID “F0001”, “A” is stored in the name, “General Affairs Department” is stored in the affiliation/position, and “Main job is equipment management” is stored in the other information. As person information corresponding to the face information ID “F0002”, “B” is stored in the name, “Human Resources Division” is stored in the affiliation/position, and “Main job is recruiting activity” is stored in the other information. As person information corresponding to the face information ID “F0003”, “C” is stored in the name, “Sales Department/Department Manager” is stored in the affiliation/position, and “Mainly in charge of X company” is stored in the other information.

26 FIG.(B) 2232 As illustrated in, the face feature amount tablestores a face information ID and face feature amount data in association with each other. “Feature amount data of face of A” is stored in correspondence with the face information ID “F0001”. “Feature amount data of face of B” is stored in correspondence with the face information ID “F0002”. “Feature amount data of face of C” is stored in correspondence with the face information ID “F0003”.

27 FIG. 50 204 223 is a flowchart illustrating an example of a flow of partner information acquisition processing according to Modification 1. In the partner information output processing according to Modification 1, when the agent systemdetermines that the profile of the interaction partner has not been specified in step S, the profile of the interaction partner is acquired from the person information DB. Hereinafter, processing unique to Modification 1 will be described.

204 50 400 27 FIG. In stepC in, in a case in which it is determined that the profile of the interaction partner has not been specified, the agent systemproceeds to step SC.

400 50 50 401 50 207 50 200 2232 223 In step SC, the agent systemdetermines whether or not a person whose face feature amount data has a high degree of similarity to the acquired face feature amount data has been specified from the database. In a case in which it is determined that a person having a high similarity of the face feature amount data has been specified from the database, the agent systemproceeds to step SC. On the other hand, in a case in which it is determined that a person having a high similarity of the face feature amount data has not been specified from the database, the agent systemproceeds to step SC. The agent systeminquires the face feature amount data acquired in step Sfrom the face feature amount tableof the person information DBto specify a person having a high similarity of the face feature amount data.

401 50 50 400 2231 50 50 206 In step SC, the agent systemacquires a profile of the specified person from the database. Specifically, the agent systemacquires the profile corresponding to the person specified in step SC from the person information table. As an example, the agent systemacquires the name “Mr. A” and the affiliation/position “General Affairs Department”. The agent systemproceeds to step SC.

206 50 50 201 203 50 204 400 401 50 50 In step SC, the agent systemoutputs the acquired emotion value, post content, and profile. Specifically, the agent systemoutputs the emotion value acquired in step SC, the post content acquired in step SC, and the profile. Here, the agent systemoutputs the profile specified from the post in a case in which the profile has been specified on the basis of the specified post (in the case of YES in step SC), and outputs the acquired profile in a case in which the profile has been acquired from the database (in the case of YES in step SC). In the case of outputting the profile acquired in step SC, the agent systemoutputs, as an example, information such as “being gloomy”, “Recently, my favorite cup has been broken”, “Mr. A”, and “belonging to the general affairs department”. The agent systemends the partner information output processing.

50 506 223 50 60 In the agent systemof Modification 1, the person information acquisition unitacquires the profile of the interaction partner from the person information DB. Therefore, according to the agent systemof Modification 1, the usercan ascertain the profile of the highly reliable interaction partner registered in the database.

50 76 72 50 76 72 10 FIG. 10 FIG. In the agent systemaccording to the fourth embodiment, the output destination of the partner information output processing is the display(see) of the smart glasses. The agent systemaccording to Modification 2 is characterized in that the information output through the partner information output processing is summarized and then output to the display(see) of the smart glasses. Hereinafter, contents unique to Modification 2 will be described.

508 55 508 55 22 FIG. The information output unitaccording to Modification 2 inhas a function of outputting information acquired from the sentence generation model. Specifically, the information output unitoutputs the summary acquired from the sentence generation model.

28 FIG. is a flowchart illustrating an example of a flow of summary output processing of outputting summarized information.

500 50 55 50 206 55 50 50 55 28 FIG. 23 FIG. In step SC in, the agent systeminputs the output emotion value, post content, and profile to the sentence generation model. Specifically, the agent systeminputs the information output in step SB (see) to the sentence generation model. As an example, the agent systeminputs information such as “being gloomy”, “Recently, my favorite cup has been broken. Bad things are continuing. I want to change my mood.”, and “Mr. A”. The agent systemmay input at least two of the emotion value, the post content, and the profile of the interaction partner to the sentence generation model.

501 50 55 50 55 500 50 55 In step SC, the agent systeminputs the information to the sentence generation modelto generate a summary from the input information. Specifically, the agent systeminstructs the sentence generation modelto summarize the information input in step SC. As an example, the agent systeminputs text such as “Summarize the input information” to the sentence generation model.

502 50 55 50 In step SC, the agent systemacquires the summary output from the sentence generation model. As an example, the agent systemacquires text such as “Mr. A seems to be feeling gloomy due to continuous unpleasant events recently.”.

503 50 50 503 76 72 50 10 FIG. In step SC, the agent systemoutputs the acquired summary. Specifically, the agent systemoutputs the summary acquired in step SC to the display(see) of the smart glasses. The agent systemends the summary output processing.

50 506 50 60 In the agent systemof Modification 2, the information processing output unitoutputs summarized information regarding the interaction partner on the basis of at least two of the emotion value, the post content, and the profile of the interaction partner. Therefore, according to the agent systemof Modification 2, the time for the userto check the information regarding the interaction partner can be shortened compared with the case in which the information regarding the interaction partner is not summarized.

50 76 72 50 76 72 10 FIG. 10 FIG. The agent systemaccording to Modification 2 of the fourth embodiment summarizes the information output in the partner information output processing and then outputs the summarized information to the display(see) of the smart glasses. The agent systemaccording to Modification 3 outputs a topic generated on the basis of the information output in the partner information output processing to the display(see) of the smart glasses. The topic is an example of “conversation information”. Hereinafter, contents unique to Modification 3 will be described.

508 55 508 55 22 FIG. The information output unitaccording to Modification 3 inhas a function of outputting information acquired from the sentence generation model. Specifically, the information output unitoutputs a topic acquired from the sentence generation model.

29 FIG. is a flowchart illustrating an example of a flow of conversation information output processing of outputting a topic.

600 500 50 55 29 FIG. 28 FIG. Step SC inis similar to step SC (see), and thus description thereof will be omitted. The agent systemof Modification 3 may input at least one of the emotion value, the post content, or the profile of the interaction partner to the sentence generation model.

601 50 55 50 55 600 50 55 In step S, the agent systeminputs the information to the sentence generation modelto generate a topic from the input information. Specifically, the agent systeminstructs the sentence generation modelto generate a topic on the basis of the information input in step S. As an example, the agent systeminputs text such as “Propose a topic on the basis of the input information” to the sentence generation model.

602 50 55 50 In step S, the agent systemacquires the topic output from the sentence generation model. As an example, the agent systemacquires text “Why don't you explain that there is a technique called metal joining as a method of repairing a broken cup?”

603 50 76 72 50 10 FIG. In step S, the agent systemoutputs the acquired topic to the display(see) of the smart glasses. The agent systemends the conversation information output processing.

50 506 50 60 In the agent systemof Modification 3, the information processing output unitoutputs a topic with the interaction partner on the basis of at least one of the emotion value, the post content, or the profile of the interaction partner. Therefore, according to the agent systemof Modification 3, the usercan treat the topic according to the situation of the interaction partner.

50 50 The agent systemaccording to Modification 1 of the fourth embodiment specifies the profile of the interaction partner from the database in a case in which the profile of the interaction partner cannot be specified on the basis of the specified post. The agent systemaccording to Modification 4 of the fourth embodiment acquires the profile of the interaction partner from the database, and specifies the post of the interaction partner from the acquired profile. Hereinafter, contents unique to Modification 4 different from Modification 1 will be described.

504 504 223 232 504 223 504 232 504 The published information acquisition unitaccording to Modification 4 has a function of acquiring information regarding an interaction partner from information on the Internet on the basis of the profile and the emotion value. Specifically, the published information acquisition unitspecifies a poster on the SNS on the basis of the profile acquired from the person information DB, specifies a post of the specified poster on the basis of the emotion value acquired by the emotion determination unit, and acquires post content of the specified post. As an example, the published information acquisition unitspecifies a poster whose name matches the name of “A” acquired from the person information DB. The published information acquisition unitspecifies a post in which the emotion value estimated from the post content of the post within the predetermined period of the specified poster matches the emotion value of the interaction partner acquired by the emotion determination unit. The published information acquisition unitmay acquire post content of a post within a predetermined period without using an emotion value.

30 FIG. 50 400 201 223 is a flowchart illustrating an example of a flow of partner information output processing according to Modification 4. In the partner information output processing according to Modification 4, the agent systemexecutes step SD after executing the emotion acquisition processing in step SD, and specifies the interaction partner from the person information DB. Hereinafter, processing specific to Modification 4 will be described.

201 50 50 400 50 30 FIG. In step SD in, the agent systemexecutes emotion acquisition processing. The agent systemproceeds to step SD. As an example, the agent systemacquires that the emotion of the interaction partner is “pleasant”.

400 50 202 50 401 In step SD, in a case in which the agent systemdetermines that the person having the high similarity of the feature amount data of the face has not been specified from the database, the processing proceeds to step SD. On the other hand, in a case in which it is determined that a person having a high degree of similarity of the feature amount data of the face is specified from the database, the agent systemproceeds to step SD.

401 50 50 400 2231 50 50 700 In step SD, the agent systemacquires a profile of the specified person from the database. Specifically, the agent systemacquires the profile corresponding to the person specified in step SD from the person information table. As an example, the agent systemacquires a name “Mr. A”. The agent systemproceeds to step SD.

700 50 50 701 50 702 In step SD, the agent systemdetermines whether or not the post of the interaction partner has been specified from the post on the SNS on the basis of the acquired profile and the emotion value. In a case in which it is determined that the post of the interaction partner has been specified, the agent systemproceeds to step SD. On the other hand, in a case in which it is determined that the post of the interaction partner has not been specified, the agent systemproceeds to step SD.

701 50 50 700 50 50 206 In step SD, the agent systemacquires post content of the specified post. Specifically, the agent systemacquires the post content of the post specified in step SD. As an example, the agent systemacquires post content such as “Today, it was pleasant to have flower-viewing.” of “Mr. A”. The agent systemproceeds to step SD.

206 50 50 201 701 700 50 50 In step SD, the agent systemoutputs the acquired emotion value, post content, and profile. Specifically, the agent systemoutputs the emotion value acquired in step S, the post content acquired in step SD, and the profile acquired in step SD. As an example, the agent systemoutputs information such as “looks pleasant”, “We went to flower viewing the other day.”, and “Mr. A”. The agent systemends the partner information output processing.

702 50 50 50 In step SD, the agent systemoutputs the acquired emotion value and profile. As an example, the agent systemoutputs information such as “looks pleasant” and “Mr. A”. The agent systemends the partner information output processing.

50 506 223 504 232 50 60 223 In the agent systemaccording to Modification 4 of the fourth embodiment, the person information acquisition unitacquires the profile of the interaction partner from the person information DB. The published information acquisition unitacquires post content of a post on the SNS on the basis of the acquired profile and the emotion value acquired by the emotion determination unit. Therefore, according to the agent systemaccording to Modification 4, the usercan ascertain the latest information published regarding the interaction partner. Since the information published regarding the interaction partner is acquired on the basis of the profile registered in the person information DB, it is possible to acquire information with high accuracy compared with the case of acquiring the information on the basis of the face feature amount.

506 223 50 50 60 223 In a case in which the person information acquisition unitdoes not specify a person whose degree of similarity to the face feature amount data of the interaction partner is high from the person information DB, the agent systemaccording to Modification 4 of the fourth embodiment specifies the post of the interaction partner from the post on the SNS and acquires the profile of the interaction partner. Therefore, according to the agent systemaccording to Modification 4, the usercan know the information regarding the interaction partner even in a case in which the information regarding the interaction partner is not registered in the person information DB.

502 502 The face information acquisition unitaccording to the fourth embodiment acquires face feature amount data as face information. However, the disclosure is not limited thereto, and the face information acquisition unitmay acquire a face image as the face information. In the fourth embodiment, the processing using the face feature amount data may be performed by using a face image.

60 60 50 The partner information output processing of the fourth embodiment is executed in a case in which it is recognized that the interaction partner is present in the userand in a case in which it is recognized that the interaction partner of the userhas been changed. However, the disclosure is not limited thereto, and in a case in which the information regarding the interaction partner is newly acquired, only the corresponding information may be updated and output. As an example, the agent systemmay output only the emotion value to be updated in a case in which the emotion value of the interaction partner changes.

50 200 50 86 72 82 2 FIG. 10 FIG. In the fourth embodiment, the agent systemacquires the face feature amount data from the sensor unit(see). However, the disclosure is not limited thereto, and the agent systemmay acquire the face feature amount data by using the cameraof the smart glassesor the sensor group(see). The face feature amount data may be acquired from an external device such as a smartphone or an imaging device.

50 50 In the fourth embodiment, the agent systemspecifies a post of an interaction partner from a post on a social networking service (SNS), and acquires post content of the specified post. However, the disclosure is not limited thereto, and the agent systemmay specify a post of the interaction partner from a post on social media such as a blog, a moving image sharing site, a word-of-mouth site, or a shopping site. A post is not limited to a post of the interaction partner, and may be news, an article, or the like in which the interaction partner is covered.

50 502 50 In the fourth embodiment, the agent systemspecifies a poster whose face feature amount data acquired by the face information acquisition unitis similar to the face feature amount data included in the image indicating the poster, and specifies the post of the interaction partner. However, the disclosure is not limited thereto, and the agent systemmay specify the information regarding the interaction partner by executing web search using the feature amount data of the face of the interaction partner.

50 76 72 50 80 72 80 72 10 FIG. In the fourth embodiment, the agent systemoutputs information to the display(see) of the smart glasses. However, the disclosure is not limited thereto, and the agent systemmay output information from the speakerof the smart glasses. The output from the speakerincludes vocal sound output using an earphone or bone conduction. An output destination is not limited to the smart glasses, and may be a mobile terminal, a wearable terminal, or the like.

72 72 50 Next, as a fifth embodiment, an execution example of a fast reading application in which a wearable terminal (specifically, the smart glasses) is applied as an electronic apparatus will be described. An electronic apparatus is not limited to the smart glasses, and may be other electronic apparatuses (robots, dolls, stuffed toys, pendants, smart watches, smartphones, smart speakers, earphones, personal computers, and the like) on which the agent systemcan be mounted.

74 10 FIG. The execution program of the fast reading application of the present embodiment is stored in the recording medium of the control deviceillustrated inand executed by the CPU.

72 In the fast reading application, the smart glassesare controlled to execute the following (Operation 1) to (Operation 7).

72 (Operation 1) The smart glassesare activated when the user spreads a book and says “Read the book quickly”.

72 (Operation 2) A camera (not illustrated) of the smart glassesimages letter information of a page of a book or the like read by the user.

(Operation 3) The captured image is subjected to OCR to acquire a sentence.

(Operation 4) A prompt preset to “Summarize the acquired sentence” is input to ChatGPT to acquire a summary of the page being read. Depending on the function of the CHATGPT, an image or an illustration of the page may be captured to analyze the image (using the function of the GPT-4).

72 (Operation 5) The summary content is output by using vocal sound from the speaker of the smart glassesor the connected earphone.

72 (Operation 6) Whether or not the user has performed a page turning operation is recognized from the image of the camera of the smart glasses.

(Operation 7) The above (Operation 2) to (Operation 6) are repeatedly executed.

31 FIG.(A) 72 is a perspective view illustrating a fast reading example of a book in a fast reading application using the smart glasses.

31 FIG.(A) 10 72 illustrates a state in which the userwearing the smart glassesis reading a book B while holding the book B.

10 72 In accordance with the timing at which the useris reading, a camera (not illustrated) of the smart glassesimages letter information of a page that is opened (normally two left and right pages), and the text information is subjected to OCR to acquire a sentence written on the page.

72 10 Since the answer from ChatGPT based on the fast reading instruction is output from the speaker (not illustrated) of the smart glasseswith vocal sound, the usercan recognize the outline before reading the letters on the page. The output is generated as text such as “This page describes OOO for CCC indicating that AAA is BBB.”.

10 10 Therefore, the usercan ascertain an outline of the content written on the page before reading the page. The utterance may be arranged to increase the reader's (user) reading motivation as in a preview. For example, in the case of a reasoning novel, utterance such as “Finally, the truth of OO becomes clear on this page. The person who holds the key is . . . ” is made.

32 FIG. 74 72 is a control flowchart illustrating a flow of processing of the fast reading application illustrated in (Operation 1) to (Operation 7) described above, executed by the control deviceof the smart glasses.

10 410 72 32 FIG. When the usersays “Read fast”, the fast reading application inis activated, and in stepE, imaging is started by the camera function of the smart glasses.

412 414 414 412 412 414 31 FIG.(A) In next stepE, a region (hereinafter, referred to as a book region) of the book B (see) within the imaging angle of view is searched, and it is determined whether or not the book region is specified in stepE. In a case in which a negative determination is made in stepE, the processing returns to stepE, and stepsE andE are repeatedly executed until the book region is specified.

414 416 418 In a case in which a positive determination is made in stepE (book region specification), the processing proceeds to stepE to image the book region (double spread), and then the processing proceeds to stepE to perform OCR processing. The sentence of the double spread pages is acquired through OCR processing.

420 10 32 FIG. In the next stepE, text such as “summarize the acquired sentence” is input to ChatGPT. As a result, ChatGPT analyzes the sentence to create a summary of double spread pages with a predefined number of letters. The predefined number of letters may be fixed or may be a certain ratio (for example, 10%) based on the number of letters of the sentence to be analyzed. The user(see) may add a request such as “at a high school student level” or “in English” to the input of “Summarize the acquired sentence”.

422 424 10 72 31 FIG.(A) In the next stepE, it is determined whether or not there is an answer from ChatGPT, and in a case in which an affirmative determination is made, the processing proceeds to stepE, and the summary content is output by using vocal sound. For example, the output is performed at the ear of the userfrom the speaker of the smart glasses, the connected earphone, or the like (not illustrated in).

426 428 In the next stepE, it is determined whether or not the page turning has been checked, and in a case in which a negative determination is made, the processing proceeds to step, and it is determined whether or not there is a fast reading end instruction.

10 Examples of the fast reading end instruction include utterance of “fast reading end” by the user, page turning not being performed for a certain period of time, the book B being closed, and automatic recognition of the last page.

426 428 426 428 426 410 In a case in which a negative determination is made in each of stepE and stepE, stepE and stepE are repeatedly executed. Here, in a case in which an affirmative determination is made in stepE, the processing returns to stepE to repeat the above processing in order to continue the fast reading.

428 430 In a case in which an affirmative determination is made in stepE, the processing proceeds to stepE, and end processing (at this point, the imaging by the camera is ended) of the fast reading application is executed, and this routine is ended.

31 FIG.(A) 31 FIG.(B) 31 FIG.(B) 72 10 10 In, the sound is output from the speaker of the smart glassesor the like at the ear of the user, but as illustrated in, the fast reading result may be output from a speaker of another electronic apparatus (in, the smartphone SP) via a communication function. A communication destination may be another electronic apparatus such as a smart speaker or a PC. As a result, it is possible to make another person located in the vicinity of the userlisten to the reservation of the book B.

31 32 FIGS.and 10 11 FIGS.and 12 FIG. 72 92 10 In the embodiment described with reference to, the smart glassesillustrated inare applied as an electronic apparatus, but another wearable terminal such as the smart pendantor the headset (VR goggle) illustrated inmay be used. In the case of VR goggles, the book B itself may be a virtual reality image. That is, for example, in a case in which the userutters identification information for specifying a book such as a title in a state in which the palm is opened toward the user, a virtual reality book image may be displayed and the book may be read, and the summary sentence may be uttered in synchronization with the page turning operation of the user.

100 102 10 100 102 10 1 FIG. 31 FIG. 31 FIG. An electronic apparatus is not limited to a wearable terminal, and may be robotsand(see) that closely follow the userside (see). The robotsandcontrol themselves to have an imaging optical axis that coincides with the destination (book region) of the line of sight of the user(see).

10 The electronic apparatus may be a camera alone (a speaker function as an output unit is separately required) or another electronic apparatus such as a smartphone or a tablet terminal as long as the imaging optical axis coincides with the destination (book region) of the line of sight of the user.

10 10 72 10 As described above, in the present embodiment (fast reading application), when the userreads, by using the fact that the userfaces the book B, a series of flows such as the letter information of the book being imaged by the camera mounted on the smart glasses, subjected to OCR, a summary being created by ChatGPT, and the summary being output by using vocal sound is executed, so that the usercan receive the assistance of reading ability without any special measure.

72 60 72 10 11 FIGS.and Next, as a sixth embodiment, an execution example of an emotion positivity application in which a wearable terminal (specifically, the smart glasses) is applied as an electronic apparatus will be described. That is, as illustrated in, the present embodiment assumes a state in which the userwears the smart glasses.

72 50 An electronic apparatus is not limited to the smart glasses, and may be any of other electronic apparatuses (robots, dolls, stuffed toys, pendants, smart watches, smartphones, smart speakers, earphones, personal computers, and the like) on which the agent systemcan be mounted.

74 10 FIG. The execution program of the emotion positivity application of the present embodiment is stored on the recording medium of the control deviceillustrated inand executed by the CPU.

50 In the emotion positivity application, the agent systemis controlled to execute the following (Operation 1) to (Operation 7).

60 60 72 (Operation 1) An action of the userand a surrounding situation (image and vocal sound) of the userare acquired by a sensor (a camera or a microphone that is “not illustrated”) of the smart glasses.

60 (Operation 2) An event related to the useris recognized from the sensor output.

60 (Operation 3) An emotion of the userduring the execution of the event is analyzed by using the sensor output.

60 55 (Operation 4) In a case in which the emotion of the useris negative, “How can I positively view the current event?” is input to the sentence generation model.

55 (Operation 5) A way of thinking for positive viewing is acquired from the sentence generation model.

72 (Operation 6) A way of thinking for positive viewing is output by using vocal sound from a speaker of the smart glassesor a connected earphone. It is also possible to output the event and the way of thinking by using vocal sound later through setting.

(Operation 7) The above (Operation 1) to (Operation 6) are repeatedly executed.

33 33 FIGS.(A) to(C) 72 are front views of images relating to a specific event (here, “cleaning” is selected as the event) captured by the camera of the smart glasses.

33 33 FIGS.(A) to(C) 33 FIG.(A) 33 FIG.(B) 33 FIG.(C) 60 1400 1 1402 2 1404 3 60 are images captured when the userstops by each room in the house.is a children's room image(captured image),is a kitchen image(captured image), andis a living room image(captured image). The image may be captured when the useris cleaning (during execution of an event) or when each room is checked before execution of the event.

50 60 33 33 FIGS.(A) to(C) The agent systemanalyzes an emotion of the userat the time of capturing the images of.

60 60 In this case, since the room is messy, the agent collates usual activities of the userand determines that “Each room is messy, and cleaning is very difficult.” and “Other work such as washing and preparation for dinner cannot be performed.”, and the emotion of the useris “negative”.

50 55 Therefore, the agent systeminputs “How can I positively view the current event?” to the sentence generation modelas a prompt (advice request for positivity).

55 72 60 60 Since the answer from the sentence generation modelbased on the advice request for positivity is output from the speaker (not illustrated) of the smart glassesby using vocal sound while the userexecutes the event, it is possible to help the emotion of the usershift to a direction of positively viewing the event. The output is generated, for example, as a sentence such as “Why don't you consider it for OO?”.

The output may be “Next, how about AAA?” together with the content of the event after the end of the event instead of during the execution of the event (real time). As a result, even if a similar situation (each room is messy) is encountered in the same event (cleaning), the possibility of making emotions positive can be increased. In the present embodiment, “messy” with respect to “cleaning” is used as a relationship between the specific event and the negative, but a relationship between the specific event and the negative is not limited to “messy” with respect to “cleaning”, and “noise” with respect to “sound”, “damage” with respect to “decorative article”, “deviation or shortage” with respect to “schedule”, and the like can be considered.

34 FIG. 74 72 is a control flowchart illustrating a flow of processing of the emotion positivity application illustrated in (Operation 1) to (Operation 7) described above, executed by the control deviceof the smart glasses.

60 72 60 For example, the emotion positivity application is activated when the userwears the smart glassesand turns on the power, and basically maintains the operation state, but the operation may be activated or stopped by the speech or manual operation of the user.

410 60 72 72 60 60 5 6 FIGS.and In stepF, sensor information is sequentially acquired in accordance with the action of the userwearing the smart glasses. Specifically, the camera of the smart glassescaptures an image at an angle of view corresponding to a line of sight of the user, and the microphone acquires information corresponding to a user action such as recording surrounding environmental sound. Although the sensor information is the captured image information and the environmental sound information here, the sensor information includes all kinds of information regarding all of happiness, anger, sadness, and pleasure felt by the user(see).

412 60 414 60 33 33 FIGS.(A) to(C) In the next stepF, an event related to the useris recognized, and then the processing proceeds to stepF to read sensor information belonging to the event.illustrate image information read as an event on the basis of a situation in which the useris cleaning the room, and it can be seen that each room is messy.

416 60 60 33 33 FIGS.(A) to(C) In the next stepF, the emotion (positive or negative) of the useris analyzed on the basis of the sensor information. From the images in, emotions such as “Each room is messy, and cleaning is very difficult.” and “Other work such as washing and preparation for dinner cannot be performed.” are analyzed, and it is predicted that the emotion of the useris negative.

418 60 420 60 418 418 410 33 33 FIGS.(A) to(C) 33 FIG.(A) In the next stepF, it is determined whether or not the emotion is negative, and in a case in which an affirmative determination (analysis result based on the images in) is made, it is determined that the emotion of the useris negative, and the processing proceeds to step. In a case in which the sensor information indicates that each room is clean and tidy unlike in the image illustrated in, the emotion of the useris positive and an affirmative determination is made in stepF, and thus the processing returns from stepF to stepF to repeatedly execute the above steps.

420 55 422 In stepF, “How can I positively view the current event?” is input to the sentence generation model(as an example, ChatGPT), and the processing proceeds to stepF.

422 55 424 In stepF, it is determined whether or not there is an answer from the sentence generation model(as an example, ChatGPT), and in a case in which an affirmative determination is made, the processing proceeds to step, and the way of thinking of positively viewing the current event is output by using vocal sound. The vocal sound output may be performed in real time during the event or after the end of the event.

Examples of real-time vocal sound output include “Why don't you consider it for OO?”. On the other hand, examples of the vocal sound output after the end of the event include “Next, how about AAA?” together with the content of the event.

426 414 60 In the next stepF, it is determined whether or not the event has been ended. In a case in which a negative determination is made, the processing proceeds to stepF, sensor information (latest information) belonging to the event is read, and the above-described steps are repeatedly executed. For example, in a case in which an image in which a child starts cleaning (tidying up) by himself/herself in a child's room is acquired, the emotion of the usermay change positively.

426 410 In a case in which an affirmative determination is made in stepF, the processing proceeds to stepF and the above steps are repeatedly executed in order to cope with the next event.

72 92 10 11 FIGS.and 12 FIG. In the present embodiment, the smart glassesillustrated inare applied as an electronic apparatus, but another wearable terminal such as the smart pendantor a headset (VR goggle) illustrated inmay be used.

100 102 An electronic apparatus is not limited to a wearable terminal, and may be robotsandthat function as an agent system. The sensor information may be acquired from a single device such as a camera or a microphone installed in each portion in the house. In the case of a single device, it is necessary to transmit information (sensor information) acquired by the single device to another installed agent system.

60 60 55 60 As described above, in the present example (emotion positivity application), in a case in which the userexecutes an event, the emotion (positive or negative) of the userrelated to the event is analyzed on the basis of sensor information (captured image information obtained by the camera, vocal sound information obtained by the microphone, and the like) belonging to the event, and in the case of negative, appropriate advice can be obtained by asking about a countermeasure for positivity as a prompt to the sentence generation model, and it is possible to help the usercope with the event in a positive manner. When the emotion is classified into negative and positive, the emotion may be classified by using an absolute threshold or may be classified relatively. The term “relative” indicates that (1) the emotion is more positive this time than the emotion determined yesterday, (2) the emotion is negative this time from a transition situation for a certain period of time, and (3) the emotion is more negative than that of the spouse and siblings living together.

Next, as a seventh embodiment, an embodiment of a ToDo management application that manages an action history of an action history target designated by a user in an agent system will be described below.

10 100 101 102 100 2400 10 1 FIG. 35 FIG. The user and the action history target have a predetermined relationship. In the present embodiment, the useris an administrator (parent) of the robots,, and(hereinafter, collectively referred to as a robot) illustrated in, while the action history target is set to a child(see) of the user, and a parent-child relationship is assumed.

2 FIG. 2 FIG. 200 210 220 230 232 234 236 238 250 252 280 An execution program of the ToDo management application of the present embodiment is executed by the function of the CPU (each functional configuration illustrated in). As described above, the functional configuration includes the sensor unit, the sensor module unit, the storage unit, the user state recognition unit, the emotion determination unit, the action recognition unit, the action determination unit, the storage control unit, the action control unit, the control target, and the communication processing unit, and the ToDo management application is executed by cooperation of the functional configuration units (see).

100 2400 10 In the ToDo management application, the robotis controlled to execute the following (Operation 1) to (Operation 5) in order to monitor an action of the action history target (child) in response to a command from the user(parent).

100 2400 (Operation 1) The robotsequentially recognizes actions of the action history target (child) and stores an action history.

10 100 (Operation 2) From a communication terminal (a smartphone or a mobile phone) of the user(parent) to the robot, by using the SNS, a specific action is selected from a ToDo list registered in advance as an instruction for determining whether or not the specific action has been achieved, and a check message is transmitted (for example, “Have you done your homework?”). The specific action is not limited to homework, and may be any action registered in the ToDo list in advance, such as whether the child has cleaned, napped, or fed a pet. In a case in which it is desired to check a specific action that is not registered in the ToDo list, the action may be additionally registered. In this case, in a case in which there is an action history related to the additionally registered specific action, the action can be handled.

100 2400 (Operation 3) The robotdetermines whether or not the childhas done the homework from the action history of the child.

2400 55 35 FIG.(A) Operation 4-1: In a case in which the childis doing homework (see the action history in), a prompt such as “How should I praise the child in a case in which the child is doing his/her homework by himself/herself?” is input to the sentence generation model, to give an instruction for creating a sentence regarding a way of speaking (words of praise). 2400 55 3 FIG.(B) Operation 4-2: In a case in which the childis not doing homework (see the action history in), a prompt such as “How can I encourage the child to do his/her homework?” is input to the sentence generation model, to give an instruction for creating a sentence regarding a way of speaking (words of encouragement). (Operation 4 “Operation 4-1 or Operation 4-2”)

100 2400 (Operation 5) A speaker mounted on the robotor a connected earphone outputs words of praise (in the case of Operation 4-1) or words of encouragement (in the case of Operation 4-2) to the childby using vocal sound.

35 FIG. 2 FIG. 402 402 203 200 100 is a front view of image groupsA andB captured in time series for determination of a history of a specific event (an action of a designated action history target) captured by a camera (for example, the 2D camerawhich is one of the sensor unitsmounted on the robotillustrated in) managed and controlled by the agent system.

35 FIG. 2400 10 illustrates an image group extracted in a case in which there is an instruction regarding the action history of whether or not the action history target (child) has done the homework from the user.

35 FIG.(A) 35 FIG.(B) 2400 2400 illustrates an example of an image group in a case in which the childhas done the homework, andillustrates an example of an image group in a case in which the childhas not done the homework. In practice, one of the image groups exists in the same time period.

203 100 100 100 2400 A camera is not limited to the 2D cameramounted on the robot, and the robotmay acquire an image captured by a camera installed in each indoor room through communication. As a result, it is possible to acquire an image for the action history even if the robotdoes not follow the action of the child.

35 FIG.(A) 2400 2400 2406 2400 2400 2408 As illustrated in, in a case in which the image group at the time of doing the homework is checked in time series, it can be checked that the childis lying and operating the smartphone SP in the action history in the initial stage, and the childviews a televisionin the action history in the middle stage, but the childis studying (homework) in the action history in the end stage by synthesizing a state in which the childfaces the desk.

100 55 55 This case corresponds to the above (Operation 4-1), the robotinputs “How should I praise the child in a case in which the child is doing his/her homework by himself/herself?” to the sentence generation model. In response to this input, a way of speaking (words of praise) is acquired from the sentence generation model.

35 FIG.(B) 2400 2400 2400 2408 2400 2408 2410 As illustrated in, in a case in which the image group when the child is not doing the homework is checked in time series, it can be checked that the childhas not done the homework when summing up the states in which the childis lying down and operating the smartphone SP in the action history in the initial stage, the childis dozing while facing the deskin the action history in the middle stage, and the childmoves away from the deskwith the soccer ballin the action history in the final stage.

100 55 55 This case corresponds to the above (Operation 4-2), the robotinputs “How can I encourage the child to do his/her homework?” to the sentence generation model. In response to this input, a way of speaking (words of encouragement) is acquired from the sentence generation model.

36 FIG. is a control flowchart illustrating a flow of action history acquisition in the processing of the ToDo management application executed by the control device of the robot.

420 100 10 2400 10 In stepG, it is determined whether or not an action history start instruction has been given. The action history start instruction may be synchronized with activation of the robot, may be a time period set by a timer or the like, or may be an instruction from the user. In the case of the present embodiment (action history determination of the child), the time at which the parent who is the usergoes out may be set as an activation timing.

420 420 422 2400 10 In a case in which a negative determination is made in stepG, this routine is ended. In a case in which an affirmative determination is made in stepG, the processing proceeds to stepG, and information for specifying the action history target person (the childin the present embodiment) designated by the useris acquired. Examples of the specifying information include a feature used for face authentication, a feature of appearance, and a feature of sound. The number of action history target persons may be plural (siblings or the like).

424 426 2400 428 2400 In the next stepG, imaging is started. In next stepG, the action history target (child) is retrieved from the captured image, and then the processing proceeds to stepG to execute follow-up imaging processing for the retrieved action history target (child). As the follow-up imaging processing, time-lapse imaging (frame capturing) is performed, and in a case in which there are a plurality of action history targets, the action history targets are classified for each action history target, and the captured images are stored in time series. A still image or a moving image may be obtained through the time-lapse imaging capturing.

430 100 10 2400 10 In the next stepG, it is determined whether or not there is an action history end instruction. The action history end instruction may be synchronized with the activation end of the robot, or may be a time period set by a timer or the like, or an instruction from the user. In the case of the present embodiment (action history determination of the child), the time at which the parent who is the userreturns home from a place outside the house may be set as an activation end timing.

430 426 430 432 In a case in which a negative determination is made in stepG, the processing returns to stepG and the above-described steps are repeatedly executed in order to continue the action history. In a case in which an affirmative determination is made in stepG, the processing proceeds to stepG, the imaging is ended, and this routine is ended.

In other words, the action history (follow-up imaging processing) is continued until an action history end instruction is given.

37 FIG. 36 FIG. is a control flowchart illustrating a flow of an interrupt routine that is interrupted when an action history determination request is made during execution of the action history in(during follow-up imaging processing).

100 10 In the present embodiment, the action history determination request is that the robotreceives information such as “Have you done your homework?” from the going-out user(parent) via the SNS, and when the information is received, the interrupt routine at the time of giving an action history determination request is activated.

440 2400 In stepG, an action history target (child) to be determined is specified. For example, in a case in which there are a plurality of action history targets, identification information such as “first son” and “OO kun” is necessary.

442 2400 35 FIG.(A) 35 FIG.(B) In the next stepG, time-series image data of the specified action history target (child) is read. That is, in the present embodiment, any of the time-series images inoris actually stored, and any of the time-series image data is read.

444 In the next stepG, a result of the requested determination (determination of achievement or non-achievement of a specific action) is analyzed. Here, in response to the request “Have you done your homework?”, a positive determination (achievement of the homework) or a negative determination (non-achievement of the homework) is made.

35 FIG.(A) 2400 2406 2408 In a case in which the read time-series image data is the data illustrated in, it can be checked that the childis studying (doing homework) from the time-series images of lying down and operating the smartphone SP (initial stage), viewing the television(middle stage), and going to the desk(final stage), and thus, a positive determination is made.

35 FIG.(B) 2400 2408 2408 2410 On the other hand, in a case in which the read time-series image data is that data illustrated in, it can be checked that the childhas not done the homework from the time-series images of lying down and operating the smartphone SP (initial stage), moving toward the deskbut dozing (middle stage), and moving away from the deskwith the soccer ball(final stage), and thus, a negative determination is made.

446 444 In the next stepG, it is determined whether the determination result in stepG is a positive determination or a negative determination.

446 448 55 452 In a case in which it is determined that a positive determination is made in stepG, the processing proceeds to stepG, to input “How should I praise the child in a case in which the child is doing his/her homework by himself/herself?” to the sentence generation model(as an example, ChatGPT), and the processing proceeds to step.

446 450 55 452 In a case in which it is determined that a negative determination is made in stepG, the processing proceeds to stepG, to input “How can I encourage the child to do his/her homework?” to the sentence generation model(as an example, ChatGPT), and the processing proceeds to stepG.

452 55 454 55 In stepG, it is determined whether or not there is an answer from the sentence generation model(as an example, ChatGPT). In a case in which an affirmative determination is made, the processing proceeds to stepG, and the answer received from the sentence generation model(as an example, ChatGPT) is output from a speaker or the like by using vocal sound, and this routine is ended.

454 55 In stepG, an example of an answer from the sentence generation model(as an example, ChatGPT) ChatGPT to the positive determination (having done the homework) includes “Good job. There are snacks in the refrigerator, so eat some.”.

454 55 On the other hand, in stepG, an example of an answer from the sentence generation model(as an example, ChatGPT) to the negative determination (not having done the homework) includes “There are snacks in the refrigerator, so eat some after you finish your homework.”.

10 100 2400 2400 As described above, in the present embodiment (ToDo management application), instead of the user(parent) who is out of the house, the robotequipped with the agent system determines whether or not the action history target (child) who is staying home has done the homework, and inquires of ChatGPT about the respective situations of doing the homework (positive) and not doing the homework (negative), so that advice (appropriate words) can be received, and appropriate words can be spoken to the childby using vocal sound.

100 72 92 10 FIG. 12 FIG. An electronic apparatus is not limited to the robot, and may be another electronic apparatus (a doll, a stuffed toy, or smart glasses(see), the smart pendant(see), a smart watch, a smartphone, a smart speaker, an earphone, a personal computer, and the like) on which the agent system can be mounted.

10 In the present embodiment, a relationship between the userand the action history target is a parent-child relationship (guardian and protected person), but is not limited to the parent-child relationship, and can be applied as long as a relationship between a manager (guardian) and a managed person (protected person), such as a superior and a subordinate, a master and a student, an ordering side and an order-receiving side, and an owner and a pet, is established.

2400 10 10 10 100 10 10 10 100 In the present embodiment, the action history target (child) is distinguished from user, but in a broad sense, the action history target may also belong to user. That is, the usercan also designate himself/herself as an action history target and set himself/herself as a target of the action history by the robot. Alternatively, among a plurality of users, one usermay designate the other useras an action history target, and set the other user as a target of the action history by the robot.

5 5 50 38 39 FIGS.and Next, a systemaccording to an eighth embodiment will be described with reference to. The systemof the eighth embodiment is a system using a device on which the agent systemis mounted, similarly to the first embodiment.

5 50 50 In addition to smartphones, types of portable devices that can be owned by users, such as smart watches, smart glasses, and smart pendants, have been diversified, and one user usually owns a plurality of types of devices. The systemof the eighth embodiment is configured to be able to hand over data stored in the agent systemincluded in one device among a plurality of devices owned by one user to the agent systemincluded in another device. The term “owned” as used herein is not limited to a case in which a user properly purchases a device that is movable property and is an owner in a narrow sense under the Civil Code, and is also used in a meaning including a case in which the user has a legitimate right to use the device by a method such as lending, renting, or leasing.

400 50 400 5 39 FIG. As an example, the handover operation can be executed by a management serverthat manages the agent system.is a flowchart illustrating a handover operation of the agent system under the management of the management serverin the systemof the eighth embodiment.

39 FIG. 10 173 174 50 173 10 400 173 174 121 58 50 173 174 10 illustrates, as an example, a procedure in a case in which the userwants to change the device to be used from a smart watchto a smart pendantwhile interacting with a character set in an agent systemA in the smart watchowned by the user. In this case, the useraccesses the management serverand makes a handover request for change from the old device (smart watch) to the new device (smart pendant) (step S). The handover request can be made via, for example, a handover request input unitprovided in the agent systemof the smart watchor the smart pendant. The handover request may include, in addition to user information (a user ID, a password, and the like) indicating the userhimself/herself, device IDs of new and old devices to be transferred, and the like. The handover request may be performed through an operation of the old device, may be performed through an operation of the new device, or may be performed by a device other than the old and new devices (for example, a personal computer or a smartphone). The handover request may also be made offline.

400 122 Upon receiving this request, the management serverchecks the user information (the user ID and the password) indicated in the handover request, performs user authentication, and checks user registration of the presented device ID (step S). In a case in which the user registration has not been completed, a message indicating that the change according to the change request is performed after the user registration is completed is output. In a case in which the user registration is completed, the processing proceeds to the next step.

400 50 50 173 400 400 400 123 In a case in which the user authentication and the user registration are completed, in the management server, various data stored in the agent systemA is transmitted from the agent systemA of the smart watch, which is an old device, to the management serverunder the management of the management server, and is temporarily stored in the management server(step S).

400 400 50 174 124 400 10 125 400 174 10 50 50 173 53 174 400 Subsequently, the data temporarily stored in the management serveris transferred from the management serverto the agent systemB of the smart pendantthat is a new device (step S). In a case in which the transfer operation is completed, the management servertransmits a message reporting completion of the transfer to the account of the user(step S). Instead of the management serverreporting the completion of the transfer, the smart pendantthat is a transfer destination may perform a notification of completion of the transfer. With the above operation, the usercan use the data of the agent systemA of the old device used by the user in the agent systemB of the new device. For example, information regarding a character set in the smart watchof the old device, the interaction history performed at that time, and the emotion data generated by the emotion enginecan also be used in the smart pendantthat is a new device. Even when a plurality of owned devices are appropriately switched and used, it is possible to continue the interaction in the old device even in the new device. In the present embodiment, an example in which data of the agent system is handed over under the management of the management serverhas been described as an example, but the handover operation is not limited thereto. Under certain regulations, the handover operation of data may be directly executed between new and old devices through, for example, short-range wireless communication or wired connection. The same applies to the following embodiments.

5 5 50 100 40 41 FIGS.and Next, a systemaccording to a ninth embodiment will be described with reference to. The systemof the ninth embodiment is a system using a device on which the agent systemis mounted, similarly to the first embodiment, and has a feature related to handover of data in a case of interacting with the robotor an external device.

40 FIG. 100 10 10 100 100 5 50 100 50 173 10 50 100 50 173 10 100 10 10 As illustrated in, although the robotand the userare interacting with each other, the usermay leave the robotto go home or move to the next destination, and move out of the field of view of the robot. In this case, the systemof the ninth embodiment operates to transfer data of an agent systemX included in the robotto an agent systemA of the device (for example, the smart watch) owned by the user. That is, the data of the agent systemX of the robotis handed over by the agent systemA of the device (smart watch) owned by the user, and the interaction between the robotand the usercan be continued through the device owned by the user.

41 FIG. 400 10 173 100 73 is a flowchart for describing a handover operation of the agent system under the management of the management server. Here, it is assumed that the userowns the smart watchas an example, and is interacting with the robotwithout using the smart watch.

10 100 100 10 100 400 50 100 173 131 10 In this situation, it is assumed that the userleaves the robotafter interaction with the robotand moves to another place where interaction is not possible. In this case, in a case in which the userdesires to continue the interaction with the robot, the user may request the management serverto hand over the data of the agent systemX of the robotto a handheld device (for example, the smart watch) (step S). The handover request may include the device ID of the device that is a handover target and the like in addition to the user information (the user ID and the password) indicating the userhimself/herself.

400 132 10 100 10 Upon receiving this request, the management serverchecks the user information (the user ID and the password) indicated in the handover request, performs user authentication, and checks user registration of the presented device ID (step S). In a case in which the user registration has not been completed, a message indicating that handover is to be performed in accordance with the handover request after the user registration is completed is output, and in a case in which the user registration is completed, the processing proceeds to the next step. In this flowchart, the handover of the agent system is performed starting from the handover request from the user, but instead, the robotmay detect leaving of the userand start the handover operation.

400 50 50 100 400 400 400 133 In a case in which the user authentication and the user registration of the management serverare completed, various types of data stored in the agent systemX are transmitted from the agent systemX of the robotto the management serverunder the management of the management server, and is temporarily stored in the management server(step S).

400 400 50 173 134 400 10 125 400 173 10 50 100 50 173 100 100 53 173 100 100 Subsequently, the data temporarily stored in the management serveris transferred from the management serverto the agent systemB of the smart watch(step S). In a case in which the transfer operation is completed, the management servertransmits a message reporting completion of the transfer to the account of the user(step S). Instead of the management serverreporting completion of transfer, the smart watchthat is a transfer destination may perform a notification of completion of the transfer. With the above operation, the usercan use the data of the agent systemA of the robothaving been interacting in the agent systemA of the smart watchwhich is a new device to be used after leaving the robot. For example, information regarding the character of the robot, a history of the interaction performed at that time, and emotion data generated by the emotion enginecan also be used in the smart watchthat is a new device. Therefore, even after leaving the robot, it is possible to continue the interaction with the robotin the new device.

5 5 100 5 42 43 FIGS.and Next, a systemaccording to a tenth embodiment will be described with reference to. The systemof the tenth embodiment has a feature related to handover of data in the case of interacting with the robotor an external device, similarly to the ninth embodiment. The systemof the tenth embodiment is particularly effective in a case in which a user owns a plurality of types of devices.

42 FIG. 10 173 174 175 100 10 100 100 5 173 174 175 100 10 50 100 100 10 10 As illustrated in, the userinitially owns a plurality of types of devices, for example, a smart watch, a smart pendant, and smart glasses, but interacts with the robotwithout using the devices. However, thereafter, the usermay leave the robotto return home or move to the next destination, and move out of the field of view of the robot. In this case, the systemof the tenth embodiment determines which device is to be used among a plurality of devices (,,) owned by one user is most suitable for the robotand the userto have the smoother and more meaningful conversation. Here, a numerical value representing the degree to which the interaction can be smoothly and effectively performed is referred to as a “priority”, and a device having a high priority is selected as a handover target. The device determined to be the handover target with high priority hands over the data of the agent systemX of the robot, similarly to the above embodiment. As a result, the interaction between the robotand the usercan be continued through the device owned by the user.

43 FIG. 400 10 173 174 175 100 173 175 is a flowchart for describing a handover operation of the agent system under the management of the management server. Here, it is assumed that the userowns a plurality of smart devices such as the smart watch, the smart pendant, and the smart glassesas an example, and interacts with the robotwithout using these devicesto.

10 100 100 400 10 100 100 141 400 10 142 In this situation, it is assumed that the userleaves the robotafter interaction with the robotand moves to another place where interaction is not possible. In this case, the management serverchecks the leaving of the userfrom the robotaccording to a detection signal by the robotor a request from the user (step S). The management serverchecks user authentication of the userrelated to the leaving and user registration of the owned device (step S).

400 10 143 144 10 100 Interaction history data between the userand the robotbefore leaving 10 100 Environment of a position where the userand the robothave interacted before leaving 10 Emotion data of userbefore leaving Thereafter, the management serverdetermines priorities of a plurality of devices owned by the userwho has left (step S), and selects a device having a high priority (step S). As an example, the priority can be calculated using the following items as a determination material.

173 174 10 174 10 174 10 175 175 For example, in a case in which there is a high possibility that the content of the subsequent interaction is related to the physical condition of the user from the interaction history data, the smart watchthat can easily take the physical condition data of the user can be determined to have a high priority. In a case in which it is determined that the subsequent interaction is highly likely to be related to the surrounding environment from the environment of the position where the interaction has been performed or the like, the smart pendantthat easily captures or detects the surrounding environment can be determined to have a high priority. Alternatively, a priority for determining which device is used by the userto have a smoother and more meaningful conversation may be calculated from the history of the interaction or the like. For example, when it is determined that selecting the smart pendantimproves the appearance of the user and improves the reaction of the robot, the determination may be made by increasing the priority of the smart pendant. In a case in which it is determined that an intelligent appearance is given to the userby selecting the smart glasses, the determination can be made by increasing the priority of the smart glasses.

400 50 50 100 40 400 400 144 In a case where a device having a high priority is selected, the management servertransmits various types of data stored in the agent systemX from the agent systemX of the robotto the management serverunder the management of the management serverand temporarily stores the data in the management server(step S).

400 400 50 173 145 400 10 10 146 400 174 Subsequently, the data temporarily stored in the management serveris transferred from the management serverto the agent systemA of the selected device (for example, the smart watch) (step S). When the transfer operation is completed, the management servertransmits, to the account of user, a message reporting to which device of the userthe data has been transferred (step S). Instead of the management serverreporting completion of transfer, the devicethat is transfer destination may perform a notification of completion of the transfer.

10 50 100 100 100 10 10 10 With the above operation, the usercan use the data of the agent systemA of the robothaving been interacting in the agent system of the new device to be used after leaving the robot. In a case in which a plurality of devices are included, the data of the agent system of the robotis automatically transferred to a device having a high priority, and thus, the usercan continue the interaction with the roboteven after leaving the robot.

100 50 250 2 FIG. Hereinafter, as an eleventh embodiment, action control executed at the time of a sudden emotion change in action control programmed in advance for a robotas an electronic apparatus on which an agent systemis mounted will be described. The action control executed at the time of the emotion change of the present embodiment is executed by the action control unitillustrated in.

100 50 The electronic apparatus is not limited to the robot, and may be any of other electronic apparatuses (dolls, stuffed toys, pendants, smart watches, smartphones, smart speakers, earphones, personal computers, and the like) on which the agent systemcan be mounted.

100 100 10 100 The action of the robotis determined according to the degree of emotion including “happy”, “angry”, “sad”, and “pleasant”. For example, as described above, a value (0 to 5 levels) indicating the intensity of each of “happy”, “angry”, “sad”, and “pleasant” is determined, and the robotacts in conjunction with the emotion of the useror acts according to the situation (brightness of a surrounding environment, a remaining battery level, or the like) in which the robotis placed.

The value (0 to 5 levels) indicating the intensity of each emotion of “happy”, “angry”, “sad”, and “pleasant” is a value determined at the time of action execution, and it is possible to impart a so-called sharpness to the action of the robot by considering the process of reaching the value indicating the intensity of the emotion.

For example, in a case in which the emotion of “pleasant” changes to the emotion of “angry”, when an action is determined only with a value indicating the intensity of the current emotion, the same action is performed regardless of whether degree of the change is gradual or abrupt.

Therefore, in the present embodiment, the process until the value indicating the intensity of the emotion is determined is reviewed, and an action for clarifying the switching of the emotion is interposed between the action changes according to the degree of change in the emotion. As an example, in a case in which the emotion changes suddenly, the following action (vocal sound output is primarily performed) can be taken. The determination as to whether the change in emotion is sudden will be described later.

(Action 1) In a case in which a sudden change occurs from an emotion of “pleasant” to an emotion of “angry”, an action expressing that the emotion is switched is executed before the emotion is switched. Examples of the action expressing switching include vocal sound output such as “hum”.

(Action 2) In a case in which a sudden change occurs from an emotion of “sad” to an emotion of “pleasant”, an action expressing that the emotion is switched is executed before the emotion is switched. Examples of the action expressing switching include vocal sound output such as “Puh!” and “humming”.

In the above description, the example of expressing the switching of the emotions by using the vocal sound has been described. However, in addition to the vocal sound, the intonation of the onomatopoeia or speech, the facial expression (changes in eyebrows, eyes, mouth, and the like), the temperature (in particular, it is effective in a case of being in contact with a user like a wearable device), the color (complexion or color development state of some light sources), and the movement (restless actions such as foot tapping, bruxism, and breathing) may be simultaneously executed. Different types of expressions may be executed simultaneously (for example, dancing while humming a tune).

55 In the present embodiment, it is assumed that an inquiry about an action is made to the sentence generation model(for example, ChatGPT). In this case, the speech is essential, but the action may be an action not including vocal sound as long as it is possible to notify the user of a change in emotion.

The current emotion can be sufficiently determined as long as the values (0 to 5) indicating the intensities of “happy”, “angry”, “sad”, and “pleasant” are known. However, it is difficult to determine the degree of change in emotion with a numerical value at the present time.

44 FIG. Therefore, in the present embodiment, as illustrated in, the intensity of each emotion (“happy”, “angry”, “sad”, and “pleasant”) is acquired and stored at regular time intervals, and the degree of change in the emotion is determined by comprehensively verifying each plot point.

44 FIG. 44 FIG. In, the horizontal axis represents the time axis, and the vertical axis represents the intensity of each emotion. In, the emotion intensity is 0 (weakest) to 100 (strongest), but may be 8 bits (0 to 255).

44 FIG. 44 The value (0 to 5 levels) indicating the intensity of each of “happy”, “angry”, “sad”, and “pleasant” described above can be determined, for example, by the relative relationship between the intensities of each emotion (numerical values on the vertical axis in) every minute. The determination of 0 to 5 levels is not limited to the characteristic diagram of FIG., and may be determined by using another means.

44 FIG. On the other hand, in the present embodiment, the degree of emotion is determined on the basis of the characteristics inseparately from the values (0 to 5 levels) indicating the intensities of “happy”, “angry”, “sad”, and “pleasant”.

As a determination parameter of the degree of emotion, in addition to the real-time value of each emotion, a change rate, an average value in a past certain period, a relative difference from each emotion, and the like are calculated, and the degree of change is obtained on the basis of a predetermined emotion determination function.

In a case in which there is a change (for example, a change from “pleasant” to “angry”) in the emotion and the calculated degree of change is larger than a predetermined threshold, the change is determined as a sudden change (corresponding to Action 1 described above).

45 FIG. 2 FIG. 44 FIG. 45 FIG. 600 232 600 is a functional block diagram of an emotion sudden change determination unitthat determines the presence or absence of a sudden change in emotion, set as a part of the function of the emotion determination unit(see), based on the characteristic diagram of. Each control block of the emotion sudden change determination unitinis classified by function, and does not limit the hardware configuration, and may be an IC chip such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) in which some or all of the control blocks operate on the basis of a control program, or a software program.

602 604 44 FIG. A data collection unitcollects the value of the intensity of each emotion (read value on the vertical axis in) at regular time intervals (for example, 1 second) and stores the value in the database.

606 604 604 608 A determination parameter calculation unitis connected to the database, and reads the value of the intensity of each emotion stored in the databaseevery 1 minute, for example, and calculates each determination parameter. In the present embodiment, in addition to the real-time value of each emotion, a change rate, an average value in a past certain period, a relative difference value from each emotion, and the like are calculated, and each value is sent to a function calculation unit.

608 610 610 612 608 614 The function calculation unitobtains a degree of change on the basis of a predetermined emotion determination function, and sends the degree of change to a comparison unit. The comparison unitreads a predetermined threshold from a threshold storage unitand compares a calculated value calculated by the function calculation unitwith the threshold. The comparison result is sent to a determination unit, it is determined whether or not the change in emotion is sudden, and the determination result is sent to the action control system.

236 50 55 Here, in the case of determining that there is a sudden change in emotion, the action determination unitof the agent systeminputs “What action should be interposed in switching of the sudden change in emotion (for example, in the case of Action 1, “pleasant” to “angry”)?” to the document generation model(for example, ChatGPT).

55 50 In response to this input, an action (mainly vocal sound) at the time of switching is acquired from the sentence generation model. Examples of the answer include “Please say “Huh?” and, in accordance with this, perform an action of making a gesture such as a gesture of raising both hands slightly with the palm facing upward. “. In the agent system, this answer is analyzed and reflected in the action.

236 250 On the basis of the answer result, the action determination unitdetermines an action expressing switching before the emotion is switched, and the action control unitcontrols execution of the action.

As the action (vocal sound) expressing switching, in the case of Action 1, vocal sound such as “Huh?” is output. At the same time as the vocal sound output, it is preferable to execute an action such as gesture or facial expression as an action.

46 FIG. 236 250 is a control flowchart illustrating a flow of action execution processing executed by the action control system (mainly, the action execution unitand the action control unit).

420 420 422 In stepH, it is determined whether or not there is a change in emotion, and in a case in which a negative determination is made, this routine is ended. In a case in which an affirmative determination is made in stepH, it is determined that there is a change in emotion, and the processing proceeds to stepH.

422 44 FIG. In stepH, emotion level determination processing is executed. The emotion level determination processing is executed by using the characteristic diagram of.

45 FIG. 402 404 That is, as illustrated in, the data collection unitcollects the value of the intensity of each emotion at regular time intervals (for example, 1 second) and stores the value in the database.

406 404 408 The determination parameter calculation unitreads the value of the intensity of each emotion stored in the databaseevery minute, for example, and calculates each determination parameter (a real time value, a change rate, an average value in a past certain period, a relative difference value with each emotion, or the like). The function calculation unitobtains the degree of change on the basis of a predetermined emotion determination function, and compares the degree of change with a threshold to determine whether or not the change in emotion is sudden.

424 422 426 424 434 In the next stepH, it is determined whether or not the degree of change is sudden on the basis of the determination result in stepH, and in a case in which an affirmative determination (sudden change in emotion) is made, the processing proceeds to stepH. In a case in which a negative determination (no sudden change in emotion) is made in stepH, the processing proceeds to stepH (that will be described later).

426 55 In stepH, “What action should be interposed in switching of a sudden change?” is input to the sentence generation model(ChatGPT).

428 55 428 430 55 432 434 In the next stepH, it is determined whether or not there is an answer from the sentence generation model. In a case in which an affirmative determination is made in stepH, the processing proceeds to stepH where the answer received from the sentence generation modelis analyzed to determine an action. Next, the processing proceeds to stepH where an emotion sudden change notification action is executed before the action based on the change in emotion, and the processing proceeds to stepH.

10 100 As a result, the usercan clearly recognize the change in emotion of the robot, and can recognize that some irregular situation (abnormal situation, emergency, or the like) has occurred.

434 430 424 In stepH, an action based on the change in emotion is executed. That is, in a case in which the emotion suddenly changes, the action based on the change in emotion is executed through the emotion sudden change notification action in stepH. In a case in which the emotion does not suddenly change, the processing proceeds from stepH and the action based on the change in emotion is executed without performing the emotion sudden change notification action.

As described above, in the present embodiment (the action control at the time of sudden change in emotion), in a case in which the degree of change in the emotion is determined to be sudden when the emotion changes to different types of emotions, it is possible to clearly notify the user that the change in emotion has occurred by interposing an action for appealing the change in emotion rather than simply performing an action in accordance with the changing emotion. Whether or not the change in emotion is sudden can be determined on the basis of the intensity of each of happiness, anger, sadness, and pleasure in consideration of parameters such as a real-time value, a change rate, an average value in a past certain period, and a relative difference from each emotion.

100 In the above embodiment, in a case in which there is a sudden change in the action on the basis of only the emotion of the robotitself, the action indicating the switching of the emotion is interposed. However, in consideration of the emotion of the user, an action indicating a trigger for the switching of the emotion may be added.

For example, in a case in which the user is likely to be intimidated or show fear, an action to reduce the rate of the change in emotion is added. This can reduce the sense of discouragement or fear of the user.

100 50 In the present embodiment, the change in emotion of the robothas been described. However, in a case in which the agent systemis a wearable device, a trigger of the change in emotion may be reported by using vibration or a temperature change.

In the present embodiment, the following embodiments can be considered. There may be a part overlapping with the above embodiment.

(Embodiment 1) The emotion sudden change notification action includes notification through hearing, such as sound (sound, intonation, and onomatopoeia). For example, “All right!!” is exemplified.

100 100 100 (Embodiment 2) The emotion sudden change notification action includes visual notification (action). For example, there is a change in the facial expression (eyebrows, eyes, mouth, and the like) of the robot. The visual notification may be a color change of a part of the complexion, the lamp, or the like of the robot, or the robotmay be restless or execute a peculiar gesture such as foot tapping.

72 100 (Embodiment 3) The emotion sudden change notification action includes a notification through tactile sense (temperature change). For example, a contact temperature in the case of the wearable device, or hot air blowing in the case of the robotmay be used.

(Embodiment 4) The expression of a state change notification function is made different according to the direction of the change in emotion (pleasant->angry->pleasant) and the degree of change.

(Embodiment 5) The degree of emphasis of expression is adjusted according to the degree of change in emotion. That is, the greater the degree of change, the higher the degree of emphasis of the expression. The smaller the degree of change, the lower the degree of emphasis of the expression. A primary or numerical proportional relationship or a stepwise relationship may be used.

10 10 10 10 (Embodiment 6) The emotion of the useris taken into consideration. In other words, when the userfeels intimidated or scared, the change in emotion that alleviates the intimidation or scare is executed. When the userenjoys, a change in emotion is executed to encourage and excite the user.

47 FIG. 50 50 Hereinafter, with reference to, arbitration in a case in which two or more agent systemsconflict in action control programmed in advance with respect to the agent systemwill be described as a twelfth embodiment.

47 FIG. 7 FIG. 1 FIG. 60 50 60 72 92 60 10 As illustrated in, in the present embodiment, it is assumed that the user(see) is under the service provision of two or more kinds of agent systems. As an example, an example in which the userwears the smart glassesand the smart pendanteach having a function as an agent system will be described. The useris equivalent to the userillustrated in.

72 92 60 50 50 50 7 FIG. The smart glassesand the smart pendantworn by the userrespectively include agent systemsA andB having functions similar to those of the agent systemillustrated in.

50 50 51 52 53 54 55 56 57 50 7 FIG. That is, the agent systemsA andB include, as common functions, an interaction processing unit, a character setting unit, an emotion engine, an interaction history storage unit, a sentence generation model, a command acquisition unit, and a robotic process automation (RPA). Since the details of each unit are the same as those of the agent systemillustrated inand have been described above, the detailed description thereof will be omitted here.

72 74 76 78 80 82 84 86 88 The hardware configuration of the smart glassesincludes a control device, a display, a touch panel, a speaker, a sensor group, a microphone, a camera, and a communication I/F.

92 93 94 95 96 On the other hand, the hardware configuration of the smart pendantincludes a control device, a speaker, a microphone, and a communication I/F.

72 86 92 In implementing the present embodiment, the major difference is that the smart glassesinclude the camera, whereas the smart pendantdoes not include the camera.

89 50 72 50 89 50 89 Here, in the present embodiment, an arbitration unitis provided in the agent systemA mounted on the smart glasses. In other words, the agent systemA including the arbitration unitis an “arbitration agent”, and the agent systemB not including the arbitration unitis a “non-arbitration agent”. Although the arbitration agent is single, there may be a plurality of non-arbitration agents.

89 60 50 50 60 The arbitration unithas a function of arbitrating an optimum answer as an answer to the userwhen the agent systemsA andB provide answers to the speech of the userin a case in which two answers are different.

60 84 72 95 92 50 50 60 55 That is, when the usermakes speech, the microphoneof the smart glassesand the microphoneof the smart pendantcollect sound, and the agent systemsA andB acquire answers to the speech of the userby using the function of the sentence generation model.

50 50 60 60 The agent systemsA andB speak the respectively acquired answers to the user, but the content of the speech may prompt different decision-making (for example, giving an instruction for different actions for one event) of the user(answer mismatch).

89 Therefore, when there is a mismatch between the answers, the arbitration unitderives an optimum answer (arbitration processing) on the basis of a predetermined condition, and executes speech based on the arbitration result.

60 As the predetermined condition, for example, an adoption history of the userwhen there was a mismatch between answers in the past is used. An answer of an agent having the highest adoption ratio is selected in the adoption history.

60 As another condition, for example, in a case in which the speech of the userrelates to a type of dish, it is conceivable to provide a recommended dish in consideration of the weather, temperature, time period, and the like of the day.

60 50 50 The usercan execute a satisfactory action while listening to the opinions of the plurality of agent systemsA andB and receiving the optimum answer.

48 FIG. 60 74 50 93 50 is a control flowchart illustrating a flow of response processing to speech of the user, executed by each of the control deviceof the agent systemA and the control deviceof the agent systemB.

48 FIG.(A) 60 74 50 is a control flowchart illustrating a flow of response processing to speech of the user, executed by the control deviceof the agent systemA.

4001 60 In step, it is determined whether or not speech of the userhas been received (whether or not sound has been collected), and in a case of a negative determination, standby is performed until an affirmative determination is made.

4001 4021 404 55 In a case in which an affirmative determination is made in step, the processing proceeds to stepto execute voice recognition on the collected speech, and then the processing proceeds to stepI to input “How to answer the speech content?” to the sentence generation model(for example, ChatGPT).

406 55 In the next stepI, it is determined whether or not there is an answer from the sentence generation model, and in the case of a negative determination, standby is performed until an affirmative determination is made.

406 4081 55 60 410 410 60 72 55 In a case in which the affirmative determination is made in stepI, the processing proceeds to step, the answer received from the sentence generation modelis analyzed, speech content to the useris determined, and the processing proceeds to stepI. In stepI, speech directed to the useris executed. The speech content may be determined on the basis of the information regarding the user stored in the smart glasseswithout using the sentence generation model.

412 60 462 48 FIG.(B) 48 FIG. Here, in the next stepI, an answer (speech) to the userby another agent (non-arbitration agent) is acquired as a function of the arbitration agent. This acquisition is based on the processing in stepin(see the dotted arrow A in).

414 414 416 414 48 FIG. In the next stepI, it is determined whether the answers match. In stepI, in a case in which a negative determination (speech mismatch determination) is made, the processing proceeds to stepI, and arbitration processing is executed. In a case in which an affirmative determination (speech match determination) is made in stepI, arbitration is not necessary, and thus the routine is ended. Whether or not to execute the arbitration processing is reported to the non-arbitration agent (see the dotted arrow B in).

416 60 418 420 48 FIG. In the arbitration processing in stepI, an optimum answer is derived on the basis of a predetermined condition (for example, the adoption history information of the user). In the next stepI, the arbitration result information is output to the other agent (see the dotted arrow C in), and the processing proceeds to stepI, and the speech based on the arbitration result is output, and this routine is ended.

48 FIG.(B) 60 93 50 is a control flowchart illustrating a flow of response processing to speech of the user, executed by the control deviceof the agent systemB.

450 60 In stepI, it is determined whether or not speech of the userhas been received (whether or not sound has been collected), and in a case of a negative determination, standby is performed until an affirmative determination is made.

450 452 454 55 In a case in which an affirmative determination is made in stepI, the processing proceeds to stepI to execute voice recognition for the collected speech, and then the processing proceeds to stepI to input “How to answer the speech content?” to the sentence generation model(for example, ChatGPT).

456 55 In the next stepI, it is determined whether or not there is an answer from the sentence generation model, and in the case of a negative determination, standby is performed until an affirmative determination is made.

456 4581 55 60 460 460 60 92 55 In a case in which the affirmative determination is made in stepI, the processing proceeds to step, the answer received from the sentence generation modelis analyzed, speech content to the useris determined, and the processing proceeds to stepI. In stepI, speech directed to the useris executed. The speech content may be determined on the basis of the information regarding the user stored in the smart pendantwithout using the sentence generation model.

4621 60 48 FIG. Here, in the next step, an answer (speech) for the useris output to the adjustment agent as a function of the non-arbitration agent (see the dotted arrow A in).

464 416 464 464 466 418 48 FIG.(A) 48 FIG. 48 FIG.(A) 48 FIG. In the next stepI, the presence or absence of arbitration is determined. This determination material is based on the processing in stepI in(see the dotted arrow B in). In a case in which a negative determination is made in stepI, it is determined that there is no arbitration, and this routine is ended. In a case in which an affirmative determination is made in stepI, it is determined that arbitration has been performed, and the processing proceeds to stepI to acquire an arbitration result. This acquisition is based on stepI in(see the dotted arrow C in).

466 468 55 55 454 In a case in which a negative determination is made in stepI, standby is performed until an affirmative determination is made, and in a case in which an affirmative determination is made, the processing proceeds to stepI, and speech based on the arbitration result information is output, and this routine is ended. Although the detailed description of the speech based on the arbitration result information is omitted here, speech content indicating agreement with the arbitration result may be prepared in advance, or “The result of arbitration was oo, but how to say?” or the like may be input to the sentence generation modeland the speech may be made on the basis of an answer of the sentence generation modelas in stepI.

60 72 92 60 50 In the present embodiment, the processing of responding to the userusing the two devices (the smart glassesand the smart pendant) has been described, but the usermay be provided with three or more types of devices (the agent system).

60 60 60 As described above, in the present embodiment, when the usermurmurs for some answer, a plurality of agents collect vocal sound, and in a case in which answers obtained through vocal sound analysis show different results, arbitration is performed on the agent system side before the usermakes a final selection. As a result, it is possible to eliminate the user's hesitation in selection and to increase the probability of selecting an optimal action.

The arbitration in the present embodiment does not mean arbitration for conflict between users, but refers to determining a priority in a case in which agents give different pieces of advice.

50 50 50 50 72 86 50 92 60 72 86 72 92 50 50 For example, in a case in which the proposal of the agentA has a higher adoption ratio than the proposal of the agentB, the proposal of the agentA is preferentially adopted. In a case in which arbitration is performed between the agentA of the smart glasses(with the camera) and the agentB of the smart pendant, the expression of the useris acquired from the smart glasses, and which proposal to adopt can be determined. In this case, by sharing the image from the cameranot only with the smart glassesbut also with the smart pendant, it is possible to improve the proposal accuracy of each of the agentsA andB.

50 In a case in which the image from the camera is not shared with the smart pendant, the proposal of the agentA may be prioritized from the beginning.

49 FIG. 48 FIG. illustrates an execution example of the flowchart of, and illustrates a situation in which frames A to F are executed in time series in this order.

60 50 50 60 4001 4021 450 4521 48 FIG.(A) 48 FIG.(B) (Frame A) The usermurmurs, “I'm hungry . . . what should I eat . . . ” The agentsA andB each collect murmur of the userand perform voice recognition (stepand stepin, and stepI and stepin).

50 60 55 404 4101 48 FIG.(A) (Frame B) The agentA makes an inquiry about the murmur of the userto the sentence generation model, and replies “Let's go to eat hamburgers!!” (stepsI toin).

50 60 454 4601 48 FIG.(B) (Frame C) The agentB replies “On the second thought, ramen noodles!” to the murmur of the user(stepsI toin).

50 50 60 In this example, the agentB has answered (frame C) in consideration of the answer (frame B) of the agentA, but the order of the answers may be reversed, or the answers may not affect the answers of other agents. In the frame B and the frame C, simultaneous speaking or repeated answers may be performed, and in a certain sense, it is possible to form a situation in which a plurality of people are discussing, and to give enjoyment to the process up to the decision making of user.

50 416 48 FIG.(A) (Frame D) Arbitration is performed by the agentA (stepI in).

50 50 In this case, the arbitration is performed in a state in which there is an answer from the agentA and the agentB one by one in the frame B and the frame C. However, in a case in which there is a process of discussion, the arbitration may be started after a predetermined discussion time has elapsed.

50 50 4201 48 FIG.(A) (Frame E) In a case in which the answer of the agentA is adopted as a result of the arbitration, the agentA says “Hamburger is the best choice now!! “. (stepin)

50 4681 48 FIG.(B) The side rejected through the arbitration (the agentB) may make speech accepting the arbitration result such as “That's right . . . ” (stepin).

60 50 50 60 (Frame F) When the usersays “I decided to eat hamburgers!!” according to the arbitration result, the agentsA andB collect the speech of the userand perform voice recognition, thereby ending the series of processing.

49 FIG. 50 60 60 86 72 92 86 72 In the execution example in, the agentA determines an answer on the basis of a predetermined condition (such as the adoption history of the userwhen there was a mismatch between answers in the past). However, the current state of the usermay be imaged by the cameramounted on the smart glasses, and arbitration may be performed by using the imaged information. The camera function may be provided in each device, or in a case in which a camera is not mounted like the smart pendant, information imaged by another device (here, the cameraof the smart glasses) may be shared.

60 In addition to the current state or instead of the current state, a past action history image of the userup to the current state may be used (yesterday's meal menu information, exercise amount, dining partner, and the like).

3400 50 3400 3400 7 FIG. 2 FIG. In a thirteenth embodiment, an order terminalusing an agent system(see) configured by using some or all of the functions of the action control system (see) will be described. Portions having configurations similar to those of the above-described embodiment will be denoted by the same reference numerals, and description thereof will be omitted or simplified. Hereinafter, a case in which the order terminalis installed at a guest seat of a restaurant will be described. The order terminalof the present embodiment is a terminal that transmits a user's order to an order processing apparatus (not illustrated) that processes the order.

50 FIG. 3400 3400 60 3400 3410 3420 3430 34340 3450 3460 3470 3480 schematically illustrates an example of a hardware configuration of the order terminalaccording to the thirteenth embodiment. The order terminalis a tablet-type smart device, and is used by the usersimilarly to a general smart device. The order terminalincludes a control device, a display, a touch panel, a speaker, a sensor group, a microphone, a camera, and a communication I/F.

3410 3400 3411 3412 The control deviceis a device that controls the entire order terminal, and includes, for example, a storage, a CPU, and a memory (not illustrated).

51 FIG. 3411 3400 3411 3411 3411 schematically illustrates an example of a configuration of the storageof the order terminalaccording to the thirteenth embodiment. The storageaccording to the thirteenth embodiment includes a product information databaseA and an order programB.

3411 The product information databaseA is a database that stores information regarding menus provided at restaurants. Examples of the information regarding the menu include a menu name, an image photograph of the menu, a description of the menu, and a price. The menu provided at a restaurant is an example of a “product”.

3411 3411 3411 The order programB is a program that executes processing including order placement processing and emotion determination processing that will be described later. The product information databaseA and the order programB may be stored in a memory (not illustrated).

3412 50 FIG. The CPUinis a central processing unit, and executes various programs and controls each unit.

3420 3420 3400 60 3430 60 34340 60 The displayis, for example, a liquid crystal display. The displayis provided, for example, in a front portion of the order terminal, and the display content can be visually recognized by the user. The touch panelreceives an input from the user. The speakeroutputs vocal sound indicating various types of information to the user.

3450 60 3450 3450 60 The sensor groupdetects a state of the user. The sensor groupincludes a plurality of sensors. For example, the sensor groupincludes a 3D depth sensor, a distance sensor, an acceleration sensor, and a temperature sensor (not illustrated). These sensors are merely examples, and it is a matter of course that other sensors may be mounted in order to detect the state of the user.

3460 60 3400 3470 3400 3470 60 3470 3480 3400 3410 3420 3430 34340 3450 3460 3470 3480 3490 The microphoneacquires vocal sound output by the useror environmental sound around the order terminal. The cameracan image surroundings of the order terminal. The cameracan also image the face of the user. The camerais, for example, a CCD camera. The communication I/Fhas a communication function between the order terminaland the outside. The control device, the display, the touch panel, the speaker, the sensor group, the microphone, the camera, and the communication I/Fare connected to a bus.

52 FIG. 7 FIG. 3400 3400 3412 3412 3412 3412 3412 3411 3400 50 50 3410 50 3400 3480 50 3400 schematically illustrates an example of a functional configuration of the order terminalaccording to the thirteenth embodiment. The order terminalof the present embodiment functions as a reception unitA, an input unitB, a presentation unitC, and an order placement unitD when the CPUexecutes an order programB. The order terminalcan use the function of the agent system(see). Here, using the agent systemincludes that the control devicehas a function similar to that of the agent systemand uses the function, and also includes an aspect in which the order terminalcommunicates with the outside (for example, the server) via the communication I/Fto use the agent systemstored outside the order terminal.

3412 3412 60 3460 60 3412 60 3430 The reception unitA has a function of receiving information regarding menu selection. Specifically, the reception unitA receives selection of a menu using vocal sound of the uservia the microphone. The information regarding menu selection includes information regarding a menu name, a quantity, a provision timing, and the like of a menu selected by the user. The reception unitA may receive an input from the userthrough the touch panel. The menu selection is an example of “ordering a product”. The information regarding menu selection is an example of “order information”.

3412 60 3412 60 3412 60 The reception unitA has a function of receiving a reply of the user. Specifically, the reception unitA receives whether or not a reply of the userto information presented by the presentation unitC that will be described later is affirmative. The reply of the useris an example of an “answer of the user”.

3412 55 3412 60 3412 55 50 7 FIG. The input unitB has a function of inputting information regarding menu selection to a sentence generation model(see) that will be described later. Specifically, the input unitB inputs the information regarding the menu selection from the userreceived by the reception unitA to the sentence generation modelof the agent system.

55 50 55 60 3412 3411 55 60 3412 7 FIG. 51 FIG. The sentence generation modelof the agent systemof the present embodiment inhas a function of generating a menu name corresponding to information regarding menu selection. Specifically, the sentence generation modelgenerates a menu name corresponding to the information regarding menu selection of the userinput from the input unitB with reference to the product information databaseA (see). The sentence generation modelcan generate information regarding a quantity corresponding to the information regarding the menu selection of the userinput from the input unitB.

3412 3412 51 50 420 440 4312 3440 3420 51 52 FIG. 7 FIG. The presentation unitC inhas a function of presenting information for checking the information regarding menu selection. Specifically, the presentation unitC displays lines of the agent determined by the interaction processing unit(see) of the agent systemthat will be described later on the displayor outputs the lines from the speaker. The presentation unitC may output lines of the agent from the speakerand display information regarding a menu related to the lines on the display. The information for checking the information regarding the menu selection is an example of “check information”. The interaction processing unitis an example of a “conversation processing unit”.

51 50 51 60 55 60 53 51 60 53 7 FIG. The interaction processing unitof the agent systemaccording to the present embodiment inhas a function of determining lines of the agent on the basis of a menu name and an emotion value. Specifically, the interaction processing unitdetermines the lines of the agent for checking the menu selected by the useron the basis of the menu name generated by the sentence generation modeland the emotion values of the userand the agent estimated by the emotion enginethat will be described later. The interaction processing unitmay determine the lines of the agent on the basis of only the emotion value of the userestimated by the emotion engineor only the emotion value of the agent. The lines of the agent for checking the menu are an example of “speech content”.

53 50 60 60 53 60 60 230 53 232 53 2 FIG. 2 FIG. The emotion engineof the agent systemaccording to the present embodiment has a function of determining emotions of the userand the agent who has a conversation with the user. Specifically, the emotion enginedetermines an emotion value indicating the emotion of the userand an emotion value of the agent from a state of the userrecognized by the user state recognition unit(see) of the action control system that will be described later. The emotion engineincludes the emotion determination unit(see) of the action control system. The emotion engineis an example of an “emotion determination unit”.

230 50 60 230 60 3450 3460 3470 230 2 FIG. The user state recognition unit(see) of the action control system used by the agent systemof the present embodiment has a function of acquiring the state of the user. Specifically, the user state recognition unitacquires the state of the useron the basis of information analyzed by the sensor group, the microphone, and the camera. The user state recognition unitis an example of an “acquisition unit”.

3412 60 60 412 3412 3412 52 FIG. The order placement unitD inhas a function of ordering a menu according to a reply of the user. Specifically, in a case in which the reply of the userreceived by the reception unitA to the information presented by the presentation unitC is affirmative, the order placement unitD orders a menu corresponding to the information.

53 FIG. 3400 60 schematically illustrates an example of a flow of order placement processing according to the thirteenth embodiment. The order placement processing of the thirteenth embodiment is processing executed when the order terminalreceives an order from the user.

200 3412 60 3412 60 3460 53 FIG. In step SJ in, the CPUreceives information regarding menu selection from the user. As an example, the CPUreceives information “Four beer, fried” from the userin vocal sound via the microphone.

201 3412 55 3412 200 55 In step SJ, the CPUinputs the received information to the sentence generation model. Specifically, the CPUinputs the information regarding the menu selection received in step SJ to the sentence generation model.

202 3412 55 3412 55 201 3412 In step SJ, the CPUacquires menu names corresponding to the received information from the sentence generation model. Specifically, the CPUacquires the menu names generated by the sentence generation modelon the basis of the information input in step SJ. As an example, the CPUacquires “draft beer, medium mug” and “bottled beer” as menu names corresponding to information regarding “beer”, and acquires “fried young chicken” and “fried chicken wings” as menu names corresponding to information regarding “fried”.

203 3412 3412 60 3412 60 54 FIG. In step SJ, the CPUexecutes emotion determination processing (see) that will be described later. The CPUdetermines emotion values of the userand the agent through the emotion determination processing. As an example, the CPUdetermines that the emotion value of the useris “pleasant” and the emotion value of the agent is “delighted”.

204 3412 3412 202 203 3412 3412 60 In step SJ, the CPUdetermines lines of the agent on the basis of the acquired menu names and the determined emotion values. Specifically, the CPUdetermines the lines of the agent on the basis of the menu names acquired in step SJ and the emotion values determined in step SJ. As an example, the CPUdetermines the lines of the agent as “It seems that you had a good time today! I'll check your order details! Would you like four medium mugs of draft beer and a piece of fried chicken? For beer, there is bottled beer, and for fried chicken, there is also fried chicken wings.”. The CPUmay determine the lines of the agent on the basis of only the emotion value of the useror only the emotion value of the agent.

205 3412 60 3412 60 204 3440 3412 3420 In step SJ, the CPUpresents the menu names to the useron the basis of the determined lines. Specifically, the CPUpresents the menu names to the userby outputting the lines of the agent determined in step SJ by using vocal sound from the speaker. The CPUmay display, on the display, a menu name to be presented, an image of a menu corresponding to the menu name, a description of the menu, a quantity, and the like.

206 3412 60 60 206 3412 208 60 206 3412 207 In step SJ, the CPUdetermines whether or not the reply of the userto the presented menu names is affirmative. In a case in which it is determined that the reply of the useris affirmative (step SJ: YES), the CPUproceeds to step SJ. On the other hand, in a case in which it is determined that the reply of the useris not affirmative (step SJ: NO), the CPUproceeds to step SJ.

207 3412 55 60 207 3412 60 200 201 3412 202 In step SJ, the CPUinputs to the sentence generation modelthat the presented menu names are different from the intention of the user. Instead of step SJ, the CPUmay again receive and input information regarding menu selection from the user(steps SJ and SJ). The CPUreturns to step SJ.

208 3412 3412 202 3412 3412 In step SJ, the CPUorders menus corresponding to the presented menu names. Specifically, the CPUtransmits the menus corresponding to the menu names acquired in step SJ to an order processing apparatus (not illustrated). As an example, the CPUorders “four” “draft beer medium mugs” and “one” “fried young chicken”. The CPUends the order placement processing.

54 FIG. 53 FIG. 203 schematically illustrates an example of a flow of emotion determination processing according to the thirteenth embodiment. The emotion determination processing is processing executed in step SJ (see).

300 3412 60 3412 60 3450 3460 3470 54 FIG. In step SK in, the CPUrecognizes a state of the user. Specifically, the CPUrecognizes the state of the useron the basis of the information analyzed by the sensor group, the microphone, and the camera.

301 3412 60 3412 60 3450 3460 3470 60 300 In step SK, the CPUdetermines an emotion value of the user. Specifically, the CPUdetermines an emotion value indicating the emotion of the useron the basis of the information analyzed by the sensor group, the microphone, and the camera, and the state of the userrecognized in step SK.

302 3412 3412 3450 3460 3470 60 300 3412 In step SK, the CPUdetermines an emotion value of the agent. Specifically, the CPUdetermines an emotion value indicating the emotion of the agent on the basis of the information analyzed by the sensor group, the microphone, and the camera, and the state of the userrecognized in step SK. The CPUends the emotion determination processing.

3400 60 55 55 50 60 3400 60 3400 3400 60 3400 60 3400 The order terminalof the thirteenth embodiment receives information regarding menu selection from the userby using vocal sound, inputs the received information to the sentence generation model, presents a menu name output from the sentence generation modelto the user, and orders a menu according to a reply of the user. Therefore, according to the order terminalof the thirteenth embodiment, it is possible to save time and effort for the userto operate the order terminalfor selecting a menu. According to the order terminalof the present embodiment, an order can be received even when the information regarding the menu selection received from the useris not a formal name of the menu. According to the order terminalof the present embodiment, the usercan quickly order a target menu compared with a case in which the order terminalis operated to order the target menu from a large number of menus.

3400 60 60 60 3400 60 3400 60 60 The order terminalaccording to the thirteenth embodiment acquires the state of the user, determines the emotion values of the userand the agent on the basis of the acquired state of the user, and determines the lines of the agent on the basis of the determined emotion values. The order terminalpresents the determined lines of the agent to the user. Therefore, according to the order terminalof the thirteenth embodiment, the agent can serve as a clerk, and the order placed by the usercan be checked by using lines that closely follow the emotion of the user.

60 60 In the thirteenth embodiment, the order placement processing is performed by receiving information regarding menu selection from the user. Modification 1 of the thirteenth embodiment is characterized in that the order placement processing is performed by acquiring information regarding a remaining amount of a menu ordered by the user. Hereinafter, contents unique to Modification 1 will be described.

3412 55 4312 60 53 55 3412 53 55 52 FIG. The input unitB (see) of Modification 1 has a function of inputting an emotion value to the sentence generation model. Specifically, the input unitB inputs the emotion value of the useroutput from the emotion engineto the sentence generation model. The input unitB may input the emotion value of the agent output from the emotion engineto the sentence generation model.

230 60 230 60 3450 3460 3470 60 2 FIG. The user state recognition unit(see) of the action control system of Modification 1 has a function of acquiring information regarding the remaining amount of the menu ordered by the user. Specifically, the user state recognition unitacquires information regarding the remaining amount of the menu ordered by the useron the basis of the information analyzed by the sensor group, the microphone, and the camera. The information regarding the remaining amount of the menu ordered by the useris an example of “consumption information”.

55 50 55 60 412 411 7 FIG. 51 FIG. The sentence generation model(see) of the agent systemof Modification 1 has a function of generating a menu name corresponding to an emotion value. Specifically, the sentence generation modelgenerates a menu name corresponding to the emotion value of the userinput from the input unitB with reference to the product information databaseA (see).

55 50 55 60 230 411 7 FIG. 51 FIG. The sentence generation model(see) of the agent systemof Modification 1 has a function of generating a menu name corresponding to the remaining amount of the menu. Specifically, the sentence generation modelgenerates a menu name corresponding to the remaining amount of the menu ordered by the useracquired by the user state recognition unitwith reference to the product information databaseA (see).

51 50 55 60 53 51 60 51 60 53 7 FIG. The interaction processing unit(see) of the agent systemof Modification 1 has a function of determining a menu proposal as lines. Specifically, on the basis of the menu name generated by the sentence generation modeland the emotion values of the userand the agent estimated by the emotion engine, the interaction processing unitdetermines lines of the agent for proposing a menu to the user. The interaction processing unitmay determine the lines of the agent on the basis of only the emotion value of the useror only the emotion value of the agent, estimated by the emotion engine.

55 FIG. 53 FIG. 3400 60 schematically illustrates an example of a flow of order placement processing according to Modification 1 of the thirteenth embodiment. The order placement processing of Modification 1 is processing executed by the order terminalwhen the userwho is eating gives an order. In the order placement processing of Modification 1, the same processing as that in the order placement processing inis denoted by the same reference sign, and detailed description thereof will be omitted.

200 4312 60 4312 60 3460 3412 400 55 FIG. In stepJ in, the CPUreceives information regarding menu selection from the user. As an example, the CPUreceives information such as “I want to give an additional order” from the uservia the microphone. The CPUproceeds to step SL.

400 3412 60 3412 60 3470 3412 60 3412 3450 3460 3412 203 In step SL, the CPUacquires information regarding the remaining amount of the menu ordered by the user. Specifically, the CPUacquires information regarding the remaining amount of the menu that the useris eating or drinking from the information analyzed by the camera. As an example, the CPUacquires that the remaining amount of a medium mug of draft beer that useris drinking is 20% or less. The CPUmay acquire the information regarding the remaining amount of the menu from information analyzed by the sensor groupand the microphone. The CPUproceeds to step SJ.

203 3412 60 3412 401 In step SJ, the CPUdetermines, for example, that the emotion value of the useris “hesitating” and the emotion value of the agent is “kind”. The CPUproceeds to step SL.

401 3412 60 55 3412 400 60 203 55 3412 55 In step SL, the CPUinputs the acquired information and the determined emotion value of the userto the sentence generation model. Specifically, the CPUinputs the information acquired in step SL and the emotion value of the userdetermined in step SJ to the sentence generation model. As an example, the CPUinputs information such as “additional order”, “medium mug of draft beer, remaining amount 20% or less”, and “hesitating” to the sentence generation model.

402 3412 60 55 3412 55 200 401 3412 3412 204 In step SL, the CPUacquires a menu name corresponding to the acquired information and the emotion value of the userfrom the sentence generation model. Specifically, the CPUacquires the menu name generated by the sentence generation modelon the basis of the information received in step SJ, and the information and the emotion value input in step SL. As an example, the CPUacquires “highball” as a menu name corresponding to “additional order”, “medium mug of draft beer, remaining amount 20% or less”, and “hesitating”. The CPUproceeds to step SJ.

204 3412 402 203 3412 In step SJ, the CPUdetermines lines of the agent on the basis of the menu name acquired in step Sand the emotion value determined in step SJ. As an example, the CPUdetermines the lines of the agent as “Are you hesitating about what drink to order next? How about a highball?”.

206 3412 60 60 206 3412 208 60 206 3412 3412 200 60 In step SJ, the CPUdetermines whether or not a reply of the userto the presented menu name is affirmative. In a case in which it is determined that the reply of the useris affirmative (step SJ: YES), the CPUproceeds to step SJ. On the other hand, in a case in which it is determined that the reply of the useris not affirmative (step SJ: NO), the CPUends the order placement processing. The CPUmay return to step SJ without ending the order placement processing, and may receive information regarding menu selection from the user.

3400 60 60 3400 60 The order terminalaccording to Modification 1 of the thirteenth embodiment presents a menu proposal based on the emotion value of the userto the useras lines of the agent. Therefore, according to the order terminalof Modification 1, it is possible to propose a menu that closely follows the emotion of the user.

3400 60 60 3400 60 60 3400 3400 60 60 The order terminalaccording to Modification 1 of the thirteenth embodiment acquires information regarding the remaining amount of the menu ordered by the user, and presents a menu proposal based on the acquired information to the useras lines of the agent. Therefore, according to the order terminalof Modification 1, it is possible to propose an additional order to the useraccording to the remaining amount of the menu ordered by the user. Information acquired by the order terminalis not limited to information regarding the remaining amount of a drink. As an example, the order terminalmay acquire information regarding the remaining amount of a meal of the userand make a proposal regarding the order of dessert at a timing at which the meal of the useris likely to end.

60 60 60 60 60 In the thirteenth embodiment, information regarding menu selection is received from the user. Modification 2 of the thirteenth embodiment is characterized by receiving information regarding menu selection from a plurality of users. The usersof Modification 2 include usersA andB. Hereinafter, contents unique to Modification 2 will be described.

412 60 412 60 3460 412 60 60 60 60 60 52 FIG. The reception unitA (see) of Modification 2 has a function of simultaneously receiving information regarding menu selection from a plurality of users. Specifically, the reception unitA simultaneously receives menu selection based on pieces of vocal sound of a plurality of usersvia the microphone. The reception unitA distinguishes and receives an order of the userA and an order of the userB by analyzing the pieces of received vocal sound of the plurality of the users. The phrase “simultaneously receiving” includes a case of receiving in a state in which all or some of the pieces of vocal sound of the plurality of usersoverlap and a case of sequentially receiving the pieces of vocal sound of the plurality of users. Simultaneously receiving is an example of “receiving at a time”.

55 50 55 60 3412 3411 55 7 FIG. 51 FIG. The sentence generation model(see) of the agent systemof Modification 3 has a function of aggregating a plurality of pieces of information regarding menu selection to generate a menu name. Specifically, the sentence generation modelgenerates a menu name corresponding to the information regarding the menu selection of the plurality of usersreceived by the reception unitA with reference to the product information databaseA (see). In a case in which a plurality of the same menu names exist in the generated menu names, the sentence generation modelleaves only one menu name, and determines the number of menu names that exist as a quantity.

56 FIG. 53 FIG. 3400 60 schematically illustrates an example of a flow of order placement processing according to Modification 2 of the thirteenth embodiment. The order placement processing of Modification 2 is processing executed when the order terminalreceives orders from a plurality of users. In the order placement processing of Modification 2, the same processing as that in the order placement processing inis denoted by the same reference sign, and detailed description thereof will be omitted.

500 3412 60 3412 60 60 3460 3412 60 60 56 FIG. In step SM in, the CPUsimultaneously receives information regarding menu selection from the plurality of users. Specifically, the CPUreceives information regarding menu selection from the userA and the userB by vocal sound via the microphone. As an example, the CPUreceives information such as “Oolong tea and a dish using ingredients produced in Hokkaido” from the userA and information such as “Oolong tea and a dish using seasonal ingredients” from the userB.

501 3412 60 3412 60 60 60 In step SM, the CPUanalyzes the received information for each user. Specifically, the CPUdistinguishes between the information from the userA and the information from the userB, and analyzes that there are orders from the two users.

502 3412 60 55 3412 202 In step SM, the CPUinputs information analyzed for each userto the sentence generation model. The CPUproceeds to step S.

202 3412 3412 60 60 In step SJ, as an example, the CPUacquires “Carpaccio of Hokkaido scallops” as the menu name corresponding to the information such as “a dish using ingredients produced in Hokkaido”, and acquires “rice cooked with bamboo shoots” as the menu name corresponding to the information regarding “a dish using seasonal ingredients”. The CPUacquires “oolong tea” and “two” as the menu name and the quantity corresponding to the information such as “oolong tea” from the userA and the userB.

203 3412 60 60 In step SJ, as an example, the CPUdetermines that the emotion value of the userA is “reassuring”, the emotion value of the userB is “anxious”, and the emotion value of the agent is “proud”.

204 3412 205 In step SJ, as an example, the CPUdetermines the lines of the agent as “We will suggest carefully selected menus, so please do not worry! How about two oolong tea, one carpaccio of scallops produced in Hokkaido, and one rice cooked with bamboo shoots?”. Hereinafter, step SJ and subsequent steps are the same as those in the thirteenth embodiment.

3400 60 55 55 3400 60 The order terminalaccording to Modification 2 of the thirteenth embodiment simultaneously receives information regarding menu selection from a plurality of users, inputs the received information to the sentence generation model, and presents the aggregated menu name and quantity output from the sentence generation model. Therefore, according to the order terminalof Modification 2, it is possible to finish the menu order in a short time compared with the case in which the plurality of usersorder menus one by one.

60 60 60 60 60 60 Modification 1 of the thirteenth embodiment performs the order placement processing by acquiring the information regarding the remaining amount of the menu ordered by the user. Modification 3 of the thirteenth embodiment is characterized in that order placement processing is performed by acquiring information regarding configurations of a plurality of users. The plurality of usersof Modification 3 include a userC, a userD, and a userE. Hereinafter, contents unique to Modification 3 will be described.

230 60 230 60 450 3460 3470 230 60 60 60 2 FIG. The user state recognition unit(see) of the action control system of Modification 3 has a function of acquiring information regarding the number, ages, and genders of the plurality of users. Specifically, the user state recognition unitacquires information regarding the configurations of the plurality of userson the basis of the information analyzed by the sensor group, the microphone, and the camera. Information acquired by the user state recognition unitis not limited to the information regarding the number, ages, and genders of the plurality of users, and may be any information regarding the configuration of the plurality of users. The information regarding the number, the ages, and the genders of the plurality of usersis an example of “configuration information”.

55 50 60 55 60 230 411 7 FIG. 51 FIG. The sentence generation model(see) of the agent systemof Modification 3 has a function of generating a menu name corresponding to configurations of a plurality of users. Specifically, the sentence generation modelgenerates a menu name corresponding to the information regarding the configurations of the plurality of usersacquired by the user state recognition unitwith reference to the product information databaseA (see).

57 FIG. 53 FIG. 3400 60 schematically illustrates an example of a flow of order placement processing according to Modification 3 of the thirteenth embodiment. The order placement processing of Modification 3 is processing executed when the order terminalproposes a menu to a plurality of users. In the order placement processing of Modification 3, the same processing as that in the order placement processing inis denoted by the same reference sign, and detailed description thereof will be omitted.

200 3412 60 3412 60 3460 3412 600 57 FIG. In step SJ in, the CPUreceives information regarding menu selection from the user. As an example, the CPUreceives information such as “What is today's recommendation?” from the userC via the microphone. The CPUproceeds to step S.

600 3412 60 3412 60 450 3460 3470 3412 In step S, the CPUacquires information regarding the number, ages, and genders of the plurality of users. Specifically, the CPUacquires information regarding the number, ages, and genders of the plurality of usersfrom the information analyzed by the sensor group, the microphone, and the camera. As an example, the CPUacquires information such as “1 female in 30's, 1 female around 8 years old, and 1 male around 6 years old”.

601 3412 55 3412 200 600 55 3412 55 In step S, the CPUinputs the acquired information to the sentence generation model. Specifically, the CPUinputs the information received in step Sand the information acquired in step Sto the sentence generation model. As an example, the CPUinputs information such as “today's recommendation” and “1 female in 30's, 1 female around 8 years old, and 1 male around 6 years old” to the sentence generation model.

602 3412 55 3412 55 601 3412 3412 203 In step S, the CPUacquires a menu name corresponding to the acquired information from the sentence generation model. Specifically, the CPUacquires a menu name generated by the sentence generation modelon the basis of the information input in step S. As an example, the CPUacquires “pasta lunch” and “one” and “children's lunch” and “two” as menu names and quantities corresponding to “today's recommendation” and “1 female in 30's, 1 female around 8 years old, and 1 male around 6 years old”. The CPUproceeds to step S.

203 3412 60 60 60 In step SJ, as an example, the CPUdetermines that the emotion value of the userC is “safe”, the emotion value of the userD is “pleasant”, the emotion value of the userE is “happy”, and the emotion value of the agent is “kind”.

204 3412 602 203 3412 In step SJ, specifically, the CPUdetermines lines of the agent on the basis of the menu names acquired in step Sand the emotion values determined in step SJ. As an example, the CPUdetermines the lines of the agent as “Welcome! Please enjoy your meal! Today's recommendations are one pasta lunch and two children's lunches. How about this?”.

206 3412 60 60 206 3412 208 60 206 3412 3412 200 60 In step SJ, the CPUdetermines whether or not the reply of the userto the presented menu name is affirmative. In a case in which it is determined that the reply of the useris affirmative (step SJ: YES), the CPUproceeds to step S. On the other hand, in a case in which it is determined that the reply of the useris not affirmative (step SJ: NO), the CPUends the order placement processing. The CPUmay return to step SJ without ending the order placement processing, and may receive information regarding menu selection from the user.

3400 60 60 3400 60 The order terminalaccording to Modification 3 of the thirteenth embodiment acquires information regarding the number, ages, and genders of the plurality of users, and presents a menu proposal based on the acquired information to the plurality of usersas lines of the agent. Therefore, according to the order terminalof Modification 3, it is possible to propose a menu in a short time compared with a case in which a menu is proposed to a plurality of usersone by one.

3400 3400 3400 In the thirteenth embodiment, the case in which the order terminalis installed and used at a guest seat of a restaurant has been described. However, the disclosure is not limited thereto, and the order terminalmay be used in a case in which an article is purchased online shopping. The order terminalis not limited to a tablet type smart device, and includes a case in which a similar function is installed in a computer.

3400 60 3400 3400 3400 60 In the thirteenth embodiment, the order terminalreceives information regarding menu selection by using vocal sound of the user. However, the disclosure is not limited thereto, and the order terminalmay receive information regarding menu selection by using text, an image, a moving image, or the like. As an example, the order terminalcan receive an image of an ingredient. Therefore, according to the order terminalof the present embodiment, even in a case in which the userdoes not know the name of the ingredient, the order of the menu can be received.

3400 60 3400 In Modification 1 of the thirteenth embodiment, the order terminalproposes a menu based on the emotion value of the user. However, the disclosure is not limited thereto, and the order terminalmay propose a menu based on the emotion value of the agent.

3400 60 Therefore, according to the order terminalof Modification 1, the range of menu selection by the usercan be expanded.

3400 60 3400 60 3400 60 In Modification 1 of the thirteenth embodiment, the order terminalproposes a menu based on the remaining amount of the menu ordered by the user. The order terminalmay propose a menu based on a menu order history of the user. Therefore, according to the order terminalof Modification 1, it is possible to propose a menu according to a preference of the user.

3400 60 3400 60 3400 60 3400 60 In Modification 3 of the thirteenth embodiment, the order terminalproposes a menu based on information regarding the number, ages, and genders of the plurality of users. The order terminalmay propose a menu based on information regarding menu selection of the user. As an example, the order terminalproposes addition or reduction of a menu in a case in which the number of menus selected with respect to the number of the plurality of usersis not appropriate. Therefore, according to the order terminalof Modification 3, the usercan select an appropriate number of menus.

8 9 FIGS.and 50 50 50 1 6 50 10 10 50 55 50 10 50 50 10 800 10 In addition to the description with reference toand the like, as an example of implementation of the agent system, the agent systemcan be mounted on an imaging device such as a camera. The agent systemmounted on the imaging device can execute processing through the above Stepsto. Specifically, the agent systemmounted on the imaging device acquires, for example, vocal sound output from the userat the time of imaging. The vocal sound output by the userincludes, for example, instruction information indicating an instruction of setting at the time of imaging. The agent systemmounted on the imaging device inputs the acquired vocal sound including the instruction information to the sentence generation model, determines speech content of the agent system, and outputs the determined speech content. For example, it is assumed that the usersays “Set so that an emotional picture can be taken” to the imaging device equipped with the agent system. In this case, the agent systemdetermines and outputs speech content for the user, for example, “Is it OK to set the ISO sensitivity toand the F value to 2.8? It is possible to take a picture with noise and a blurred background.” according to the acquired instruction information based on the vocal sound (speech) content of the user.

50 5 10 50 800 50 10 55 50 55 10 50 10 50 10 53 50 10 The agent systemmounted on the imaging device determines whether or not to execute a command for controlling the imaging device on the basis of Stepdescribed above. For example, it is assumed that the usersays “No noise is needed, so please darken” in response to the speech of the agent system, that is, “Is it OK to set the ISO sensitivity toand the F value to 2.8? It is possible to take a picture with noise and a blurred background.” in the above example. In this case, the agent systemmounted on the imaging device inputs speech (vocal sound) of the userincluding the instruction information to the sentence generation model. The agent systemmounted on the imaging device checks the instruction information output from the sentence generation model, and receives the checked instruction of setting for the imaging device. Specifically, on the basis of the instruction information using the vocal sound of the user, “No noise is needed, so please darken”, the agent systemmounted on the imaging device executes a command for setting to darken without noise. In a case in which the reaction of the userto the previous proposal is a nodding swing or an affirmative reaction, a command is executed to the imaging device so that the setting as proposed by the agent systemis performed. On the other hand, in a case in which the userhas a negative reaction to the previous proposal, for example, the proposal has been determined by using the emotion engine, the agent systemmounted on the imaging device determines the speech content so as to make another proposal to the user, and does not execute the command to the imaging device.

50 10 10 10 50 53 50 56 10 72 50 10 10 10 10 50 10 10 As described above, since the agent systemis mounted on the imaging device, even in a case in which the userdoes not know the meaning of the numerical value necessary for setting the imaging device, the imaging setting according to the request of the useris performed on the basis of the vocal sound content or the like of the user. For example, in the case of imaging a subject of a person, the agent systemmounted on the imaging device can propose a recommended setting on the basis of an emotion of the subject estimated by the emotion engine. The agent systemmounted on the imaging device can automatically perform imaging on the basis of the emotion of the subject by associating the emotion of the subject with the imaging command. For example, the command acquisition unitcan acquire a state of the user, a surrounding situation of the user, and the like, as described below with reference to the smart glasses. The agent systemmounted on the imaging device can propose an imaging timing to a subject such as the useror a person according to a state of the userand a surrounding situation including the user. For example, when the userspeaks the imaging timing to the imaging device in advance, in other words, inputs the imaging timing as vocal sound, the agent systemmounted on the imaging device can set the imaging timing on the basis of the instruction information included in the vocal sound of the user. As a result, the usercan prevent forgetting to capture an image or can capture an image at an appropriate timing.

50 In a fourteenth embodiment, processing of outputting speech information to another electronic apparatus that enables communication with a computer network and acquiring response information corresponding to the speech information from the other electronic apparatus in a case in which an electronic apparatus including an agent systemcannot communicate with the computer network will be described. Portions having the same configurations as those of the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

50 60 60 60 50 50 50 50 122 122 122 122 122 500 52 122 122 500 52 122 The agent systemis a computer system that performs a series of actions according to the intention of the userthrough an interaction performed with the user. The interaction with the usercan be performed by using vocal sound or text. The agent systemis mounted on an electronic apparatus. The agent systemcan be mounted on, for example, a robot, a doll, a stuffed toy, a pendant, smart glasses, a smart watch, a smartphone, a smart speaker, an earphone, a personal computer, or the like. The agent systemmay be implemented in a web server and used via a web browser operating on a communication terminal such as a smartphone owned by the user. Hereinafter, an example in which the agent systemis mounted on smartphonesA andB will be described. The smartphoneA is an example of an electronic apparatus, and the smartphoneB is an example of another electronic apparatus. The smartphoneA is owned by Mr. A and functions as an agentA which is an agentfor the smartphoneA. The smartphoneB is owned by Mr. B and functions as an agentB that is an agentfor the smartphoneB.

50 50 1212 1230 1224 51 52 53 54 55 56 57 51 511 512 511 512 58 FIG. 58 FIG. 58 FIG. Next, a functional configuration of an agent systemaccording to the fourteenth embodiment will be described with reference to. As illustrated in, in the agent systemaccording to the fourteenth embodiment, a CPUexecutes a program stored in a ROMor a storage deviceto function as an interaction processing unit, a character setting unit, an emotion engine, an interaction history storage unit, a sentence generation model, a command acquisition unit, and a robotic process automation (RPA). As illustrated in, the interaction processing unitincludes an acquisition unitand an output unit. Hereinafter, functions of the acquisition unitand the output unitunique to the fourteenth embodiment will be described.

511 60 511 60 The acquisition unitacquires information regarding the state of the user. Specifically, the acquisition unitacquires vocal sound output by the user, text to be input, and an expression imaged by the camera.

512 51 60 512 122 60 122 512 122 122 122 512 3 500 122 500 122 59 FIG. The output unitoutputs the speech content generated by the interaction processing unitto the user. In a case in which communication with the computer network is disabled, the output unitoutputs speech information to another electronic apparatus that enables communication with the computer network. As a specific example, it is assumed that the smartphoneA of the useris temporarily unable to communicate with the computer network. In this case, when Mr. A asks the smartphoneA how to get to the station, the output unitof the smartphoneA outputs speech information to the smartphoneB that “Oh, signal lost. How can I get to the station from here?” as illustrated in. On the other hand, since communication with the computer network is not cut off in the smartphoneB, the output unitexecutes Internet search or the like and outputs speech information such as “For Shinjuku station, it is good to enter from Aexit ahead, right, Mr. B.”. As a result, the agentB of the smartphoneB answers the route to the station to Mr. A instead of the agentA of the smartphoneA that has become unable to communicate.

59 FIG. 60 FIG. 511 122 122 511 122 3 122 Following, as illustrated in, on the other hand, the acquisition unitof the smartphoneA acquires response information corresponding to the speech information from the smartphoneB. Specifically, the acquisition unitof the smartphoneA acquires response information “For Shinjuku station, it is good to enter from Aexit ahead, right, Mr. B.” from the smartphoneB.

122 512 122 60 122 60 512 122 60 52 122 54 60 FIG. In addition to the response information acquired from the smartphoneB, the output unitof the smartphoneA outputs response information updated on the basis of the history of the interaction with the userstored in the smartphoneA to the user. Specifically, as illustrated in, the output unitof the smartphoneA outputs updated response information such as “If Mr. A goes home, it is good to get on the Chuo line from Shinjuku station.” to Mr. A who is the user. The agentA of the smartphoneA acquires the information of Mr. A from the interaction history storage unit.

50 50 500 122 500 122 50 1212 511 512 61 FIG. 62 FIG. Next, an operation of the agent systemaccording to the fourteenth embodiment will be described. In the agent system, the interaction processing of the agentA of the smartphoneA illustrated inand the interaction processing of the agentB of the smartphoneB illustrated inare executed. Each processing in the agent systemis executed by the CPUfunctioning as the acquisition unitand the output unit.

500 122 400 1212 122 122 400 1212 122 400 1212 402 61 FIG. The interaction processing of the agentA of the smartphoneA will be described with reference to. In step SM, the CPUof the smartphoneA determines whether or not communication with the computer network is possible. In a case in which it is determined that the smartphoneA can perform communication (step SM: YES), the CPUends the processing. On the other hand, in a case in which it is determined that the smartphoneA cannot perform communication (step SM: NO), the CPUproceeds to step S.

402 1212 122 In step SM, the CPUof the smartphoneA acquires the speech of the user.

404 1212 122 60 In step SM, the CPUof the smartphoneA analyzes the intention of the speech of the user.

406 1212 122 In step SM, the CPUof the smartphoneA generates speech information.

408 1212 122 122 In step SM, the CPUof the smartphoneA outputs the speech information to the smartphoneB, which is another electronic apparatus.

410 1212 122 122 In step SM, the CPUof the smartphoneA acquires the response information from the smartphoneB, which is another electronic apparatus.

412 1212 122 412 1212 416 412 1212 414 In step SM, the CPUof the smartphoneA determines whether or not the acquired response information is sufficient. In a case in which it is determined that the response information is sufficient (step SM: YES), the CPUproceeds to step S. On the other hand, in a case in which it is determined that the response information is not sufficient (step SM: NO), the CPUproceeds to step S.

414 1212 122 60 54 In step SM, the CPUof the smartphoneA updates the acquired response information on the basis of the history of the interaction with the userstored in the interaction history storage unit.

416 1212 122 122 60 In step SM, the CPUof the smartphoneA generates the speech information by using the response information acquired from the smartphoneB and the response information updated on the basis of the history of the interaction with the user.

418 1212 122 60 In step SM, the CPUof the smartphoneA outputs speech information to the user.

420 1212 122 60 60 420 1212 402 60 420 1212 In step SM, the CPUof the smartphoneA determines whether or not there is speech of the user. In a case in which it is determined that there is speech of the user(step SM: YES), the CPUreturns to step SM. On the other hand, in a case in which it is determined that there is no speech of the user(step SM: NO), the CPUends the processing.

500 122 62 FIG. The interaction processing of the agentB of the smartphoneB will be described with reference to.

600 1212 122 122 60 In step S, the CPUof the smartphoneB acquires speech from the smartphoneA of the user.

602 1212 122 122 In step S, the CPUof the smartphoneB analyzes the intention of the speech information acquired from the smartphoneA.

604 1212 122 In step S, the CPUof the smartphoneB generates response information.

606 1212 122 122 60 In step S, the CPUof the smartphoneB outputs the response information to the smartphoneA of the user.

608 1212 122 122 60 608 1212 600 608 1212 In step S, the CPUof the smartphoneB determines whether or not there is speech from the smartphoneA of the user. In a case in which it is determined that there is speech (step S: YES), the CPUreturns to step S. On the other hand, in a case in which it is determined that there is no speech (step S: NO), the CPUends the processing.

122 122 60 500 60 60 122 60 As described above, according to the present embodiment, the agent can be used by using the smartphoneB instead of the smartphoneA that has become unable to communicate. That is, even if one electronic apparatus is unable to communicate, the usercan obtain information. Since the agentA outputs, to the user, response information updated on the basis of the past interaction history with the userwith respect to the response information from the smartphoneB, it is possible to provide information intended by the user.

500 500 500 500 500 500 500 Mr. A does not directly speak to the agentB, but Mr. A acquires an answer through mediation of interaction from the agentA to the agentB. In this case, as in the speech of “That's right, Mr. B” of the agentB, the agentB can request Mr. B for confirmation or agreement. That is, according to the present embodiment, it is possible to realize an interaction among four parties of Mr. A, the agentA, Mr. B, and the agentB, and there is no stress on the situation in which communication is disabled for Mr. A.

63 FIG. 63 FIG. 60 50 60 60 60 50 60 60 50 60 50 60 60 50 80 76 50 50 232 50 50 60 Next, a fifteenth embodiment will be described with reference to.illustrates an aspect in which, in a case in which there are a plurality of users, the agent systemis provided for each of the plurality of users. In this aspect, one userA of the plurality of userscan notify his/her agent systemA that he/she wants to know an emotion of another userB in a remote place, and can acquire information regarding a state of another userB from the agent systemB of the another userB. In this case, the agent systemB of the other userB acquires information regarding an emotion and a situation of the other userB and transmits the information to the agent systemA. Various output units (the speaker, the display, and the like) of the agent systemA output the emotion of the other userB determined by the emotion determination unitof the agent systemB as the speech content of the agent systemA on the basis of the state of the other userB.

60 50 50 60 60 50 60 For example, in a case in which the userA asks a question “The first sunrise of the year viewed from Mt. Fuji must be beautiful” to the agent systemA of the user, the agent systemA executes data exchange with another agent system in order to search for another usernear Mt. Fuji. As a result of the data exchange, in a case in which another userB who is near Mt. Fuji is searched for, the data exchange is executed with the corresponding another agent systemB, and information regarding the emotion and the situation of the userB who is in Mt. Fuji is searched for.

50 60 60 50 60 As a result of the search, the agent systemA of the userA acquires information regarding the emotion of the userB via the agent systemB of the userB in Mt. Fuji.

50 60 60 For example, the agent systemA can obtain information such as “Of course, the first sunrise of the year viewed from Mount Fuji is unusually great!!” as the emotion of the userB. As a result, the userA can feel as if the user himself/herself is experiencing a remote place.

50 60 60 60 50 50 60 60 The agent systemA can acquire an image (for example, images around the userB, such as Mt. Fuji viewed by the userB) around the userB from another agent systemB. The agent systemA can present the emotion information of another user as described above while showing the image data to the userA. As a result, the userA can obtain a realistic experience as if looking at a remote place with his/her own eyes.

60 60 50 232 60 80 76 60 60 In a case in which emotion information of a plurality of other users is acquired, information regarding an emotion having the highest occurrence frequency can be acquired and presented. For example, in a case in which a plurality of other usersB . . .N who are near Mt. Fuji are searched for, if a large number of users among the other users have an emotion such as “beautiful like heaven”, information based on the emotion information can be presented. In a case in which a plurality of types of emotion information of a plurality of other users are obtained, the agent systemA can determine an emotion corresponding to an emotion of the user determined by the emotion determination unitas the emotion information most corresponding to the emotion of the userA, and output the emotion from the output unit such as the speakeror the displayas the speech content of the agent. For example, in a case in which the userA has an emotion such as “The scenery of Mt. Fuji is beautiful” and expects a corresponding emotion, such emotion information is adopted and presented. Conversely, if the userA has a positive emotion toward Mt. Fuji, emotion information (for example, “there is a lot of garbage”, or “there are too many people”) of the user having a negative emotion toward Mt. Fuji is not adopted.

50 72 11 FIG. The agent systemdescribed above can be applied to the smart glassesdescribed inand the like.

The disclosure is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the disclosure, and are not necessarily limited to those having all the described configurations. A part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. It is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Japanese Patent Application No. 2023-073806 (filing date: Apr. 27, 2023) Japanese Patent Application No. 2023-136435 (filing date: Aug. 24, 2023) Japanese Patent Application No. 2023-152212 (filing date: Sep. 20, 2023) Japanese Patent Application No. 2023-165251 (filing date: Sep. 27, 2023) Japanese Patent Application No. 2023-174709 (filing date: Oct. 6, 2023) Japanese Patent Application No. 2023-176974 (filing date: Oct. 12, 2023) Japanese Patent Application No. 2023-177748 (filing date: Oct. 13, 2023) Japanese Patent Application No. 2023-184939 (filing date: Oct. 27, 2023) Japanese Patent Application No. 2023-185845 (filing date: Oct. 30, 2023) Japanese Patent Application No. 2023-185846 (filing date: Oct. 30, 2023) Japanese Patent Application No. 2023-193182 (filing date: Nov. 13, 2023) Japanese Patent Application No. 2023-195488 (filing date: Nov. 16, 2023) Japanese Patent Application No. 2023-214129 (filing date: Dec. 19, 2023) Japanese Patent Application No. 2023-223427 (filing date: Dec. 28, 2023) The disclosures of the patent applications listed below are incorporated herein by reference.