Electronic Apparatus and Controlling Method Thereof

This application is a continuation application of U.S. patent application Ser. No. 18/244,714, filed on Sep. 11, 2023, which is a by-pass continuation application of International Application No. PCT/KR2023/009684, filed on Jul. 7, 2023, which is based on and claims priority to Korean Patent Application No. 10-2022-0150957, filed on Nov. 11, 2022, in the Korean Intellectual Property Office, the disclosures of which are incorporated by reference herein their entireties.

This disclosure relates to an electronic apparatus and a controlling method thereof and, more particularly, an electronic apparatus capable of using a plurality of voice assistance functions or a plurality of voice assistants by switching or integrating them and a controlling method thereof.

Recent electronic apparatuses provide assistance functions. Here, an assistance function refers to a function that helps a user to perform a desired function using an artificial intelligence technology even if the user does not provide an exact instruction.

Such assistance function is applied to various fields such as content playback, content search, schedule management, and information search, etc.

According to an aspect of the disclosure, an electronic apparatus includes: a display; a memory configured to store a plurality of assistance instructions respectively corresponding to a plurality of voice assistants; and at least one processor operatively connected to the display and the memory, wherein the at least one processor is configured to, based on a user voice command, generate response information by using the at least one voice assistant and control the display to display the generated response information, based on a predetermined voice command received during a first voice assistant operation, activate a second voice assistant operation corresponding to the predetermined voice command.

According to another aspect of the disclosure, a controlling method of an electronic apparatus, includes: receiving a user voice command; based on the user voice command, generating response information by using at least one voice assistant of a plurality of voice assistants; displaying the generated response information; and based on a predetermined voice command being input during a first voice assistant operation, activating a second voice assistant operation corresponding to the predetermined voice command.

According to another aspect of the disclosure, a method performed by an integration engine communicating with a display device having a plurality of voice assistants, includes: transferring, to the display device, a first command to activate a first voice assistant of the plurality of voice assistants; receiving, from the display device, a first request to determine a new voice assistant to be used by the display device; determining the new voice assistant to be used by the display device; and transferring, to the display device, a second command to inform the determined new voice assistant.

Since the disclosure may be variously modified and have several exemplary embodiments, specific exemplary embodiments of the disclosure will be illustrated in the drawings and be described in detail in the detailed description. However, it is to be understood that the disclosure are not limited to specific exemplary embodiments, but include all modifications, equivalents, and substitutions according to exemplary embodiments of the disclosure. Throughout the accompanying drawings, similar components will be denoted by similar reference numerals.

In describing the disclosure, when it is decided that a detailed description for the known functions or configurations related to the disclosure may unnecessarily obscure the gist of the disclosure, the detailed description therefor will be omitted.

In addition, the following exemplary embodiments may be modified in several different forms, and the scope and spirit of the disclosure are not limited to the following exemplary embodiments. Rather, these exemplary embodiments make the disclosure thorough and complete, and are provided to completely transfer the spirit of the disclosure to those skilled in the art.

Terms used in the disclosure are used only to describe specific exemplary embodiments rather than limiting the scope of the disclosure. Singular forms are intended to include plural forms unless the context clearly indicates otherwise.

In the disclosure, an expression “have,” “may have,” “include,” “may include,” or the like, indicates existence of a corresponding feature (for example, a numerical value, a function, an operation, a component such as a part, or the like), and does not exclude existence of an additional feature.

3 In the disclosure, an expression “A or B,” at least one of “A or/and B,” “one or more of A or/B,” or the like, may include all possible combinations of items enumerated together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may indicate all of 1) a case in which at least one A is included, 2) a case in which at least one B is included, or) a case in which both of at least one A and at least one B are included.

st nd Expressions “first”, “second”, “1,” “2” or the like, used in the disclosure may indicate various components regardless of a sequence and/or importance of the components, will be used only in order to distinguish one component from the other components, and do not limit the corresponding components.

When it is mentioned that any component (for example: a first component) is (operatively or communicatively) coupled with/to or is connected to another component (for example: a second component), it is to be understood that any component is directly coupled to another component or may be coupled to another component through the other component (for example: a third component).

On the other hand, when it is mentioned that any component (for example, a first component) is “directly coupled” or “directly connected” to another component (for example, a second component), it is to be understood that the other component (for example, a third component) is not present between any component and another component.

An expression “~configured (or set) to” used in the disclosure may be replaced by an expression “suitable for,” “having the capacity to,” “~designed to,” “~adapted to,” “~made to,” or “~capable of” depending on a situation. A term “~configured (or set) to” may not necessarily mean “specifically designed to” in hardware.

Instead, an expression “~an apparatus configured to” may mean that the apparatus “is capable of” together with other apparatuses or components. For example, a “processor configured (or set) to perform A, B, and C” may mean a dedicated processor (for example, an embedded processor) for performing the corresponding operations or a generic-purpose processor (for example, a central processing unit (CPU) or an application processor) that may perform the corresponding operations by executing one or more software programs stored in a memory apparatus.

In exemplary embodiments, a “module” or a “unit” may perform at least one function or operation, and be implemented by hardware or software or be implemented by a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “units” may be integrated in at least one module and be implemented by at least one processor except for a ‘module’or a ‘unit’that needs to be implemented by specific hardware.

Operations performed by the modules, the programs, or the other components according to the diverse embodiments may be executed in a sequential manner, a parallel manner, an iterative manner, or a heuristic manner, or at least some of the operations may be performed in a different order or be omitted, or other operations may be added.

Various components and areas in the drawings are schematically drawn. Therefore, the technical spirit of the present disclosure is not limited by the relative size or spacing drawn in the accompanying drawings.

The electronic apparatus according to one or more embodiments may include at least one of smartphones, tablet personal computers (PCs), desktop PCs, laptop PCs, or wearable devices. Here, the wearable device may include at least one of an accessory type of a device (e.g., a timepiece, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, or a head-mounted-device (HMD)), one-piece fabric or clothes type of a circuit (e.g., electronic clothes), a body-attached type of a circuit (e.g., a skin pad or a tattoo), or a bio-implantable type of a circuit.

According to some embodiments, the electronic apparatus may include at least one of refrigerators, air-conditioners, cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, home automation control panels, security control panels, media boxes (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), game consoles (e.g., Xbox™ or Play Station™), electronic dictionaries, electronic keys, camcorders, electronic picture frames, or the like. In addition to the above-described embodiments, an electronic apparatus according to the present disclosure may be any apparatus as long as it includes a display.

Hereinafter, an embodiment according to the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement it.

1 FIG. illustrates an electronic apparatus according to one or more embodiments.

1 FIG. 100 Referring to, an electronic apparatusperforms a voice assistant function. Here, the voice assistant function is a function that uses an AI technology to help a user perform a desired function even if the user does not input a voice command accurately. Such an assistance function is being applied to various fields such as content reproduction, schedule management, information search, and the like.

For example, in the case of wanting to listen to music, a user had to designate the music he or she wanted to hear accurately. However, if the assistance function is used, music corresponding to the user's taste may be automatically selected and played.

1 2 3 Recent electronic apparatuses may be equipped with a plurality of voice assistants instead of one voice assistant. For example, various voices assistances such as a voice assistant (VA) provided by a manufacturer, a voice assistant (VA) provided by an OS platform, a voice assistant (VA) provided by a specific application, etc. may be installed.

As such, when a plurality of voice assistants are installed, the user has difficulty in selecting the plurality of voice assistants. For example, the user had to directly press a button corresponding to the voice assistant to be used or select an utterance trigger indicating the corresponding voice assistant.

In order to change a voice assistant in a situation where the executed voice assistant is not the voice assistant intended by the user, the currently executed voice assistant must be terminated and the intended voice assistant must be selected and activated, which causes inconvenience.

In addition, in the prior art, only response information (or response contents) of one voice assistant could be obtained in response to a user voice command. However, since each voice assistant has advantages and disadvantages, a method for integrating and using a plurality of voice assistants has been required.

100 Accordingly, the electronic apparatusaccording to the present disclosure capable of using a plurality of assistance functions by easily switching or integrating them.

210 For example, when a user activates a voice assistant function by pressing a voice assistant button, a voice assistant to be used is determined based on the user's uttered command, the current user, the currently executed application, and the like, a voice command can be processed using the determined voice assistant.

100 In addition, the electronic apparatusaccording to the present disclosure may switch to another voice assistant only with the user's uttered command while using a specific voice assistant.

100 Further, the electronic apparatusaccording to the present disclosure may provide response information corresponding to a voice command using response information of a plurality of voice assistants rather than a result of once voice assistant.

4 10 FIGS.to 2 3 FIGS.and 100 Detailed functions of the integrated voice assistant will be described in greater detail with reference to. Further, specific configuration and operation of the electronic apparatuswill be described with reference to.

100 As described above, the electronic apparatusaccording to the present disclosure integrates and manages (or processes) a plurality of voice assistants, and the user can easily select/switch voice assistants and can be provided with various and more accurate information through a single voice command.

1 FIG. 200 100 200 illustrates that a user voice is input through a remote controller, but in the implementation, it is possible not only to receive a user voice directly from the electronic apparatusbut also to receive a voice command through a user terminal device (e.g., smartphone, tablet, etc.) other than the remote controller.

1 FIG. In addition,describes the feature of processing a voice command only for the voice assistant selected or determined by a user, but in the implementation, all installed voice assistants may process a user voice command, and only the activated (or selected) voice assistant may display (or perform) the result.

In other words, being activated in the present disclosure means not only being in a state where a user's uttered command is received and can be processed but also being in a state where a response result processed according to a voice command can be displayed (or executed).

2 FIG. illustrates a configuration of an electronic apparatus according to one or more embodiments.

2 FIG. 100 110 120 130 Referring to, the electronic apparatusmay include a memory, a displayand a processor.

110 110 100 100 The memorymay store data necessary for one or more embodiments of the disclosure. The memorymay be implemented in the form of a memory embedded in the electronic apparatus, or may be implemented in the form of a memory that can communicate with (detachable from) the electronic apparatus, based on a data storing purpose.

100 100 100 100 100 100 For example, data for driving the electronic apparatusmay be stored in the memory embedded in the electronic apparatus, and data for an extension function of the electronic apparatusmay be stored in the memory that can communicate with the electronic apparatus. The memory embedded in the electronic apparatusmay be implemented as at least one of a volatile memory (for example, a dynamic random access memory (DRAM), a static RAM (SRAM), or a synchronous dynamic RAM (SDRAM)), a non-volatile memory (for example, an one time programmable read only memory (OTPROM), a programmable ROM (PROM), an erasable and programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a mask ROM, or a flash ROM), a flash memory (for example, a NAND flash or a NOR flash), a hard drive, or a solid state drive (SSD)). In addition, the memory that can communicate with the electronic apparatusmay be implemented in the form of a memory card (for example, a compact flash (CF), a secure digital (SD), a micro secure digital (Micro-SD), a mini secure digital (Mini-SD), an extreme digital (xD), or a multi-media card (MMC)), or an external memory which may be connected to a universal serial bus (USB) port (for example, a USB memory).

110 100 100 According to one or more embodiments, the memorymay store at least one instruction or a computer program including instructions for controlling the electronic apparatus. Such a computer program may be a plurality of assistance programs (or assistance instructions) corresponding to each of a plurality of voice assistants, an integrated engine program for integrated management of a plurality of voice assistants, an application program installed in the electronic apparatus, and the like.

110 According to other embodiments, the memorymay store information regarding an artificial intelligence model including a plurality of layers. Here, storing an artificial intelligence model may mean storing various information related to an operation of an artificial intelligence model, for example, information regarding a plurality of layers included in an artificial intelligence model, information regarding a parameter (e.g., filter coefficients, bias, etc.) used in each of a plurality of layers, and the like. For example, such an artificial intelligence model may be a model for voice recognition or a model for an assistance function.

110 110 The memorymay store information regarding a voice assistant corresponding to a user, an application, and the like or history information related to use of the voice assistant. For example, the memorymay store history information indicating that a specific user uses a first voice assistant or history information indicating that a second voice assistant is used while a specific application is executed, and the like.

120 130 120 The displaymay receive a signal from the processor, and display information regarding the received signal. For example, the displaymay be implemented as a display including a self-light emitting element or a display including a non self-light emitting element and a backlight.

120 120 For example, the displaymay be implemented in various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a light emitting diode (LED) display, a micro light emitting diode (micro LED) display, a mini LED display, a plasma display panel (PDP), a quantum dot (QD) display, a quantum dot light-emitting diode (QLED) display, a projector, and the like. The displaymay also include a driving circuit, a backlight unit, or the like, which may be implemented in a form such as an a-si thin film transistor (TFT), a low temperature poly silicon (LTPS) TFT, or an organic TFT (OTFT).

120 The displaymay be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a three-dimensional (3D) display, a display in which a plurality of display modules are physically connected with each other, or the like.

120 120 The displaymay display a screen corresponding to an executed application. In addition, when a plurality of applications is running, the displaymay display a plurality of screens corresponding to each of the plurality of applications together.

120 120 6 10 FIGS.to The displaymay display a UI screen according to a voice assistant operation, or display the above-described UI screen together with an executed application. The above-described UI screen may display an operation state of a voice assistant or information regarding a voice assistant that is currently operating (or activated), or may display a response result that is a voice recognition result. Examples of various UI screens that can be displayed on the displaywill be described later with reference to.

130 100 130 The processorcontrols a configuration of the electronic apparatus. Such a processormay also be referred to as a controller, a control device, etc.

130 130 100 The processormay consist of one or a plurality of processors. Specifically, the processormay perform an operation of the electronic apparatusaccording to one or more embodiments by executing at least one instruction stored in a memory.

130 130 130 According to one or more embodiments, the processormay be implemented as a digital signal processor (DSP) processing a digital video signal, a microprocessor, a graphics processing unit (GPU), an artificial intelligence (AI) processor, or a neural processing unit (NPU), or a timing controller (T-CON). However, the disclosure is not limited thereto, and the processormay include at least one of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an advanced RISC machine (ARM) processor, or may be defined by this term. In addition, the processormay be implemented in a system-on-chip (SoC) or a large scale integration (LSI) in which a processing algorithm is embedded, or may be implemented in the form of an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

130 In addition, the processorfor executing an integrated voice assistant function according to one or more embodiments may be implemented as a general-purpose processor such as CPU, AP, Digital Signal Processor (DSP), etc., a graphic-only processor such as GPU and Vision Processing Unit (VPU), or a combination of an AI-only processor such as NPU and software.

130 110 130 130 The processormay control input data according to predefined operation rules or artificial intelligence models stored in the memory. Alternatively, if the processoris a dedicated processor (or an AI-only processor), it may be designed as a hardware structure specialized for processing a specific artificial intelligence model. For example, hardware specialized for processing a specific artificial intelligence model may be designed as a hardware chip such as ASIC, FPGA, etc. When the processoris implemented as a dedicated processor, it may be implemented to include a memory for implementing an embodiment of the present disclosure or to include a memory processing function for using an external memory.

130 130 120 100 When a user command is input, the processormay perform a function corresponding to the user command. For example, when a content playback command is input, the processormay control the displayto display an image corresponding to the content selected by the user. Such a user command may be input through a button provided in the electronic apparatusor through an external device.

130 100 When there are a plurality of currently running applications, the processormay be controlled so that a plurality of applications (corresponding to each of the plurality of applications) is displayed together. In other words, the electronic apparatusmay support multi-view.

130 200 300 100 When a voice assistant call event occurs, the processormay determine a voice assistant to process a user's uttered voice. Here, the voice assistant call event may be input through an external device (e.g., a remote controller, a user terminal device, etc.), and it may be input through a button provided in the electronic apparatusor may be a case where a user utters a keyword (e.g., “Hi, Bixby”) calling a specific voice assistant.

130 Specifically, when a voice assistant call event occurs, the processormay determine at least one voice assistant based on at least one of a user's uttered keyword, a user account, a currently running application, or voice assistant history information.

For example, when a user generates a voice assistant call event by specifying a specific voice assistant, the user may determine the specific voice assistant as a voice assistant to be used. In other words, when a user inputs a voice command including a specific keyword (e.g., “Bixby, please recommend a movie”, wherein “Bixby” is the call name (or keyword) of the voice assistant), a voice assistant corresponding to the keyword may be determined as a voice assistant to be used. Alternatively, even when a voice assistant is called by pressing a button for calling a specific assistance on a remote controller (or a user terminal device), a voice assistant corresponding to the button may be determined as a voice assistant to be used as described above.

130 100 If the user does not specify a voice assistant, the processormay specify the user currently using the electronic apparatus, identify the current user and the currently running application, and determine a voice assistant corresponding to the current user and the current application using pre-stored history information.

130 If it is difficult to specify the user, for example, if a voice command is input through a remote controller, the processormay check history information and determine a voice assistant corresponding to the currently running application as a voice assistant to be used. In this case, if a plurality of applications are currently running and the voice assistant corresponding to each of the plurality of applications is different, the plurality of voice assistants may be determined as the voice assistants to be used.

130 On the other hand, if it is possible to specify the user but there is no currently running application, the processormay determine a voice assistant corresponding to most frequently used by the corresponding user based on the user's history information.

130 130 In the process of determining a voice assistant, the processormay determine that a plurality of voice assistants rather than one voice assistant are used. For example, when a voice command is input, the processormay check the user voice command using a first voice assistant, when the checked user voice command is a voice command that can be processed in another second voice assistant, activate the second assistance, and determine that response information is generated using each of the first voice assistant and the second voice assistant.

The operation of determining the voice assistant to be used may be implemented in various ways. For example, a voice assistant that performs voice recognition on a user's uttered voice may be determined, or a voice assistant that provides a response result according to the recognition result may be determined.

In other words, in implementation, when a voice assistant call event occurs, only the voice assistant determined in the above manner may process the input user voice data, or all installed voice assistants may process the input user voice data and only the voice assistant previously determined in the process of providing a corresponding response result may provide the response result.

130 In addition, processing voice data using a plurality of assistances may be implemented in various ways. For example, the processormay be configured not only that all processes such as voice recognition using voice data input by each of a plurality of voice assistants, confirming a user request and generating a response result according to a request are performed but also that a single voice assistant performs only voice recognition and another voice assistant receives and operates text information according to the voice recognition result.

130 130 130 120 As such, when the voice assistant to be used is determined, the processormay process a user voice command using the determined voice assistant. Specifically, the processormay convert a user's uttered voice into a text using a voice recognition engine, confirm the user's intent and/or an object using the converted text, confirm a user command, and generate response information corresponding to the confirmed user command. In this case, the processormay control the displayso that the voice recognition result corresponding to the user's uttered voice is displayed.

130 100 In one embodiment, the processormay perform the above-described operation by the electronic apparatusitself or may obtain response information according to a voice recognition result using an external server.

130 120 130 130 When response information using a voice assistant is generated through the above-described process, the processormay control the displayto display the response information. In this case, when a plurality of response information is generated by the operation of a plurality of applications, the processormay display a UI screen including response information regarding each of the plurality of voice applications. Alternatively, the processormay select only common response information from among the plurality of response information and display a UI screen including only the selected response information.

130 120 When a user voice command is input while a plurality of screens are displayed by the operation of a plurality of applications, the processormay control the displayto display response information of the voice assistant corresponding to each of the plurality of applications on an area corresponding to the screen of the plurality of applications.

130 130 When a predetermined voice command is input during the operation of the first voice assistant, the processormay activate the operation of the second voice assistant corresponding to the predetermined voice command. For example, when a voice command is input, the processormay check the user voice command using the first voice assistant, when the checked user voice command is a command for calling the second voice assistant, may activate the second voice assistant, and when the second voice assistant is activated, may process the input voice command using the second voice assistant. In other words, the subject to process the voice command may be switched to the second voice assistant.

In one embodiment, when all of a plurality of voice applications perform a voice recognition operation and a selected voice application provides response information, the above-described conversion may be to switch the voice application that provides response information.

130 120 When the checked user voice command is a command for calling the first voice assistant, the processormay control the displayto display a UI screen informing that the voice assistant is changed.

2 FIG. 100 100 Althoughbriefly illustrates the configuration of the electronic apparatus, the electronic apparatusmay further include various configurations.

3 FIG. illustrates a configuration of an electronic apparatus according to one or more embodiments.

3 FIG. 100 110 120 130 140 150 160 170 Referring to, an electronic apparatus′ may include a memory, a display, a processor, a communication device, an input device, a microphone, and a speaker.

110 120 130 2 FIG. 2 FIG. Since the operation of the memory, the displayand the processorhas been described above with reference to, only other operations not described inwill be described.

140 140 140 The communication deviceincludes at least one circuit, and may perform communication with various types of external devices. Such a communication devicemay be implemented as various interfaces according to one or more embodiments. For example, the communication devicemay include at least one interface among various types of digital interfaces, AP-based Wi-Fi (Wireless LAN Network), Bluetooth, Zigbee, wired/wireless Local Area Network (LAN), Wide Area Network (WAN), Ethernet, Near Field Communication (NFC), and IEEE1394.

140 In addition, the communication devicemay include at least one interface among High Definition Multimedia Interface (HDMI), Mobile High-Definition Link (MHL), Universal Serial Bus (USB), Display Port (DP), Thunderbolt, Video Graphics Array (VGA) port, RGB port, D-Subminiature (D-SUB), Digital Visual Interface (DVI), Audio Engineering Society/European Broadcasting Union (AES/EBU), Optical, and Coaxial.

140 The communication devicereceives a content. Here, the content may be a content having sound source data such as a music content, a video content, and the like, or a content not including sound source data (e.g., photos, texts, e-books, etc.).

140 140 140 140 The communication devicemay communicate with an external server, and transmit/receive various data for performing an assistance function. For example, the communication devicemay transmit an audio signal corresponding to a user's uttered voice or text information obtained by converting the corresponding audio signal into a text to an external server. In addition, the communication devicemay receive a response content corresponding to the transmitted information. For example, when a user utters a voice such as “please play music A”, the communication devicemay receive a content corresponding to music A.

In one embodiment, when such an operation is performed, the external server that transmits the above-described uttered voice or text may be different from the external server that provides the corresponding response content. In other words, the external server that transmits the above-described uttered voice or text may be a server that performs a voice recognition function, and the external server that provides the content may be a content providing server.

100 100 100 100 100 As such, when different external servers operate in conjunction with each other, the electronic apparatusmay receive a content directly from a content providing server without separate intervention, receive a response message from the server that performs a voice recognition function and receive a content by providing a request message corresponding to the received response message to the content providing server. For example, the external server that performs voice recognition may generate a response message indicating that music content A is reproduced and provide the response message to the electronic apparatus. In this case, the electronic apparatusmay request and receive music content A from the content providing server. Alternatively, the external server that performs voice recognition may directly transmit a command to provide music content A to the content providing server to the electronic apparatusto the content providing server. In this case, the electronic apparatusmay directly receive music content A.

140 100 In addition, the communication devicemay transmit information request for performing a command corresponding to a voice recognition result. For example, when a user requests today's weather, the electronic apparatusmay recognize that todays' weather information needs to be output and request and receive information regarding today's weather from an external server.

140 140 In addition, the communication devicemay transmit a command corresponding to a voice recognition result to another device. For example, when a user voice command is a command to control another device (e.g., “please lower the temperature of the air conditioner”), the communication devicemay transmit a control command corresponding to the user command to another device.

140 100 Further, the communication devicemay receive a voice command (specifically, sound source data, voice data including a user utterance) from another device. Specifically, when a user's uttered command is input through a remote controller or a user terminal device rather than the electronic apparatus, user voice data may be received from the corresponding device.

140 In this case, when receiving the above-described user voice data from a user terminal device, the communication devicemay also receive user information of the user terminal device or voice assistant identification information. Here, when the user determines a voice assistant in the user terminal device, the voice assistant identification information may be identification information indicating the determined voice assistant.

150 150 120 The input devicemay be implemented as a device for receiving a user's control command. Alternatively, the input devicemay be implemented as a touch screen that also performs the function of the display.

160 130 160 160 The microphoneis configured to receive a user voice or other sounds and convert them into audio data. The processormay perform a voice assistant function using a user voice input through the microphone. In addition, the microphonemay be implemented as a stereo microphone that receives sound input from a plurality of locations.

170 170 170 The speakeroutputs a sound corresponding to a content. In addition, the speakermay output various guide messages or response information corresponding to received response messages as a sound. Such a speakermay consist of a plurality of speaker units.

300 130 When receiving voice data from an external device (e.g., when receiving voice data from a user terminal device), if user information or voice assistant information included in the voice data is received together, the processormay determine a voice assistant to be used using the corresponding information.

4 FIG. illustrates an operation of an electronic apparatus according to one or more embodiments.

4 FIG. illustrates various operational examples of an integrated voice assistant function according to the present disclosure. The examples illustrate that each operation is performed sequentially, but in implementation, the order may be applied differently from the illustrated examples.

410 410 Firstly, a standby stateis described. In the standby state, a plurality of voice assistants may be in standby. The examples illustrate a case in which three voice assistants are installed, but in implementation, there may be a case in which two voice assistants are installed or a case in which four voice assistants are installed.

410 420 When an event to call a voice assistant occurs in the standby state, the voice assistant to be used is determined. As described previously, the voice assistant to be used may be determined by determining a currently running application, user information, etc. For example, in the case of a TV viewing state, the first voice assistant corresponding to an application related to TV viewing may be determined as the voice assistant to be used.

100 As such, when an assistance function using the first voice assistant is performed, the electronic apparatusmay display response information by performing the processing of voice recognition regarding a user's uttered voice using the first voice assistant.

430 100 430 In one embodiment, when the user requests switching of the second voice assistant while an assistance function is being performed using the first voice assistant, it may become the switch state. For example, when the user inputs a switching command including a keyword of the second voice assistant, the electronic apparatusmay switch the voice assistant to be used to the second voice assistant. In this case, the switch statemay display a screen indicating that the voice assistant is being switched.

420 430 In one embodiment, one voice assistant is used in each of the TV viewing stateand the switch state, but a plurality of voice assistants may be used simultaneously in response to a user voice command.

440 Specifically, when the user does not specify the voice assistant to be used or the user's uttered command can be processed in a plurality of voice assistants, individual response information may be obtained using the plurality of voice assistants simultaneously. In addition, response information using the obtained response information can be displayed. For example, a response result of each of the plurality of assistances may be displayed on the screen together as in a response information providing state.

100 450 In addition, when the electronic apparatusoperates in a multi-view, that is, when screens of a plurality of applications are displayed simultaneously as the plurality of applications are executed, a voice assistant corresponding to each application may operate individually, and a result thereof may also be displayed individually ().

5 FIG. 5 FIG. 100 131 100 300 illustrates an operation of an electronic apparatus according to one or more embodiments. Specifically,illustrates an operation of the electronic apparatusin case where an integrated engineand a user voice (or a user's voice command) are received by the electronic apparatusthrough a user terminal device.

510 1 2 3 100 In stage, a plurality of voice assistants (e.g., VA, VA, and VA) may be installed in the electronic apparatus.

300 131 100 131 131 520 131 100 100 520 100 When user voice data is received from the user terminal device, the integrated enginemay determine the voice assistant to be used (e.g., by the electronic apparatusor the integrated engine) based on user account information or currently running application, etc. The integrated enginemay perform an assistance function using the determined voice assistant (stage). Alternatively, the integrated enginemay notify the determined voice assistant to the electronic apparatus, thus the electronic apparatusmay perform an assistance function using the determined voice assistant. In stage, the electronic apparatusmay display information indicating the determined voice assistant on the screen.

530 1 131 1 2 100 In stage, when the user inputs a command to switch from the current voice assistant (e.g., VA) to another voice assistant while an assistance function is being performed using the first voice assistant, the integrated enginereceives a request to switch from one VA (e.g., VA) to another VA (e.g., VA) from the electronic apparatus.

131 1 2 The integrated enginemay determine that the voice assistant to be used is switched from the first voice assistant (e.g., VA) to the second voice assistant (e.g., VA) corresponding to the user's switch command.

540 131 100 2 100 6 FIG. In stage, the integrated enginemay inform the electronic apparatusof the switched voice assistant, which is the second voice assistant (e.g., VA). The electronic apparatusmay display that the voice assistant has switched. In this case, a screen indicating the conversion may be displayed and the switching of the voice assistant may be output by sound. A user interface window that may be displayed in the process of switching the voice assistant as described above will be described later with reference to.

131 When a user voice command is input, the integrated enginemay determine the voice assistant to process the corresponding voice command by analyzing the corresponding voice command. As in the illustrated example, it may be determined that a plurality of voice assistants are used when a user voice command is a request that does not specify a specific voice assistant or when “TV” is called as a subject to perform an operation. Such an operation may be performed not only at a time of an event for calling a voice assistant function for the first time but also while an assistance function using a specific voice assistant is being executed.

2 131 131 For example, while an assistance function using the second voice assistant (e.g., VA) is being executed, the integrated enginemay receive a voice recognition result from the second voice assistant and determine whether it is possible to perform the corresponding user request in another voice assistant based on the received voice recognition result. In this case, the integrated enginemay perform the above-described operation through context analysis, and may make the above-described determination based on whether a keyword (e.g., TV) indicating an integrated search is included in the corresponding voice recognition result(s).

131 Alternatively, when it is determined that the corresponding operation is possible only in a voice assistant other than the current voice assistant, the integrated enginemay cause the corresponding operation to be performed only in the voice assistant capable of performing the corresponding voice recognition result. For example, if a user's uttered request is a command to control other devices and the corresponding operation can be performed only in a voice assistant other than the currently running voice assistant, it may be determined that only the voice assistant capable of executing the corresponding command is used.

131 131 131 If such a determination is made, the integrated enginemay determine that a plurality of voice assistants are used, and as a user provides a voice message to the plurality of assistances, response information may be received from each of the plurality of voice assistants. To this end, the integrated enginemay store the user's voice data for a certain period of time and then, provide voice data stored in another voice assistant. Alternatively, the integrated enginemay provide a voice recognition result (e.g., text information or a user request for which context analysis, etc. has been completed) output from the currently operating voice assistant to another voice assistant.

131 550 131 131 8 10 FIGS.to In addition, the integrated enginemay display response information received from each of a plurality of voice assistants (stage). In this case, the integrated enginemay provide all identified response information or only common response information. Alternatively, the integrated enginemay provide only a plurality of response information, but may process and display common responses more distinctively than other responses. Various examples regarding a user interface window that can be displayed in the integrated response process using a plurality of voice assistants will be described later with reference to.

131 560 7 FIG. When a plurality of multi-view screens are being provided, the integrated enginemay separately display response information corresponding to each multi-view screen (stage). For example, if the first to third applications are being executed, a response of a voice assistant corresponding to the first application may be displayed on a screen corresponding to the first application, a response of a voice assistant corresponding to the second application may be displayed on a screen corresponding to the second application, and a response of a voice assistant corresponding to the third application may be displayed on a screen corresponding to the third application. An example of a user interface window that can be displayed on a display in such a multi-view environment will be described later with reference to.

6 10 FIGS.to In the above description, various functions according to the present disclosure have been explained and hereinafter, an operation of each function will be described in greater detail with reference to.

6 FIG. illustrates an operation of switching a voice assistant according to one or more embodiments.

6 FIG. 610 100 Referring to, a first screenmay display a content and a voice recognition result corresponding to a user voice. As such, when a user request is to switch to another voice assistant, the electronic apparatusmay switch the voice assistant. The illustrated example describes that the voice assistant is switched according to a user request but in implementation, the voice assistant may be switched by an event other than a user's uttered command.

For example, when a function according to a user's uttered command cannot be performed in the current voice assistant, but it can be performed in another voice assistant, switching to another voice assistant capable of performing the function according to the user's uttered command may be determined regardless of the user utterance.

In the above-described example, a user's uttered command is to control an external device, but a case in which a corresponding function cannot be performed in the current voice assistant but can be performed in another voice assistant may be an example. In addition, each voice assistant may perform a voice assistant function using an external server corresponding to each voice assistant. A case in which there is a communication failure between the first voice assistant and the corresponding first external server but there is no communication failure between the second voice assistant and the corresponding second external server may be an example.

620 630 In such a process, a second screenindicating that the voice assistant is being switched may be displayed. When the voice assistant is switched, a third screenindicating that the conversion is completed and it is ready to receive a user voice command may be displayed.

100 610 630 620 The illustrated example displays only an operation state of the electronic apparatusbut in implementation, information (or icon, thumbnail, symbol, shortcut) regarding the currently operating voice assistant may be displayed on each screen. In other words, the first screenmay display the first icon corresponding to the first voice assistant on one side of the screen, and the third screenmay display the second icon corresponding to the second voice assistant on one side of the screen. In addition, the second screenmay display the first icon corresponding to the first voice assistant on the left and the second icon corresponding to the second voice assistant on the right, thereby intuitively indicating that the voice assistant is being switched from the first voice assistant to the second voice assistant.

7 FIG. illustrates an assistance function during a multi-view operation according to one or more embodiments.

7 FIG. 100 711 713 715 710 Referring to, the electronic apparatusmay display UIs,,corresponding each of a plurality of applications on a single screen. In the illustrated example, a multi-view screen displaying three UIs is illustrated but in implementation, two UIs may be displayed or more than four UIs may be displayed.

In such a multi-view state, conventionally, when response information according to a user's uttered command is provided, the response information is displayed without considering the multi-view environment. In other words, in the conventional multi-view environment, it is difficult to individually control each application corresponding to each multi-view.

In order to solve the above problem, the present disclosure provides an assistance function for specifying an application corresponding to each multi-view or each screen during a process or operation of providing response information, etc.

Specifically, in a multi-view state, when a user inputs a voice command to request an operation of a specific application, the corresponding response result may be displayed only on the screen corresponding to the specific application. In addition, even when the user does not specify an application, a UI screen or application to which the current voice command is to be applied may be determined in consideration of the user's focus, history, etc.

724 723 721 723 725 720 In other words, response informationmay be displayed only on a screencorresponding to an application requested by the user from among UIs,,corresponding to each of a plurality of applications as shown on a second screen. As such, response information is displayed only on a specific screen, and it is possible to individually control a specific application from among a plurality of running applications through a voice command.

8 10 FIGS.to are views illustrating an example of UI screen that displays an integration result according to one or more embodiments.

8 FIG. Specifically,illustrates an example of UI screen that displays only common response information among response information generated by a plurality of voice assistants in the process of providing response information using the plurality of voice assistants.

810 811 For example, when a user inputs a voice command “Recommend a program for the user to watch”, the integrated engine may obtain response information corresponding to the above-described voice command using a plurality of voice assistants. When such response information is obtained, the integrated engine may display only common response information on a screenas a response result.

As such, an electronic apparatus according the present disclosure displays response information commonly recommended by a plurality of voice assistants, so the electronic apparatus may provide information that is more suitable for the user's intention or more accurate.

9 FIG. 9 FIG. 910 illustrates a case where there is no common result or a case where a response result generated in each of a plurality of voice applications is individually output based on user setting, etc. Referring to, it can be seen that a screenclassifies and displays a response result of each voice assistant. Each response result is illustrated to include on result but in implementation, a response result of each voice assistant may include a plurality of contents (or information).

As such, an electronic apparatus according to the present disclosure displays the response results of a plurality of voice assistants at once and thus, a user can easily check desired information. In other words, in the existing environment in which only one voice assistant can be controlled, a user has to check the response information of each voice assistant one by one while individually calling the voice assistant, but according to the present disclosure, an integrated result is displayed through one operation, so it is possible for the user to check desired information more easily.

10 FIG. illustrates a case in which only a specific voice assistant can perform a response operation according to a user request.

10 FIG. For example, when the user utters a command “Adjust the wind strength of the air conditioner” and only the air conditioner of the first voice assistant can be controlled, only the response information of the first voice assistant can be displayed as illustrated in.

As such, even though the user does not specify a specific voice assistant, the voice assistant corresponding to the user request is determined and operated and thus, the user convenience is improved.

In other words, in the prior art, the user had to determine the voice assistant capable of controlling an air conditioner and call the corresponding voice assistant. If another voice assistant is in use, the function of the corresponding voice assistant had to be terminated, and the voice assistant capable of controlling the air conditioner must be called to perform the above-described operation. However, by using the integrated engine according to the present disclosure, the user can obtain a response according to the user request without specifying a voice assistant.

11 FIG. illustrates a control operation of an electronic apparatus according to one or more embodiments.

11 FIG. 1110 1120 Referring to, firstly, when a user voice command is input (S), response information is generated by using at least one voice assistant from among different voice assistants (S).

In the above-described operation, the voice assistant to be used is determined in advance, but when the voice assistant to be used is not determined in advance, for example, when a voice assistant call event occurs, at least one voice assistant may be determined based on at least one of the user's uttered keyword, the user account, the currently running application, or voice assistant history information.

In this case, a plurality of voice assistants instead of one voice assistant may be determined. When a plurality of voice assistants is determined, a UI screen including response information of each of the plurality of voice assistants may be displayed. In this case, when a user voice command is input while a plurality of screens corresponding to each of the plurality of applications are displayed, response information of the voice assistant corresponding to each of the plurality of applications may be displayed on the corresponding area. Alternatively, when a plurality of voice assistants are determined, a UI screen including only common response from among response information of each of the plurality of voice assistants may be displayed.

1130 1140 When a predetermined voice command is input during the operation of a first voice assistant (S), a second voice assistant corresponding to the predetermined voice command is activated (S). Specifically, when a voice command is input, the user voice command is identified using the first voice assistant, when the identified user voice command is a command to activate or invoke the second voice assistant, the second voice assistant is activated, and when the second voice assistant is activated, the input voice command may be processed using the second voice assistant. In this case, when the identified user voice command is a command to activate or invoke the first voice assistant, a UI screen informing that the voice assistant is changed may be displayed.

Alternatively, when a voice command is input, the user voice command is identified using the first voice assistant, and when the identified user voice command is a voice command that can be processed in another second voice assistant, the second voice assistant may also be activated and response information may be generated using each of the first voice assistant and the second voice assistant.

As described above, a controlling method in an electronic apparatus according to the present disclosure integrates and manages (or processes) a plurality of voice assistants and thus, it is possible for the user to easily select/convert the voice assistant and to receive more diverse or accurate information through a single voice command.

12 FIG. illustrates an operation of an integrated voice assistant according to one or more embodiments.

12 FIG. 1205 1240 Referring to, it is identified whether a voice assistant call event occurs. Specifically, such a voice assistant call event may occur when the user presses or selects a microphone button provided on an electronic apparatus, a remote controller or a user terminal device (S) or when the user utters a certain keyword (a trigger command) (S).

1205 1210 In case where the user presses a microphone button (S-Y), it may be identified whether there is a history for grasping the context of the current screen (S). Here, the history information may include a TV watching history, whether a specific application has been performed, whether it is a search process, whether the focus is on a specific application, and the like.

1210 1215 1220 When there is such history information (S-Y), the voice assistant to be used may be determined based on the history information (S). Also, an assistance function using the determined voice assistant may be performed (S).

1210 1225 When there is no such history information (S-N), the voice assistant to be used may be determined by using a pre-stored algorithm in addition to the above-described history information (S). Also, an assistance function using the determined voice assistant may be performed.

1230 1235 In one embodiment, when the user inputs a command to switch to another voice assistant while using the first voice assistant (S), the voice assistant may be switched and the conversion process may be displayed (S).

When the conversion is completed, an assistance function may be performed using the converted voice assistant.

1240 If the user does not press the microphone button, it may be identified whether a specific keyword is included in the user's uttered voice (S). For example, when a keyword calling an integrated engine (e.g., “Hey, TV”, etc.) or a keyword corresponding to a specific voice assistant (e.g., “Hi Bixby”, etc.) is included, it may be determined that a voice assistant function is called.

100 1245 When such a voice assistant function is called, it is determined whether the electronic apparatusis operating in a multi-view (S).

100 1250 When the electronic apparatusis operating in a multi-view, a voice assistant function may be performed and the result may be displayed according to the content of each multi-view (S).

100 1260 When the electronic apparatusis not operating in a multi-view, the voice assistant to perform a voice assistant function is determined. When there is one determined voice assistant, a voice assistant function may be performed using the corresponding voice assistant (S).

1260 When it is determined that a plurality of voice assistants are used, an assistance function may be performed using the plurality of voice assistants and it may be provided to the user through the response result of each voice assistant (S).

The methods according to one or more embodiments of the disclosure described above may be implemented in the form of an application that can be installed in the existing electronic apparatus.

In addition, the methods according to one or more embodiments of the disclosure described above may be implemented only by software upgrade or hardware upgrade for the existing electronic apparatus.

In addition, the methods according to one or more embodiments of the disclosure described above may be performed through an embedded server provided in the electronic apparatus or an external server of at least one of the electronic apparatuses.

According to one or more embodiments of the disclosure, the embodiments described above may be implemented by software including an instruction stored in a storage medium readable by a machine (e.g., a computer). The machine may be an apparatus that invokes the stored instruction from the storage medium and may be operated based on the invoked instruction, and may include an electronic apparatus (e.g., electronic apparatus A) according to the disclosed embodiments. In case that the instruction is executed by the processor, the processor may directly perform a function corresponding to the instruction or other components may perform the function corresponding to the instruction under control of the processor. The instruction may include codes provided or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term ‘non-transitory’ indicates that the storage medium is tangible without including a signal (e.g., electromagnetic waves), and does not distinguish whether data are semi-permanently or temporarily stored in the storage medium. For example, ‘non-transitory storage medium’ may include a buffer in which data is stored temporarily.

According to one or more embodiments, the methods may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a purchaser. The computer program product may be distributed in the form of a storage medium (e.g., compact disc read only memory (CD-ROM)) that is readable by devices, may be distributed through an application store (e.g., Play Store TM) or directly between two user devices (e.g., smartphones), or may be distributed online (e.g., by downloading or uploading). In the case of an online distribution, at least part of the computer program product (e.g., a downloadable application) may be at least temporarily stored in a storage medium readable by a machine such as a server of the manufacturer, a server of an application store, or the memory of a relay server or may be temporarily generated.

The one or more embodiments of the present disclosure may be implemented by software including an instruction stored in a storage medium readable by a machine (e.g., a computer). The machine may be an apparatus that invokes the stored instruction from the storage medium and may be operated based on the invoked instruction, and may include an electronic apparatus (e.g., electronic apparatus A) according to the disclosed embodiments.

In case that the above-described instruction is executed by the processor, the processor may directly perform a function corresponding to the instruction or other components may perform the function corresponding to the instruction under control of the processor. The instruction may include codes provided or executed by a compiler or an interpreter.

In the above, preferred embodiments have been illustrated and described, but the disclosure is not limited to the specific embodiments described above, and those of ordinary skill in the art pertaining to the disclosure may modify the embodiments without departing from the gist of the claims and these modifications should not be individually understood from the technical spirit or perspective of the disclosure.