Display Device Capable of Displaying a Customized Recommendation Utterance

Pursuant to 35 U.S.C. § 119(a), this application claims the benefit of the earlier filing date and the right of priority to PCT Patent Application No. PCT/KR2024/096818, filed on Dec. 12, 2024, the contents of which are incorporated by reference herein in their entirety.

The present disclosure relates to a display device capable of displaying a customized recommendation utterance. More particularly, the present disclosure relates to an AI system including a display device capable of displaying a customized recommendation utterance and a server.

In recent years, digital TV services using wired or wireless communication networks have grown in popularity. The digital TV services can provide various services that existing analog broadcast services could not provide.

For example, Internet Protocol Television (IPTV) and smart TV services, which are categorized as types of the digital TV services, provide bidirectional communication functionality that enables users to actively select types of programs to be viewed, specify viewing times, and perform viewing-related operations.

In recent years, display devices such as digital TVs have provided various services based on artificial intelligence (AI). In this regard, a display device installed within a household is capable of providing AI services customized for a plurality of users. However, there exists a problem in that providing user-specific recommendations is difficult due to the challenge of distinguishing usage data among users who share the same display device installed in a household.

From the perspective of personal information exposure, a privacy problem exists regarding recommendations, because viewing histories, utterance records, or search records may be exposed to users who share the same display device. In addition, in terms of search time and operational complexity, a problem arises in that direct input of, or searching for, a keyword for viewing requires significant time and effort, thereby hindering the use of various functions provided by the display device.

One object of the present disclosure, which is intended to address the above-mentioned problems and other related problems, is to provide an AI system that includes a display device capable of displaying a customized recommendation utterance and a server.

Another object of the present disclosure is to distinguish usage data associated with each of a plurality of users who share the same display device installed in a household, thereby enabling provision of user-specific recommendations.

A further object of the present disclosure is to protect privacy in providing recommendations, without exposing viewing histories, utterance records, or search records to users who share the same display device.

Another object of the present disclosure is to reduce time and effort required for direct input of, or searching for, a keyword for viewing in terms of search time and operational complexity, thereby facilitating the use of various functions provided by a display device.

According to one aspect of the present disclosure, there is provided a display device capable of displaying a customized recommendation utterance, the display device including: a user input interface unit configured to receive a first input, associated with a recommendation utterance request, and a second input, associated with recognition of utterance data of a user, in conjunction with a remote control device; a display configured to display either information or a recommendation utterance, each of which is associated with the utterance data; and a processor configured to enable the display to display a keyword associated with either the utterance data or previous utterance data of the user.

According to an embodiment, when a key value corresponding to a button of the remote control device is received, the processor may generate the recommendation utterance request, may request a history of the previous utterance data from a database in response to the recommendation utterance request, and may request the recommendation utterance from a server on the basis of a history of the utterance data and account information of the user. The processor may receive the recommendation utterance from the server and may control the output of the received recommendation utterance, including the keyword, on the display.

According to an embodiment, in the display device, the processor may include: a recommendation utterance output module configured to output the recommendation utterance by executing an application program when a first input, having a duration shorter than a threshold time, is applied to the button; and a history management module configured to receive the recommendation utterance request from the recommendation utterance output module and receive the history of the utterance data from the database. The history management module may transfer the recommendation utterance, received from the server, to the recommendation utterance output module.

According to an embodiment, in the display device, when a key value corresponding to a power button of the remote control device is received, the recommendation utterance output module may transfer the recommendation utterance request to the history management module. The history management module may transfer a greeting request to a second server and may receive, from the second server, a greeting based on temporal information. The history management module may transfer the received greeting to the recommendation utterance output module. The recommendation utterance output module may output a screen, including the received greeting, on the display.

According to an embodiment, in the display device, when a switch from a powered-off state to a powered-on state first occurs on a per-date basis, the processor may output a screen, including a first greeting, on the display on the basis of the temporal information. When the switch to the powered-on state occurs two or more times, the processor may output, on the display, a screen including a second greeting different from the first greeting. When a first threshold time elapses in the powered-on state, the processor may extract keywords according to utterance frequency derived from the history of the utterance data for each corresponding week. The processor may display the extracted keywords in order of the utterance frequency.

According to an embodiment, in the display device, the processor may further include a voice recognition processing module configured to transfer the recognized utterance data from the remote control device to the server. The voice recognition processing module may perform processing for analysis of the user's intention associated with the recognized utterance data in conjunction with the server when a second input, having a duration equal to or longer than the threshold time, is applied to the button. The voice recognition processing module may control the display of a search result, associated with the recognized utterance data, on a screen of the display on the basis of the result of the processing for analysis of the user's intention.

According to an embodiment, in the display device, the voice recognition processing module may determine, in conjunction with the server, whether the recognized utterance data is voiceprint data of a second user other than the user. When the second user is a registered user, the processor may display an account switch screen, associated with whether to a switch from the user's login-required account to the second user's second login-required account. When a switch input for the switch to the second login-required account is received from the account switch screen, the processor may switch to the second login-required account.

According to an embodiment, in the display device, the processor may request a second recommendation utterance from the server on the basis of a second history of utterance data of the second user and second account information of the second user. The processor may receive the second recommendation utterance from the server and may control the output of the received second recommendation utterance, including a second keyword, on the display.

According to an embodiment, in the display device, the history management module may store the history of the utterance data in the database on the basis of the result of the processing for analysis of the user's intention. The history management module may store a history of content information in the database on the basis of the content information selected from a home screen of the display. The history management module may store a history of second content information in the database on the basis of the second content information selected from an integration search screen of the display.

According to an embodiment, in the display device, the utterance data, the content information, and the second content information, which are to be stored in the database, may each include temporal information including a start time and an end time, a keyword, a genre code, an intention type, and information on an application being executed on the display device. The keyword may be one of the following: a keyword derived from the utterance data, the content, or the name of the second content.

According to an embodiment, in the display device, the voice recognition processing module may control the storage of the history of the utterance data in the database. The history of the utterance data may include the user's account, an STT result, the date and time of an utterance, the corresponding day of the week, and temporal information.

According to an embodiment, in the display device, when the key value corresponding to the button is received, the processor may extract information about a currently logged-in account from the database, may extract keyword information associated with an utterance used by the user, on the basis of the current day of the week and the temporal information. The processor may control the output of the received recommendation utterance, including the keyword, on the display on the basis of the keyword information.

According to an embodiment, in the display device, the processor may transfer information about a currently logged-in account, the current time, and the user's access record to the second server. The second server may transfer the information about the currently logged-in account to a UPP server, may receive customer segment information from the UPP server, and may transfer, to the processor, a welcome message generated on the basis of the current time, the customer segment information, and the user's access record. The processor may control the output of the welcome message on the display.

According to an embodiment, in the display device, when the key value corresponding to the button is received, the processor may transfer, to the second server, the current time, information about a connected apparatus, information about a program currently being viewed, and information about an application currently in use. The processor may receive, from the second server, usage tip information customized on the basis of the current time, the information about the connected apparatus, the information about the program currently being viewed, and the information about the application currently in use. The processor may display at least one sentence associated with a usage tip on the display on the basis of the customized usage tip information.

According to an embodiment, in the display device, the processor may acquire the information about the application in use and the account information of the user. The processor may output at least one sentence, associated with searching for and playing back content items provided by the application, on a screen of the display on the basis of the information about the application currently in use, the history of the utterance data, and the account information of the user.

According to an embodiment, in the display device, the processor may acquire state information of the connected apparatus and environmental information of a region where the display device and the connected apparatus are arranged. The processor may output at least one sentence, associated with the usage tip for controlling the operation of the connected apparatus, on a screen of the display on the basis of the state information and the environmental information.

According to an embodiment, in the display device, the processor may receive, from the server, broadcast information associated with a program to be viewed by the user. The processor may transfer the broadcast information to a third server that extracts an association keyword associated with the broadcast information. The third server may receive detailed program information associated with the broadcast information from a fourth server of an integrated broadcasting information system. The processor may output the association word associated with the broadcast information on the display on the basis of the detailed program information.

According to an embodiment, in the display device, when a first user is detected in a state where the first user's first login-required account, among a plurality of user accounts, is logged in, the processor may request a first recommendation utterance from the server on the basis of a first history of first utterance data of the first user and first account information of the first user. When a second user is detected in a state where the second user's second login-required account, among the plurality of user accounts, is logged in, the processor may request a second recommendation utterance from the server on the basis of a second history of second utterance data of the second user and second account information of the second user. The first user may be detected by detecting a first identifier of the first user's first device or through the first user's voiceprint data. The second user may be detected by detecting a second identifier of the second user's second device or through the second user's voiceprint data.

According to an embodiment, in the display device, when the second user is detected instead of the first user in the state where the first login-required account is logged in, the processor may request the second recommendation utterance from the server on the basis of the second history and the second account information. When the first user and the second user are detected in the state where the first login-required account is logged in, the processor may request a first recommendation utterance and the second recommendation utterance from the server on the basis of the first history, the second history, the first account information, and the second account information. The processor may display a common recommendation utterance on the display on the basis of a keyword common to the first recommendation utterance and the second recommendation utterance. The processor may display the first recommendation utterance after the common recommendation utterance on the basis of the first user's first keyword. The second may be registered as having a familial relationship with the first user.

According to the present disclosure, a recommendation customized for each user can be provided by managing usage data for each account. Furthermore, user identification through voiceprint recognition enables data learning for each user solely through utterance without the need for a separate login process.

According to the present disclosure, the customized recommendation can be immediately received by pressing an AI key at any time and in any location. Enhancing recommendation accuracy through advanced logic can significantly reduce search time, thereby enabling execution of a desired function or access to desired content.

According to the present disclosure, either a setting or association information optimized for content currently being viewed can be provided. Furthermore, a usage tip for enhancing the usefulness of an application currently in use can be provided, thereby making the TV viewing experience more immersive and enjoyable.

Technical terms used in the present specification are intended solely to describe specific embodiments and are not intended to limit the present disclosure. In addition, terms expressed in the singular form in the present specification should be construed to include the plural form unless the context clearly dictates otherwise. The terms “module” and “unit” used in the following description are assigned or used interchangeably solely for convenience in drafting the present specification and are not intended to denote any distinct meanings or roles in and of themselves.

The expression “is configured from” or “includes constituent elements or steps,” used in the present specification, should not be construed to require inclusion of all of the constituent elements or steps. Rather, this expression should be interpreted to mean that some of the constituent elements or steps may be omitted, and additional elements or steps may be included.

In addition, in describing the technology disclosed in the present disclosure, when it is determined that a detailed description of known technology in the art may obscure or detract from the nature and gist of the present disclosure, such description is omitted from the present specification.

In addition, the accompanying drawings are merely provided to facilitate understanding of the embodiments disclosed in the present specification and shall not be construed as limiting the technical idea disclosed in the present specification. Furthermore, any modifications, equivalents, or substitutions of constituent elements, to the extent falling within the technical scope of the present disclosure, are intended to be encompassed within the scope of the present disclosure. In addition, not only the embodiments described below but also combinations of such embodiments, as modifications, equivalents, or substitutions, may be encompassed within the technical idea and scope of the present disclosure.

The embodiments disclosed in the present specification may be described in detail below with reference to the accompanying drawings.

1 FIG. 100 is a block diagram illustrating a configuration of a display deviceaccording to an embodiment of the present disclosure.

1 FIG. 100 130 135 140 150 170 110 175 180 185 190 With reference to, the display devicemay include a broadcast reception unit, an external device interface unit, a database, a user input interface unit, a processor, a wireless communication unit, a voice recognition processing module, a display, an audio output unit, and a power supply unit.

130 131 132 133 The broadcast reception unitmay include a tuner, a demodulation unit, and a network interface unit.

131 131 The tunermay select a broadcast station with a specific broadcast channel according to a command for channel-based broadcast station selection. The tunermay receive a broadcast signal over a specific broadcast channel from the selected broadcast station.

132 The demodulation unitmay separate the received broadcast signal into a video signal, an audio signal, and a data signal associated with a broadcast program and may demodulate the video signal, the audio signal, and the data signal, which result from this separation, into an output-enabled format.

133 100 133 The network interface unitmay provide an interface for connecting the display deviceto a wired or wireless network of which examples include an Internet network. Through the network interface unit, data may be transmitted or received to or from another user or another electronic device over a connected network or another network linked to the connected network.

133 133 133 The network interface unitmay have access to a predetermined web page over a connected network or another network linked to the connected network. That is, the network interface unitmay have access to a predetermined web page. Through the network interface unit, data may be transmitted or received to or from a specific server.

133 133 Then, through the network interface unit, content or data that are provided by a content provider or a network manager may be received. That is, content, such as a movie, an advertisement, a game, VOD, or a broadcast signal, along with information associated with the content, may be received over a network through the network interface unitfrom a content provider or a network manager.

133 In addition, through the network interface unit, firmware update information and an update file that are provided by the network manager may be received, and data may be transmitted to an Internet or content provider, or to the network manager.

133 Through the network interface unit, a desired application may be selected and received over a network from among applications that are available to the public.

135 170 140 Through the external device interface unit, an application or an application list may be received from an adjacent external device, and the received application or application list may be transferred to the processoror the storage unit.

135 100 135 170 100 135 The external device interface unitmay establish a connection path between the display deviceand the external device. Through the external device interface unit, one or more of the following may be received and transferred to the processor: image or audio data, each of which is output from the external device connected to the display devicein a wireless or wired manner. The external device interface unitmay include a plurality of external input terminals. The plurality of external input terminals may include an RGB terminal, one or more High Definition Multimedia Interfaces (HDMI) terminals, and a component terminal.

135 180 135 185 An image signal of the external device that is input through the external device interface unitmay be output through the display. A voice signal of the external device that is output through the external device interface unitmay be output through the audio output unit.

135 The external device connectable to the external device interface unitmay, for example, be one of the following: a set-top box, a Blu-ray player, a DVD player, a game device, a sound bar, a smartphone, a PC, a USB memory device, or a home theater.

100 100 In addition, some of the content data stored in the display devicemay be transmitted to a user selected from among other users pre-registered with the display unitor to an electronic device selected from among other electronic devices pre-registered therewith.

170 140 140 A program for processing and controlling each signal within the processormay be stored in the database. Furthermore, an image, voice, or data signal, each of which is signal-processed, may be stored in the database.

135 133 140 140 In addition, an image, voice, or data signal, each of which is input from the external device interface unitor the network interface unit, may also be stored in the databasethrough a temporary storage function. Furthermore, information about a predetermined image may also be stored in the databasethrough a channel memory function.

135 133 140 An application or an application list, each of which is input from the external device interface unitor the network interface unit, may be stored in the storage unit.

140 100 Content files (a moving image file, a still image file, a music file, a text file, an application file, and the like) stored in the storage unitmay be played back on the display devicefor user viewing.

150 170 170 150 200 150 170 200 Through the user input interface, a signal that is input by a user may be transferred to the processor, and a signal may be transferred from the processorto the user. For example, the user input interfacemay receive control signals for powering on or powering off, channel selection, screen setting, and similar functions from a remote control deviceand process the received control signals in compliance with various communication standards, such as Bluetooth, Ultra-Wideband (UWB), ZigBee, Radio Frequency (RF), and Infrared (IR). Alternatively, the user input interfacemay process a control signal from the processorfor transmission to the remote control device.

150 170 In addition, through the user input interface, a control signal that is input from local keys (not illustrated), such as a power key, a channel key, a volume key, and a setting key, may be transferred to the processor.

170 180 170 135 An image signal that is image-processed in the processormay be input into the display, and an image corresponding to the image signal may be displayed thereon. In addition, the image signal that is image-processed in the processormay be input into an output device through the external device interface unit.

170 185 170 135 The voice signal that is processed in the processormay be audio-output to the audio output unit. In addition, the voice signal that is processed in the processormay be input into the external output device through the external device interface unit.

170 100 Moreover, the processormay control the overall operation of the display device.

170 100 150 170 100 170 100 10 20 In addition, the processormay control the display deviceby executing either a user command that is input through the user input deviceor an internal program. The processormay enable an application or an application list, each of which is desired by the user, to be downloaded into the display deviceby making a connection to a network. The processormay be configured to control the display deviceby executing at least one application program. Execution of a first application programmay enable determination, on the basis of the viewing history and playback time for specific content, as to whether such content has been purchased, played back, refunded, resold, or subjected to similar actions. Execution of a second application programmay enable information about the playback history and playback time for specific content and other related information to be stored and managed.

170 180 185 The processorenables channel information or the like, which is selected by the user, together with the processed image or voice signal, to be output on the displayor through the audio output unit.

150 170 135 180 185 In addition, in response to an external device image playback command received through the user input interface, the processormay enable an image signal or a voice signal, each of which is input from an external device, for example, a camera or a camcorder through the external device interface unit, to be output on the displayor through the audio output unit.

170 180 170 180 131 135 140 180 The processormay control the displaysuch that an image is displayed thereon. For example, the processormay control the displaysuch that a broadcast image input through the tuner, an external input image input through the external device interface unit, an image input through the network interface unit, or an image stored in the storage unitis displayed thereon. In this case, the image displayed on the displaymay be a still image or a moving image and may be a 2D image or a 3D image.

170 180 100 In addition, the processorcontrols the display unitsuch that content stored within the display device, received broadcast content, or external input content, input from the outside, is played back. Such content may be in various formats, such as a broadcast image, an external input image, an audio file, a still image, an image from an accessed web source, and a text file.

110 110 110 110 100 100 100 100 100 The wireless communication unitmay perform communication with an external apparatus through wired or wireless communication. The wireless communication unitmay perform short-range communication with the external apparatus. To this end, the wireless communication unitmay support short-range communication using at least one of the following technologies: Bluetooth™, Bluetooth Low Energy (BLE), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct, or Wireless Universal Serial Bus (USB). The wireless communication unitmay support wireless communication over short-range wireless area networks between the display deviceand a wireless communication system, between the display deviceand a different display device, or between the display deviceand a network on which the display deviceor an external server is located. The short-distance wireless communication may be short-range Wireless Personal Area Networks.

100 100 110 100 100 170 100 110 100 The different display devicehere may be a mobile terminal, such as a wearable device (for example, a smartwatch, smart glasses, or a head-mounted display (HMD)) or a smartphone, each of which is capable of interchanging data with (or operating in conjunction with) the display deviceaccording to the present disclosure. The wireless communication unitmay detect (or recognize) a communication-capable wearable device in the vicinity of the display device. Moreover, in a case where the detected wearable device is a device authenticated to communicate with the display deviceaccording to the present disclosure, the processormay transmit at least one portion of the data processed in the display deviceto the wearable device through the wireless communication unit. Therefore, a user of the wearable device may use the data processed in the display devicethrough the wearable device.

175 175 100 The voice recognition processing modulemay acquire audio. The voice recognition processing modulemay include at least one microphone (not illustrated) and, through the microphone (not illustrated), may acquire audio in the vicinity of the display device.

180 170 135 The displaymay convert an image signal, a data signal, and an OSD signal, all of which are processed in the processor, into respective RGB signals, thereby generating drive signals. Alternatively, the display unit may convert an image signal, a data signal, and the like, all of which are received in the external device interface unit, into respective RGB signals, thereby generating drive signals.

100 100 1 FIG. The display deviceillustrated inis merely an embodiment of the present disclosure. Therefore, depending on the specifications of the display deviceas actually embodied, some of the illustrated constituent elements may be integrated, omitted, or separated into individual constituent elements.

That is, according to need, two or more constituent elements may be integrated into a single constituent element, or one constituent element may be separated into two or more constituent elements. In addition, the function performed in each block is merely for describing an embodiment of the present disclosure, and a specific operation or device for performing such a function does not impose any limitation on the scope of the claims of the present disclosure.

2 FIG. 300 is a diagram that is referenced to describe a serveraccording to an embodiment of the present disclosure.

300 100 The servermay provide a recommendation service for recommending content or a service favorable to a viewer who uses the display device.

300 310 320 330 The servermay include a communication interface, a memory device, and a processor.

300 100 310 The servermay transmit and receive data to and from at least one display devicethrough the communication interfaceover a wired or wireless communication network.

320 340 340 340 340 The memory devicemay be configured to include a databaseor to operate in conjunction with the database. Information associated with content played back on each device may be stored in the database. For example, content playback information, content setting information, or application installation information, each of which is associated with the identification information of each device, may be stored in the database.

100 330 340 In a case where a content recommendation request is received from the display deviceor an external device, the processormay recommend content optimized for the requesting device on the basis of data stored in the database.

3 FIG. 1000 100 is a diagram that is referenced to describe an AI systemincluding the display deviceaccording to the embodiment of the present disclosure.

1000 100 200 300 The AI systemmay include at least one display device, at least one remote control device, and the content server.

170 100 The processorof the display devicemay play back content.

170 170 In addition, the processormay generate content playback information about the played-back content. In addition, the processormay generate the content setting information, which is information about image quality, volume, and the presence or absence of a preference channel.

The content playback information may include at least one of the following: content identification information about the played-back content, content genre information, information about starting point in time of content playback, information about ending point in time of content playback, or the total content playback time.

image quality information, which is set for content during playback, volume information, or preference channel information about whether the user has registered a channel, over which content is provided, as a preference channel. The content setting information may include at least one of the following:

170 100 300 310 100 The processormay transmit the device identification information of the display device, the generated content playback information, and the generated content setting information to the serverthrough the communication interface. The device identification information may be a unique identification number for distinguishing the display devicefrom other devices.

300 100 321 The content servermay store the content playback information and the content setting information, which are received from the display device, in a content information databasein a manner that is associated with the device identification information.

170 150 175 The processormay receive a content recommendation command, as an input, through the user input unit interface unitor the voice recognition processing module.

170 100 300 310 In a case where the content recommendation command is received as an input, the processormay transmit the device identification information of the display deviceand the content recommendation request to the serverthrough the communication interface.

310 300 100 Through the communication interfaceof the server, the device identification information and the content recommendation request may be received from the display device.

330 300 100 321 The processorof the servermay acquire the content playback information and the content setting information, which are associated with the display device, from the content information databaseon the basis of the device identification information.

330 100 The processormay generate content recommendation information and recommendation setting information for the display deviceon the basis of the content playback information and the content setting information. The content recommendation information may include recommendation content identification information and recommendation content genre information about at least one recommendation content item. In addition, the recommendation setting information may include recommendation image-quality setting information and preference channel information.

330 100 310 The processormay transmit the content recommendation information and the recommendation setting information to the display devicethrough the communication interface.

330 100 310 The processormay transmit the content recommendation information and the recommendation setting information to the display devicethrough the communication interface.

170 150 175 170 The processormay display at least one recommendation content item on the basis of the received content recommendation information. In addition, when a command for playing back recommendation content is input through the user input interface unitor the voice recognition processing module, the processormay play back the recommendation content after setting the image quality of the recommendation content, which is to be played back, on the basis of the received recommendation setting information.

170 The processormay play back the recommendation content after setting the image quality thereof, and, in a case where the user inputs a change to the preference channel, may make a change to the preference channel on the basis of the preference channel information.

100 The display devicemay mirror the content whose playback is in progress to the external apparatus. Examples of the external device may include a different display device and a mobile device. In this case, content that is mirrored may be viewed through the external apparatus. Therefore, viewing information about the content that is mirrored is necessary as foundational data for recommending content for display on the external apparatus.

100 100 100 100 300 In a case where the display deviceperforms a mirroring operation for display on the external equipment, a control command for controlling the display devicemay be received from the external apparatus. Examples of the control command may include a content change command for changing playback content from first content to second content. In a case where the content change command is received, the display devicemay play back content resulting from the change. In this case, the display deviceneeds to transmit the content playback information about the content resulting from the change to the server, as information for recommending content for display on the external apparatus.

100 100 100 300 4 FIG. The display deviceaccording to the present disclosure, which displays a customized recommendation utterance, and an AI system including the display devicewill be described below. In this regard,is a flowchart illustrating an operation between each module of the display deviceaccording to the present disclosure, which displays the customized recommendation utterance, and the server.

1 4 FIGS.to 100 100 150 170 180 With reference to, the display deviceaccording to the present disclosure, which operates in conjunction with utterance data, is described. The display devicemay be configured to include the user input interface unit, the processor, and the display.

150 200 150 200 200 202 200 202 202 The user input interface unitmay be configured to receive a first input associated with a recommendation utterance request by operating in conjunction with the remote control device. The user input interface unitmay be configured to receive a second input associated with the recognition of the utterance data of the user by operating in conjunction with the remote control device. In this regard, the first input and the second input may correspond to inputs to specific buttons of the remote control device. For example, the first input and the second input may correspond to different types of inputs to an AI shortcut keyof the remote control device. The first input may correspond to a short press, lasting for a first time shorter than a threshold time of the AI shortcut key. The second input may correspond to a long press, lasting for a second time equal to or longer than the threshold time of the AI shortcut key.

180 170 180 The displaymay be configured to display either information or a recommendation utterance, each of which is associated with the utterance data. The processormay be configured to enable the displayto display a keyword associated with either the utterance data or previous utterance data of the user.

170 200 200 202 202 170 170 3 FIG. The processormay be configured to generate the recommendation utterance request when receiving a key value corresponding to a button of the remote control device. In this regard, the button of the remote control devicemay correspond to the AI short keyillustrated in. When the AI shortcut keyis pressed for the first time shorter than the threshold time, the processormay recognize this press as a first input for a short press. The processormay activate an AI concierge mode that recommends customized keywords and features in response to the first input.

202 170 170 When the AI shortcut keyis pressed for the second equal to or longer than the threshold time, the processorcan recognize this press as a second input for a long press and hold. The processormay activate an AI voice recognition mode that recognizes the utterance data (voice data) of the user in response to the second input.

170 300 100 In the AI concierge mode, the processormay acquire a history of the previous utterance data of the user and user information in order to receive the recommendation utterance request from the server. In this regard, the history of the utterance data is not limited to utterance data based on the user's voice recognized through the display device. The history of the utterance data may include a history of utterance data regarding other electronic devices. The history of the utterance data may include a history of utterance data regarding voice-recognizable home appliances or mobile terminals.

170 100 400 100 400 100 100 170 100 100 100 140 Accordingly, the processormay acquire the user's voice recognized through the display device, as well as a history of utterance data of an apparatusconnected to the display deviceand user information related to the utterance data. The apparatusesconnected to the display devicemay be a plurality of home appliances that operate in conjunction with the display device. In addition, the processormay acquire a history of utterance data of an electronic device registered through the display device, which is a user terminal, for example, a mobile terminal, and user information related to the utterance data. In this regard, the display devicemay operate as a home hub that controls the plurality of home appliances. Therefore, respective histories of utterance data of a plurality of home appliances that operate in conjunction with the display devicemay be stored in the database.

170 140 170 300 The processormay request the history of the previous utterance data from the databasein response to the recommendation utterance request. The processormay request a recommendation utterance from the serveron the basis of the history of the utterance data and account information of the user.

170 300 170 180 The processormay be configured to receive the recommendation utterance from the server. The processormay control the displayto output the received recommendation utterance that includes a keyword associated with the previous utterance data.

170 171 172 171 170 172 171 The processormay be configured to include a recommendation utterance output moduleand a history management module. In this regard, the recommendation utterance output modulemay be implemented as an AI concierge application program that outputs a recommendation utterance on a screen of the display. The history management modulemanages historical information associated with the user's previous utterance data, receives a recommendation utterance that varies according to the historical information and a user account, and transfers the received recommendation utterance to the recommendation utterance output module.

172 100 1200 1000 100 1200 100 300 1200 140 340 300 The history management moduleis not limited to being implemented in the display deviceand may be modified depending on the application. A history management module, which manages the historical information associated with the user's previous utterance data in the AI system, may be implemented as an external entity separate from the display device. The history management modulemay be implemented in an electronic device, such as a mobile terminal, a plurality of home appliances that operate in conjunction with the display device, or the server. The history management modulemay be implemented in a mobile terminal, electronic appliances, the database, the database, the servers, other databases, other servers, integration servers, or other equivalent devices.

202 200 171 10 172 171 140 172 140 172 300 171 When a button (e.g., the AI shortcut key) on the remote control deviceis pressed, the recommendation utterance output modulemay be configured to execute an application program, thereby outputting a recommendation utterance. The history management modulemay receive the recommendation utterance request from the recommendation utterance output moduleand transfer the recommendation utterance request to the database. The history management modulemay be configured to receive the history of the utterance data from the database. The history management modulemay transfer a recommendation utterance, received from the server, to the recommendation utterance output module.

202 200 200 171 10 171 172 20 172 140 30 140 172 In this regard, when the first input is applied to the AI shortcutof the remote control device, an execution input may be transferred from the remote control deviceto the recommendation utterance output module(S). The recommendation utterance output modulemay deliver the recommendation utterance request to the history management module () (S). The history management modulemay request the historical information stored in the database(S). In response to a request for the stored historical information, the databasemay transfer historical information, selected from among pieces of stored historical information, to the history management module. In this regard, the selected historical information may be historical information that varies according to an application program being currently being executed in the foreground and a login-required account (user account). For example, the selected historical information may be historical information about the previous utterance data for a specific application program that is currently being executed by a currently logged-in user, among a plurality of users.

172 300 50 300 340 300 172 60 172 300 171 70 171 180 172 80 The history management modulemay transfer the recommendation utterance request, including user history and account information, to the server(S). In response to the recommendation utterance request, the servermay select or generate a specific recommendation utterance from among recommendation utterances stored in the database. The servermay transfer a recommendation utterance, selected or generated on the basis of the user history and the account information, to the history management module(S). The history management modulemay transfer the recommendation utterance, received from the server, to the recommendation utterance output module(S). The recommendation utterance output modulemay output to a screen of the displaythe recommendation utterance including keywords, the recommendation utterance being received from the history management module(S). In this regard, a sentence (e.g., Ms. Ji-Won, please select an utterance you have frequently used over the past week) that includes the user's name may be displayed on the screen.

100 The sentence including the user's name may be displayed in a first region, and recommendation utterances, which are based on the user's utterance history, may be displayed in a second region. The recommendation utterances may be displayed with the keyword being combined with a hashtag. The recommendation utterances may be displayed such that a sentence including the keyword is displayed in combination with the hashtag. Respective keywords of the recommendation utterances may be first content and second content (a first program and a second program) that may be viewed through the display device. Additionally, the keyword of the recommendation utterance may be a sentence uttered by the user, for example, “What's the weather like tomorrow?”

100 100 100 Therefore, the display device according to the present disclosure, which displays the customized recommendation utterance, may be referred to as an AI concierge that performs an AI function. The AI concierge may provide a user segment or a situation-dependent customized greeting when the display deviceis powered on. In addition, while the display deviceis in use, a recommendation utterance, including user-specific and situation-based customized keywords, may be recommended according to the user's intention and the usage context. A recommendation utterance may be displayed on a screen of the display devicein order to recommend a setting, content, and similar items, as well as a keyword and a specific function.

100 200 100 300 5 FIG. The display deviceaccording to the present disclosure may be configured to recognize utterance data and a customized recommendation utterance through the remote control device. In this regard,is a flowchart illustrating an operation between each module of the display deviceaccording to the present disclosure, which is configured to recognize utterance data or display the customized recommendation utterance, and the server.

1 5 FIGS.to 100 With reference to, the display deviceaccording to the present disclosure, which operates in conjunction with the utterance data, is described.

100 200 300 2000 Regarding the recognition and loading of the utterance data, each constituent element of the display devicemay operate in conjunction with the remote control deviceand the server. In a state where the second input, having a duration equal to or longer than the threshold time, is applied to a specific button of the remote control device, the user may utter a first keyword of a first program. The first keyword of the first program may correspond to the first content “Three meals a day.”

175 200 2000 175 170 175 The utterance recognition processing moduleis configured to perform an utterance-recognition trigger operation to enable recognition of the user's voice through the remote control device, and to acquire utterance data corresponding to the user's voice in response to the utterance-recognition trigger operation. In a state where the second input, having a duration equal to or longer than the threshold time, is applied to a specific button of the remote control device, the voice recognition processing modulemay acquire the utterance “Three meals a day.” The processormay display the sentence “Here is a search result for ‘Three meals a day’” in conjunction with the voice recognition processing module.

170 170 180 The processormay display, on the screen, utterance data and the result of processing for analysis of the utterance data with respect to intention. The processormay display, on a first screen of the display, the sentence “Here is a search result for ‘Three meals a day,’” and a search result for text corresponding to the acquired utterance.

170 300 300 300 300 170 300 172 140 140 172 The processormay transfer the acquired utterance data to the server. The servermay perform processing for analysis of the transferred utterance data with respect to intention. The servermay perform natural language processing (NLP) to analyze the utterance data with respect to intention. The servermay perform processing for analysis of the utterance data with respect to intention through morphological analysis, syntactic analysis, semantic analysis, and discourse analysis of the utterance data. The processormay receive, from the server, the result of the processing for analysis of the utterance data with respect to the utterance data. The history management modulemay store, in the database, the utterance data that has undergone the processing for analysis with respect to intention. Historical information regarding the utterance data stored in the databasein conjunction with the history management modulemay include utterance data, a speaker's account, speech-to-text (STT) information, the date and time of an utterance, the corresponding day of the week, and temporal information.

150 202 200 150 171 171 173 171 173 171 173 The user input interface unitmay be configured to receive the first input that has a duration shorter than the threshold time of the AI shortcut keyof the remote control device. The user input interface unitmay notify the recommendation utterance output moduleof the reception of the first input. The recommendation utterance output modulemay operate in conjunction with an account management moduleto check information about a currently logged-in account. In this regard, the recommendation utterance output modulemay transfer a request for the information about the currently logged-in account to the account management module. In response to the request for the information about the currently logged-in account, the recommendation utterance output modulemay receive the information about the currently logged-in account from the account management moduleand identify the currently logged-in account.

171 140 172 171 180 The recommendation utterance output modulemay acquire the history of the utterance data stored in the databasein conjunction with the history management module. The recommendation utterance output modulemay generate a recommendation utterance including a specific keyword on the basis of the information about a login-required account and the history of the utterance data. The recommendation utterance, including a specific keyword, may be generated on the basis of utterances or keywords that are frequently used by the user. In this regard, a sentence including the user's name may be displayed on a second screen of the display. The sentence including the user's name may be displayed in the first region, and recommendation utterances, which are based on the user's utterance history, may be displayed in the second region.

100 100 100 6 FIG. The AI system, which includes the display deviceaccording to the present disclosure that operates in conjunction with utterance data, may receive, from a separate server, greetings to be provided to the user and may display the received greetings to the screen. In this regard,is a diagram illustrating the configuration of the AI system that includes the display deviceaccording to the present disclosure, the display devicereceiving, from a separate server, and displaying the received greetings to the screen.

3 6 FIGS.and 201 200 202 180 202 201 200 171 110 With reference to, a power buttonof the remote control devicemay be pressed in a powered-off state or may switch to a powered-on state according to a timer setting. When the AI shortcut keyis pressed within a predetermined time period after being switched to the powered-on state, a greeting may be displayed on the screen of the display. When the AI shortcut keyis pressed within a predetermined time period after the power buttonis pressed in the powered-off state, an execution input may be transferred from the remote control deviceto the recommendation utterance output module(S).

171 172 120 172 300 130 300 340 300 b b b b. The recommendation utterance output modulemay transfer the recommendation utterance request to the history management module(S). The history management modulemay transfer a greeting request to a second server, which performs greeting management such that a greeting is displayed in the powered-on state (S). When the greeting request is transferred, information regarding the time at which a switch to the powered-on state occurs, and information regarding the powered-on state and a number of power-ons may be transferred to the second server. User-specific or situation-specific greetings may be stored in a greeting databaseof the second server

300 300 140 300 300 300 150 300 340 b b b b The second servermay request user segment information from a UPP serverC (S). The user (customer) segment information refers to information that is obtained by classifying a plurality of users (customers) according to the user's gender, age, viewed content, and interests, using a segment (group) classification method in accordance with a predetermined reference. The UPP serverC may be configured to store and manage information regarding user preference profiles for each user. In response to a request for the user segment information, the UPP serverC may transfer the user segment information to the second server(S). The second servermay select a specific greeting from among the greetings stored in the databaseon the basis of the user segment information, the information regarding the time at which the switch to the powered-on state occurs, and the information regarding the powered-on state and a number of power-ons.

300 201 201 300 172 b b The greeting, including “morning,” may be selected on the basis of the information regarding the time at which the switch to the powered-on state occurs. The second servermay select a first greeting, a second greeting, a third greeting, or so forth, each different from the others, according to the number of times the power buttonis pressed and an interval between pressing and releasing the power button. In a case where the switch to the powered-on state first occurs in the morning, the second servermay select the greeting “Ms. Elgini, I am delighted to be with you this morning” on the basis of the user segment information and transfer the selected greeting to the history management module.

172 300 172 171 171 180 b The history management modulemay receive, from the second server, a specific greeting selected on the basis of the user segment information, the information regarding the time at which the switch to the powered-on state occurs, and the information regarding a number of power-ons. The history management modulemay transfer the selected specific greeting to the recommendation utterance output module. The recommendation utterance output modulemay output the transferred specific greeting to the screen of the display.

1 6 FIGS.to 202 200 171 172 172 300 300 172 171 171 180 b b With reference to, when the button (e.g., the AI shortcut key) of the remote control deviceis pressed, thereby causing reception of a key value corresponding to the pressed button, the recommendation utterance output modulemay transfer the recommendation utterance request to the history management module. The history management modulemay transfer the greeting request to the second serveron the basis of the information regarding the time at which the switch to the powered-on state occurs and a number of power-ons, thereby receiving a greeting from the second server. The history management modulemay transfer the received greeting to the recommendation utterance output module. The recommendation utterance output modulemay output a screen, including the received greeting, on the display.

100 300 100 7 FIG. In the AI system that includes both the display deviceaccording to the present disclosure, which operates in conjunction with utterance data, and the server, recommendation phrases may be output based on time when the switch to the powered-on state occurs. In this regard, (a), (b), and (c) ofare views, each illustrating a screen that is output on the display deviceover the passage of time in the powered-on state.

6 FIG. 7 FIG. 180 100 180 180 With reference toand (a) of, a greeting may be output on the displayof the display devicein the powered-on state. In this regard, a greeting may be output in a portion of the lower region of the displayin a state where a home screen is fully displayed on the display.

6 FIG. 7 FIG. 180 180 180 180 With reference toand (a) and (b) of, in a state where the greeting is output, when a predetermined time period elapses without any user input, recommendation phrases based on the most recent search history or viewing history may be output on the display. In the state where the greeting is output, when no input is applied, no content application is executed, or no voice command is executed, recommendation speeds or phrases based on the most recent search history or viewing history may be output. For example, based on the recent search history, the recommendation utterance “# Search YouTube for a review of Top Gun” may be displayed with the highest priority in a portion of the lower region of the display. Based on the recent search history, the recommendation utterance “# Turn on the air purifier” may be displayed with the second-highest priority in the portion of the lower region of the display. In addition, based on the recent search history, the content “# Yonhap News” or a recommendation phrase associated with the channel may be displayed with the third-highest priority in the portion of the lower region of the display.

6 FIG. 7 FIG. 180 180 180 180 With reference toand (b) and (c) of, after another predetermined period elapses in a state where the recommendation utterance based on the most recent search or viewing history is output, the user's search or viewing history may be extracted on a per-day, per-week, or per-month basis. Therefore, recommendation phrases, based on the user's search or viewing history on a per-day, per-week, or per-month basis, may be output on the display. For example, based on the search history for the past one day, the recommendation utterance “# What's the weather like tomorrow?” may be displayed with the highest priority in the portion of the lower region of the display. For example, based on the search history for the past one day, a recommendation utterance associated with the content “# Three meals a day” may be displayed with the second highest priority in the portion of the lower region of the display. In addition, based on the viewing history for the past one day, a recommendation phrase associated with the content “# Good Partner” may be displayed with the third-highest priority in the portion of the lower region of the display.

100 300 100 180 1 180 2 8 FIG. The AI system that includes both the display deviceaccording to the present disclosure, which operates in conjunction with utterance data, and the server, may be configured to analyze the user's intention through the utterance data. In this regard,is a diagram illustrating the configuration of the display deviceconfigured to store historical information regarding the analysis of the user's intention through the utterance data and historical information regarding content selected from a home screen-and an integration search screen-.

5 8 FIGS.and 175 300 300 175 172 172 140 With reference to, the voice recognition processing modulemay recognize the user's voice command “Search for a movie” and perform processing for analysis of the intention of the recognized voice in conjunction with the server. The result of the analysis of the intention of the recognized utterance may be transferred from the serveror the utterance recognition processing moduleto the history management module. The history management modulemay store, in the database, temporal information, a keyword, a genre code, an intention type, and execution information of an application on the basis of the result of the analysis of the intention. In response to the voice command “Find a movie,” the keyword “movie” may be extracted, and the intention type (intentType) may be designated as “search.”

180 1 180 200 172 180 2 180 200 172 180 1 180 2 172 140 First content information regarding first content selected from the home screen-of the displaythrough the remote control devicemay be transferred to the history management module. In addition, second content information regarding second content selected from the integration search screen-of the displaythrough the remote control devicemay be transferred to the history management module. Content information regarding content selected from the home screen-or the integration search screen-may be transferred to the history management moduleand stored in the database.

180 1 180 2 180 1 180 2 100 A keyword associated with the content selected from the home screen-or the integration search screen-may be selected as “PPORORO,” or “Baby Shark,” and the intention type (IntentType) may be designated as search. The intention type (IntentType) of a keyword selected through voice recognition and that of a keyword selected from the home search screen-or the integration search screen-may both be classified and designated as search. The keyword “Turn off the TV” may be extracted through voice recognition, and the intention type (IntentType) of the keyword “Turn off the TV” may be classified and designated as tv_control for powering off the display device.

180 1 100 100 Searching for movies and playing back the first content may be performed through a first application program, such as YouTube. In this regard, in a state where the first application program is executed in the foreground, a voice utterance may be received and recognized. From the home screen-, the first application program may be selected. The playback of the second content may be performed through a second application program, such as WebOS. The playback of the second content may be performed through the second application on a channel service that is operated by a manufacturer of the display device. Therefore, either the result of the analysis of the intention or history management may vary depending on an application that is currently being executed on the display device.

1 5 FIGS.to 8 FIG. 170 175 175 200 300 With reference toand, processes of loading either historical information regarding a voice utterance or historical information regarding content selected from many screens are described in detail below. In this regard, the processormay be configured to recognize and process voice data uttered by the user, in conjunction with the voice recognition processing module. The voice recognition processing modulemay be configured to transfer the recognized utterance data from the remote control deviceto the server.

202 200 175 300 175 180 When the second input, having a duration equal to or longer than the threshold time, is applied to the button (e.g., the AI shortcut key) of the remote control device, the voice recognition processing modulemay be configured to perform processing for analysis of the user's intention on the basis of the utterance data in conjunction with the server. The voice recognition processing modulemay control the display of a search result, which is associated with the recognized utterance data, on the screen of the displayon the basis of the result of the processing for analysis of the user's intention.

172 140 300 172 140 180 172 140 802 180 The history management modulemay be configured to store the history of the utterance data in the databaseon the result of the processing for analysis of the user's intention, the result being received from the server. The history management modulemay be configured to store a history of content information in the databasebased on the content information selected from the home screen of the display. The history management modulemay be configured to store a history of second content information in the databaseon the basis of the second content information selected from the integration search screenof the display.

140 140 100 The utterance data, the content information, and the second content information that are to be stored in the databasemay each include various types of information. Various types of information that are to be stored in the databasemay each include temporal information including a start time (FromTime) and an end time (ToTime), a keyword, a genre code (genreCode), an intention type (IntentType), and information on an application (App) being executed on the display device.

100 100 The intention type (IntentType) may include an operation (tv_control) associated with the voice utterance, the search from the home screen or the integration search screen, or control of the display deviceon an application associated with WebOS-based live TV. The operation (tv_control) associated with the control may be associated with a voice utterance associated with the powering-off of the display deviceon the application associated with the live TV, or with the playback of specific content. The keyword may be at least one of the following: a keyword derived from the utterance data of the user, the content, or the name of the second content.

175 140 202 200 170 140 170 170 180 The voice recognition processing modulemay control the storage of the history of the utterance data in the database. The history of the utterance data may include the user's account, an STT result, the date and time of an utterance, the corresponding day of the week, and temporal information. In this regard, when the button (e.g., the AI shortcut key) of the remote control deviceis pressed, the processormay extract the information about the currently logged-in account through the database. The processormay extract keyword information associated with an utterance used by the user, on the basis of the current day of the week and temporal information. The processormay control the output of the received recommendation utterance, including the keyword, on the displayon the basis of the keyword information.

100 9 FIG.A 9 FIG.B The display deviceaccording to the present disclosure, which displays the customized recommendation utterance, may be configured to switch among login-required accounts of users on the basis of voiceprint data of the user who is uttering. In this regard,is a view illustrating an initial screen that is displayed when a specific user logs into their login-required account.is a view illustrating a screen for changing profiles of a plurality of users.

8 9 FIGS.andA 183 180 1 180 183 180 1 180 180 a b With reference to, an iconassociated with a first user's first login-required account may be displayed in a portion of the home screen-of the display. An iconthat enables the execution of the second application program, such as WebOS, may be displayed in another portion of the home screen-of the display. Icons corresponding to services provided by a plurality of content servers may be displayed on the displayin a distinctively categorized manner. The icons corresponding to the services provided by the plurality of content servers may be displayed in mutually distinct categories including home office, gaming, music, home hub, and sports.

9 FIG.B 184 184 184 184 180 183 184 184 184 184 184 185 184 a b c d a a b c d a a a With reference to, icons,,, and, which are associated with respective user profiles of a plurality of users, may be in specific regions of the display. Depending on a selection input to the iconassociated with the first login-required account, the icons,,, and, each representing a respective login-required account of a family member, may be displayed on the screen. The iconassociated with the first user's first login-required account may be displayed on the screen, the first user corresponding to Dad. For example, the first user's first login-required account may be aaa@gmail.com. An iconfor logging out from the first user's first login-required account may be displayed in the lower portion of the iconassociated with the first user's profile.

184 184 184 b c d The iconassociated with a second user's second login-required account may be displayed on the screen, the second user corresponding to Chloe. For example, the second user's second login-required account may be bbb@gmail.com. The iconassociated with a third user's third login-required account may be displayed on the screen, the third user corresponding to Mom. In addition, the iconfor generating a fourth user's fourth login-required account may be displayed on the screen.

1 FIG. 8 9 FIGS.toA 100 With reference toand, a configuration and operation of the display devicethat displays the customized recommendation utterance and switches among the users'login-required accounts on the basis of voiceprint data of the user who is uttering are described.

175 300 170 170 9 FIG.B 9 FIG.B The voice recognition processing modulemay determine, in conjunction with the server, whether the recognized utterance data is voiceprint data of the second user other than the user in a logged-in state. When the second user is a registered user, the processormay display an account switch screen in, which is associated with whether to perform a switch from the user's login-required account to the second user's second login-required account. When a switch input for the switch from the account switch screen into the second login-required account is received, the processormay switch to the second login-required account.

185 184 185 a a a A first selection boxassociated with logging out of the first login-required account may be displayed under the first iconfor the first user's first login-required account, the first user corresponding to Dad. When the first selection boxis selected, the first login-required account may be switched to a logged-out state, and the second user's second login-required account may be automatically switched to a logged-in state.

185 184 185 b b b A second selection boxassociated with logging into the second login-required account may be displayed under the second iconfor the second user's second login-required account, the second user corresponding to Chloe. When the second selection boxis selected, a switch to the second user's second login-required account may be performed, and the first login-required account may be automatically switched to a logged-out state.

100 100 170 170 300 3 9 FIGS.toB The display deviceaccording to this disclosure may recognize the user and display a recommendation utterance customized on a per-user basis. With reference to, the display deviceaccording to the present disclosure, which displays the recommendation utterance customized on a per-user basis, is described. The processormay detect whether the first user is an authenticated user in a state where the first user's first login-required account, among a plurality of user accounts, is logged in. When the first user is detected, the processormay request a first recommendation utterance from the serveron the basis of both a first history of the first user's first utterance data and first account information. In this regard, a first identifier of the first user's first device and the first user's voiceprint data or face may be detected, thereby enabling detection of the first user.

170 170 300 The processormay detect whether the second user is an authenticated user in a state where the second user's second login-required account, among the plurality of user accounts, is logged in. When the second user is detected, the processormay request a second recommendation utterance from the serveron the basis of both a second history of second utterance data of the second user and second account information of the second user. In this regard, a second identifier of the second user's second device and the second user's voiceprint data or face may be detected, thereby enabling detection of the second user.

170 170 170 300 The processormay determine whether the second user is detected instead of the first user in a state where logging into the first login-required account is performed. When the second user is detected instead of the first user, the processormay display a screen for determining whether to switch to the second user's second login-required account. When the second user is detected instead of the first user, the processormay request the second recommendation utterance from the serveron the basis of the second history and second account information.

170 170 300 170 The processormay determine whether the first user and the second user are detected in a state where logging into the first login-required account is performed. When the first and second users are detected, the processormay request the first and second recommendation utterances from the serveron the basis of the first history, the second history, the first account information, and the second account information. In this regard, the processormay determine, through user profile information, whether the second user is registered as having a familial relationship with the first user.

170 180 170 170 When the second user is registered as having a familial relationship with the first user, the processormay display a common recommendation utterance on the displayon the basis of a keyword common to the first recommendation utterance and the second recommendation utterance. The processormay display the first recommendation utterance after the common recommendation utterance on the basis of the first user's first keyword. The processormay display the second recommendation utterance after the first recommendation utterance on the basis of the second user's second keyword.

170 180 170 170 When the first and second users are detected and the second user is not registered as having a familial relationship with the first user, the processormay display the common recommendation utterance on the displayon the basis of the keyword common to the first recommendation utterance and the second recommendation utterance. The processormay determine, on the basis of a first user input, whether to display the first recommendation utterance on the basis of the first user's first keyword. When the first user input is received, the processormay display the first recommendation utterance on the screen after the common recommendation utterance.

100 300 202 180 202 200 170 10 FIG. 3 6 10 FIGS.,, and The AI system that includes both the display device, according to the present disclosure, which operates in conjunction with the utterance data, and the server, may provide a recommendation service customized based on a user or situation, and a usage tip service. In this regard,is a diagram illustrating a recommendation service, according to an embodiment, which is customized based on the user or situation. With reference to, when the AI shortcut keyis pressed within a predetermined time period after the switch to the powered-on state, a greeting may be displayed on the screen of the display. When the AI shortcut keyis pressed, current login-acquired account information, the current time, and user access record information may be transferred from the remote control deviceto the processor.

300 300 300 300 300 170 300 180 180 180 b b b b The second servermay transfer the login-acquired account information to a UPP serverC. The UPP serverC may acquire customer segment information on the basis of the login-acquired account information and transfer the acquired customer segment information to the second server. The second servermay generate and recommend a welcome message that varies depending on the current time, the customer segment information, and the user access record information. The processormay transfer the welcome message received from the second serverto the displayand control the displaysuch that the welcome message is output on the screen. Therefore, the welcome message, “Ms. Elgini, I am delighted to be with you this morning,” may be displayed on the screen of the display.

100 170 170 180 1 6 FIGS.to 6 FIG. The display devicemay display a greeting, that is, a welcome message that varies depending on the time at which the switch to the powered-on state occurs and a number of power-ons. With reference to, the processordetermines whether the switch from the powered-off state to the powered-on state first occurs on a per-date basis. When the switch to the powered-on state first occurs, the processormay output a screen including a first greeting on the displayon the basis of temporal information. When the switch to the powered-on state first occurs, the first greeting may be generated in the form of the sentence “Ms. Elgini, I am delighted to be with you this morning,” as illustrated in.

170 180 10 FIG. When the switch to the powered-on state occurs two or more times, the processormay display, on the screen, a screen including a second greeting different from the first greeting. When the switch to the powered-on state occurs two or more times, the second greeting on temporal information may be generated in the form of the sentence “Ms. Elgini, I am delighted to be again with you this morning,” as illustrated in.

4 5 FIGS.and 1 5 FIGS.to 170 When a predetermined time period elapses in a state while the first or second greeting is displayed in the powered-on state, recommendation utterances inmay be displayed on the screen. With reference to, the processordisplays, on the screen, a sentence that includes the user's name (e.g., Ms. Ji-Won, please select an utterance that is frequently used during the past week). The sentence including the user's name may be displayed in the first region, and recommendation utterances, which are based on the user's utterance history, may be displayed in the second region.

170 170 100 In this regard, when a first threshold time elapses in the powered-on state, the processormay extract keywords according to utterance frequency derived from the history of the utterance data for each corresponding week. The processormay display the extracted keywords in order of the utterance frequency. The recommendation utterances may be displayed with the keyword being combined with a hashtag. The recommendation utterances may be displayed such that a sentence including the keyword is displayed in combination with the hashtag. Respective keywords of the recommendation utterances may be the first content and the second content (the first program and the second program) that may be viewed through the display device. Additionally, the keyword of the recommendation utterance may be a sentence uttered by the user, for example, “What's the weather like tomorrow?”

1 10 FIGS.to 170 300 300 300 300 300 170 170 180 b b c c b With reference to, the processormay transfer the information about the currently logged-in account, the current time, and the user's access record to the second server. The second servermay transfer the account information to the UPP serverand then receive the customer segment information from the UPP server. The second servermay transfer a welcome message, which is generated on the basis of the current time, the customer segment information, and the user's access history, to the processor. The processormay control the output of the welcome message on the display.

170 100 100 Therefore, the processormay provide a situation-customized recommendation function. In this regard, when the display deviceis powered on, a situation-customized welcome greeting may be provided based on a time span of the powered-on state, the frequency of powering on or powering off, an interval between powering on and powering off, and segment information for each time span during which the display deviceis most used.

100 100 In addition, a function that is performable immediately in the powered-on state of the display devicemay be preferentially recommended. While the display deviceis in use, a keyword most suitable for a corresponding situation may be recommended by analyzing an application in use or content being viewed. Another application or setting having a high correlation may be proposed as a usage tip, according to the characteristics of the application currently in use. Through the title, genre, and metadata information of the content being watched, association keywords associated with viewing, such as cast, director, viewership ratings, and OST, may be provided.

100 100 202 180 202 170 300 170 11 FIG. 3 11 FIGS.and b The display device, according to the present disclosure, that displays the customized recommendation utterance, and the AI system, including the display device, may provide a usage tip service customized based on the user or situation. In this regard,is a diagram illustrating a configuration, according to an embodiment, which provides the usage tip service customized based on the user or a situation. With reference to, when the AI shortcut keyis pressed, a usage tip phrase that is obtained by considering a program being viewed and an application in use may be displayed on the screen of the display. When the AI shortcut keyis pressed, the processormay transfer the current time, information about the connected apparatus, information about a program currently being viewed, and information about an application currently in use to the second server. In this regard, the processormay receive usage tip information selected based on the identifier of the connected device and state information thereof.

300 170 300 b b. The second servermay generate and recommend the usage tip information customized on the basis of the current time, the information about the connected apparatus, the information about the program currently being viewed, and the information about the application currently in use. The processormay receive the customized usage tip information or a usage tip keyword associated therewith from the second server

170 300 180 180 180 b The processormay transfer the usage tip keyword received from the second serverto the displayand control the displaysuch that the usage tip keyword is output on the screen. Therefore, the usage tip information items “Please find a Top Gun review on YouTube,” “Please turn off the air purifier,” and “What's the weather like tomorrow?” may be displayed on the screen of the display.

100 180 In this regard, when the first application program, YouTube, is currently being executed in the foreground on the display device, the user tip information “Find Top Gun review on YouTube” may be displayed on the screen of the display.

100 180 100 100 12 12 FIGS.A andB When the display deviceis connected to an air purifier, the usage tip information “Please turn off the air purifier” may be displayed on the screen of the display. In this regard,are views, each illustrating the screen of the display devicethat displays state information of an apparatus connected to the display device.

12 12 FIGS.A andB 180 100 180 With reference to, state information items of an air purifier, a washing machine, a dryer, and a refrigerator may be displayed on the displayof the display device. In addition, state information items of a light, a switch, and blinds may be displayed on the display.

12 FIG.A 12 FIG.B 11 FIG. 400 100 180 400 400 100 180 400 180 a b With reference to, the air purifier may be one of the apparatusesconnected to the display device. As displayed on a control screen of a home hub of the display, the air purifier may be in an operating state. With reference to, the operating states of the apparatusesconnected to the display devicemay be changed. As displayed on the control screen of the home hub of the display, the air purifier may be in an off state. In this regard, as illustrated in, the user tip information “Turn off the air purifier” may be selected from the screen of the display.

11 FIG. 12 12 FIGS.A andB 202 200 202 400 100 202 200 180 400 100 400 400 a b. As illustrated in, when the AI shortcut keyof the remote control deviceis pressed, a user-customized menu, such as a usage tip, may be provided either on the screen or via voice. Alternatively, user voice may be recognized depending on the manner in which the AI shortcut keyis pressed. Therefore, among the apparatusesconnected to the display device, the operation of an apparatus that may cause interference with voice recognition or audio output may be temporarily suspended. Therefore, when the AI shortcut keyof the remote control deviceis pressed, the usage tip information “Please turn off the air purifier” may be displayed on the screen of the display. Accordingly, when the user tip information “Please turn off the air purifier” is selected, as illustrated in, the air purifier, among the apparatusesconnected to the display device, may switch from the operating stateto the off state

180 400 11 FIG. 12 FIG.B b. In this regard, when a user input for selecting the usage tip information “Please turn off the air purifier” is applied, the displaymay switch from the screen into the screen in. A screen switch and a state switch are not limited to being triggered by the user input. When a cancellation input is not received within a predetermined time period, the air purifier may also automatically switch to the off state

180 180 The usage tip information “Please turn off the air purifier” may be displayed on the screen of the displayon the basis of state information of the air purifier. In this regard, when it is determined that the air purifier has been operating for a predetermined time period and that the current concentration of fine particles in the indoor air is equal to or lower than a threshold value, the state information of the air purifier and an air purification state may be determined to be normal. When the state information of the air purifier and the air purification state are normal, the usage tip information “Please turn off the air purifier” may be displayed on the screen of the display.

180 180 11 FIG. 12 FIG.A When the air purifier is in an off state and the concentration of fine particles in the indoor air is determined to be higher than the threshold value, the usage tip information “Please turn on the air purifier” may also be displayed on the screen of the display. When the usage tip information “Please turn on the air purifier” is selected, the screen of the displaymay switch from the screen into the screen in.

1 12 FIGS.toB 170 202 200 220 170 300 170 300 170 180 b b With reference to, the processormay detect whether the button (e.g., the AI shortcut key) of the remote control deviceis pressed. When the first input through the AI shortcut keyis detected, the processormay transfer, to the second server, the current time, the information about the connected apparatus, the information about the program currently being viewed, and the information about the application currently in use. The processormay receive, from the second server, the usage tip information customized on the basis of the current time, the information about the connected apparatus, the information about the program currently being viewed, and the information about the application currently in use. The processormay control the output of at least one sentence associated with a usage tip on the displayon the basis of the customized usage tip information.

The customized usage tip information and the recommendation utterance may be provided on the basis of a user-customized keyword for each account. Regarding the user-customized keyword for each account, user data may be managed based on the user account, and, during a voice search, a preference for each account may be learned through voiceprint recognition. The user-customized keyword may be recommended through integrated inference from user data, the user data including: a history of voice utterances; content for viewing; TV settings, data relating to home IoT apparatus connected in a household; and preference segments classified according to usage patterns. User preference may be analyzed based on the user data accumulated on a per-day or per-time basis, and the recent information of interest may be continuously updated based on the latest data. With respect to content, to provide a recommendation based on the user's preferred genre, an optimal recommendation may be generated by utilizing content preference information of a cluster of users who view the same content.

100 300 13 FIG. The AI system that includes both the display device, according to the present disclosure, that operates in conjunction with utterance data, and the servermay be configured to provide an association keyword recommendation service based on broadcast information. In this regard,is a diagram illustrating the configuration of an AI system, according to an embodiment, that provides a service for recommending an association keyword based on the broadcast information.

13 FIG. 100 300 500 600 100 170 180 With reference to, the AI system may be configured to include the display device, the server, the third serverfor extracting an association keyword, and the fourth serverassociated with an integrated broadcasting information system (IBIS). The display devicemay include the processorthat requests and receives information in conjunction with the servers, and the displaythat displays the association keyword.

170 300 140 210 300 140 100 220 170 300 140 300 140 100 The processormay request broadcast information about a program to be viewed from the serveror the database(S). The broadcast information received from the serveror the databasemay be transferred to the display device(S). The processormay receive the broadcast information from the serveror the database. In this regard, an entity that manages the broadcast information of the viewed program may be the serveror the databaseof the display device.

170 500 230 500 600 140 240 600 140 500 250 500 100 260 170 180 180 The processormay transfer the broadcast information to the third serverthat extracts the association keyword (S). The third servermay request detailed program information from the fourth serverassociated with the IBIS or from the database(S). The fourth serveror the databasemay transfer the detailed program information to the third server(S). The third servermay transfer an association keyword based on the broadcast information, the association keyword being extracted on the basis of the detailed program information, to the display device(S). The processormay transfer the association keyword based on the broadcast information to the display(or a screen output unit) such that the association keyword is output on the display.

1 13 FIGS.to 170 300 170 500 500 600 500 600 170 180 With reference to, the processormay be configured to receive broadcast information, associated with a program to be viewed by the user, from the server. The processormay be configured to transfer the broadcast information to the third serverthat extracts an association keyword related to the broadcast information. The third servermay request the detailed program information associated with the broadcast information from the fourth serverof the integrated broadcasting information system. In response to the request, the third servermay be configured to receive the detailed program information associated with the broadcast information from the fourth server. The processormay be configured to output the association keyword related to the broadcast information on the displayon the basis of the detailed program information.

100 100 100 100 100 100 170 100 14 FIG. 15 FIG. 1 15 FIGS.to A method of displaying the customized recommendation speed, the method being performed by the display deviceaccording to the present disclosure and by the AI system including the display device, is described. In this regard,is a flowchart illustrating the method of displaying the customized recommendation utterance, the method being performed by the display deviceand by the AI system including the display device.is a flowchart illustrating a method of displaying an association search result based on the analysis of the user's intention identified through voice recognition, the method being performed by the display deviceand by the AI system including the display device. With reference to, the method of displaying the result of the association search may be performed by the processorof the display device.

220 200 10 170 171 172 20 172 140 30 172 140 40 When the button (the AI shortcut key) of the remote control deviceis pressed for the first time shorter than the threshold time, and thus the first input is applied (S), the processormay generate the recommendation utterance request. The recommendation utterance output modulemay transfer the generated recommendation utterance request to the history management module(S). In response to the recommendation utterance request, the history management modulemay request the history of the previous utterance data from the database(S). The history management modulemay receive the history of the previous utterance data from the database(S).

172 300 50 172 300 60 172 171 70 171 180 The history management modulemay request the recommendation utterance from the serveron the basis of the history of the utterance data and the account information of the user (S). The history management modulemay receive the recommendation utterance from the server(S). The history management modulemay transfer the received recommendation utterance to the recommendation utterance output module(S). The recommendation utterance output modulemay control the display of the received recommendation utterance, including a keyword, on the screen of the display.

202 200 10 170 175 175 300 20 b b When the button (the AI shortcut key) of the remote control deviceis pressed for the second time equal to or longer than the threshold time, and thus the second input is applied (S), the processormay recognize the utterance data (voice) of the user through the voice recognition processing module. The voice recognition processing modulemay transfer the recognized utterance data to the server(S).

175 171 300 30 171 180 40 175 171 140 172 50 172 30 140 172 40 b b b The voice recognition processing moduleand the recommendation utterance output modulemay receive, from the server, the result of the processing for analysis of the user's intention associated with the recognized utterance data (S). The recommendation utterance output modulemay control the display of a search result, associated with the recognized utterance data, on the screen of the displayon the basis of the result of the processing for analysis of the user's intention (S). The voice recognition processing moduleand the recommendation utterance output modulemay control the loading of the utterance data onto the databasethrough the history management module(S). The history management modulemay request the history of the utterance data (S). In response to this request, the history of the utterance data stored in the databasemay be transferred to the history management module(S).

100 1 12 FIGS.to The method of displaying the result of the association search, according to another aspect of the present disclosure, the method being performed by the display devicethat displays the customized recommendation utterance, is not limited to the foregoing description, and may be variously applicable in combination with embodiments in.

100 100 The display device, according to the present disclosure, that displays the customized recommendation utterance and the AI system that includes the display devicehave been described above. The technical effects of the display device, according to the present disclosure, that displays the customized recommendation utterance and of the AI system, including the display device, may be summarized as follows, but are not limited to this summary.

According to the present disclosure, a recommendation customized for each user can be provided by managing usage data for each account. Furthermore, user identification through voiceprint recognition enables data learning for each user solely through utterance without the need for a separate login process.

According to the present disclosure, the customized recommendation can be immediately received by pressing an AI key at any time and in any location. Enhancing recommendation accuracy through advanced logic can significantly reduce search time, thereby enabling execution of a desired function or access to desired content.

According to the present disclosure, either a setting or association information optimized for content currently being viewed can be provided. Furthermore, a usage tip for enhancing the usefulness of an application currently in use can be provided, thereby making the TV viewing experience more immersive and enjoyable.

The display device, according to the present disclosure, that displays a customized recommendation utterance may be referred to as an AI concierge that performs an AI function. The AI concierge can provide a user segment or a situation-dependent customized greeting when the display device is powered on. In addition, while the display device is in use, the recommendation utterance, including user-specific and situation-based customized keywords, can be recommended according to the user's intention and the usage context. The recommendation utterance can be displayed on a screen of the display device in order to recommend a setting, content, and similar items, as well as a keyword and a specific function.

Regarding a user-customized keyword for each account, the user data can be managed based on a user account, and, during a voice search, a preference for each account may be learned through voiceprint recognition. The user-customized keyword may be recommended through integrated inference from the user data including: a history of voice utterances; content for viewing; TV settings, data relating to home IoT apparatus connected in a household; and preference segments classified according to usage patterns. User preference can be analyzed based on the user data accumulated on a per-day or per-time basis, and the recent information of interest may be continuously updated based on the latest data. With respect to content, to provide a recommendation based on the user's preferred genre, an optimal recommendation can be generated by utilizing content preference information of a cluster of users who view the same content.

In this regard, when the display device is powered on, a situation-customized welcome greeting can be provided based on a time span of a powered-on state, the frequency of powering on or powering off, an interval between powering on and powering off, and segment information for each time span during which the display device is most used. In addition, a function that is performable immediately in the powered-on state of the display device can be preferentially recommended. While the display device is in use, a keyword most suitable for a corresponding situation can be recommended by analyzing an application in use or content being viewed. Another application or setting having a high correlation can be proposed as a usage tip, according to the characteristics of the application currently in use. Through the title, genre, and metadata information of the content being watched, association keywords associated with viewing, such as cast, director, viewership ratings, and OST, can be provided.

The present disclosure may be embodied in the form of computer-readable code on a medium on which a program is recorded. The computer-readable media include all types of recording devices capable of storing data that are readable by a computer system. Examples of media readable by computers include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the examples encompass forms implemented as carrier waves (e.g., transmission via the Internet). In addition, the computer may include a control unit of a terminal. Therefore, the detailed description should not be interpreted in a limited manner in all respects and should be considered as serving the purpose of illustration. The scope of the present disclosure should be determined by the proper interpretation of the following claims. All equivalent modifications to the embodiments of the present disclosure fall within the scope of the present disclosure.