Systems and Methods for a Machine Learning Adaptable User Interface

This application is a Continuation of U.S. application Ser. No. 18/478,337, filed Sep. 29, 2023, the entirety of which is hereby incorporated herein by reference.

The present disclosure generally relates to an artificial intelligence and machine learning system for adapting a user interface to user inputs, and more particularly to systems and methods for using artificial intelligence and machine learning to provide a user interface adapting to a user's neurological condition, user experience, and emotional state.

When designing user interfaces, developers generally cater the user interface to the ability and skill of the average expected user. By setting the user interface to the ability and skill of the average expected user, a user with neurological conditions (e.g., autism, attention deficit hyperactivity disorder, etc.) may be unfairly disadvantaged when operating the user interface because of his or her difference in ability of memory or concentration. Users with neurological conditions may require additional assistance, or additional tools to accomplish the same tasks on a user interface as someone with neurotypical ability. Current user interfaces are also not adaptable to a user's emotional state, such as adapting when the user is frustrated or confused.

According to certain embodiments, a method for providing an adaptable user interface for users based on the user's neurological condition, emotional state, and experience is provided. The method includes: selecting a neurological condition associated with a user; determining, based on historical user data, a user experience level; generating a user interface based on the selected neurological condition and the user experience level; receiving, from a tracer configured to log a user's activity, a tracking log comprising information regarding the user's activity; determining a behavioral metric by analyzing the tracking log using a machine learning model trained by processing prior user activity, wherein the behavioral metric represents an emotional state of the user; and modifying one or more elements of the user interface based on the behavioral metric.

In a further embodiment, the neurological condition is selected from a list of neurological conditions comprising one or more of autism, Parkinson's disease, epilepsy, and attention-deficit hyperactivity disorder.

In an additional embodiment, the historical user data includes a measurement of time the user has operated an application.

In another embodiment, the tracking log includes one or more of: mouse movements, mouse hover, periods of mouse inactivity, typing speed, spelling errors, abandoned sessions, and pageviews.

In a further embodiment, the method for providing an adaptable user interface further includes recording a user facial expression; and evaluating the user facial expression, using a machine learning model trained by processing user facial expressions, to update the behavioral metric.

In an additional embodiment, the method for providing an adaptable user interface further includes recording a user's voice; and evaluating the user's voice, using a machine learning model trained by processing user voice for tone, to update the behavioral metric.

In another embodiment, each neurological condition of the list of neurological conditions is associated with a template user interface; and the user interface is generated using the template.

The above methods can be implemented as computer-executable program instructions stored in a non-transitory, tangible computer-readable media and/or operating within a processor or other processing device and memory.

Reference will now be made in detail to various and alternative illustrative examples and to the accompanying drawings. Each example is provided by way of explanation, and not as a limitation. It will be apparent to those skilled in the art that modifications and variations can be made. For instance, features illustrated or described as part of one example may be used in another example to yield a still further example. Thus, it is intended that this disclosure include modifications and variations as come within the scope of the appended claims and their equivalents.

In one illustrative embodiment, an adaptable user interface system comprises an application for generating a user interface that adapts to a user's neurological condition, experience level, and emotional state. The application receives an input including a selection of a user's neurological condition and uses user inputs from past and current operation of the user interface to determine a user's experience level and a user's emotional state. The system may execute the application on a user device or in cloud service provider infrastructure. Further, in some embodiments, the application may be a microservice.

The user's neurological condition may include various conditions, disorders, and cognitive impairing diseases, such as autism, attention-deficit hyperactivity disorder, or Parkinson's disease. The neurological conditions may further include conditions from the Diagnostic and Statistical Manual of Mental Disorders such as DSM-5-TR or another edition of the manual.

User interfaces include various elements providing visual representations of information and allowing for user input to and navigation through the user interface. For example, elements of the user interface may include input fields, navigational components, and informational components such as visual representations (e.g., graphics, images, visuals) of information.

The adaptable user interface system includes a repository or database including a list of neurological conditions. Each neurological condition is associated with a user interface template. The user interface template may vary in appearance and function based on the neurological condition. For example, the adaptable user interface system may associate a neurological condition of post-traumatic stress disorder with a user interface template including graphics with muted and less vibrant colors. The user interface template may also use slow transitions between moving elements or adjust advertisement permissions of the user interface to prevent potentially triggering content. The user interface may also lower the default volume of actions or remove options from the user interface. In another example, for a user with Parkinson's disease, the user interface may disable motion controls to prevent unintended triggering of motion commands during tremors. The adaptable user interface system may also adjust the size of text font and buttons to allow users lacking precise movement to interact with the user interface more easily.

The adaptable user interface system further adapts the user interface by adding, removing, or modifying functions and changing the appearance of the user interface based on the user's experience level and emotional state. The application may use a tracer, e.g., a tracer application, to track user inputs and a machine learning model to determine a user's experience level and the user's emotional state from the user inputs. For example, the tracer may track the user's mouse movements, mouse hovering, typing speed, typing accuracy, abandoned sessions, and page views. In other examples, the application may use a microphone of a user device to record a user's voice to determine an emotional state of the user from the user's tone, inflections, and choice of words. In further examples, the application may use a camera of the user interface to record a user to determine an emotional state of the user from the user's facial expressions, gesticulations, or body language (e.g., a user putting his or her hands over his or her face, chewing fingernails, or gritting teeth).

For example, the user's emotional state may be upset or frustrated. The adaptable user interface system modifies elements of the user interface based on the user's emotional state, such as by adjusting language used in the user interface. In one such example, when a user is confused, instead stating “Input questions here”, the input field may state “Need help? Ask us a question here.”

The adaptable user interface system further adjusts the user interface based on the experience level of the user. For example, the adaptable user interface system may edit the user interface based on past user inputs and tasks completed by the user on the user interface. As one example, in one embodiment where the user interface is associated with a banking application which the user has previously used to apply for a loan, the application may modify the user interface based on the user's prior application. In such an embodiment, for example, when the user applies for another loan, the adaptable user interface may remove graphics including tutorial steps for applying for the loan. In another embodiment, the application may edit the user interface to include more information regarding the transaction in place of a simplified step by step guide requesting user information.

The adaptable user interface system may use a machine learning model to determine the experience level of the user and the emotional state of the user. The machine learning model may be a pretrained model, pretrained on data mapping user inputs to a set of emotional states and experience levels. In some examples, users may provide feedback to the machine learning model, such as requesting the system add or remove features or elements from the user interface. Based on the user's feedback, the machine learning model may adjust future modifications to the user interface for experience levels and emotional states.

illustrates an adaptable user interface system. The adaptable user interface system includes an applicationwith a machine learning module; one or more user devicessuch as a smartphone, laptop, or tablet executing a browser; a repository; and cloud service provider (CSP) infrastructure, such as various servers, computing devices, processors, and databases.

As shown in, the applicationmay run on the cloud service provider (CSP) infrastructure. In other embodiments, the user devicemay execute the application. Users may interact with the applicationthrough an adaptable user interface such as a guided user interface (GUI) on a browseror application running locally on the user device.

The user devicemay include various peripherals providing for user input, such as a microphone, a camera, a touch screen, and buttons. For example, the microphone may record inputs such as voice commands and the camera may record facial expressions of the user.

Users may provide further inputs to the applicationincluding selecting a neurological condition. Based on the user input, the applicationqueries the repositoryfor a user interface template associated with the selected neurological condition.

The applicationmay download the user interface template from the repositoryand upload data to repository. The applicationmay run on cloud service provider (CSP) infrastructure and comprise various servers and databases to store information associated with user interactions with the application and user interface, such as historical user data. Historical user data includes information associated with the user's past operation of the application, such as past tasks performed by the user within the application, efficiency of the user in performing the past tasks (e.g., how long it takes the user to complete tasks), and the amount of time the user has spent operating the user interface of the application.

In some examples, repositorymay store user profiles, which may include collections of historical user data associated with the user. The repositorymay also store information such as user interface templates associated with neurological conditions from the list of neurological conditions. The repository may include a preset mapping of user interface template to one or more neurological conditions. In one embodiment, the adaptable user interface systemmay associate a first user interface to the neurological conditions of autism and post-traumatic stress disorder, and a second user interface to Parkinson's disease.

In one embodiment, when the application is a finance and banking application, the repository may include a user profile indicating the types of loans that the user has applied for using the application, past questions of the user, the efficiency of the user in performing tasks, and the amount of time the user has spent operating the user interface of the application. The applicationmay query the repositoryfor the historical user data from users.

The applicationmay further include a tracer application or program, which may generate a tracking log representing user inputs from a current session of the application. The machine learning modulemay use the tracking log to determine an emotional state of the user and may use the historical user data to determine an experience level of the user. Further description of the machine learning moduleis provided in the description of.

is a flowchart showing illustrative methodfor operating an adaptable user interface system. In some embodiments, some of the steps in the flow chart ofare implemented in program code executed by a processor, for example, the processor in a general-purpose computer, mobile device, or server. In some examples, these steps are implemented by a group of processors. In some examples the steps shown inare performed in a different order or one or more steps may be skipped. Alternatively, in some examples, additional steps not shown inmay be performed.

At step, an application, such as applicationfrom, selects a neurological condition associated with a user. The application may select the neurological condition from a list of neurological conditions, which may include various neurological conditions such as autism, attention-deficit hyperactivity disorder, Parkinson's disease, epilepsy, Alzheimer's, and post-traumatic stress disorder. The list may further include various neurological conditions from the Diagnostic and Statistical Manual of Mental Disorders (DSM).

The neurological conditions are associated with a user interface template preset to cater to the needs of users with the associated neurological condition. For example, a user interface associated with the neurological condition of epilepsy may include muted colors, removing flashing graphics, and using slower transitions between pages within the user interface (e.g., causing the user interface to transition between pages of the user interface by fading pages in or out instead of using an abrupt transition between pages).

At step, the application using a machine learning module, determines a user experience level. The user experience level may be an alphanumeric representation of a user's experience with the application. The machine learning module uses historical user data to determine the user experience level. Historical user data includes information associated with the user's past operation of the application, such as past tasks performed by the user within the application, efficiency of the user in performing the past tasks (e.g., how long it takes the user to complete tasks), and the amount of time the user has spent operating the user interface of the application. For example, historical user data may include information associated with tasks performed by the user within the application and the user's efficiency of performing said tasks. In one such example, user efficiency may include how quickly users complete tasks within the application, which may demonstrate whether the user is able to effectively operate the user interface and application. Historical user data may also include information associated with the amount of time the user has spent within an application, whether the user has requested help, and whether the user has completed a tutorial.

Historical user data may further include past emotional states of the user. For example, past historical user data may indicate that a configuration of the user interface is associated with the user being frustrated. The machine learning module may learn from past emotional states of a user to determine whether modifications should be made to the user interface.

At step, the application generates a user interface. The application generates the user interface based on the selected neurological condition and associated user interface template, as well as the user experience level. The application uses the user interface template and adds, removes, or modifies elements and features of the user interface template based on the user experience level.

At step, the application receives a tracking log. The tracking log includes information regarding user inputs to the user interface. A tracer program, such as a tracer application, may track the user's inputs to the user interface, and represent the inputs in the tracking log. The tracking log may include various inputs of the user and measured metrics of the user interface, such as movements of the mouse, errors in typing, repeated page views, amount of time spent completing a task, amount of time the user is within the user interface, and incomplete tasks.

At step, the machine learning module of the application determines a behavioral metric representing an emotional state of the user based on the tracking log. For example, the tracking log may include information associated with user inputs including mouse movements and viewed pages of the user interface. In one such example, the slow mouse movements and reviewing the same page multiple times may indicate that the user is confused. In another example, rapid clicking and fast mouse movements may indicate the user is frustrated.

At step, the application adds, removes, or modifies one or more elements of the user interface based on the behavioral metric. In some examples, the machine learning module frommay determine the user is frustrated and modify elements of the user interface to try to calm the user, such as by asking the user if he or she needs help. By way of example and not to limit the method, below is a sample list of behavioral metrics that may be analyzed by the machine learning module when determining a user's emotional state.

is a flowchart showing illustrative methodfor operating an adaptable user interface system. In some examples, the steps in the flow chart ofare implemented in program code executed by a processor, for example, the processor in a general-purpose computer, mobile device, or server. In some examples, these steps are implemented by a group of processors. In some examples the steps shown inare performed in a different order or one or more steps may be skipped. Alternatively, in some examples, additional steps not shown inmay be performed.

At step, the user may access a digital asset (e.g., the application) to request service. For example, a user may access the application using a user device, such as the user device in, to request a service, such as requesting a loan.

At step, the application uses a machine learning module to analyze user inputs. For example, the application may use various machine learning techniques such as neural networks, trained and untrained classifiers, support-vector machines, decision trees, and Bayesian networks to analyze user inputs to determine the user's emotional state. In one example, user inputs may be assigned numerical weights to indicate that the user input is associated with various emotional states. The machine learning module may weigh a combination of user inputs to determine a likely emotional state of the user.

At step, the machine learning module determines, based on the emotional state of the user, modifications to the user interface to improve the user's experience. The adaptable user interface system may include preset modifications to be implemented for emotional states. For example, when the adaptable user interface system identifies the user is frustrated, a preset modification may include a popup message asking for the user's feedback. In other examples, the machine learning module may determine the modification. In such an example, the machine learning module may identify that the user is frustrated, and that the user's frustration is associated with a particular element, such as by identifying the user has repeatedly and rapidly clicked an element of the user interface. The machine learning module may identify the element as being a source of the user's frustration and modify the element such providing a pop-up box explaining the element or simplifying the element by removing information provided by the element.

Modifications to the user interface may include the individual modifications and solutions represented in steps,, andor various combinations thereof.

Stepincludes modifications to tools and functionality of the user interface. For example, the machine learning module may determine to add, remove, or modify individual tools and functions of the user interface, such as by adjusting the arrangement of elements of a user interface, and adjusting the elements present within the user interface.

Stepincludes providing educational materials. The educational materials may include tutorials for operating the user interface, or educational materials for tasks the user is attempting to complete on the user interface. For example, when the task is completing a loan application, the educational materials may include an explanation of credit scores, or a warning to the user of actions that may affect approval of a loan.

Stepincludes connecting the user to an operator, such as by starting an instant message conversation, or a phone call between the user and an operator of the application (e.g., information technology staff or customer service department associated with the adjustable user interface system).

When the modifications or responses to the user's emotional state have been executed or implemented, the machine learning module may analyze the user inputs to determine whether the modifications or responses were effective in assisting the user operate the user interface, such as by identifying whether the user completed a task he or she was working on, or whether the user's efficiency in operating the user interface improved. The analysis of the user inputs after implementing the modification or response is represented at stepas post-transactional behavioral analytics. The post-transactional behavioral analytics are then used in a feedback loop to improve the machine learning module's analysis at step.

illustrates an example of logic used by the machine learning module of the adjustable user interface system in determining modifications to the user interface. The adjustable user interface system includes a user interface template for conditions of the user. The conditions of the userincludes various neurological conditions, such as autism, post-traumatic stress disorder, and attention deficit hyperactivity disorder.

The templates for the conditions of the user are adjustable based on user experience and user's emotional state, represented by an experience metricand behavioral metricrespectively.represents the experience metricand the behavioral metricas numerical values. For example, a machine learning module, such as the machine learning module from, may determine a numerical value for the experience metric based on historical user data and a numerical value for the behavioral metric based on user inputs during operation of the user interface.

andillustrates an example adaptable user interface system transitioning from user interface 1A to user interface 2B based on a user's emotional state and experience level. User interface 1A and user interface 2B include a toolbarAB, graphicAB, and an input fieldAB respectively. User interface 1A may be generated from a user interface template, such as the user interface template stored in repositoryof. By way of example,illustrates a modified user interface for a frustrated user lacking experience with user interface 1A from.