Methods to infer content relationships from user actions and system automations

This application is a continuation-in-part of U.S. patent application Ser. No. 17/707,888, filed Mar. 29, 2022, which is incorporated herein by reference in its entirety for all purposes.

This application claims priority to U.S. Provisional Patent Application No. 63/255,401, filed Oct. 13, 2021, which is incorporated herein by reference in its entirety for all purposes.

The present invention relates to data communication and, more particularly, to systems and methods for a collaborative platform.

As the processing power of computers allow for greater computer functionality and the internet technology era allows for interconnectivity between computing systems, electronic content have become more ubiquitous. Content providers use various platforms, such as websites or other applications, to provide their content.

However, since the implementation of more sophisticated online tools, several shortcomings in these technologies have been identified and have created a new set of impediments. For example, electronic content for a given user or organization now tends to be highly fragmented across a variety of different tools and systems. The high level of fragmentation makes it more difficult to monitor the electronic content consumed by users, to extract knowledge from the content consumed by users, and to augment electronic content with other useful knowledge.

For the aforementioned reasons, there is a desire for an efficient system and method to identify, extract, and improve access to knowledge and capabilities across fragmented systems by analyzing electronic content that has been, is being, or will be consumed by users (e.g., is presented to users). More specifically, there is a growing need for methods and systems that facilitate how users interact with computer systems in order to, for example, enable users to easily identify relevant actions for and perform relevant actions on any given piece of data across a variety of systems and interfaces, and facilitate users' access to relevant knowledge for any given piece of data. Methods and systems described herein can monitor and help manage electronic content, actions, and relationships between and across data and actions.

The methods and systems described herein can be provided through independently built, managed, and/or distributed system and applications, as well as be integrated into existing features and/or windows presented by an operating system and/or application (e.g., provided by the analytics server) on a computer. For example, a user viewing a webpage associated with a given medication might be able to see related units of work and actions in a contextual panel on the side of the screen, including references to the medication across sources such as WebMD, the user's electronic health record, any indexed files and messages, etc. Similarly, the contextual side panel could show relevant information specific to the user such as whether the user has previously taken this specific medication, other similar medications, the doctors that prescribed them, the instructions for how the medication was prescribed to the user, common side effects, actions related to the medication such as refilling prescriptions and asking the prescribing doctor questions, etc.

Moreover, the methods and systems described herein can also be implemented, such that the analytics server can present context/relevant data associated with any given unit of work (e.g., chat message, email, file, mailing address, calendar event, event venue, movie, actor, script, purchase order, patient record, medication, etc.). For instance, instead of analyzing the content displayed (or otherwise outputted) by the user's computer, the analytics server may receive a selection of a unit of work from the user (e.g., the user may right-click on a file name or may otherwise select the file, a system might automatically send a selection of one or more products, etc.). In response, the analytics server may analyze the selected unit of work and display relevant data and actions as described herein. In some configurations, the user may launch a search interface with contextual data as an overlay/pop-up which will automatically use the electronic content as the selected unit(s) of work described above.

The user can easily interact (e.g., keyboard shortcut, icon click, voice command, API endpoint, etc.) with the electronic device to pull up searchable relevant context data and/or actions about the electronic content. When using the search feature, if no results are found to be directly associated with the electronic content, the analytics engine can recommend results from elsewhere in the nodal data structure and/or other sources. The analytics server may use the electronic content (and the various connections, and/or similar attributes such as specific fields or vectors, that it may have to units of work in nodal data structure) to influence the ranking of search results such that nodes that are related to both the search query and the electronic content are ranked higher than nodes with only an association to the search query.

In some configurations, the user may launch a search interface with contextual data as an overlay/pop-up which will automatically use a combination of the electronic content and information about the user, such as information about the user's role or intent, as the selected unit(s) of work described elsewhere herein (e.g., create or otherwise use vector representations of the selected unit or units of work to perform nearest neighbor searches). In this way, the user is able to easily interact (e.g., keyboard shortcut, icon click, voice command, etc.) with the electronic device to pull up searchable data that is relevant to the electronic content and to the user performing the search. When using the search feature, if no results are found to be directly associated with the electronic content, the analytics engine can recommend results from elsewhere in the nodal data structure. The analytics server may use the electronic content (and the various connections, and/or similar attributes such as specific fields or vectors, that it may have to units of work in nodal data structure) to influence the ranking of search results such that nodes that are related to both the search query, the electronic content, and specific associations with the user entering the query are ranked higher than nodes with only an association to the search query.

In a non-limiting embodiment, a method comprises monitoring, by a processor, interactions by a user associated with first electronic content and second electronic content presented on a computing device; linking, by a processor, a pair of nodes within a set of nodes of a nodal data structure based the monitored interactions, the pair of nodes comprising a first node associated with the first electronic content and a second node previously unlinked to the first node and corresponding to the second electronic content; revising, by a processor, a link between a pair of nodes within a set of nodes of a nodal data structure based the monitored interactions, the pair of nodes comprising a first node associated with the first electronic content and a second node associated with the second electronic content; in response to receiving a request for electronic content from the computing device: determining, by the processor, at least one node within the set of nodes of the nodal data structure that corresponds to the request; and providing, by the processor, data associated with the at least one node and additional data associated with any other node linked to the at least one node.

In another embodiment, a system comprises a non-transitory computer-readable medium having a set of instructions, that when executed, cause a processor to: monitor interactions by a user associated with first electronic content and second electronic content presented on a computing device; revising, by a processor, a link between a pair of nodes within a set of nodes of a nodal data structure based the monitored interactions, the pair of nodes comprising a first node associated with the first electronic content and a second node associated with the second electronic content; in response to receiving a request for electronic content from the computing device: determine at least one node within the set of nodes of the nodal data structure that corresponds to the request; and provide data associated with the at least one node and additional data associated with any other node linked to the at least one node.

In another embodiment, a system comprises a computing device configured to output a first electronic content and a second electronic content; and a server in communication with the computing device, the server configured to monitor interactions by a user associated with first electronic content and second electronic content presented on the computing device; revising, by a processor, a link between a pair of nodes within a set of nodes of a nodal data structure based the monitored interactions, the pair of nodes comprising a first node associated with the first electronic content and a second node associated with the second electronic content; in response to receiving a request for electronic content from the computing device: determine at least one node within the set of nodes of the nodal data structure that corresponds to the request; and provide data associated with the at least one node and additional data associated with any other node linked to the at least one node.

While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various aspects and embodiments disclosed are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

displays a method executed by the analytics server that allows a user to search and/or navigate electronic content displayed on a computer. For instance, the analytics server may execute the methodto allow users to navigate a website using their keyboard and without needing to use any other input devices/elements, such as a mouse. The analytics server, as used herein, may include a browser extension, any code running on the user's computer (implemented by the analytics server or a third party), etc. This code and/or related configuration could also be semi-self-generating (e.g., as it learns from user behavior to do new types of searches and/or selections).

The methodmay describe the analytics server acting on a “webpage hosted by a webserver.” The webpage could also be hosted on the user's personal computer (e.g., local files opened in the browser with the file:// protocol). Moreover, the methods and systems described herein are not limited to webpages. They also apply to any electronic content (e.g., it could be an NSView in an iOS app, an Apple TV Markup Language document, WPF or XAML document for windows desktop applications, etc.).

These methods and systems also apply to speech input which gets translated into text then follows the same process as keyboard text input. They also apply to other forms of human-computer interaction devices such as cameras, haptic input devices, and brain scanning devices that allow the translation of user generated inputs into text commands and/or parameters.

At step, the analytics server may, receive an input from a computing device displaying a webpage responsive to displaying an input element as an overlay on the webpage hosted by a webserver.

Referring now to, a non-limiting example of a website and the overlay is provided. The analytics server may display a graphical element on the user's computer. The graphical element may be controlled by the analytics server through a browser extension. For instance, the user may download the browser extension and, as a result, the analytics server may display the graphical element on the user's computer. Alternatively, the computing device's operating system or the browser application may have the graphical element and analytics server included, thereby removing the need for an extension. Regardless of the exact configuration, the analytics server may display the graphical element as an overlay over the electronic content being displayed on the user's computer. The graphical element may include an input element, as depicted in the graphical elementdisplayed as an overlay to the webpage. The graphical elementmay also include the “enter” button indicating that the user would like to proceed with searching for the inputted term. In other words, the primary command for the graphical elementis “enter” or “select.” The graphical elementallows the user to input a desired term (e.g., “read” as depicted in).

Even though certain aspects of the present disclosure describe and depict a graphical element having an input element that is displayed throughout the process, in some configurations, the graphical element may be dormant and hidden from the user (e.g., operating as a background process). For instance, the graphical element may not be displayed, however, the analytics server may be monitoring the user's key stokes (or other input devices such as a microphone) to identify any input received from the user. In that way, the same services described herein (e.g., method) can be provided without obfuscating the user's view of the electronic content (e.g., webpage).

The input element is not limited to alphanumerical strings and characters inputted by the user using a keyboard. In alternative embodiments, the user may input the desired terms using audio input elements or other input elements, such as eye-controlled input elements, special keyboards, sip and puff (SNP) input devices, and the like.

In some embodiments, the user may use voice commands for their input. In those embodiments, the analytics server may utilize a voice recognition protocol to identify the voice command received from the user.

In some embodiments, the analytics server may use translation services to allow a user to input commands in one language and convert them into commands in the detected language used by the webpage and select results accordingly. This is particularly relevant for webpages or web browser which do not offer translation of the text on a webpage.

Referring back to, at step, the analytics server may identify electronic content associated with the input received. Using various methods and systems described herein, the analytics server may analyze the electronic content and determine one or more parts of the electronic content that correspond to the input received at step.

In a non-limiting example, when a user types in the input element provided by the analytics server (e.g., search bar on a webpage or the graphical user interface), the analytics server recursively scans through the webpage's document object model (DOM) node tree to find all DOM nodes which match the user's query. As used herein, the DOM is an interface that treats HTML or XML documents as a tree structure, where each DOM node is an object of the document. DOM also provides a set of methods to query the tree and/or alter the structure or style. Therefore, a DOM node is different than the nodes described in relation to the nodal data structure discussed herein.

Whether a DOM node matches the query or not may be determined by detecting if any text within the DOM node includes the user's query or if an attribute of the DOM node includes the user's query. The analytics server may use this method of scanning DOM node attributes hidden to the user to identify buttons and/or actions displayed on the webpage which, for example, only display an icon and no text (e.g., based on the buttons functionality rather than text displayed on the button or action). Not all attributes of a DOM node may be relevant to the user's query or relevant to be used by the analytics server executing the method. Thus, the analytics server may only search a specific list of attributes per DOM node based on its tag name. These tag data may be found in files associated with the webserver hosting the electronic content being displayed on the user's computer. For instance, the analytics server can analyze JSON files and execute corresponding code to analyze DOM nodes, such as the following:

In addition to matching nodes against the user's exact query (e.g., input received in step), the analytics server may also match the input received against synonyms of the user's query, as well as other words and phrases with similar meaning. In this way, the user who has expressed their intention in a slightly different manner may still see results generated by the analytics server, even though the webpage may not include the exact term inputted by the user. For instance, a user may input “trash” where the user would like to “delete” a file and where the webpage uses the term “discard” to mean the same thing. In this way, the analytics server may identify the user's intention and identify the text “discard” within a DOM node or a DOM node's attributes.

The analytics server may use a precompiled synonym library or a word similarity algorithm (e.g., word2vec libraries, sentence2vec libraries). Alternatively, the analytics server may use one or more configuration files (generated by the analytics server or a third party) per URL host (e.g., webserver) with synonyms specific to that host. Effectively, each configuration file can be thought of as mapping to an “App” used in the browser (e.g., mail.google.com for Gmail, news.ycombinator.com for Hacker News). More specifically, the analytics server may choose from various configuration files related to a specific URL host, depending on what specific website the user is viewing. For example, the analytics service might use different configuration files when the user is viewing his email inbox, when he is reading an email, and when he is writing an email draft. In some embodiments, the specific configurations may be identified by the attributes of the webpage the user is viewing, such as attributes of the webpage's full URL, attributes of the webpage's header, or attributes of the webpage's page source (e.g., inside the head tags, using schema.org information).

Some embodiments may consider configuration files based on other criteria, such as the identification of a particular unit of work on the webpage or the identification of a certain type of functionality on the webpage, rather than only the URL host the webpage belongs to. For example, various email tools are likely to offer similar functionality and use similar terms. These various email tools may therefore also have a high likelihood of successfully sharing the same configuration file(s) across URL hosts. In this way, if a particular configuration file is missing for a URL host, then the analytics server might be able to rely on generic configuration files for “email” tools, or even configuration file(s) from other “email” tools with specific URL host data.

In some embodiments, the analytics server may ask the user to provide the missing information in order to best understand what types of actions are important. In other words, the analytics server may ask a user to provide a “word” or “description” and to correlate them with specific action(s) and/or interactions(s) that he would like the analytics server to simulate and/or provide via the graphical input. The user may, for example, provide the necessary configuration file(s), be guided through a no-code set-up wizard for a given webpage that relates an action with one or more words, and the like.

After finding all DOM nodes corresponding to (e.g., matching) the user's query, the analytics server may filter those nodes down to only those which are likely to be able to be selected or otherwise acted upon by the user. The analytics server may use various factors to identify this subset of the DOM nodes. The following is a non-limiting list of factors used to determine whether a DOM node that has been identified as corresponding to the user's query is able to be selected or otherwise acted upon:

If the DOM node's tag name specifies that it's a button, link, or input;

If any of the DOM node's attributes specifies that it's a button, link, or input; or

If the DOM node matches an additional selector defined in the configuration file (e.g., under the additional_button_selectors config).

Using various scoring algorithms, the analytics server may score each of the matched buttons, links, and inputs (e.g., corresponding to the DOM nodes identified within the webpage), which will be selected. The analytics server may use these scores to determine which DOM node is the “best”, or most relevant, matching DOM node that will be selected first. Of course the “best” matching DOM node is highly contextual based on several factors. For instance, a DOM node's score may be increased if:

The node matches one of the selectors in the list of “relevant selectors” in the configuration file.

Each of the above-described factors may have a related weight which determines how much it effects the DOM node's score.

Using the above-described methods and schemes, the analytics server may determine which identified DOM nodes is “better” than other identified DOM nodes.

At step, the analytics server may visually highlight the identified electronic content. Once the matching buttons, links, and inputs (corresponding to identified DOM nodes) are found and sorted according to their respective score, the analytics server may visually highlight the corresponding content on the electronic content displayed on the user's computer. For instance, the analytics server may add a selection box around each node and focus and automatically scroll to the one with the highest score. Referring now to, when the user enters “read,” the analytics sever displays the boxaround “read” displayed on the webpage.

In some embodiments, a user can then press the tab key (e.g., or any other pre-configured key) to move through the matches. For instance, if the analytics server has identified five instance of “read” within the webpage, the user may be able to navigate through all choices.

Additionally or alternatively, the analytics server may use the HTML structure to return results related to a given query that does not necessarily match the query's exact terms. For example, a group of related buttons that are contained in the same HTML element (e.g., a div or container), could be selected and toggled through by their association to each other, given that at least one of the results matches the query with some level of relevance. This type of selection could be provided through configuration for a given webpage type (e.g., by domain, by type, by some combination of multiple attributes, by a custom classifier, etc.)

Referring back to, at step, the analytics server may, in response to receiving a selection of the identified electronic content and a command, instruct the web server to execute the command corresponding to the identified content.

The analytics server may receive a selection of the visually highlighted content and a corresponding command. As a result, the analytics server instructs the webserver to execute the command. For instance, and referring back to, the user can press the enter key to click or focus the selected button, link, or input (e.g., box). As a result, the analytics server activates the hyperlink depicted within the boxand directs the user to the corresponding new webpage.

depicts a second non-limiting example. In this embodiment, the analytics server displays the graphical elementas an overlay on the webpageusing a browser extension (not shown). When the user inputs “arc” (using the keyboard or by speaking “arc” into a microphone), the analytics server identifies two DOM nodes that correspond to the boxesand. As depicted, the boxcorresponds to a button and not the word archived as the box. When the user interact with the “enter” button of the graphical element, the analytics server instructs the web server to activate the archive button and archive the email displayed on the user's computer.

depicts yet another non-limiting example. In the depicted example, the user is viewing the webpage, which is a list of emails received or that is associated with a user's account. When the user inputs “quo” into the input element of the graphical element, the analytics server highlights the identified content (e.g., displays the contentas larger than the rest of the content within the webpage). When the user interacts with the “enter” button, the analytics server instructs the web server to open that selected email.

Additionally, the analytics server may combine the methoddescribed herein with speech recognition protocols to allow users to navigate most webpages without a mouse or other input elements. To achieve this result, the analytics server may execute the following steps:

In some embodiments, the analytics server may use a Web Speech API to identify the user's voice commands. For instance, when the user audibly inputs a command, the analytics server may open a search bar, such as the graphical element discussed herein. For instance, when the user says “YipYip,” the analytics server opens the search bar and actively listens to the user to receive an input. For instance, the analytics server receives an input from the user and identifies the input using one or more speech recognition protocols.

The analytics server may use the methods and systems discussed herein to identify the content and visually highlight the identified content. Then the analytics server may actively listen to the user, such that the user can navigate through the highlighted content to make a selection. For instance, the user may say “YipYip next” to select a next match or say “YipYip last” to select a previous match.

When the user is satisfied with the selection, the user may say “YipYip select” to press enter and the analytics server may instruct the webserver to execute the corresponding command for the selected content.