Active Analytics

This application is a continuation of U.S. application Ser. No. 17/959,757 filed Oct. 4, 2022 filed, the disclosure of which is incorporated herein by reference in its entirety.

In the process of developing executable software components, multiple systems and environments can be involved. Software components (also referred to as “components”) can include applications, web pages, and other such content executable by a computer system. Software components can be deployed in a range of environments, such as secure environments, where limited access is provided to a select group of users, or cloud environments that may support wider-scale deployment with publicly available content. Testing of components can be performed prior to making the components available. Some components may have dependencies on data sources and/or other components. As such, a component that was previously functional may experience issues where linked components or data sources change or access/connectivity events occur. Accordingly, it can be challenging to verify that software components continue to operate as intended after the software components are deployed.

Some aspects of software component performance relate to service level objectives, where responsiveness and other performance indicators are expected to be within a predetermined range during operation to meet the service level objectives. Performance indicators can change due to factors beyond the control of software component developers, such as availability of computing/cloud resources. Monitoring for performance after deployment and verifying that system security is maintained can be challenging particularly where third-party data feeds are used by the software components. Traditional testing approaches may consider whether a system is up or down and have a limited ability to understand whether service level objectives or other such objectives are being met over a period of time.

According to an embodiment, a system for active analytics is provided. The system can improve computer system performance by supporting testing of software components prior to deployment and after deployment to ensure that security and other aspects are met. Performance monitoring provided by active analytics can support multiple script languages, security models, levels of failures, and notifications. Rather than simply providing a pass/fail indicator, embodiments provide a range of testing and reporting options. Further, test scripts can be reused with environment variables set and changed to test multiple scenarios and/or rapidly generate test cases based on similar configurations. The use of higher-level management scripts can configure a sequence of test cases beyond single test script execution. Further, multiple tests can be executed at the same time to monitor for load effects, interactions, and/or resource contention issues. Reporting can include providing a range of warnings and/or alerts to one or more selected recipients using one or more selected reporting channels.

The system may also enhance security through security vault integration. Using a framework, as further described herein, users and/or system owners can manage the security of required credentials for system access, allow differing levels of scripts to be run in a pipeline targeting specific functionality, and produce reports and/or metrics defined by weighted scoring. The framework can provide for the management and measurement of execution of test scripts, such as synthetic user scripts, to test and simulate interactions with software components. The system supports a modular capability, where the scripts can be triggered for specific functionality as part of automated testing, manage script variability, such as user credentials and triggered inputs, and store variables and credentials using secure vault technology.

Rather than only validating component availability, embodiments can assess multiple performance factors of software components. Examples of performance factors can include specific performance objectives and associated service level indicators, among other factors. Additionally, various layout aspects, such as web page content, can be monitored, as well as the associated processing and system security for interactions with the web page content. Monitoring performed while executing test scripts can include interrogation of responses and system execution to ensure execution is optimized for performance and third-party data feeds. Utilizing active analytics during component development as a system for managing script performance can provide a more robust user-focused measure of system changes during the software development lifecycle. Traditional unit testing typically focuses on a specific unit of code that has changed. In contrast, embodiments may employ test scripts that synthesize user interactions, which can be executed and monitored by the active analytics system to allow a development team to better understand how a change may impact an end user experience.

In embodiments, the system can manage scripts for specific functions and a range of executed functions. Active analytics can include a script modularization capability that provides a single point of configuration for development teams to choose the conditions in which a script will execute in a target environment. For example, a specific script to handle logging-in may measure the impact of an authentication change to validate the change in a development environment. However, a full regression script may be run and measured criteria passed in order to move code from a development environment to a production environment. Active analytics can provide both warehousing and selective execution capability. Thus, the system provides multiple improvements to computer technology and practical applications which can improve existing system performance along with the future performance of software components under development.

1 FIG. 100 100 102 104 106 108 110 112 110 112 102 112 108 100 102 104 106 105 110 Turning now to, a systemis depicted upon which active analytics may be implemented. The systemincludes a plurality of server systems, such as one or more component development servers, one or more component execution servers, a data warehouse system, and an active analytics systemcoupled to a network. A plurality of user systemscan access content and/or interfaces through the network. For example, user systemA may be configured as a developer system operable to interface with the component development servers, while user systemN may be configured as an analyst system operable to interface directly with the active analytics systemand access other elements of the systemindirectly, such as the component development servers, component execution servers, and data warehouse system. Data feeds for some components may be sourced from third parties through external sources, which may be accessible through the network.

1 FIG. 102 104 106 106 104 106 102 114 114 In the example of, the component development serversare operable to develop software components through one or more code files for subsequent execution on the component execution servers. Various records associated with component development, execution, and/or problems can be collected and stored in the data warehouse system. The data warehouse systemcan also include data sources that are produced and consumed during execution of components on the component execution servers. For example, the data warehouse systemcan manage access to files and databases and may be further partitioned to control access to the files and data based on user permissions. During a software development process, code to implement components can be developed on the component development serversand stored in component repository. The component repositorycan include check-in/check-out version management control of component files.

118 108 124 120 114 104 104 118 124 118 122 124 126 128 130 112 122 124 126 128 130 120 122 124 126 128 130 100 120 118 112 1 FIG. An active analytics toolof the active analytics systemcan execute one or more test scriptsfrom a storage systemto test one or more components of the component repository. Component testing can be performed prior to release of a component to the component execution servers. Alternatively, after a component is deployed for execution by the component execution servers, the active analytics toolcan be used to run one or more test scriptsthat test aspects of the component, for example, to confirm that service level performance is maintained and/or linked content continues to function as expected. The active analytics toolcan use one or more management scriptsto configure test aspects, such as setting environment variables, setting a frequency of execution of the test scripts, identifying layout templatesto confirm display aspects, identifying test criteria, and managing other such aspects. Test results can be stored in logsas a test execution summary and used to trigger alert notifications, such as transmitting alerts to one or more targeted users of the user systems. In the example of, the management scripts, test scripts, layout templates, test criteria, and logsare depicted as being stored in the storage system. However, it will be understood that one or more of the management scripts, test scripts, layout templates, test criteria, and/or logscan be distributed elsewhere within the system. For instance, the storage systemcan be implemented as a storage managed within an enterprise or distributed in whole or in part within a cloud environment. The active analytics toolcan generate user interfaces for display and to support configuration by users of the user systemshaving corresponding permissions.

1 FIG. 102 104 106 108 112 100 100 102 104 106 108 112 102 104 106 108 112 In the example of, each of the component development servers, component execution servers, data warehouse system, active analytics system, and user systemscan include a processor (e.g., a processing device, such as one or more microprocessors, one or more microcontrollers, one or more digital signal processors) that receives instructions (e.g., from memory or like device), executes those instructions, and performs one or more processes defined by those instructions. Instructions may be embodied, for example, in one or more computer programs and/or one or more scripts. In one example, the systemexecutes computer instructions for implementing the exemplary processes described herein. Instructions that implement various process steps can be executed by different elements of the system, such as elements of the component development servers, component execution servers, data warehouse system, active analytics system, and/or user systems. Although depicted separately, one or more of the component development servers, component execution servers, data warehouse system, active analytics system, and/or user systemscan be combined or further subdivided.

112 112 112 The user systemsmay each be implemented using a computer executing one or more computer programs for carrying out processes described herein. In one embodiment, the user systemsmay each comprise a personal computer (e.g., a laptop, desktop, etc.), a network server-attached terminal (e.g., a thin client operating within a network), or a portable device (e.g., a tablet computer, personal digital assistant, smart phone, etc.). In an embodiment, the user systemsare operated by users having the role of a system architect, a software developer, an analyst, or a non-developer (e.g., a manager or administrator) with respect to an application development and troubleshooting process, and the role designations may change.

102 104 106 108 112 Each of the component development servers, component execution servers, data warehouse system, active analytics system, and user systemscan include a local data storage device, such as a memory device. A memory device, also referred to herein as “computer-readable memory” (e.g., non-transitory memory devices, as opposed to transmission devices or media), may generally store program instructions, code, and/or modules that, when executed by a processing device, cause a particular machine to function in accordance with one or more embodiments described herein.

110 100 100 110 110 100 112 106 114 120 The networkcan include any type of computer communication technology within the systemand can extend beyond the systemas depicted. Examples include a wide area network (WAN), a local area network (LAN), a global network (e.g., Internet), a virtual private network (VPN), and an intranet. Communication within the networkmay be implemented using a wired network, an optical network, a wireless network, and/or any kind of physical network implementation known in the art. The networkcan be further subdivided into multiple sub-networks that may provide different levels of accessibility or prevent access to some elements of the system. For example, some users of user systemsmay have limited (e.g., read-only) access to the data warehouse system, component repository, and/or the storage system.

2 FIG. 2 FIG. 1 FIG. 200 200 201 200 102 104 105 106 108 112 depicts a block diagram of a systemaccording to an embodiment. The systemis depicted embodied in a computerin. The systemis an example of one of the component development servers, component execution servers, external sources, data warehouse system, active analytics system, and/or user systemsof.

2 FIG. 201 205 210 215 235 235 235 201 In an exemplary embodiment, in terms of hardware architecture, as shown in, the computerincludes a processing deviceand a memory devicecoupled to a memory controllerand an input/output controller. The input/output controllermay comprise, for example, one or more buses or other wired or wireless connections, as is known in the art. The input/output controllermay have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the computermay include address, control, and/or data connections to enable appropriate communications among the aforementioned components.

250 255 235 201 225 230 In an exemplary embodiment, a keyboardand mouseor similar devices can be coupled to the input/output controller. Alternatively, input may be received via a touch-sensitive or motion sensitive interface (not depicted). The computercan further include a display controllercoupled to a display.

205 220 210 205 201 The processing devicecomprises a hardware device for executing software, particularly software stored in secondary storageor memory device. The processing devicemay comprise any custom made or commercially available computer processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the computer, a semiconductor-based microprocessor (in the form of a microchip or chip set), a macro-processor, or generally any device for executing instructions.

210 220 210 220 210 220 240 205 210 220 205 The memory devicecan include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and nonvolatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, programmable read only memory (PROM), or the like, etc.). Secondary storagecan include any one or combination of tape, compact disk read only memory (CD-ROM), flash drive, disk, hard disk drive, diskette, cartridge, cassette or the like, etc. Moreover, the memory deviceand/or secondary storagemay incorporate electronic, magnetic, optical, and/or other types of storage media. Accordingly, the memory deviceand/or secondary storageare examples of a tangible computer readable storage mediumupon which instructions executable by the processing devicemay be embodied as a computer program product. The memory deviceand/or secondary storagecan have a distributed architecture, where various components are situated remotely from one another, but can be accessed by one or more instances of the processing device.

210 210 211 216 211 201 205 210 210 201 216 2 FIG. The instructions in memory devicemay include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. In the example of, the instructions in the memory deviceinclude a suitable operating system (O/S)and program instructions. The operating systemessentially controls the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. When the computeris in operation, the processing deviceis configured to execute instructions stored within the memory device, to communicate data to and from the memory device, and to generally control operations of the computerpursuant to the instructions. Examples of program instructionscan include instructions to implement the processes as further described herein.

201 260 110 260 2 FIG. 1 FIG. The computerofalso includes a network interfacethat can establish communication channels with one or more other computer systems via one or more network links of the networkof. The network interfacecan support wired and/or wireless communication protocols known in the art.

3 FIG. 1 3 FIGS.- 1 FIG. 3 FIG. 3 FIG. 300 118 300 300 300 302 304 306 308 300 310 312 314 316 300 depicts an example of a block diagram of a frameworkfor active analytics according to an embodiment and is described in reference to. In embodiments, the active analytics toolofcan include one or more elements of the frameworkor may have access to one or more elements of the framework. In the example of, the frameworkcan include a configuration tool, a script runner, and scriptsto test one or more components. The frameworkcan also include user interfaces, frontend application programming interfaces (APIs), configuration data, and backend APIs. Although the example ofdepicts a specific configuration and directional flows, it will be understood that elements of the frameworkmay be combined, modified, or further subdivided.

310 112 310 310 314 312 302 310 302 310 302 304 306 314 304 306 306 310 306 316 316 314 306 306 316 318 1 FIG. The user interfacescan be accessible through the user systemsofwith various levels of access provided, depending upon the permissions of the user. The user interfacescan be used to set up script execution parameters and/or view script execution results. For example, the user interfacescan include a configuration interface that sets values of the configuration datathrough the frontend APIsfor use by the configuration tool. The user interfacescan also access the configuration toolto view current settings and/or modify settings. The user interfacescan also include a scheduling interface that may establish when the configuration toolshould invoke the script runnerto execute one or more scriptsas configured through the configuration data. The script runnercan support multiple script languages, such that the scriptscan be written in one or more scripting languages and may support sequencing of scriptsin two or more different scripting languages. The user interfacescan also monitor results through a results dashboard that displays results of executing the scriptsafter result processing is performed by the backend APIs. The backend APIscan update values of the configuration data, such that results of executing the scriptscan be used to modify configuration conditions in subsequent execution of the scripts. Thus, more complex behaviors can be tested through feedback loops and conditional inputs. The backend APIscan also include condition monitors to trigger one or more alertswhen one or more conditions are met.

312 306 312 130 316 306 316 126 316 308 128 124 122 306 316 130 105 314 316 306 308 314 1 FIG. 1 FIG. 1 FIG. 1 FIG. 1 FIG. Example functions of the frontend APIscan include setting scheduling and configuration values, as well as getting configuration and scriptsto be executed next in a sequence. The frontend APIscan also access execution results, for example, to review previous results captured in the logsof. Example functions of the backend APIscan include capturing execution status of the scripts, such as start conditions, stop conditions, and any errors or warning conditions. The backend APIsmay also perform screen captures to compare layout performance to layout templatesof. The backend APIscan also perform monitoring to compare the response of the componentsto the test criteriaassociated with the test scriptsor the management scriptsofas the scripts. The backend APIsmay also support storing results to the logsofand fetching supplemental values from other sources, such as the external sourcesof, for comparisons or making updates to the configuration data. The backend APIsmay also interface with other monitoring tools to determine the effects of executing the scriptson the components. The configuration datacan include information such as execution parameters, environment variables, log conditions, thresholds, and other such values.

320 300 312 316 320 312 320 316 318 308 312 316 310 320 External toolscan interface with various elements of the framework, such as the frontend APIsand backend APIs. For example, external toolscan use the frontend APIsto configure tests and trigger test execution. The external toolscan access the backend APIsto set trigger conditions for the alert, to access test results, and verify that performance goals are achieved before allowing deployment of new or updated versions of the components. Thus, the frontend APIsand backend APIscan support both direct user interactions through user interfacesand/or automated interactions through the external tools.

4 FIG. 4 FIG. 1 FIG. 1 FIG. 1 FIG. 1 FIG. 3 FIG. 1 FIG. 1 FIG. 400 402 404 406 404 408 408 402 122 408 408 124 410 118 402 408 408 410 112 402 408 408 408 412 414 402 416 418 406 408 408 406 404 414 308 114 104 412 414 408 408 414 414 depicts a block diagram of a test configurationaccording to some embodiments. In the example of, a management scriptcan interface with a test script repositoryand a secure vault. The test script repositorycan store a plurality of test scriptsA-N. The management scriptis an example of one of the management scriptsofand the test scriptsA-N are examples of the test scriptsof. A sequencerof the active analytics toolofcan be used to configure the management scriptto determine which of the test scriptsA-N should be performed in a particular order. For example, the sequencermay have a user interface accessible through the user systemsofto allow a user to select a script execution flow for the management script. For instance, test scriptA may comprise a login script, test scriptB may be directed to testing a display layout under a first set of conditions, and test scriptC may be directed to testing the display layout under a second set of conditions. To establish an environmentfor testing a component, the management scriptmay access credentialsand variablesof the secure vaultas part of configuring and executing one or more of the test scriptsA-N. The secure vaultcan use encryption and additional layers of security beyond that of the test script repository. The componentcan be one of the componentsof, which can be extracted from the component repositoryofor accessed after deployment, for instance, on component execution serversof. Further, the environmentcan include multiple components, and tests of the test scriptsA-N can interact with multiple components, as well as test interactions between components.

402 416 418 412 414 408 408 414 402 416 412 414 408 416 414 128 1 416 418 408 408 404 408 408 416 418 Upon execution of the management script, the credentialsand variablescan be accessed to configure the environmentfor the componentto align with one or more of the test scriptsA-N used to test the component. For example, the management scriptmay use the credentialsto configure the environmentin a valid state to test the componentunder normal expected operating conditions by a test scriptand test for an error condition by using a modified version of the credentialsto ensure that the componentdetects and handles invalid credentials according to the test criteriaof FIG.. Securely storing the credentialsand variablesseparately from the test scriptsA-N of the test script repositorycan allow the test scriptsA-N to be copied and reused to test other components without exposing details of the credentialsand/or variables.

400 420 402 408 408 410 408 408 420 408 408 420 418 402 418 406 412 408 414 420 118 410 The test configurationmay also include a script analyzerthat can assist the management scriptin determining whether the test scriptsA-N rely upon any environment variables for operation. For example, when the sequenceradds one of the test scriptsA-N to a planned test execution sequence, the script analyzercan be invoked to analyze the newly added test scriptto determine whether one or more environment variables are used by the test script. The script analyzercan prompt the user to select a value for the environment variable to support the test or to use a default value. The selected value can then be stored in the variables. When the management scriptruns, the values of the variablescan then be extracted from the secure vaultand used to configure the environmentto support test cases for the test scriptassociated with the corresponding test of the component. The script analyzercan comprise part of the active analytics tooland/or can be combined with the sequencer.

404 406 120 100 404 406 408 408 1 FIG. 1 FIG. The test script repositoryand the secure vaultcan comprise part of the storage systemofor may be implemented elsewhere within the systemof. Although depicted separately, in some embodiments, the test script repositoryand the secure vaultmay be combined, for example, where a higher level of security is desired for the test scriptsA-N.

5 FIG. 1 FIG. 4 FIG. 5 FIG. 500 500 126 500 502 414 414 502 504 504 504 500 506 506 504 504 105 504 504 504 504 502 500 504 502 depicts a layout templateaccording to some embodiments. The layout templateis a visual example of one of the layout templatesof. The layout templatemay define a display regionthat can be displayed by the componentof. For example, the componentcan comprise an application or a webpage that may be viewed through another application, such as a web browser. Within the display region, there can be multiple regions defined at targeted locations. For instance, regionA can comprise a header or banner location, regionB can comprise a left side full panel location, and regionC can comprise a right side partial panel location. The layout templatecan also include one or more undefined regions, where the content at a location of an undefined regiondoes not impact the test results. For the regionsA-C, the content can be defined as a particular graphic, an embedded object, text, or a link to other content, such as the external sources. Testing can include verifying that all of the content of regionsA-C is placed at the target locations. The regionsA-C can be defined through variable dimensions that scale with resizing of the display region. For instance, locations can be defined in terms of relative ratios to support various levels of display sizing. Although the example ofappears as panels on a static display configuration, the layout templatecan be defined as a larger size than can be displayed in a single view to support scrolling and dynamic content. Thus, the regionA can be expected to be initially displayed but would also be expected to scroll off of the visible portion of the display regionas scrolling occurs. Further additional regions for testing may become visible as scrolling occurs. It will be understood that many configuration options are possible with any number of regions defined for testing.

6 FIG. 3 FIG. 4 FIG. 4 FIG. 4 FIG. 600 600 310 402 600 602 604 402 606 608 610 612 614 604 402 604 606 416 414 606 606 608 414 610 610 612 408 408 408 408 408 408 408 depicts a user interfaceaccording to some embodiments. The user interfaceis an example of one of the user interfacesoffor configuring the management scriptof. The user interfaceallows general information, such as a nameand team, to be defined for the management script. Other configuration items can include a security model, an environment definition, a number of retries, a script list, and environment variables. The teamcan define a group of users with access to edit and/or execute the management script. Membership in the group of users associated with teamcan be used to restrict access for viewing, creating, and/or editing various parameters. The security modelcan be used in combination with the credentialsto establish secure execution of the componentof. The security modelcan be any known in the art, such as Basic Auth, Bearer Tokens, Web Forms, Single Sign On (SSO), and the like. Further, various sub-options of implementation variations can be supported through the security model. The environment definitioncan define whether the componentwill be tested in a development environment, a production environment, or other such environment. The number of retriescan define how many sequential failures are allowed before triggering an alert notification. For example, if the number of retriesis 2, then there can be two sequential failures before an alert is generated to cut down on excessive notifications for transient conditions, if desired. The script listcan define which of the test scriptsA-N ofare being called and can establish a sequence of execution. The selected instances of the test scriptsA-N can be executed sequentially, in parallel, or in a mixed combination (e.g., execute a sequence of test scriptsA andB in parallel with test scriptC).

614 402 418 614 614 600 600 312 320 4 FIG. 3 FIG. 6 FIG. The environment variablescan identify the variable names and values used for the management scriptand can be stored in variablesof. The environment variablesalso illustrate an example where some values may be viewable and others may be secure and hidden. Hidden values for sensitive information may appear masked, for instance, as a string of replacement characters (e.g., “********” or other such masking). The environment variablescan be edited, made secure/hidden, and/or deleted from the user interface. Values depicted in the user interfacecan be directly populated, pulled from another source, accessed from a repository, dragged/dropped from another source, or updated in other ways. For instance, values can be populated through the frontend APIsby one or more of the external toolsof. It will be understood that the example ofrepresents an example user interface and other features can be included or features omitted in embodiments.

7 FIG. 7 FIG. 4 FIG. 7 FIG. 11 FIG. 7 FIG. 700 700 408 408 702 704 408 706 710 712 714 716 708 408 710 408 712 714 408 716 408 700 720 720 706 722 depicts a user interfaceaccording to some embodiments. In the example of, the user interfacedepicts a summary of the configuration and results of executing a test script, such as one of the test scriptsA-N of. The summary can be associated with a particular script configuration. The summary can include schedule information, such as an execution frequency and a number of retries. The summary can also include run-time configuration information, such as a username, a password, a security model, and a resource location to execute the test scriptwith user permissions set to match permissions of a user account associated with the username. The summary can also include results, such as a start time, an end time, a duration, a result status, and other such information. Various commands can be selectable, such as run now 708, pause, edit, mute, and/or view script. Run nowcan initiate execution of the test scriptthat may otherwise be configured to run at a scheduled execution frequency. Pausecan temporarily halt the test scriptduring execution. Editcan support changing operational parameters (e.g., environment variables) used during script execution. Mutecan allow the test scriptto run without triggering an alert notification. View scriptcan open the test scriptfor viewing and/or editing. Further, the user interfacecan include links to other tools, such as an application dashboard tool link. The application dashboard tool linkcan pass information to another tool or be a targeted link that closely aligns with other data relevant to execution of the script associated with. The resultscan include further options, for instance, to open detailed result logs, for a particular script execution (e.g., as further described in reference to). It will be understood that the example ofrepresents an example user interface and other features can be included or features omitted in embodiments.

8 FIG. 8 FIG. 4 FIG. 4 FIG. 8 FIG. 800 800 408 408 802 402 418 802 802 800 800 408 408 408 408 800 804 804 408 408 804 depicts a user interfaceaccording to some embodiments. In the example of, the user interfacedepicts a configuration interface associated with executing a test script, such as one of the test scriptsA-N of, and a notification selector. Environment variablescan identify the variable names and values used for the management scriptand can be stored in variablesof. The environment variablescan also store access and security information. The environment variablescan be edited, made secure/hidden, and/or deleted from the user interface. The user interfacecan support settings for a single one of the test scriptsA-N or a group of test scriptsA-N. The user interfacecan also include an alert profile selector. The alert profile selectorcan identify one or more services to invoke when the associated test scriptsA-N fail or meet a notification condition. Examples of services of the alert profile selectorcan include an IT service management system, a collaborative tool, e-mail, a paging system, and/or other such services. Services can have different types of content and delivery options. For instance, IT service management may trigger an investigation request for an IT group to further analyze the results, while the collaborative tool may open a chat session for a group of users. For each of the services, a default reporting message and distribution group can be defined. Thus, one group of users may be notified via a first service, and another group of users can be notified via a second service. It will be understood that the example ofrepresents an example user interface and other features can be included or features omitted in embodiments.

9 FIG. 9 FIG. 4 FIG. 7 FIG. 9 FIG. 900 900 902 408 408 902 900 902 900 904 902 906 depicts a user interfaceaccording to some embodiments. In the example of, the user interfacedepicts a summary of the resultsof executing multiple test scripts, such as the test scriptsA-N of. Viewing of the resultsthrough the user interfacecan be limited based on the permissions of the user and/or team membership. The resultscan identify the test script, the last result, the time/date of the last execution of the script, a scheduled frequency of execution, and other such information. The last result can comprise a visual indicator of a pass, a failure, and/or warning. A warning may be used to indicate a deviation in performance that is not considered a failure, such as meeting some but not all of the service level objectives. The user interfacecan also include an option to add a new scriptto the list of scripts currently being executed. The resultscan include further options, for instance, to open a summary for a particular script (e.g., as previously described in reference to). It will be understood that the example ofrepresents an example user interface and other features can be included or features omitted in embodiments.

10 FIG. 10 FIG. 4 FIG. 9 FIG. 4 FIG. 10 FIG. 1000 1000 408 408 1000 904 1002 404 1000 1004 1004 1006 depicts a user interfaceaccording to some embodiments. In the example of, the user interfacedepicts an option of adding a new script for execution, such as the test scriptsA-N of. The user interfacemay be invoked when a user selects the option to add a new scriptof. General script execution informationcan include a script name, an execution frequency, and a script file to be uploaded. The uploading of the script file can support linking to an existing test script or adding a new test script to the test script repositoryof. The user interfacecan also include environment variables. The environment variablescan identify information associated with an environment, such as a username, a password, a security model, and a resource location to execute the test script with user permissions set to match permissions of a user account associated with the username. The user interfacecan also include command options to support saving or canceling the request to add a script. It will be understood that the example ofrepresents an example user interface and other features can be included or features omitted in embodiments.

11 FIG. 11 FIG. 4 FIG. 1 FIG. 5 FIG. 5 FIG. 11 FIG. 1100 1100 408 408 1102 1104 1106 1108 1100 1110 1110 126 1110 500 504 1110 504 1110 1100 1112 1112 1112 1100 1114 depicts a user interfaceaccording to some embodiments. In the example of, the user interfacedepicts a test execution summary of a test script, such as one of the test scriptsA-N of. Summary information can include, for instance, a start time, an end time, a duration, a result status, and other such information. The summary information may be from a most recent execution of the test script and may halt upon a failure condition. Alternatively, a user may be able to scroll through a list of results and view associated summary information for a selected execution. The user interfacecan also depict captured screen shotscollected during execution of the test script. The screen shotscan be used in making comparisons to one or more layout templatesofto detect issues with a display result of a component under test. The screen shotscan also mark differences detected relative to an associated layout templateof. For instance, if the difference is detected in regionC ofin one of the screen shots, a corresponding area of the screen shot that aligns with regionC may be marked with a visual overlay (e.g., a box or highlighting). Further, detected differences in the screen shotscan be logged with descriptive text to explain a detected mismatch. The user interfacecan also display one or more records of a test history log. The test history logmay provide general warnings of detected minor problems (e.g., an unexpected non-critical image) and identify failure conditions that are considered an error, such as a login failure. The test history logmay include links to view additional details about warnings and errors. The user interfacemay also include other linksto support navigation to other user interfaces. It will be understood that the example ofrepresents an example user interface and other features can be included or features omitted in embodiments.

12 FIG. 4 FIG. 4 FIG. 5 FIG. 1200 1200 1202 1204 1206 1210 1212 1202 1204 1204 1206 1206 1208 1209 1208 408 408 414 1209 414 1206 1210 1208 1212 500 1209 1206 depicts a training and prediction processaccording to some embodiments. The training and prediction processcan include a training processthat analyzes training datato develop trained modelsfor a personally identifiable information (PII) predictorand a variance predictor. The training processcan use labeled or unlabeled data in the training datato learn features, such as a mapping of words and phrases to PII and mapping images to image variations. The training datacan include name lists and image content to establish a ground truth for learning coefficients/weights and other such features known in the art of machine learning to develop trained models. Images for training can include intentionally modified images that change the scale, color, orientation, contrast, and other such parameters of baseline training images. The trained modelscan include a family of models to identify specific types of features from text dataand/or image data. The text datacan be collected during execution of test scriptsA-N that result in a text output of the componentof. The image datacan include portions of images extracted from image output of the componentof. The trained modelscan include the PII predictorto identify likely PII in text data. The variance predictorcan be used to determine a variance score based on a difference between an observed layout and a layout template, such as layout templateoffor image data. Detection of image differences can use any known image processing technique and may pass image data to a separate image processing service to produce summarized results. Other such models and further subdivision of the trained modelscan be incorporated in various embodiments.

1206 1214 1216 1214 1216 504 504 500 1214 1214 1204 1206 1200 100 100 5 FIG. 5 FIG. 1 FIG. The trained modelscan output a confidence determinationindicating a confidence level of a PII prediction or variance prediction. For example, where a text pattern is detected that maps to a pattern similar to other PII, the confidence determination may be above a threshold as matching a known pattern. Result postprocessingcan determine an action to take based on the confidence level identified by the confidence determination, such as verifying that a security operation is performed on the PII. Result postprocessingcan include determining whether content in one of the regionsA-C ofis considered sufficiently similar to the expected content when comparing observed images of a screen shot to the layout templateof. Where the image similarity is below a confidence threshold from the confidence determination, an image fault can be declared in the test results. As results are processed and interpreted, the results or user responses to the results can be used as feedback to adjust the confidence determinationor trigger retraining. For example, where tests incorrectly fail, the results can be used to add to the training datato periodically retrain the trained modelswith higher quality examples. Different image processing effects can be overlayed depending on the confidence level. For instance, a lower confidence difference that exceeds a minimum threshold can have a different image effect overlay than a higher confidence difference that also exceeds the minimum threshold (e.g., a thinner border around the difference area that expands to a thicker border with higher confidence). It will be understood that the training and prediction processcan be performed by any portion of the systemofand/or may be performed by another server (not depicted) which may be accessible by the system.

13 FIG. 1 FIG. 1 13 FIGS.- 1 FIG. 1300 1300 1300 100 1300 118 Turning now to, a process flowis depicted according to an embodiment. The process flowincludes a number of steps that may be performed in the depicted sequence or in an alternate sequence. The process flowmay be performed by the systemof. The process flowis described in reference toand can be performed by the active analytics toolofin combination with other tools.

1302 118 402 408 404 418 408 408 414 404 408 408 402 408 402 408 At block, the active analytics toolcan configure a management scriptassociated with a test scriptof the test script repositoryto set one or more environment variables (e.g., variables) used by the test script, where the test scriptis configured to test a software component (e.g., component). The test script repositorycan include a plurality of test scriptsA-N configured to synthesize user interactions with one or more software components. Configuring the management scriptcan include setting a start condition and an end condition to execute a portion of the test script. Configuring the management scriptcan also include setting a username, a password, a security model, and a resource location to execute the test scriptwith user permissions set to match permissions of a user account associated with the username.

1304 118 402 408 304 3 FIG. At block, the active analytics toolcan execute the management scriptat a scheduled frequency to trigger testing of the software component by the test script. Script execution can be performed by the script runnerof.

1306 118 408 1308 118 128 At block, the active analytics toolcan access one or more results of executing the test script. At block, the active analytics toolcan compare the one or more results to a plurality of test criteriato determine a test execution summary.

1310 118 318 804 318 At block, the active analytics toolcan trigger an alertto notify a service based on the test execution summary indicating a target condition. The service can be selected from a group of services through the alert profile selector. For instance, the service can comprise a message service, an email service, a meeting service, a maintenance request service, or other such service. The service to send the alertcan be selectable through a user interface, and an alert recipient list can be configurable on a per service basis.

118 408 130 408 In some embodiments, the active analytics toolcan track a plurality of execution statistics associated with executing the test script, store the execution statistics with the test execution summary in a test history log (e.g., as part of the logs), and display one or more records of the test history log on a user interface based on a request received through a user interface. The test history log can be configured to track a start time, an end time, a duration, a pass/fail status of a plurality of test runs of the test script, and other such information.

118 408 408 402 408 In some embodiments, the active analytics toolcan perform an analysis of the test scriptto search for at least one environment variable used by the test scriptand generate a list of environment variables to be populated by the management scriptprior to executing the test script.

402 408 408 In some embodiments, the management scriptcan be configured to change at least one input used by the test scriptduring execution of the test script. This can support dynamic testing, for instance, to test a range of conditions, such as maximum and minimum values of parameters. Here, “during execution”, can include time between sequential script execution as part of a larger script execution process. Further, changes made during script execution can include insertion of error conditions to determine whether the errors are handled properly.

402 408 408 In some embodiments, a plurality of management scriptscan be sequenced to execute and collect test data from execution of a plurality of test scriptsA-N.

118 In some embodiments, the active analytics toolcan analyze one or more results to check one or more of: a search engine optimization result, tracking module parameters, cookies, text content, and a document object model validation status.

14 FIG. 1 FIG. 1 14 FIGS.- 13 FIG. 1400 1400 1400 100 1400 118 100 1400 1300 Turning now to, a process flowis depicted according to an embodiment. The process flowincludes a number of steps that may be performed in the depicted sequence or in an alternate sequence. The process flowmay be performed by the systemof. The process flowis described in reference toand can be performed by the active analytics tooland/or other components of the system. The process flowcan be performed as part of the process flowofto support test script configuration and execution.

1402 118 105 110 1404 118 105 1406 118 105 105 105 1408 118 130 At block, the active analytics toolcan access one or more external sourcesthrough a networkto determine at least one value of the one or more environment variables. At block, the active analytics toolcan set the at least one value of the one or more environment variables based on accessing the one or more external sources. At block, the active analytics toolcan set the at least one value of the one or more environment variables to a default value based on detecting an error when attempting to access the one or more external sources. For example, if the external sourcescannot be reached, then a default value can be used in place of data from the external sources. At block, the active analytics toolcan record the error in a test log (e.g., in logs) associated with testing of the software component.

15 FIG. 1 FIG. 13 FIG. 1 15 FIGS.- 1500 1500 1500 100 1500 1300 1500 Turning now to, a process flowis depicted according to an embodiment. The process flowincludes a number of steps that may be performed in the depicted sequence or in an alternate sequence. The process flowmay be performed by the systemof. The process flowcan expand upon the process flowof. The process flowis described in reference to.

1502 118 1110 408 1504 118 1110 1506 118 1110 500 500 504 504 1508 118 500 1510 118 118 118 1210 1212 500 500 500 At block, the active analytics toolcan capture one or more screen shotsassociated with testing of the software component as the test scriptis executing. At block, the active analytics toolcan analyze the one or more screen shotsfor text, graphic, and object content. At block, the active analytics toolcan compare an observed layout of the one or more screen shotsto a layout template. The layout templatecan identify one or more regionsA-C of the observed layout to analyze for variance scoring. At block, the active analytics toolcan determine a variance score based on detecting a difference between the observed layout and the layout template. At block, the active analytics toolcan log a variance notification based on the variance score indicating a deviation beyond a threshold level. In some embodiments, the active analytics toolcan scan the text for personally identifiable information and verify that the display of personally identifiable information complies with one or more security rules. In some embodiments, the active analytics toolcan use the PII predictorto scan the text for PII and use the variance predictorto determine a variance score based on detecting a difference between the observed layout and the layout template. The layout templatecan be learned or adapted. For example, an initial combination of images on a web page may be captured as the layout templatewithout specifically identifying regions within the web page. If a change to the web page is detected, a user may be notified to confirm whether the change should be the basis for a new/adjusted layout template or whether it is an unexpected variance. Change detection can be performed for various viewable regions or can span larger areas that extend beyond a currently viewable area. For example, layout templates can be compared as scrolling is performed and previously non-viewable content moves into a viewable area.

16 FIG. 1 FIG. 13 FIG. 1 16 FIGS.- 1600 1600 1600 100 1600 1300 1600 Turning now to, a process flowis depicted according to an embodiment. The process flowincludes a number of steps that may be performed in the depicted sequence or in an alternate sequence. The process flowmay be performed by the systemof. The process flowcan expand upon the process flowof. The process flowis described in reference to.

1602 118 408 700 1604 118 408 408 1606 118 1608 118 408 1610 118 1612 118 408 320 312 312 316 320 320 3 FIG. At block, the active analytics toolcan execute the test scripton demand in response to a start command received through a user interface, such as user interface. At block, the active analytics toolcan pause the test scriptduring execution of the test scriptin response to a pause command received through the user interface. At block, the active analytics toolcan edit at least one of the one or more environment variables through the user interface. At block, the active analytics toolcan display contents of the test scriptin response to a view script command received through the user interface. At block, the active analytics toolcan set the scheduled frequency for executing the test script through the user interface. At block, the active analytics toolcan set a number of retries of executing the test scriptbefore declaring a failure, where the failure can comprise a target condition. Test script execution can also be triggered by one or more external tools, for example, through frontend APIsof. Adjustment to test configuration may also be performed through frontend APIsand/or backend APIsby one or more external tools. For instance, the scheduled frequency for executing a test script and/or a number of retries may be modified by the external toolsin an automated mode of operation.

Technical effects include providing a framework to manage and measure synthetic user activity within a system. Embodiments support script modularization by allowing a same script to be reused in different environments and execution contexts. Modularization can allow part of a script, such as only a step, to run where less than a complete script is needed. Embodiments can also obtain or call external values from environment variables and/or external sources for added flexibility. Response and results of running test scripts for one or more software components can be interrogated to analyze content, consistency, and other such features. Image variance analysis can score differences between observed content output by a software component as compared to expected content. A variety of script languages can be used, including a sequence of scripts written in different script languages as part of a larger-scale test process.

It will be appreciated that aspects of the present invention may be embodied as a system, method, or computer program product and may take the form of a hardware embodiment, a software embodiment (including firmware, resident software, micro-code, etc.), or a combination thereof. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

One or more computer readable medium(s) may be utilized. The computer readable medium may comprise a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may comprise, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In one aspect, the computer readable storage medium may comprise a tangible medium containing or storing a program for use by or in connection with an instruction execution system, apparatus, and/or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may comprise any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, and/or transport a program for use by or in connection with an instruction execution system, apparatus, and/or device.

The computer readable medium may contain program code embodied thereon, which may be transmitted using any appropriate medium, including, but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. In addition, computer program code for carrying out operations for implementing aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.

It will be appreciated that aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products, according to embodiments of the invention. It will be understood that each block or step of the flowchart illustrations and/or block diagrams, and combinations of blocks or steps in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer program instructions may also be loaded on to a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

In addition, some embodiments described herein are associated with an “indication”. As used herein, the term “indication” may be used to refer to any indicia and/or other information indicative of or associated with a subject, item, entity, and/or other object and/or idea. As used herein, the phrases “information indicative of” and “indicia” may be used to refer to any information that represents, describes, and/or is otherwise associated with a related entity, subject, or object. Indicia of information may include, for example, a code, a reference, a link, a signal, an identifier, and/or any combination thereof and/or any other informative representation associated with the information. In some embodiments, indicia of information (or indicative of the information) may be or include the information itself and/or any portion or component of the information. In some embodiments, an indication may include a request, a solicitation, a broadcast, and/or any other form of information gathering and/or dissemination.

Numerous embodiments are described in this patent application, and are presented for illustrative purposes only. The described embodiments are not, and are not intended to be, limiting in any sense. The presently disclosed invention(s) are widely applicable to numerous embodiments, as is readily apparent from the disclosure. One of ordinary skill in the art will recognize that the disclosed invention(s) may be practiced with various modifications and alterations, such as structural, logical, software, and electrical modifications. Although particular features of the disclosed invention(s) may be described with reference to one or more particular embodiments and/or drawings, it should be understood that such features are not limited to usage in the one or more particular embodiments or drawings with reference to which they are described, unless expressly specified otherwise.

Devices that are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. On the contrary, such devices need only transmit to each other as necessary or desirable, and may actually refrain from exchanging data most of the time. For example, a machine in communication with another machine via the Internet may not transmit data to the other machine for weeks at a time. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more intermediaries.

A description of an embodiment with several components or features does not imply that all or even any of such components and/or features are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention(s). Unless otherwise specified explicitly, no component and/or feature is essential or required.

Further, although process steps, algorithms or the like may be described in a sequential order, such processes may be configured to work in different orders. In other words, any sequence or order of steps that may be explicitly described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order practical. Further, some steps may be performed simultaneously despite being described or implied as occurring non-simultaneously (e.g., because one step is described after the other step). Moreover, the illustration of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to the invention, and does not imply that the illustrated process is preferred.

“Determining” something can be performed in a variety of manners and therefore the term “determining” (and like terms) includes calculating, computing, deriving, looking up (e.g., in a table, database or data structure), ascertaining and the like.

It will be readily apparent that the various methods and algorithms described herein may be implemented by, e.g., appropriately and/or specially-programmed computers and/or computing devices. Typically a processor (e.g., one or more microprocessors) will receive instructions from a memory or like device, and execute those instructions, thereby performing one or more processes defined by those instructions. Further, programs that implement such methods and algorithms may be stored and transmitted using a variety of media (e.g., computer readable media) in a number of manners. In some embodiments, hard-wired circuitry or custom hardware may be used in place of, or in combination with, software instructions for implementation of the processes of various embodiments. Thus, embodiments are not limited to any specific combination of hardware and software.

A “processor” generally means any one or more microprocessors, CPU devices, computing devices, microcontrollers, digital signal processors, or like devices, as further described herein.

The term “computer-readable medium” refers to any medium that participates in providing data (e.g., instructions or other information) that may be read by a computer, a processor or a like device. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include DRAM, which typically constitutes the main memory. Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves and electromagnetic emissions, such as those generated during RF and IR data communications. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip or cartridge, a carrier wave, or any other medium from which a computer can read.

The term “computer-readable memory” may generally refer to a subset and/or class of computer-readable medium that does not include transmission media such as waveforms, carrier waves, electromagnetic emissions, etc. Computer-readable memory may typically include physical media upon which data (e.g., instructions or other information) are stored, such as optical or magnetic disks and other persistent memory, DRAM, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip or cartridge, computer hard drives, backup tapes, Universal Serial Bus (USB) memory devices, and the like.

Various forms of computer readable media may be involved in carrying data, including sequences of instructions, to a processor. For example, sequences of instruction (i) may be delivered from RAM to a processor, (ii) may be carried over a wireless transmission medium, and/or (iii) may be formatted according to numerous formats, standards or protocols, such as Bluetooth™, TDMA, CDMA, 3G, 4G, 5G, etc.

Where databases are described, it will be understood by one of ordinary skill in the art that (i) alternative database structures to those described may be readily employed, and (ii) other memory structures besides databases may be readily employed. Any illustrations or descriptions of any sample databases presented herein are illustrative arrangements for stored representations of information. Any number of other arrangements may be employed besides those suggested by, e.g., tables illustrated in drawings or elsewhere. Similarly, any illustrated entries of the databases represent exemplary information only; one of ordinary skill in the art will understand that the number and content of the entries can be different from those described herein. Further, despite any depiction of the databases as tables, other formats (including relational databases, object-based models and/or distributed databases) could be used to store and manipulate the data types described herein. Likewise, object methods or behaviors of a database can be used to implement various processes, such as the described herein. In addition, the databases may, in a known manner, be stored locally or remotely from a device that accesses data in such a database.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one more other features, integers, steps, operations, element components, and/or groups thereof.