Test Recommendation Based on a Source Code Dependency Graph and Code Coverage Results

Software code can be tested to identify errors in the code.

The following presents a simplified summary of the disclosed subject matter in order to provide a basic understanding of some of the various embodiments. This summary is not an extensive overview of the various embodiments. It is intended neither to identify key or critical elements of the various embodiments nor to delineate the scope of the various embodiments. Its sole purpose is to present some concepts of the disclosure in a streamlined form as a prelude to the more detailed description that is presented later.

An example system can operate as follows. The system can create a dependency graph that represents dependencies within computer code, wherein nodes of the dependency graph represent files of the computer code, wherein edges of the dependency graph represent respective relationships between respective files of the files, and wherein respective nodes of the nodes are associated with respective control flow graphs for respective code blocks of the computer code that are stored in the respective files. The system can add respective attributes to the respective nodes that identify respective check tests that are associated with the respective files. The system can, for a pull request regarding the computer code, determine a first subgroup of the files that is modified by the pull request, and a group of functions that is modified by the pull request. The system can, based on the dependency graph, the first subgroup of the files, and the group of functions, identify call chains of the computer code that are affected by the group of functions, and a second subgroup of the files that is affected by the group of functions. The system can, based on the dependency graph, the call chains, and the second subgroup of the files, determine a subgroup of the check tests that corresponds to a portion of the computer code that is modified by the pull request. The system can store an indication of the subgroup of the check tests.

An example method can comprise creating, by a system comprising at least one processor, a dependency graph that represents dependencies within computer code, wherein nodes of the dependency graph represent files of the computer code, wherein edges of the dependency graph represent respective relationships between respective files of the files, wherein respective nodes of the nodes are associated with respective control flow graphs for respective code blocks of the computer code that are stored in the respective files, and wherein respective attributes of the respective nodes identify respective check tests that are associated with the respective files. The method can further comprise, for a pull request regarding the computer code, determining, by the system, a first subgroup of the files that is modified by the pull request, and a group of functions that is modified by the pull request. The method can further comprise, based on the dependency graph, the first subgroup of the files, and the group of functions, identifying, by the system, call chains of the computer code that are impacted by the group of functions, and a second subgroup of the files that is impacted by the group of functions. The method can further comprise, based on the dependency graph, the call chains, and the second subgroup of the files, determining, by the system, a subgroup of the check tests that corresponds to a portion of the computer code that is modified by the pull request.

An example non-transitory computer-readable medium can comprise instructions that, in response to execution, cause a system comprising a processor to perform operations. These operations can comprise creating a dependency graph that represents dependencies within computer code, wherein nodes of the dependency graph represent files of the computer code, wherein edges of the dependency graph represent respective relationships between respective files of the files, wherein respective nodes of the nodes are associated with respective control flow graphs for respective code blocks of the computer code that are stored in the respective files, and wherein respective attributes of the respective nodes identify respective check tests that are associated with the respective files. These operations can further comprise, for a pull request regarding the computer code, determining a first subgroup of the files that is modified by the pull request, and a group of functions that is modified by the pull request. These operations can further comprise, based on the dependency graph, the first subgroup of the files, the group of functions, call chains of the computer code that are affected by the group of functions, and a second subgroup of the files that are affected by the group of functions, determining a subgroup of the check tests that corresponds to a portion of the computer code that is modified by the pull request.

When developing code, developers can want to ensure that changes that are made do not cause any sort of regression. Prior approaches to addressing this generally involve writing new unit tests, confirming no regression in existing unit tests (e.g., failing tests that previously passed), and some level of manual testing. These prior approaches can be useful for the developer to sanity check, but are generally not exhaustive.

The present techniques can be implemented to provide a method where, based on a code diff, there can be a determination of a set of automated functional tests (that is, test lists) and unit tests that are to be executed to ensure that a regression has not been introduced.

A graph from each source code file and its dependencies can be generated based on a source scanning tool that creates a dependency graph of a source tree. Each of those files can be augmented with flow information from a control-flow-graph that captures defined functions and their subsequent call chains.

A dependency graph can generally comprise nodes that represent files, and directed edges between the nodes that represent dependencies between the files (e.g., Function A of File A calls Function B of File B). A control flow diagram (CFG) can generally represent the flow control within a program. A CFG can be structured a directed graph where code blocks are represented as nodes and flow paths are represented by edges. A CFG can be used in analysis and optimization of an execution flow within a software program.

A pull request (PR) impact generator; A code coverage report analyzer; and A test recommender. This approach can generally involve three components:

An enhanced dependency graph using a control flow graph (CFG), which can be referred to as a dependency graph (DG). A pull request difference (diff) relative to an existing code base. The PR impact generator can take two inputs:

The PR impact generator can analyze the pull request diff and determine which files are touched and which functions within the files were modified.

Using the DG, the PR impact generator can then find call chains and respective files within the source tree that leverage the functions that were modified. From here, an analysis can be performed on each of the impacted call chains to determine which check tests are impacted. This can be done by adding an attribute to nodes in the DG that identify which files point to check tests. Once that is in place, the graph can be queried in a manner that facilitates looking for any node that contains the functions used in its CFG.

Files that are touched by the diff; Functions within that file that are modified; and Relevant check tests that can be executed to validate no regression has been introduced. This information can be included in a report that can comprise:

This information can be included in a report (e.g., a document in a JavaScript Object Notation (JSON) structure) to be used as an input for the test recommender.

The other component according to the present techniques is the code coverage report analyzer. Unlike the PR impact generator that can be intended to be run every time a PR is generated, it can be that there no hard requirement as to how often the code coverage report analyzer should run. In some examples, it can be that it needs to be run often enough to ensure the code coverage report still provides valuable results against the PR impact report. That is, the code coverage report can serve as a baseline, and, if the code base deviates enough over time, a new baseline can be determined.

The code coverage report analyzer can take, as an input, the code coverage reports from test lists executed on different platforms, and outputs a baseline coverage report.

Name of the test list; Platform the list was executed on; and The coverage report for the execution of the test list on the given platform. The baseline coverage report can provide the following details for each test list executed:

Compare coverage reports from different platforms running the same test list. If the reports match (that is, lines tested match), those platforms can be deemed as being equivalent. Any non-matching platforms can be assumed to be testing different code paths, and can be deemed not equivalent to others. Analyze each of the coverage reports and determine which files are touched and what functions are impacted by the test list. Using the coverage report, the analyzer can:

Using the baseline coverage report and the PR impact report, this information can be passed to a test recommender. The test recommender can look at the PR impact report and cycle through the files that are touched by the diff. Knowing this, the test recommender can cycle through the baseline coverage report and walk through each of the lists looking to find tests that touch those files. If the files are touched, the test recommender can then look at functions modified from the diff. If those same functions are touched in the baseline coverage report, then the tests can be appended to the test recommendation.

This can come from the PR impact generator; Check tests to be executed; If a test list identified has equivalency defined, this information can be included in this report, and it can be left to the test executor to determine which platform it should run on. That test executor can know to only execute on one of the supported platforms, and can inspect the infra to see what is the “most” available. Which tests to execute and on what platforms; After cycling through the files within the PR impact report, and the tests from the baseline coverage report, a new test recommendation report can be generated that includes:

The present techniques can facilitate using a CFG to determine check unit tests impacted by a code change. The present techniques can also facilitate determining a minimum test matrix (test and platform) by cross-referencing code coverage data and a code diff.

1 FIG. 100 illustrates an example system architecturethat can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure.

100 102 104 106 102 108 110 112 114 System architecturecomprises computer system, communications network, and repository. In turn, computer systemcomprises test recommendation based on a source code dependency graph and code coverage results component, code base, diff, and tests.

102 106 1000 104 10 FIG. Each of computer systemand/or repositorycan be implemented with part(s) of computing environmentof. Communications networkcan comprise a computer communications network, such as the Internet.

106 106 108 110 110 112 114 110 110 Repositorycan store source code for computer applications. When a pull request is made to a code base in repository, that can be fetched by test recommendation based on a source code dependency graph and code coverage results component, as code base. A difference (diff) between code of the pull request and code basecan be determined as diff. Testscan be check tests for code basethat can detect whether the code of code basehas errors.

108 110 112 112 114 Test recommendation based on a source code dependency graph and code coverage results componentcan use this information to build a dependency graph of code baseto determine which files contain which functions and which interdependencies (e.g., a Function A in File A calls Function B in File B). This information can be used to determine which tests test the code that is modified in diff, and then to test diff(without re-running every test of tests, which can consume computing resources on re-testing parts of the code that are unmodified in the pull request).

108 5 9 FIGS.- In some examples, test recommendation based on a source code dependency graph and code coverage results componentcan implement part(s) of the process flows ofto implement test recommendation based on a source code dependency graph and code coverage results.

100 It can be appreciated that system architectureis one example system architecture for test recommendation based on a source code dependency graph and code coverage results, and that there can be other system architectures that facilitate test recommendation based on a source code dependency graph and code coverage results.

2 FIG. 1 FIG. 200 200 100 illustrates an exampleof a pull request impact report, and that can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, part(s) of examplecan be used by part(s) of system architectureofto facilitate test recommendation based on a source code dependency graph and code coverage results.

200 Exampleshows:

{ “files”:[ { “name”: “/lib/foo/foo.c”, “functions”:[“foo_from_string”, “foo_from_int”], “check tests”: [“lib/foo/check_foo.c”], }, . . . ] }

3 FIG. 1 FIG. 300 300 100 illustrates an exampleof a baseline coverage report, and that can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, part(s) of examplecan be used by part(s) of system architectureofto facilitate test recommendation based on a source code dependency graph and code coverage results.

{ “tests”: [ { “test list name”: “test_FOO.list”, “platform”: “DuctTape”, “lcov”: “test_FOO_DuctTape.lcov” “platform equivalency”: [“H710”, “F900”] “files”: [ { “name”: “foo1.c”, “functions”: [“bar1, bar2”] }, { “name”: “/lib/foo/foo.c”, “functions”: [“foo_from_string”] }, . . . ] }, . . . ] }

4 FIG. 1 FIG. 400 400 100 illustrates an exampleof a pull request test recommendation report, and that can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, part(s) of examplecan be used by part(s) of system architectureofto facilitate test recommendation based on a source code dependency graph and code coverage results.

{ “check tests”: [“lib/foo/check_foo.c”], “functional tests”: [ { “test”: “test_FOO.list” “platforms”: [“DuctTape”, “F900”, “H710”] }, . . . ] }

5 FIG. 1 FIG. 10 FIG. 500 500 108 1000 illustrates an example process flowthat can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, one or more embodiments of process flowcan be implemented by test recommendation based on a source code dependency graph and code coverage results componentof, or computing environmentof.

500 500 600 700 800 900 6 FIG. 7 FIG. 8 FIG. 9 FIG. It can be appreciated that the operating procedures of process floware example operating procedures, and that there can be embodiments that implement more or fewer operating procedures than are depicted, or that implement the depicted operating procedures in a different order than as depicted. In some examples, process flowcan be implemented in conjunction with one or more embodiments of one or more of process flowof, process flowof, process flowof, and/or process flowof.

500 502 504 506 508 510 512 514 516 518 520 522 524 526 528 530 532 534 536 Process flowcomprises source code, source scanning tool, parse files, generate dependency graph, generate control flow graphs, enhance dependency graph using control flow graphs, PR impact generator, PR difference (diff), PR impact report, test lists, test, test infrastructure (infra), code coverage report, code coverage report analyzer, baseline coverage report, test recommender, PR test recommendation report, and test executor.

6 FIG. 1 FIG. 10 FIG. 600 600 108 1000 illustrates an example process flowfor test recommendation, that can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, one or more embodiments of process flowcan be implemented by test recommendation based on a source code dependency graph and code coverage results componentof, or computing environmentof.

600 600 700 800 5 FIG. 7 FIG. 8 FIG. 9 FIG. It can be appreciated that the operating procedures of process floware example operating procedures, and that there can be embodiments that implement more or fewer operating procedures than are depicted, or that implement the depicted operating procedures in a different order than as depicted. In some examples, process flowcan be implemented in conjunction with one or more embodiments of one or more of process flow of, process flowof, process flowof, and/or process flow of.

600 602 604 606 608 Process flowcomprises baseline coverage report, PR impact report, test recommender, and test recommendation report.

602 Baseline coverage reportidentifies:

name: Foo_A_list platform: DuctTape lcov: test_Foo_A_DuctTape.lcov platform equivalency: H710 files: [{ name: foo1.c functions: bar1, bar2 }] name: Foo_B_list platform: DuctTape lcov: test_Foo_B_DuctTape.lcov platform equivalency: none files: [{ name: foo2.c functions: bar4, bar5 }] name: Foo_A_list platform: DuctTape lcov: test_Foo_A_DuctTape.lcov platform equivalency: DuctTape files: [{ name: foo1.c functions: bar1, bar2 }]

602 In baseline coverage report, two of the coverage reports match for the same files, so an indication of this equivalency is added to the reports.

604 PR impact reportindicates: files:

files: [ { “name”: “lib/foo/foo.c” “functions”: [foo_from_string”, “foo_from_int”], “check tests”: [“lib/foo/check_foo.c”], }, { “name”: “foo1.c”, “functions”: [“bar1,” “bar2”], “check tests”: [“check_foo.c”], }, ]

604 In PR impact report, a test recommender looks for each of the files touched by the PR. In this case, foo1.c is touched by test list “Foo_A. list.” This can be added to the test recommender report.

608 Test recommendation reportindicates:

check tests: check_foo.c functional tests: [ { test: “Foo_A,list” platforms: H710, DuctTape } ]

7 FIG. 1 FIG. 10 FIG. 700 700 108 1000 illustrates another example process flowthat can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, one or more embodiments of process flowcan be implemented by test recommendation based on a source code dependency graph and code coverage results componentof, or computing environmentof.

700 700 500 600 800 900 5 FIG. 6 FIG. 8 FIG. 9 FIG. It can be appreciated that the operating procedures of process floware example operating procedures, and that there can be embodiments that implement more or fewer operating procedures than are depicted, or that implement the depicted operating procedures in a different order than as depicted. In some examples, process flowcan be implemented in conjunction with one or more embodiments of one or more of process flowof, process flowof, process flowof, and/or process flowof.

700 702 704 Process flowbegins with, and moves to operation.

704 Operationdepicts creating a dependency graph that represents dependencies within computer code, wherein nodes of the dependency graph represent files of the computer code, wherein edges of the dependency graph represent respective relationships between respective files of the files, and wherein respective nodes of the nodes are associated with respective control flow graphs for respective code blocks of the computer code that are stored in the respective files. This can be a dependency graph and control flow graphs as described herein.

704 700 706 After operation, process flowmoves to operation.

706 Operationdepicts adding respective attributes to the respective nodes that identify respective check tests that are associated with the respective files. That is, the check tests associated with the files can be added to the dependency graph at the nodes that correspond to the files.

In some examples, a file that contains check tests can be identified by a prefix in its file name.

706 700 708 After operation, process flowmoves to operation.

708 Operationdepicts, for a pull request regarding the computer code, determining a first subgroup of the files that is modified by the pull request, and a group of functions that is modified by the pull request. This can be implemented in a similar manner as a PR impact generator as described herein, and can involve analyzing a pull request diff (the diff between the updated file(s) of the pull request and the prior version of those file(s) stored in a repository) to determine which files are modified and which functions within those files are modified.

708 700 710 After operation, process flowmoves to operation.

710 Operationdepicts, based on the dependency graph, the first subgroup of the files, and the group of functions, identifying call chains of the computer code that are affected by the group of functions, and a second subgroup of the files that is affected by the group of functions. This can comprise determining the call chains (e.g., a sequence of function calls in computer code that lead up to a particular execution point in executing that code) and other files (e.g., those that contain functions used in a call chain) using the dependency graph.

710 700 712 After operation, process flowmoves to operation.

712 708 710 Operationdepicts, based on the dependency graph, the call chains, and the second subgroup of the files, determining a subgroup of the check tests that corresponds to a portion of the computer code that is modified by the pull request. That is, the information of operations-can be used to find check tests that are relevant to the pull request.

712 700 714 After operation, process flowmoves to operation.

714 Operationdepicts, storing an indication of the subgroup of the check tests.

714 714 In some examples, the indication is a first indication, implementing the pull request on the computer code produces modified computer code, and operationcomprises performing the subgroup of the check tests on the modified computer code to produce a validation result that indicates whether a regression has been introduced by the pull request, and storing a second indication of the validation result. That is, the check tests identified in operationcan be executed against the computer code to identify possible errors with the computer code.

714 714 In some examples, the indication is a first indication, and operationcomprises storing a second indication of the first subgroup of the files that is modified by the pull request. In some examples, operationcomprises storing a second indication of the group of functions that is modified by the pull request. That is, a report can comprise files that are touched by a diff, functions within those files that are modified, and relevant check tests that can be executed to validate that no regression has been introduced by the updated computer code.

In some examples, the indication of the subgroup of the check tests is stored in a structured key-value format that is human-readable. That is, a report can be in a format such as a JSON format, and can be used as an input for a test recommender.

714 In some examples, operationcomprises performing respective iterations of the determining the first subgroup of the files and the group of functions, the identifying of the call chains and the second subgroup of the files, the determining of the subgroup of the check tests, and the storing of the indication of the subgroup of the check tests for respective pull requests that comprise the pull request. That is, a PR impact generator can be run each time a PR is generated.

714 700 716 700 After operation, process flowmoves to, where process flowends.

8 FIG. 1 FIG. 10 FIG. 800 800 108 1000 illustrates another example process flowthat can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, one or more embodiments of process flowcan be implemented by test recommendation based on a source code dependency graph and code coverage results componentof, or computing environmentof.

800 800 500 600 700 900 5 FIG. 6 FIG. 7 FIG. 9 FIG. It can be appreciated that the operating procedures of process floware example operating procedures, and that there can be embodiments that implement more or fewer operating procedures than are depicted, or that implement the depicted operating procedures in a different order than as depicted. In some examples, process flowcan be implemented in conjunction with one or more embodiments of one or more of process flowof, process flowof, process flowof, and/or process flowof.

800 802 804 Process flowbegins with, and moves to operation.

804 804 704 706 7 FIG. Operationdepicts creating a dependency graph that represents dependencies within computer code, wherein nodes of the dependency graph represent files of the computer code, wherein edges of the dependency graph represent respective relationships between respective files of the files, wherein respective nodes of the nodes are associated with respective control flow graphs for respective code blocks of the computer code that are stored in the respective files, and wherein respective attributes of the respective nodes identify respective check tests that are associated with the respective files. In some examples, operationcan be implemented in a similar manner as operations-of.

804 800 806 After operation, process flowmoves to operation.

806 806 710 7 FIG. Operationdepicts, for a pull request regarding the computer code, determining a first subgroup of the files that is modified by the pull request, and a group of functions that is modified by the pull request. In some examples, operationcan be implemented in a similar manner as operationof.

806 800 808 After operation, process flowmoves to operation.

808 808 710 7 FIG. Operationdepicts, based on the dependency graph, the first subgroup of the files, and the group of functions, identifying call chains of the computer code that are impacted by the group of functions, and a second subgroup of the files that is impacted by the group of functions. In some examples, operationcan be implemented in a similar manner as operationof.

808 800 810 After operation, process flowmoves to operation.

810 810 712 7 FIG. Operationdepicts, based on the dependency graph, the call chains, and the second subgroup of the files, determining a subgroup of the check tests that corresponds to a portion of the computer code that is modified by the pull request. In some examples, operationcan be implemented in a similar manner as operationof.

810 In some examples, the subgroup of the check tests is a first subgroup of the check tests, and operationcomprises determining a baseline coverage report for the computer code, based on execution of a second subgroup of the check tests for the computer code on a platform, an identification of the platform, and a coverage report for the execution of the second subgroup of the check tests for the computer code on the platform.

810 In some examples, operationcomprises determining the baseline coverage report based on respective executions of respective second subgroups of the check tests for the computer code on respective platforms, wherein the respective second subgroups of the check tests comprise the second subgroup of the check tests, and wherein the platforms comprise the platform.

That is, a code coverage report analyzer can take, as input, code coverage reports from test lists executed on different platforms, and output a baseline coverage report. A baseline coverage report can include the following example details for each test list executed: name of test list, platform the list was executed on, and the coverage report for the execution of the test list on the given platform.

810 In some examples, operationcomprises determining that a first platform of the platforms and a second platform of the platforms are equivalent based on a first coverage report corresponding to the first platform being determined to match a second coverage report corresponding to the second platform.

810 In some examples, operationcomprises determining that a first platform of the platforms and a second platform of the platforms are inequivalent based on a first coverage report corresponding to the first platform differing from a second coverage report corresponding to the second platform.

810 In some examples, operationcomprises, based on the coverage report and the dependency graph, determining a third subgroup of the files that correspond to the coverage report.

810 In some examples, the group of functions is a first group of functions, and operationcomprises, based on the coverage report and the dependency graph, determining a second group of functions that corresponds to the coverage report.

That is, using a coverage report, an analyzer can compare coverage reports from different platforms running the same test list. If the reports match (that is, lines tested match), those platforms can be deemed as being equivalent. Any non-matching platforms can be assumed to be testing different code paths, and can be deemed not equivalent to others. The analyzer can also analyze each of the coverage reports and determine which files are touched and what functions are impacted by the test list.

In some examples, the determining of the baseline coverage report is performed independently of receiving the pull request. That is, unlike the PR impact generator that can be intended to be run every time a PR is generated, it can be that there is no hard requirement as to how often the code coverage report analyzer should run. In some examples, it can be that it needs to be run often enough to ensure the code coverage report still provides valuable results against the PR impact report. That is, the code coverage report can serve as a baseline, and, if the code base deviates enough over time, a new baseline can be determined.

810 800 812 800 After operation, process flowmoves to, where process flowends.

9 FIG. 1 FIG. 10 FIG. 900 900 108 1000 illustrates another example process flowthat can facilitate test recommendation based on a source code dependency graph and code coverage results, in accordance with an embodiment of this disclosure. In some examples, one or more embodiments of process flowcan be implemented by test recommendation based on a source code dependency graph and code coverage results componentof, or computing environmentof.

900 900 500 600 700 800 5 FIG. 6 FIG. 7 FIG. 8 FIG. It can be appreciated that the operating procedures of process floware example operating procedures, and that there can be embodiments that implement more or fewer operating procedures than are depicted, or that implement the depicted operating procedures in a different order than as depicted. In some examples, process flowcan be implemented in conjunction with one or more embodiments of one or more of process flowof, process flowof, process flowof, and/or process flowof.

900 902 904 Process flowbegins with, and moves to operation.

904 904 704 706 7 FIG. Operationdepicts creating a dependency graph that represents dependencies within computer code, wherein nodes of the dependency graph represent files of the computer code, wherein edges of the dependency graph represent respective relationships between respective files of the files, wherein respective nodes of the nodes are associated with respective control flow graphs for respective code blocks of the computer code that are stored in the respective files, and wherein respective attributes of the respective nodes identify respective check tests that are associated with the respective files. In some examples, operationcan be implemented in a similar manner as operations-of.

904 900 906 After operation, process flowmoves to operation.

906 906 710 7 FIG. Operationdepicts, for a pull request regarding the computer code, determining a first subgroup of the files that is modified by the pull request, and a group of functions that is modified by the pull request. In some examples, operationcan be implemented in a similar manner as operationof.

906 900 908 After operation, process flowmoves to operation.

908 908 710 712 7 FIG. Operationdepicts based on the dependency graph, the first subgroup of the files, the group of functions, call chains of the computer code that are affected by the group of functions, and a second subgroup of the files that are affected by the group of functions, determining a subgroup of the check tests that corresponds to a portion of the computer code that is modified by the pull request. In some examples, operationcan be implemented in a similar manner as operations-of.

908 In some examples, operationcomprises determining a baseline coverage report for the computer code, based on respective executions of respective second subgroups of tests for the computer code on respective platforms, respective identifications of the respective platforms, and respective coverage reports for the respective execution of the respective second subgroups of tests for the computer code on the respective platforms.

908 In some examples, the group of functions is a first group of functions, and operationcomprises, based on the baseline coverage report and the first subgroup of the files that are modified by the pull request, identifying a second group of functions covered by the baseline coverage report and that is affected by the pull request.

908 In some examples, the subgroup of the check tests is a first subgroup of the check tests, and operationcomprises, generating a test recommendation report that identifies a second subgroup of the check tests based on the second group of functions covered by the baseline coverage report and that is affected by the pull request.

In some examples, the test recommendation report identifies at least one platform on which to execute the second subgroup of the check tests.

908 In some examples, a first platform of the at least one platform and a second platform of the at least one platform that are identified in the test recommendation report correspond to a test of the second subgroup of the check tests, and operationcomprises selecting one of the first platform or the second platform according to an availability criterion, to produce a selected platform, and executing the test on the selected platform, while refraining running the test from the other of the first platform and the second platform.

That is, using the baseline coverage report and the PR impact report, this information can be passed to a test recommender. The test recommender can look at the PR impact report and cycle through the files that are touched by the diff. Knowing this, the test recommender can cycle through the baseline coverage report and walk through each of the lists looking to find tests that touch those files. If the files are touched, the test recommender can then look at functions modified from the diff. If those same functions are touched in the baseline coverage report, then the tests can be appended to the test recommendation.

After cycling through the files within the PR impact report, and the tests from the baseline coverage report, a new test recommendation report can be generated that includes check tests to be executed (this can come from the PR impact generator). The recommendation report can also include an indication of which tests to execute and on what platforms. If a test list identified has equivalency defined, this information can be included in this report, and it can be left to the test executor to determine which platform it should run on. That test executor can know to only execute on one of the supported platforms, and can inspect the infra to see what is the “most” available.

908 900 910 900 After operation, process flowmoves to, where process flowends.

10 FIG. 1000 In order to provide additional context for various embodiments described herein,and the following discussion are intended to provide a brief, general description of a suitable computing environmentin which the various embodiments of the embodiment described herein can be implemented.

1000 102 106 For example, parts of computing environmentcan be used to implement one or more embodiments of computer systemand/or repository.

1000 5 9 FIGS.- In some examples, computing environmentcan implement one or more embodiments of the process flows ofto facilitate test recommendation based on a source code dependency graph and code coverage results.

While the embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that the embodiments can also be implemented in combination with other program modules and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the various methods can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, as well as personal computers, hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices.

The illustrated embodiments of the embodiments herein can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

Computing devices typically include a variety of media, which can include computer-readable storage media, machine-readable storage media, and/or communications media, which two terms are used herein differently from one another as follows. Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media or machine-readable storage media can be implemented in connection with any method or technology for storage of information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

Computer-readable storage media can include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disc (BD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, solid state drives or other solid state storage devices, or other tangible and/or non-transitory media which can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” herein as applied to storage, memory, or computer-readable media, are to be understood to exclude only propagating transitory signals per se as modifiers and do not relinquish rights to all standard storage, memory or computer-readable media that are not only propagating transitory signals per se.

Computer-readable storage media can be accessed by one or more local or remote computing devices, e.g., via access requests, queries, or other data retrieval protocols, for a variety of operations with respect to the information stored by the medium.

Communications media typically embody computer-readable instructions, data structures, program modules or other structured or unstructured data in a data signal such as a modulated data signal, e.g., a carrier wave or other transport mechanism, and includes any information delivery or transport media. The term “modulated data signal” or signals refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in one or more signals. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

10 FIG. 1000 1002 1002 1004 1006 1008 1008 1006 1004 1004 1004 With reference again to, the example environmentfor implementing various embodiments described herein includes a computer, the computerincluding a processing unit, a system memoryand a system bus. The system buscouples system components including, but not limited to, the system memoryto the processing unit. The processing unitcan be any of various commercially available processors. Dual microprocessors and other multi-processor architectures can also be employed as the processing unit.

1008 1006 1010 1012 1002 1012 The system buscan be any of several types of bus structure that can further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memoryincludes ROMand RAM. A basic input/output system (BIOS) can be stored in a nonvolatile storage such as ROM, erasable programmable read only memory (EPROM), EEPROM, which BIOS contains the basic routines that help to transfer information between elements within the computer, such as during startup. The RAMcan also include a high-speed RAM such as static RAM for caching data.

1002 1014 1016 1016 1020 1014 1002 1014 1000 1014 1014 1016 1020 1008 1024 1026 1028 1024 The computerfurther includes an internal hard disk drive (HDD)(e.g., EIDE, SATA), one or more external storage devices(e.g., a magnetic floppy disk drive (FDD), a memory stick or flash drive reader, a memory card reader, etc.) and an optical disk drive(e.g., which can read or write from a CD-ROM disc, a DVD, a BD, etc.). While the internal HDDis illustrated as located within the computer, the internal HDDcan also be configured for external use in a suitable chassis (not shown). Additionally, while not shown in environment, a solid state drive (SSD) could be used in addition to, or in place of, an HDD. The HDD, external storage device(s)and optical disk drivecan be connected to the system busby an HDD interface, an external storage interfaceand an optical drive interface, respectively. The interfacefor external drive implementations can include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are within contemplation of the embodiments described herein.

1002 The drives and their associated computer-readable storage media provide nonvolatile storage of data, data structures, computer-executable instructions, and so forth. For the computer, the drives and storage media accommodate the storage of any data in a suitable digital format. Although the description of computer-readable storage media above refers to respective types of storage devices, it should be appreciated by those skilled in the art that other types of storage media which are readable by a computer, whether presently existing or developed in the future, could also be used in the example operating environment, and further, that any such storage media can contain computer-executable instructions for performing the methods described herein.

1012 1030 1032 1034 1036 1012 A number of program modules can be stored in the drives and RAM, including an operating system, one or more application programs, other program modulesand program data. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM. The systems and methods described herein can be implemented utilizing various commercially available operating systems or combinations of operating systems.

1002 1030 1030 1002 1030 1032 1032 1030 1032 10 FIG. Computercan optionally comprise emulation technologies. For example, a hypervisor (not shown) or other intermediary can emulate a hardware environment for operating system, and the emulated hardware can optionally be different from the hardware illustrated in. In such an embodiment, operating systemcan comprise one virtual machine (VM) of multiple VMs hosted at computer. Furthermore, operating systemcan provide runtime environments, such as the Java runtime environment or the .NET framework, for applications. Runtime environments are consistent execution environments that allow applicationsto run on any operating system that includes the runtime environment. Similarly, operating systemcan support containers, and applicationscan be in the form of containers, which are lightweight, standalone, executable packages of software that include, e.g., code, runtime, system tools, system libraries and settings for an application.

1002 1002 Further, computercan be enabled with a security module, such as a trusted processing module (TPM). For instance, with a TPM, boot components hash next in time boot components, and wait for a match of results to secured values, before loading a next boot component. This process can take place at any layer in the code execution stack of computer, e.g., applied at the application execution level or at the operating system (OS) kernel level, thereby enabling security at any level of code execution.

1002 1038 1040 1042 1004 1044 1008 A user can enter commands and information into the computerthrough one or more wired/wireless input devices, e.g., a keyboard, a touch screen, and a pointing device, such as a mouse. Other input devices (not shown) can include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control, or other remote control, a joystick, a virtual reality controller and/or virtual reality headset, a game pad, a stylus pen, an image input device, e.g., camera(s), a gesture sensor input device, a vision movement sensor input device, an emotion or facial detection device, a biometric input device, e.g., fingerprint or iris scanner, or the like. These and other input devices are often connected to the processing unitthrough an input device interfacethat can be coupled to the system bus, but can be connected by other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, a BLUETOOTH® interface, etc.

1046 1008 1048 1046 A monitoror other type of display device can also be connected to the system busvia an interface, such as a video adapter. In addition to the monitor, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

1002 1050 1050 1002 1052 1054 1056 The computercan operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer(s). The remote computer(s)can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer, although, for purposes of brevity, only a memory/storage deviceis illustrated. The logical connections depicted include wired/wireless connectivity to a local area network (LAN)and/or larger networks, e.g., a wide area network (WAN). Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to a global communications network, e.g., the Internet.

1002 1054 1058 1058 1054 1058 When used in a LAN networking environment, the computercan be connected to the local networkthrough a wired and/or wireless communication network interface or adapter. The adaptercan facilitate wired or wireless communication to the LAN, which can also include a wireless access point (AP) disposed thereon for communicating with the adapterin a wireless mode.

1002 1060 1056 1056 1060 1008 1044 1002 1052 When used in a WAN networking environment, the computercan include a modemor can be connected to a communications server on the WANvia other means for establishing communications over the WAN, such as by way of the Internet. The modem, which can be internal or external and a wired or wireless device, can be connected to the system busvia the input device interface. In a networked environment, program modules depicted relative to the computeror portions thereof, can be stored in the remote memory/storage device. It will be appreciated that the network connections shown are examples, and other means of establishing a communications link between the computers can be used.

1002 1016 1002 1054 1056 1058 1060 1002 1026 1058 1060 1026 1002 When used in either a LAN or WAN networking environment, the computercan access cloud storage systems or other network-based storage systems in addition to, or in place of, external storage devicesas described above. Generally, a connection between the computerand a cloud storage system can be established over a LANor WANe.g., by the adapteror modem, respectively. Upon connecting the computerto an associated cloud storage system, the external storage interfacecan, with the aid of the adapterand/or modem, manage storage provided by the cloud storage system as it would other types of external storage. For instance, the external storage interfacecan be configured to provide access to cloud storage sources as if those sources were physically connected to the computer.

1002 The computercan be operable to communicate with any wireless devices or entities operatively disposed in wireless communication, e.g., a printer, scanner, desktop and/or portable computer, portable data assistant, communications satellite, any piece of equipment or location associated with a wirelessly detectable tag (e.g., a kiosk, news stand, store shelf, etc.), and telephone. This can include Wireless Fidelity (Wi-Fi) and BLUETOOTH® wireless technologies. Thus, the communication can be a predefined structure as with a conventional network or simply an ad hoc communication between at least two devices.

As it employed in the subject specification, the term “processor” can refer to substantially any computing processing unit or device comprising, but not limited to comprising, single-core processors; single-processors with software multithread execution capability; multi-core processors; multi-core processors with software multithread execution capability; multi-core processors with hardware multithread technology; parallel platforms; and parallel platforms with distributed shared memory in a single machine or multiple machines. Additionally, a processor can refer to an integrated circuit, a state machine, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable gate array (PGA) including a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Processors can exploit nano-scale architectures such as, but not limited to, molecular and quantum-dot based transistors, switches, and gates, in order to optimize space usage or enhance performance of user equipment. A processor may also be implemented as a combination of computing processing units. One or more processors can be utilized in supporting a virtualized computing environment. The virtualized computing environment may support one or more virtual machines representing computers, servers, or other computing devices. In such virtualized virtual machines, components such as processors and storage devices may be virtualized or logically represented. For instance, when a processor executes instructions to perform “operations,” this could include the processor performing the operations directly and/or facilitating, directing, or cooperating with another device or component to perform the operations.

In the subject specification, terms such as “datastore,” data storage,” “database,” “cache,” and substantially any other information storage component relevant to operation and functionality of a component, refer to “memory components,” or entities embodied in a “memory” or components comprising the memory. It will be appreciated that the memory components, or computer-readable storage media, described herein can be either volatile memory or nonvolatile storage, or can include both volatile and nonvolatile storage. By way of illustration, and not limitation, nonvolatile storage can include ROM, programmable ROM (PROM), EPROM, EEPROM, or flash memory. Volatile memory can include RAM, which acts as external cache memory. By way of illustration and not limitation, RAM can be available in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). Additionally, the disclosed memory components of systems or methods herein are intended to comprise, without being limited to comprising, these and any other suitable types of memory.

The illustrated embodiments of the disclosure can be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

The systems and processes described above can be embodied within hardware, such as a single integrated circuit (IC) chip, multiple ICs, an ASIC, or the like. Further, the order in which some or all of the process blocks appear in each process should not be deemed limiting. Rather, it should be understood that some of the process blocks can be executed in a variety of orders that are not all of which may be explicitly illustrated herein.

As used in this application, the terms “component,” “module,” “system,” “interface,” “cluster,” “server,” “node,” or the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution or an entity related to an operational machine with one or more specific functionalities. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, computer-executable instruction(s), a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. As another example, an interface can include input/output (I/O) components as well as associated processor, application, and/or application programming interface (API) components.

Further, the various embodiments can be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement one or more embodiments of the disclosed subject matter. An article of manufacture can encompass a computer program accessible from any computer-readable device or computer-readable storage/communications media. For example, computer readable storage media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips . . . ), optical discs (e.g., CD, DVD . . . ), smart cards, and flash memory devices (e.g., card, stick, key drive . . . ). Of course, those skilled in the art will recognize many modifications can be made to this configuration without departing from the scope or spirit of the various embodiments.

In addition, the word “example” or “exemplary” is used herein to mean serving as an example, instance, or illustration. Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word exemplary is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.

What has been described above includes examples of the present specification. It is, of course, not possible to describe every conceivable combination of components or methods for purposes of describing the present specification, but one of ordinary skill in the art may recognize that many further combinations and permutations of the present specification are possible. Accordingly, the present specification is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.