Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining the provenance of source code. One of the methods includes receiving a portion of a file occurring in a source code project. For each of a plurality of windows of characters in the portion of the file, a respective provenance signature is computed. An index that maps each provenance signature to occurrences of the provenance signature in one or more files of a plurality of projects is searched to identify one or more matching files that are each associated with at least one provenance signature computed for the portion of the file. Data identifying the one or more matching files is provided in response to receiving the portion of the file occurring in the source code project.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform at least the following: receive a portion of a file occurring in a source code project; define a plurality of windows of characters within the portion of the file, wherein each window of characters of the plurality of windows of characters is defined based on one or more anchor characters, the one or more anchor characters comprising one or more reserved source code characters, and wherein each window of characters comprises an anchor character of the one or more anchor characters and a predefined number of characters at one or more particular character offsets from the anchor character; compute, for each of the plurality of windows of characters of one or more source code segments in the portion of the file, a respective provenance signature by using each window of characters as input to a content-based signature function that generates each provenance signature as a fixed-size output for any arbitrarily sized input; search an index that maps each provenance signature to occurrences of the provenance signature in one or more files of a plurality of projects to identify one or more matching files that are each associated with at least one provenance signature computed for the portion of the file; and provide data identifying the one or more matching files in response to receiving the portion of the file occurring in the source code project.
2. The system of claim 1 , wherein identifying one or more matching files comprises determining that one or more candidate signature groups are matching signature groups by determining, for each candidate signature group, that a number of matching partial signatures or matching file provenance signatures between the source code file and the candidate signature group satisfies a threshold.
3. The system of claim 1 , wherein determining that the one or more candidate signature groups are matching signature groups comprises: determining, for each candidate signature group, that a ratio of a number of matching partial signatures or matching file provenance signatures to a total number of signatures in the candidate signature group satisfies a threshold.
4. The system of claim 1 , wherein the instructions are further operable to cause the one or more computers to: determine which of the matching signature groups is associated with an oldest file among files associated with the matching signature groups; and designate an oldest particular file associated a matching signature group as being a canonical source of the source code file.
5. The system of claim 4 , wherein the instructions are further operable to cause the one or more computers to: identify a plurality of subsequent versions of the canonical source; determine which of the subsequent versions has the most matching file provenance signatures with the source code file; and designate a particular version having a highest number of matching file provenance signatures with the source code file as a most likely version of the source code file.
6. The system of claim 1 , wherein the instructions are further operable to cause the one or more computers to: receive a request to attribute a plurality of performance metrics to a developer entity responsible for the portion of the file occurring in the source code project; remove a contribution of the portion of the file occurring in the source code project that is associated with the one or more matching files from the performance metrics; and attribute a remainder of the performance metrics to the developer entity responsible for the portion of the file occurring in the source code project.
7. The system of claim 6 , wherein the performance metrics include one or more of churn, net lines of code, violations introduced, violations removed, or net violations in the source code file.
8. The system of claim 1 , wherein the instructions are further operable to cause the one or more computers to: designate the source code file as library code that was previously introduced in one or more other software projects.
9. The system of claim 8 , wherein the instructions are further operable to cause the one or more computers to: identify a library version corresponding to the source code file; determine that a newer library version is available; and provide an automatic notification to a responsible entity of the particular project that a newer version of the library code is available.
10. The system of claim 8 , wherein the instructions are further operable to cause the one or more computers to: maintain a list of library files having a known vulnerability; determine that the source code file is copied from a library file occurring in the list of library files having a known vulnerability; and provide an automatic notification to a responsible entity of the particular project that the source code file has a known vulnerability.
11. The system of claim 1 , wherein the instructions are further operable to cause the one or more computers to: identify, for a particular project, a number of other projects in which code from the particular project has been adopted; and provide, to a responsible entity for the particular project, an automatic notification identifying one or more of the other projects in which code from the particular project has been adopted.
12. The system of claim 1 , wherein the content-based signature function is a content-based hashing function.
13. The system of claim 1 , wherein the predefined number of characters is ten or more characters.
14. A computer-implemented method comprising: receiving a portion of a file occurring in a source code project; defining a plurality of windows of characters within the portion of the file, wherein each window of characters of the plurality of windows of characters is defined based on one or more anchor characters, the one or more anchor characters comprising one or more reserved source code characters, and wherein each window of characters comprises an anchor character of the one or more anchor characters and a predefined number of characters at one or more particular character offsets from the anchor character; computing, for each of the plurality of windows of characters of one or more source code segments in the portion of the file, a respective provenance signature by using each window of characters as input to a content-based signature function that generates each provenance signature as a fixed-size output for any arbitrarily sized input; searching an index that maps each provenance signature to occurrences of the provenance signature in one or more files of a plurality of projects to identify one or more matching files that are each associated with at least one provenance signature computed for the portion of the file; and providing data identifying the one or more matching files in response to receiving the portion of the file occurring in the source code project.
15. The method of claim 14 , wherein identifying one or more matching files comprises determining that one or more candidate signature groups are matching signature groups by determining, for each candidate signature group, that the number of matching partial signatures or matching file provenance signatures between the source code file and the candidate signature group satisfies a threshold.
16. The method of claim 14 , wherein determining that the one or more candidate signature groups are matching signature groups comprises: determining, for each candidate signature group, that a ratio of a number of matching partial signatures or matching file provenance signatures to a total number of signatures in the candidate signature group satisfies a threshold.
17. The method of claim 14 , further comprising: determining which of the matching signature groups is associated with an oldest file among files associated with the matching signature groups; and designating an oldest particular file associated a matching signature group as being a canonical source of the source code file.
18. The method of claim 17 , further comprising: identifying a plurality of subsequent versions of the canonical source; determining which of the subsequent versions has the most matching file provenance signatures with the source code file; and designating a particular version having a highest number of matching file provenance signatures with the source code file as a most likely version of the source code file.
19. The method of claim 14 , further comprising: receiving a request to attribute a plurality of performance metrics to a developer entity responsible for the portion of the file occurring in the source code project; removing a contribution of the portion of the file occurring in the source code project that is associated with the one or more matching files from the performance metrics; and attributing a remainder of the performance metrics to the developer entity responsible for the portion of the file occurring in the source code project.
20. The method of claim 19 , wherein the performance metrics include one or more of churn, net lines of code, violations introduced, violations removed, or net violations in the source code file.
21. The method of claim 14 , further comprising: designating the source code file as library code that was previously introduced in one or more other software projects.
22. The method of claim 21 , further comprising: identifying a library version corresponding to the source code file; determining that a newer library version is available; and providing an automatic notification to a responsible entity of the particular project that a newer version of the library code is available.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
December 21, 2018
February 23, 2021
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.