Performant and Scalable Merge Strategy for Text Indexing

1. A method of indexing data and merging indexes within a database management system, in which a need for an in-memory table tracking currency of data is eliminated, the method comprising: assigning a temporal indicator associated with a start time of a crawl to an instance of an index generated by the crawl, wherein the instance of the index comprises a first index of a plurality of indexes generated by the crawl, wherein the plurality of indexes are part of an indexing system of the database management system, wherein the crawl is a first crawl, the instance is a first instance, and the temporal indicator is a first temporal indicator; assigning the first instance of the index to a set of instances of the index to be merged; assigning a second temporal indicator associated with a start time of a second crawl to a second instance of an index generated by the second crawl; assigning the second instance of the index to the set of instances to be merged; validating the set of instances to be merged, generating a set of valid instances, wherein validating the set of instances of the index comprises: determining that the first temporal indicator is a maximum temporal indicator associated with the set of instances; in response to determining that the second temporal indicator is identical to the maximum temporal indicator merging the set of valid instances to create a merged index instance.

2. The method of claim 1 , further comprising in response to determining that the set of instances does not include a subset wherein the subset comprises a full crawl of a collection of data, selecting a third instance associated with a third temporal indicator from the set of instances and in response to determining that the third temporal indicator is not less than an instance in a priority queue of indexes not in the set, deleting the third instance.

3. The method of claim 1 , wherein merging the first instance and the second instance further comprises: for each data item identifier of a data item in the first instance and for each keyword in the data item, in response to determining that the first temporal indicator is identical to a maximum temporal indicator, selecting the first instance to generate an entry in the merged instance.

4. The method of claim 1 , wherein merging the first instance and the second instance further comprises: in response to determining that the first temporal indicator of the first instance comprising a keyword and a data item identifier is less than a maximum temporal indicator, determining that the first temporal indicator is a maximum temporal indicator for the keyword and the data item identifier in the set of instances and selecting the first instance to generate an entry in the merged instance for the keyword and data item identifier.

5. The method of claim 1 , wherein merging the first instance and the second instance further comprises: in response to determining that the first temporal indicator of the first instance comprising a keyword and a data item identifier is less than a maximum temporal indicator, determining that the first temporal indicator is not a maximum temporal indicator for the keyword and the data item identifier in the set of instances and removing the first instance from the set of instances to be merged.

6. The method of claim 5 , wherein the first instance is returned to a priority queue.

7. A method of indexing data and merging indexes within a database management system, in which a need for an in-memory table tracking currency of data is eliminated, the method comprising: assigning a temporal indicator associated with a start time of a crawl to an instance of an index generated by the crawl, wherein the instance of the index comprises a first index of a plurality of indexes generated by the crawl, wherein the plurality of indexes are part of an indexing system of the database management system, wherein the crawl is a first crawl, the instance is a first instance, and the temporal indicator is a first temporal indicator; assigning the first instance of the index to a set of instances of the index to be merged; assigning a second temporal indicator associated with a start time of a second crawl to a second instance of an index generated by the second crawl; assigning the second instance of the index to the set of instances to be merged; validating the set of instances to be merged, generating a set of valid instances, wherein validating the set of instances of the index comprises: determining that the first temporal indicator is a maximum temporal indicator associated with the set of instances; in response to determining that the second temporal indicator is not identical to the maximum temporal indicator, determining that the crawl associated with the second temporal indicator is done; and merging the set of valid instances to create a merged index instance.

8. A method of indexing data and merging indexes within a database management system, in which a need for an in-memory table tracking currency of data is eliminated, the method comprising: assigning a temporal indicator associated with a start time of a crawl to an instance of an index generated by the crawl, wherein the instance of the index comprises a first index of a plurality of indexes generated by the crawl, wherein the plurality of indexes are part of an indexing system of the database management system, wherein the crawl is a first crawl, the instance is a first instance, and the temporal indicator is a first temporal indicator, further comprising: assigning the first instance of the index to a set of instances of the index to be merged; assigning a second temporal indicator associated with a start time of a second crawl to a second instance of an index generated by the second crawl; assigning the second instance of the index to the set of instances to be merged; validating the set of instances to be merged, generating a set of valid instances; merging the set of valid instances to create a merged index instance; and in response to determining that a first subset of the set of instances associated with the first temporal indicator comprises a full crawl of a collection of data and that the second temporal indicator is less than the first temporal indicator, deleting a second subset of the set of instances associated with the second temporal indicator.