Method for Improving Search Efficiency in Enterprise Search System

1. A method executing on a processor for accessing and searching information, comprising: selecting a first subset of silos as active and a second subset of the silos as non-active based on an access pattern to the silos, each of the silos containing an index of stored documents that are associated with a different document attribute value in an attribute category; receiving a query for a search engine that is associated with a particular document attribute value; applying the query to a given index in the silo to which the particular document attribute value pertains; and returning a result set comprising stored documents that are retrieved by applying the query.

2. The method of claim 1 , wherein the active silos are stored in memory and the non-active silos are stored on a disk based device.

3. The method of claim 1 , wherein the access pattern is past access patterns to the silos.

4. The method of claim 1 , wherein the access pattern is expected future access patterns.

5. The method of claim 1 , further comprising partitioning documents into the silos based on a document attribute value.

6. The method of claim 1 , further comprising determining the document attribute values pertaining to the silos based on an observed distribution pattern.

7. The method of claim 1 , further comprising partitioning the silos into at least two dimensions.

8. The method of claim 1 , further comprising analyzing a search pattern over a period of time to determine correlated attribute values.

9. The method of claim 8 , further comprising co-locating documents based on the correlated attribute values.

10. A system for accessing and searching information, comprising: a processor and a memory; silos that each contain an index of stored documents that are associated with a different document attribute value in an attribute category; the processor configured to perform actions, comprising: selecting a first subset of the silos as active and selecting a second subset of the silos as non-active that are stored on disk storage based on access patterns to the silos; receiving a query for a search engine that is associated with a particular document attribute value; applying the query to the silo to which the particular document attribute value pertains; and returning a result set that are retrieved by applying the query.

11. The system of claim 10 , wherein the active silos are stored in a high-speed memory.

12. The system of claim 10 , wherein the access patterns are one of past access patterns to the silos and expected future access patterns to the silos.

13. The system of claim 10 , further comprising partitioning documents into the silos based on a document attribute value.

14. The system of claim 10 , further comprising observing a distribution pattern of attribute values employed in searches.

15. The system of claim 10 , further comprising partitioning the silos into at least two dimensions.

16. The system of claim 10 , further comprising limiting the search to metadata terms that are associated with the documents.

17. A computer-readable medium, excluding a signal, storing instructions that when executed perform steps, comprising: selecting a first subset of silos as active that are stored in a high-speed memory and a second subset of the silos as non-active that are stored on a disk based storage based on an access pattern to the silos, each of the silos containing information relating to stored documents that are associated with a different document attribute value in an attribute category; receiving a query for a search engine that is associated with a particular document attribute value; applying the query to a given index in the silo to which the particular document attribute value pertains; and returning a result set comprising stored documents that are retrieved by applying the query.

18. The computer-readable medium of claim 17 , wherein the access pattern is past access patterns to the silos and expected future access patterns.

19. The computer-readable medium of claim 17 , further comprising partitioning documents into the silos based on a document attribute value.

20. The computer-readable medium of claim 19 , further comprising using an observed distribution pattern of search terms to assist in partitioning the documents.