Monitoring Storage Units in a Storage Network

The present U.S. Utility Patent Application claims priority pursuant to 35 U.S.C. § 120 as a continuation of U.S. Utility application Ser. No. 18/398,361, entitled “MAINTAINING AVAILABILITY OF CRITICAL INFORMATION IN A DISTRIBUTED STORAGE NETWORK,” filed Dec. 28, 2023, which is a continuation of U.S. Utility application Ser. No. 17/827,888, entitled “STORAGE OF REBUILT DATA IN SPARE MEMORY OF A STORAGE NETWORK,” filed May 30, 2022, issued as U.S. Pat. No. 11,860,711 on Jan. 2, 2024, which is a continuation of U.S. Utility application Ser. No. 16/850,434, entitled “REBUILDING DATA IN A DISTRIBUTED STORAGE NETWORK,” filed Apr. 16, 2020, issued as U.S. Pat. No. 11,347,590 on May 31, 2022, which is a continuation-in-part of U.S. Utility application Ser. No. 16/029,898, entitled “MONITORING OF STORAGE UNITS IN A DISPERSED STORAGE NETWORK,” filed Jul. 9, 2018, issued as U.S. Pat. No. 10,628,245 on Apr. 21, 2020, which is a continuation-in-part of U.S. Utility application Ser. No. 14/613,899, entitled “GENERATING MOLECULAR ENCODING INFORMATION FOR DATA STORAGE,” filed Feb. 4, 2015, issued as U.S. Pat. No. 10,020,826 on Jul. 10, 2018, which claims priority pursuant to 35 U.S.C. § 119(e) to U.S. Provisional Application No. 61/974,110, entitled “ACCESSING DATA IN A DISPERSED STORAGE NETWORK,” filed Apr. 2, 2014, each of which is hereby incorporated herein by reference in its entirety and made part of the present U.S. Utility Patent Application for all purposes.

This invention relates generally to computer networks and more particularly to dispersed storage of data and distributed monitoring of storage resources.

Computing devices are known to communicate data, process data, and/or store data. Such computing devices range from wireless smart phones, laptops, tablets, personal computers (PC), work stations, and video game devices, to data centers that support millions of web searches, stock trades, or on-line purchases every day. In general, a computing device includes a central processing unit (CPU), a memory system, user input/output interfaces, peripheral device interfaces, and an interconnecting bus structure.

As is further known, a computer may effectively extend its CPU by using “cloud computing” to perform one or more computing functions (e.g., a service, an application, an algorithm, an arithmetic logic function, etc.) on behalf of the computer. Further, for large services, applications, and/or functions, cloud computing may be performed by multiple cloud computing resources in a distributed manner to improve the response time for completion of the service, application, and/or function. For example, Hadoop is an open source software framework that supports distributed applications enabling application execution by thousands of computers.

In addition to cloud computing, a computer may use “cloud storage” as part of its memory system. As is known, cloud storage enables a user, via its computer, to store files, applications, etc. on an Internet storage system. The Internet storage system may include a RAID (redundant array of independent disks) system and/or a dispersed storage system that uses an error correction scheme to encode data for storage.

According to an embodiment of the present invention, monitoring of a target storage unit of a dispersed storage network (DSN) is performed in a distributed manner by one or more additional storage units. Configuration information relating to DSN storage resources is utilized in selecting the target storage unit. The number of other storage units to be used in monitoring the target storage unit is also determined in view of various criteria. Selection of the monitoring storage units includes determining, for each of a plurality of storage units of the DSN, an estimated failure correlation level with the target storage unit. Based at least in part on this information, the determined number of storage units are selected and assigned to monitor, and selectively provide status information relating to, the target storage unit.

1 FIG. 10 12 14 16 18 20 22 10 24 is a schematic block diagram of an embodiment of a distributed computing systemthat includes a computing deviceand/or a computing device, a distributed storage and/or task (DST) processing unit, a distributed (or dispersed) storage and/or task network (DSTN) managing unit, a DST integrity processing unit, and a distributed storage and/or task network (DSTN) module. The components of the distributed computing systemare coupled via a network, which may include one or more wireless and/or wire lined communication systems; one or more private intranet systems and/or public internet systems; and/or one or more local area networks (LAN) and/or wide area networks (WAN).

22 36 The DSTN moduleincludes a plurality of distributed storage and/or task (DST) execution unitsthat may be located at geographically different sites (e.g., one in Chicago, one in Milwaukee, etc.). Each of the DST execution units (also referred to herein as “storage units”) is operable to store dispersed error encoded data and/or to execute, in a distributed manner, one or more tasks on data. The tasks may be a simple function (e.g., a mathematical function, a logic function, an identify function, a find function, a search engine function, a replace function, etc.), a complex function (e.g., compression, human and/or computer language translation, text-to-voice conversion, voice-to-text conversion, etc.), multiple simple and/or complex functions, one or more algorithms, one or more applications, etc.

12 14 16 18 20 26 12 16 34 Each of the computing devices-, the DST processing unit, the DSTN managing unit, and the DST integrity processing unitinclude a computing coreand may be a portable computing device and/or a fixed computing device. A portable computing device may be a social networking device, a gaming device, a cell phone, a smart phone, a personal digital assistant, a digital music player, a digital video player, a laptop computer, a handheld computer, a tablet, a video game controller, and/or any other portable device that includes a computing core. A fixed computing device may be a personal computer (PC), a computer server, a cable set-top box, a satellite receiver, a television set, a printer, a fax machine, home entertainment equipment, a video game console, and/or any type of home or office computing equipment. Computing deviceand DST processing unitare configured to include a DST client module.

30 32 33 24 30 24 14 16 32 24 12 22 16 22 33 18 20 24 With respect to interfaces, each interface,, andincludes software and/or hardware to support one or more communication links via the networkindirectly and/or directly. For example, interfacesupports a communication link (e.g., wired, wireless, direct, via a LAN, via the network, etc.) between computing deviceand the DST processing unit. As another example, interfacesupports communication links (e.g., a wired connection, a wireless connection, a LAN connection, and/or any other type of connection to/from the network) between computing deviceand the DSTN moduleand between the DST processing unitand the DSTN module. As yet another example, interfacesupports a communication link for each of the DSTN managing unitand DST integrity processing unitto the network.

10 10 20 26 FIGS.- The distributed computing systemis operable to support dispersed storage (DS) error encoded data storage and retrieval, to support distributed task processing on received data, and/or to support distributed task processing on stored data. In general and with respect to DS error encoded data storage and retrieval, the distributed computing systemsupports three primary operations: storage management, data storage and retrieval (an example of which will be discussed with reference to), and data storage integrity verification. In accordance with these three primary functions, data can be encoded, distributedly stored in physically different locations, and subsequently retrieved in a reliable and secure manner. Such a system is tolerant of a significant number of failures (e.g., up to a failure level, which may be greater than or equal to a pillar width minus a decode threshold minus one) that may result from individual storage device failures and/or network equipment failures without loss of data and without the need for a redundant or backup copy. Further, the system allows the data to be stored for an indefinite period of time without data loss and does so in a secure manner (e.g., the system is very resistant to attempts at hacking the data).

12 14 14 40 22 40 16 30 30 30 40 The second primary function (i.e., distributed data storage and retrieval) begins and ends with a computing device-. For instance, if a second type of computing devicehas datato store in the DSTN module, it sends the datato the DST processing unitvia its interface. The interfacefunctions to mimic a conventional operating system (OS) file system interface (e.g., network file system (NFS), flash file system (FFS), disk file system (DFS), file transfer protocol (FTP), web-based distributed authoring and versioning (WebDAV), etc.) and/or a block memory interface (e.g., small computer system interface (SCSI), internet small computer system interface (iSCSI), etc.). In addition, the interfacemay attach a user identification code (ID) to the data.

18 18 12 14 18 22 18 10 22 12 16 20 To support storage management, the DSTN managing unitperforms DS management services. One such DS management service includes the DSTN managing unitestablishing distributed data storage parameters (e.g., vault creation, distributed storage parameters, security parameters, billing information, user profile information, etc.) for a computing device-individually or as part of a group of computing devices (also referred to herein as user devices). For example, the DSTN managing unitcoordinates creation of a vault (e.g., a virtual memory block) within memory of the DSTN modulefor a computing device, a group of devices, or for public access and establishes per vault dispersed storage (DS) error encoding parameters for a vault. The DSTN managing unitmay facilitate storage of DS error encoding parameters for each vault of a plurality of vaults by updating registry information for the distributed computing system. The facilitating includes storing updated registry information in one or more of the DSTN module, the computing device, the DST processing unit, and the DST integrity processing unit.

The DS error encoding parameters (e.g., or dispersed storage error coding parameters) include data segmenting information (e.g., how many segments data (e.g., a file, a group of files, a data block, etc.) is divided into), segment security information (e.g., per segment encryption, compression, integrity checksum, etc.), error coding information (e.g., pillar width, decode threshold, read threshold, write threshold, etc.), slicing information (e.g., the number of encoded data slices that will be created for each data segment); and slice security information (e.g., per encoded data slice encryption, compression, integrity checksum, etc.).

18 22 The DSTN managing unitcreates and stores user profile information (e.g., an access control list (ACL)) in local memory and/or within memory of the DSTN module. The user profile information includes authentication information, permissions, and/or the security parameters. The security parameters may include encryption/decryption scheme, one or more encryption keys, key generation scheme, and/or data encoding/decoding scheme.

18 18 18 The DSTN managing unitcreates billing information for a particular user, a user group, a vault access, public vault access, etc. For instance, the DSTN managing unittracks the number of times a user accesses a private vault and/or public vaults, which can be used to generate a per-access billing information. In another instance, the DSTN managing unittracks the amount of data stored and/or retrieved by a user device and/or a user group, which can be used to generate a per-data-amount billing information.

18 10 36 10 10 Another DS management service includes the DSTN managing unitperforming network operations, network administration, and/or network maintenance. Network operations includes authenticating user data allocation requests (e.g., read and/or write requests), managing creation of vaults, establishing authentication credentials for user devices, adding/deleting components (e.g., computing devices, DST execution units, and/or DST processing units) from the distributed computing system, and/or establishing authentication credentials for DST execution units. Network administration includes monitoring devices and/or units for failures, maintaining vault information, determining device and/or unit activation status, determining device and/or unit loading, and/or determining any other system level operation that affects the performance level of the system. Network maintenance includes facilitating replacing, upgrading, repairing, and/or expanding a device and/or unit of the system.

10 20 20 22 22 20 22 16 36 To support data storage integrity verification within the distributed computing system, the DST integrity processing unitperforms rebuilding of ‘bad’ or missing encoded data slices. At a high level, the DST integrity processing unitperforms rebuilding by periodically attempting to retrieve/list encoded data slices, and/or slice names of the encoded data slices, from the DSTN module. For retrieved encoded slices, they are checked for errors due to data corruption, outdated version, etc. If a slice includes an error, it is flagged as a ‘bad’ slice. For encoded data slices that were not received and/or not listed, they are flagged as missing slices. Bad and/or missing slices are subsequently rebuilt using other retrieved encoded data slices that are deemed to be good slices to produce rebuilt slices. The rebuilt slices are stored in memory of the DSTN module. Note that the DST integrity processing unitmay be a separate unit as shown, it may be included in the DSTN module, it may be included in the DST processing unit, and/or distributed among the DST execution units.

10 18 18 18 12 14 3 19 FIGS.- To support distributed task processing on received data, the distributed computing systemhas two primary operations: DST (distributed storage and/or task processing) management and DST execution on received data (an example of which will be discussed with reference to). With respect to the storage portion of the DST management, the DSTN managing unitfunctions as previously described. With respect to the tasking processing of the DST management, the DSTN managing unitperforms distributed task processing (DTP) management services. One such DTP management service includes the DSTN managing unitestablishing DTP parameters (e.g., user-vault affiliation information, billing information, user-task information, etc.) for a computing device-individually or as part of a group of computing devices.

18 Another DTP management service includes the DSTN managing unitperforming DTP network operations, network administration (which is essentially the same as described above), and/or network maintenance (which is essentially the same as described above). Network operations include, but are not limited to, authenticating user task processing requests (e.g., valid request, valid user, etc.), authenticating results and/or partial results, establishing DTP authentication credentials for user devices, adding/deleting components (e.g., computing devices, DST execution units, and/or DST processing units) from the distributed computing system, and/or establishing DTP authentication credentials for DST execution units.

10 14 38 22 38 16 30 27 39 FIGS.- To support distributed task processing on stored data, the distributed computing systemhas two primary operations: DST (distributed storage and/or task) management and DST execution on stored data. With respect to the DST execution on stored data, if the second type of computing devicehas a task requestfor execution by the DSTN module, it sends the task requestto the DST processing unitvia its interface. An example of DST execution on stored data will be discussed in greater detail with reference to. With respect to the DST management, it is substantially similar to the DST management to support distributed task processing on received data.

2 FIG. 26 50 52 54 55 56 58 60 62 64 66 68 70 72 74 76 is a schematic block diagram of an embodiment of a computing corethat includes a processing module, a memory controller, main memory, a video graphics processing unit, an input/output (IO) controller, a peripheral component interconnect (PCI) interface, an IO interface module, at least one IO device interface module, a read only memory (ROM) basic input output system (BIOS), and one or more memory interface modules. The one or more memory interface module(s) includes one or more of a universal serial bus (USB) interface module, a host bus adapter (HBA) interface module, a network interface module, a flash interface module, a hard drive interface module, and a DSTN interface module.

76 76 70 30 14 62 1 FIG. The DSTN interface modulefunctions to mimic a conventional operating system (OS) file system interface (e.g., network file system (NFS), flash file system (FFS), disk file system (DFS), file transfer protocol (FTP), web-based distributed authoring and versioning (WebDAV), etc.) and/or a block memory interface (e.g., small computer system interface (SCSI), internet small computer system interface (iSCSI), etc.). The DSTN interface moduleand/or the network interface modulemay function as the interfaceof the computing deviceof. Further note that the IO device interface moduleand/or the memory interface modules may be collectively or individually referred to as IO ports.

3 FIG. 1 FIG. 1 FIG. 1 FIG. 34 14 16 24 1 36 22 34 80 82 1 86 84 88 90 34 n n is a diagram of an example of the distributed computing system performing a distributed storage and task processing operation. The distributed computing system includes a DST (distributed storage and/or task) client module(which may be in computing deviceand/or in DST processing unitof), a network, a plurality of DST execution units-that includes two or more DST execution unitsof(which form at least a portion of DSTN moduleof), a DST managing module (not shown), and a DST integrity verification module (not shown). The DST client moduleincludes an outbound DST processing sectionand an inbound DST processing section. Each of the DST execution units-includes a controller, a processing module, memory, a DT (distributed task) execution module, and a DST client module.

34 92 94 92 92 92 In an example of operation, the DST client modulereceives dataand one or more tasksto be performed upon the data. The datamay be of any size and of any content, where, due to the size (e.g., greater than a few Terabytes), the content (e.g., secure data, etc.), and/or task(s) (e.g., MIPS intensive), distributed processing of the task(s) on the data is desired. For example, the datamay be one or more digital books, a copy of a company's emails, a large-scale Internet search, a video security file, one or more entertainment video files (e.g., television programs, movies, etc.), data files, and/or any other large amount of data (e.g., greater than a few Terabytes).

34 80 92 94 80 92 96 80 92 80 96 80 94 98 98 96 Within the DST client module, the outbound DST processing sectionreceives the dataand the task(s). The outbound DST processing sectionprocesses the datato produce slice groupings. As an example of such processing, the outbound DST processing sectionpartitions the datainto a plurality of data partitions. For each data partition, the outbound DST processing sectiondispersed storage (DS) error encodes the data partition to produce encoded data slices and groups the encoded data slices into a slice grouping. In addition, the outbound DST processing sectionpartitions the taskinto partial tasks, where the number of partial tasksmay correspond to the number of slice groupings.

80 24 96 98 1 22 80 1 1 1 80 n 1 FIG. The outbound DST processing sectionthen sends, via the network, the slice groupingsand the partial tasksto the DST execution units-of the DSTN moduleof. For example, the outbound DST processing sectionsends slice groupand partial taskto DST execution unit. As another example, the outbound DST processing sectionsends slice group #n and partial task #n to DST execution unit #n.

98 96 102 1 1 1 1 1 1 1 Each DST execution unit performs its partial taskupon its slice groupto produce partial results. For example, DST execution unit #performs partial task #on slice group #to produce a partial result #, for results. As a more specific example, slice group #corresponds to a data partition of a series of digital books and the partial task #corresponds to searching for specific phrases, recording where the phrase is found, and establishing a phrase count. In this more specific example, the partial result #includes information as to where the phrase was found and includes the phrase count.

102 24 102 82 34 82 102 104 82 36 82 36 Upon completion of generating their respective partial results, the DST execution units send, via the network, their partial resultsto the inbound DST processing sectionof the DST client module. The inbound DST processing sectionprocesses the received partial resultsto produce a result. Continuing with the specific example of the preceding paragraph, the inbound DST processing sectioncombines the phrase count from each of the DST execution unitsto produce a total phrase count. In addition, the inbound DST processing sectioncombines the ‘where the phrase was found’ information from each of the DST execution unitswithin their respective data partitions to produce ‘where the phrase was found’ information for the series of digital books.

34 36 94 80 94 98 98 1 n. In another example of operation, the DST client modulerequests retrieval of stored data within the memory of the DST execution units(e.g., memory of the DSTN module). In this example, the taskis retrieve data stored in the memory of the DSTN module. Accordingly, the outbound DST processing sectionconverts the taskinto a plurality of partial tasksand sends the partial tasksto the respective DST execution units-

98 36 100 1 1 1 36 100 82 24 In response to the partial taskof retrieving stored data, a DST execution unitidentifies the corresponding encoded data slicesand retrieves them. For example, DST execution unit #receives partial task #and retrieves, in response thereto, retrieved slices #. The DST execution unitssend their respective retrieved slicesto the inbound DST processing sectionvia the network.

82 100 92 82 100 82 82 92 The inbound DST processing sectionconverts the retrieved slicesinto data. For example, the inbound DST processing sectionde-groups the retrieved slicesto produce encoded slices per data partition. The inbound DST processing sectionthen DS error decodes the encoded slices per data partition to produce data partitions. The inbound DST processing sectionde-partitions the data partitions to recapture the data.

4 FIG. 1 FIG. 1 FIG. 80 34 22 36 24 80 110 112 114 116 118 is a schematic block diagram of an embodiment of an outbound distributed storage and/or task (DST) processing sectionof a DST client modulecoupled to a DSTN moduleof a(e.g., a plurality of n DST execution units) via a network. The outbound DST processing sectionincludes a data partitioning module, a dispersed storage (DS) error encoding module, a grouping selector module, a control module, and a distributed task control module.

110 92 120 116 160 92 94 36 110 92 110 92 In an example of operation, the data partitioning modulepartitions datainto a plurality of data partitions. The number of partitions and the size of the partitions may be selected by the control modulevia controlbased on the data(e.g., its size, its content, etc.), a corresponding taskto be performed (e.g., simple, complex, single step, multiple steps, etc.), DS encoding parameters (e.g., pillar width, decode threshold, write threshold, segment security parameters, slice security parameters, etc.), capabilities of the DST execution units(e.g., processing resources, availability of processing recourses, etc.), and/or as may be inputted by a user, system administrator, or other operator (human or automated). For example, the data partitioning modulepartitions the data(e.g., 100 Terabytes) into 100,000 data segments, each being 1 Gigabyte in size. Alternatively, the data partitioning modulepartitions the datainto a plurality of data segments, where some of data segments are of a different size, are of the same size, or a combination thereof.

112 120 120 112 120 160 116 122 160 160 The DS error encoding modulereceives the data partitionsin a serial manner, a parallel manner, and/or a combination thereof. For each data partition, the DS error encoding moduleDS error encodes the data partitionin accordance with control informationfrom the control moduleto produce encoded data slices. The DS error encoding includes segmenting the data partition into data segments, segment security processing (e.g., encryption, compression, watermarking, integrity check (e.g., CRC), etc.), error encoding, slicing, and/or per slice security processing (e.g., encryption, compression, watermarking, integrity check (e.g., CRC), etc.). The control informationindicates which steps of the DS error encoding are active for a given data partition and, for active steps, indicates the parameters for the step. For example, the control informationindicates that the error encoding is active and includes error encoding parameters (e.g., pillar width, decode threshold, write threshold, read threshold, type of error encoding, etc.).

114 122 96 36 94 36 94 122 96 114 96 36 24 The grouping selector modulegroups the encoded slicesof a data partition into a set of slice groupings. The number of slice groupings corresponds to the number of DST execution unitsidentified for a particular task. For example, if five DST execution unitsare identified for the particular task, the group selecting module groups the encoded slicesof a data partition into five slice groupings. The grouping selector moduleoutputs the slice groupingsto the corresponding DST execution unitsvia the network.

118 94 94 98 118 118 94 36 98 118 118 98 118 98 36 The distributed task control modulereceives the taskand converts the taskinto a set of partial tasks. For example, the distributed task control modulereceives a task to find where in the data (e.g., a series of books) a phrase occurs and a total count of the phrase usage in the data. In this example, the distributed task control modulereplicates the taskfor each DST execution unitto produce the partial tasks. In another example, the distributed task control modulereceives a task to find where in the data a first phrase occurs, where in the data a second phrase occurs, and a total count for each phrase usage in the data. In this example, the distributed task control modulegenerates a first set of partial tasksfor finding and counting the first phrase and a second set of partial tasks for finding and counting the second phrase. The distributed task control modulesends respective first and/or second partial tasksto each DST execution unit.

5 FIG. 126 128 is a logic diagram of an example of a method for outbound distributed storage and task (DST) processing that begins at stepwhere a DST client module receives data and one or more corresponding tasks. The method continues at stepwhere the DST client module determines a number of DST units to support the task for one or more data partitions. For example, the DST client module may determine the number of DST units to support the task based on the size of the data, the requested task, the content of the data, a predetermined number (e.g., user indicated, system administrator determined, etc.), available DST units, capability of the DST units, and/or any other factor regarding distributed task processing of the data. The DST client module may select the same DST units for each data partition, may select different DST units for the data partitions, or a combination thereof.

130 The method continues at stepwhere the DST client module determines processing parameters of the data based on the number of DST units selected for distributed task processing. The processing parameters include data partitioning information, DS encoding parameters, and/or slice grouping information. The data partitioning information includes a number of data partitions, size of each data partition, and/or organization of the data partitions (e.g., number of data blocks in a partition, the size of the data blocks, and arrangement of the data blocks). The DS encoding parameters include segmenting information, segment security information, error encoding information (e.g., dispersed storage error encoding function parameters including one or more of pillar width, decode threshold, write threshold, read threshold, generator matrix), slicing information, and/or per slice security information. The slice grouping information includes information regarding how to arrange the encoded data slices into groups for the selected DST units. As a specific example, if the DST client module determines that five DST units are needed to support the task, then it determines that the error encoding parameters include a pillar width of five and a decode threshold of three.

132 The method continues at stepwhere the DST client module determines task partitioning information (e.g., how to partition the tasks) based on the selected DST units and data processing parameters. The data processing parameters include the processing parameters and DST unit capability information. The DST unit capability information includes the number of DT (distributed task) execution units, execution capabilities of each DT execution unit (e.g., MIPS capabilities, processing resources (e.g., quantity and capability of microprocessors, CPUs, digital signal processors, co-processor, microcontrollers, arithmetic logic circuitry, and/or any other analog and/or digital processing circuitry), availability of the processing resources, memory information (e.g., type, size, availability, etc.)), and/or any information germane to executing one or more tasks.

134 136 138 The method continues at stepwhere the DST client module processes the data in accordance with the processing parameters to produce slice groupings. The method continues at stepwhere the DST client module partitions the task based on the task partitioning information to produce a set of partial tasks. The method continues at stepwhere the DST client module sends the slice groupings and the corresponding partial tasks to respective DST units.

6 FIG. 112 112 142 144 146 148 150 116 160 is a schematic block diagram of an embodiment of the dispersed storage (DS) error encoding moduleof an outbound distributed storage and task (DST) processing section. The DS error encoding moduleincludes a segment processing module, a segment security processing module, an error encoding module, a slicing module, and a per slice security processing module. Each of these modules is coupled to a control moduleto receive control informationtherefrom.

142 120 160 116 142 120 120 152 142 120 152 In an example of operation, the segment processing modulereceives a data partitionfrom a data partitioning module and receives segmenting information as the control informationfrom the control module. The segmenting information indicates how the segment processing moduleis to segment the data partition. For example, the segmenting information indicates how many rows to segment the data based on a decode threshold of an error encoding scheme, indicates how many columns to segment the data into based on a number and size of data blocks within the data partition, and indicates how many columns to include in a data segment. The segment processing modulesegments the datainto data segmentsin accordance with the segmenting information.

144 116 152 160 116 144 152 154 144 152 146 152 146 The segment security processing module, when enabled by the control module, secures the data segmentsbased on segment security information received as control informationfrom the control module. The segment security information includes data compression, encryption, watermarking, integrity check (e.g., cyclic redundancy check (CRC), etc.), and/or any other type of digital security. For example, when the segment security processing moduleis enabled, it may compress a data segment, encrypt the compressed data segment, and generate a CRC value for the encrypted data segment to produce a secure data segment. When the segment security processing moduleis not enabled, it passes the data segmentsto the error encoding moduleor is bypassed such that the data segmentsare provided to the error encoding module.

146 154 160 116 146 154 156 The error encoding moduleencodes the secure data segmentsin accordance with error correction encoding parameters received as control informationfrom the control module. The error correction encoding parameters (e.g., also referred to as dispersed storage error coding parameters) include identifying an error correction encoding scheme (e.g., forward error correction algorithm, a Reed-Solomon based algorithm, an online coding algorithm, an information dispersal algorithm, etc.), a pillar width, a decode threshold, a read threshold, a write threshold, etc. For example, the error correction encoding parameters identify a specific error correction encoding scheme, specifies a pillar width of five, and specifies a decode threshold of three. From these parameters, the error encoding moduleencodes a data segmentto produce an encoded data segment.

148 156 160 148 156 156 158 The slicing moduleslices the encoded data segmentin accordance with the pillar width of the error correction encoding parameters received as control information. For example, if the pillar width is five, the slicing moduleslices an encoded data segmentinto a set of five encoded data slices. As such, for a plurality of encoded data segmentsfor a given data partition, the slicing module outputs a plurality of sets of encoded data slices.

150 116 158 160 116 150 158 122 150 158 158 112 116 The per slice security processing module, when enabled by the control module, secures each encoded data slicebased on slice security information received as control informationfrom the control module. The slice security information includes data compression, encryption, watermarking, integrity check (e.g., CRC, etc.), and/or any other type of digital security. For example, when the per slice security processing moduleis enabled, it compresses an encoded data slice, encrypts the compressed encoded data slice, and generates a CRC value for the encrypted encoded data slice to produce a secure encoded data slice. When the per slice security processing moduleis not enabled, it passes the encoded data slicesor is bypassed such that the encoded data slicesare the output of the DS error encoding module. Note that the control modulemay be omitted and each module stores its own parameters.

7 FIG. 142 120 1 45 160 120 160 152 is a diagram of an example of a segment processing of a dispersed storage (DS) error encoding module. In this example, a segment processing modulereceives a data partitionthat includes 45 data blocks (e.g., d-d), receives segmenting information (i.e., control information) from a control module, and segments the data partitionin accordance with the control informationto produce data segments. Each data block may be of the same size as other data blocks or of a different size. In addition, the size of each data block may be a few bytes to megabytes of data. As previously mentioned, the segmenting information indicates how many rows to segment the data partition into, indicates how many columns to segment the data partition into, and indicates how many columns to include in a data segment.

In this example, the decode threshold of the error encoding scheme is three; as such the number of rows to divide the data partition into is three. The number of columns for each row is set to 15, which is based on the number and size of data blocks. The data blocks of the data partition are arranged in rows and columns in a sequential order (i.e., the first row includes the first 15 data blocks; the second row includes the second 15 data blocks; and the third row includes the last 15 data blocks).

With the data blocks arranged into the desired sequential order, they are divided into data segments based on the segmenting information. In this example, the data partition is divided into 8 data segments; the first 7 include 2 columns of three rows and the last includes 1 column of three rows. Note that the first row of the 8 data segments is in sequential order of the first 15 data blocks; the second row of the 8 data segments in sequential order of the second 15 data blocks; and the third row of the 8 data segments in sequential order of the last 15 data blocks. Note that the number of data blocks, the grouping of the data blocks into segments, and size of the data blocks may vary to accommodate the desired distributed task processing function.

8 FIG. 7 FIG. 1 3 1 1 1 2 2 16 17 3 31 32 2 7 8 15 30 45 is a diagram of an example of error encoding and slicing processing of the dispersed error encoding processing the data segments of. In this example, data segmentincludesrows with each row being treated as one word for encoding. As such, data segmentincludes three words for encoding: wordincluding data blocks dand d, wordincluding data blocks dand d, and wordincluding data blocks dand d. Each of data segments-includes three words where each word includes two data blocks. Data segmentincludes three words where each word includes a single data block (e.g., d, d, and d).

146 148 160 1 1 1 2 1 1 2 1 16 17 16 17 1 31 32 31 32 d d d In operation, an error encoding moduleand a slicing moduleconvert each data segment into a set of encoded data slices in accordance with error correction encoding parameters as control information. More specifically, when the error correction encoding parameters indicate a unity matrix Reed-Solomon based encoding algorithm, 5 pillars, and decode threshold of 3, the first three encoded data slices of the set of encoded data slices for a data segment are substantially similar to the corresponding word of the data segment. For instance, when the unity matrix Reed-Solomon based encoding algorithm is applied to data segment, the content of the first encoded data slice (DS_&) of the first set of encoded data slices (e.g., corresponding to data segment) is substantially similar to content of the first word (e.g., d& d); the content of the second encoded data slice (DS_&) of the first set of encoded data slices is substantially similar to content of the second word (e.g., d& d); and the content of the third encoded data slice (DS_&) of the first set of encoded data slices is substantially similar to content of the third word (e.g., d& d).

1 1 1 2 The content of the fourth and fifth encoded data slices (e.g., ES_and ES_) of the first set of encoded data slices include error correction data based on the first-third words of the first data segment. With such an encoding and slicing scheme, retrieving any three of the five encoded data slices allows the data segment to be accurately reconstructed.

2 7 1 2 3 4 2 3 4 2 18 19 18 19 2 33 34 33 34 1 1 1 2 d d d The encoding and slicing of data segments-yield sets of encoded data slices similar to the set of encoded data slices of data segment. For instance, the content of the first encoded data slice (DS_&) of the second set of encoded data slices (e.g., corresponding to data segment) is substantially similar to content of the first word (e.g., d& d); the content of the second encoded data slice (DS_&) of the second set of encoded data slices is substantially similar to content of the second word (e.g., d& d); and the content of the third encoded data slice (DS_&) of the second set of encoded data slices is substantially similar to content of the third word (e.g., d& d). The content of the fourth and fifth encoded data slices (e.g., ES_and ES_) of the second set of encoded data slices includes error correction data based on the first - third words of the second data segment.

9 FIG. 160 122 160 96 114 114 1 1 15 is a diagram of an example of grouping selection processing of an outbound distributed storage and task (DST) processing in accordance with group selection information as control informationfrom a control module. Encoded slices for data partitionare grouped in accordance with the control informationto produce slice groupings. In this example, a grouping selector moduleorganizes the encoded data slices into five slice groupings (e.g., one for each DST execution unit of a distributed storage and task network (DSTN) module). As a specific example, the grouping selector modulecreates a first slice grouping for a DST execution unit #, which includes first encoded slices of each of the sets of encoded slices. As such, the first DST execution unit receives encoded data slices corresponding to data blocks-(e.g., encoded data slices of contiguous data).

114 2 16 30 114 3 31 45 The grouping selector modulealso creates a second slice grouping for a DST execution unit #, which includes second encoded slices of each of the sets of encoded slices. As such, the second DST execution unit receives encoded data slices corresponding to data blocks-. The grouping selector modulefurther creates a third slice grouping for DST execution unit #, which includes third encoded slices of each of the sets of encoded slices. As such, the third DST execution unit receives encoded data slices corresponding to data blocks-.

114 4 114 5 The grouping selector modulecreates a fourth slice grouping for DST execution unit #, which includes fourth encoded slices of each of the sets of encoded slices. As such, the fourth DST execution unit receives encoded data slices corresponding to first error encoding information (e.g., encoded data slices of error coding (EC) data). The grouping selector modulefurther creates a fifth slice grouping for DST execution unit #, which includes fifth encoded slices of each of the sets of encoded slices. As such, the fifth DST execution unit receives encoded data slices corresponding to second error encoding information.

10 FIG. 92 92 164 1 166 is a diagram of an example of converting datainto slice groups that expands on the preceding figures. As shown, the datais partitioned in accordance with a partitioning functioninto a plurality of data partitions (-x, where x is an integer greater than 4). Each data partition (or chunkset of data) is encoded and grouped into slice groupings as previously discussed by an encoding and grouping function. For a given data partition, the slice groupings are sent to distributed storage and task (DST) execution units. From data partition to data partition, the ordering of the slice groupings to the DST execution units may vary.

1 9 FIG. For example, the slice groupings of data partition #is sent to the DST execution units such that the first DST execution receives first encoded data slices of each of the sets of encoded data slices, which corresponds to a first continuous data chunk of the first data partition (e.g., refer to), a second DST execution receives second encoded data slices of each of the sets of encoded data slices, which corresponds to a second continuous data chunk of the first data partition, etc.

2 1 2 2 2 3 2 4 2 5 For the second data partition, the slice groupings may be sent to the DST execution units in a different order than it was done for the first data partition. For instance, the first slice grouping of the second data partition (e.g., slice group_) is sent to the second DST execution unit; the second slice grouping of the second data partition (e.g., slice group_) is sent to the third DST execution unit; the third slice grouping of the second data partition (e.g., slice group_) is sent to the fourth DST execution unit; the fourth slice grouping of the second data partition (e.g., slice group_, which includes first error coding information) is sent to the fifth DST execution unit; and the fifth slice grouping of the second data partition (e.g., slice group_, which includes second error coding information) is sent to the first DST execution unit.

1 5 6 10 3 7 The pattern of sending the slice groupings to the set of DST execution units may vary in a predicted pattern, a random pattern, and/or a combination thereof from data partition to data partition. In addition, from data partition to data partition, the set of DST execution units may change. For example, for the first data partition, DST execution units-may be used; for the second data partition, DST execution units-may be used; for the third data partition, DST execution units-may be used; etc. As is also shown, the task is divided into partial tasks that are sent to the DST execution units in conjunction with the slice groupings of the data partitions.

11 FIG. 169 86 88 90 34 88 is a schematic block diagram of an embodiment of a DST (distributed storage and/or task) execution unit that includes an interface, a controller, memory, one or more DT (distributed task) execution modules, and a DST client module. The memoryis of sufficient size to store a significant number of encoded data slices (e.g., thousands of slices to hundreds-of-millions of slices) and may include one or more hard drives and/or one or more solid-state memory devices (e.g., flash memory, DRAM, etc.).

96 1 169 96 1 1 2 3 88 96 174 86 9 FIG. In an example of storing a slice group, the DST execution module receives a slice grouping(e.g., slice group #) via interface. The slice groupingincludes, per partition, encoded data slices of contiguous data or encoded data slices of error coding (EC) data. For slice group #, the DST execution module receives encoded data slices of contiguous data for partitions #and #x (and potentially others between 3 and x) and receives encoded data slices of EC data for partitions #and #(and potentially others between 3 and x). Examples of encoded data slices of contiguous data and encoded data slices of error coding (EC) data are discussed with reference to. The memorystores the encoded data slices of slice groupingsin accordance with memory control informationit receives from the controller.

86 174 98 86 98 98 86 98 96 86 174 96 88 96 The controller(e.g., a processing module, a CPU, etc.) generates the memory control informationbased on a partial task(s)and distributed computing information (e.g., user information (e.g., user ID, distributed computing permissions, data access permission, etc.), vault information (e.g., virtual memory assigned to user, user group, temporary storage for task processing, etc.), task validation information, etc.). For example, the controllerinterprets the partial task(s)in light of the distributed computing information to determine whether a requestor is authorized to perform the task, is authorized to access the data, and/or is authorized to perform the task on this particular data. When the requestor is authorized, the controllerdetermines, based on the taskand/or another input, whether the encoded data slices of the slice groupingare to be temporarily stored or permanently stored. Based on the foregoing, the controllergenerates the memory control informationto write the encoded data slices of the slice groupinginto the memoryand to indicate whether the slice groupingis permanently stored or temporarily stored.

96 88 86 98 86 98 90 86 90 176 With the slice groupingstored in the memory, the controllerfacilitates execution of the partial task(s). In an example, the controllerinterprets the partial taskin light of the capabilities of the DT execution module(s). The capabilities include one or more of MIPS capabilities, processing resources (e.g., quantity and capability of microprocessors, CPUs, digital signal processors, co-processor, microcontrollers, arithmetic logic circuitry, and/or any other analog and/or digital processing circuitry), availability of the processing resources, etc. If the controllerdetermines that the DT execution module(s)have sufficient capabilities, it generates task control information.

176 90 98 90 98 86 90 The task control informationmay be a generic instruction (e.g., perform the task on the stored slice grouping) or a series of operational codes. In the former instance, the DT execution moduleincludes a co-processor function specifically configured (fixed or programmed) to perform the desired task. In the latter instance, the DT execution moduleincludes a general processor topology where the controller stores an algorithm corresponding to the particular task. In this instance, the controllerprovides the operational codes (e.g., assembly language, source code of a programming language, object code, etc.) of the algorithm to the DT execution modulefor execution.

98 90 102 88 90 90 98 102 102 88 Depending on the nature of the task, the DT execution modulemay generate intermediate partial resultsthat are stored in the memoryor in a cache memory (not shown) within the DT execution module. In either case, when the DT execution modulecompletes execution of the partial task, it outputs one or more partial results. The partial resultsmay also be stored in memory.

86 90 98 86 90 98 98 If, when the controlleris interpreting whether capabilities of the DT execution module(s)can support the partial task, the controllerdetermines that the DT execution module(s)cannot adequately support the task(e.g., does not have the right resources, does not have sufficient available resources, available resources would be too slow, etc.), it then determines whether the partial taskshould be fully offloaded or partially offloaded.

86 98 178 34 178 98 96 34 98 172 96 170 34 34 172 170 3 10 FIGS.- If the controllerdetermines that the partial taskshould be fully offloaded, it generates DST control informationand provides it to the DST client module. The DST control informationincludes the partial task, memory storage information regarding the slice grouping, and distribution instructions. The distribution instructions instruct the DST client moduleto divide the partial taskinto sub-partial tasks, to divide the slice groupinginto sub-slice groupings, and identify other DST execution units. The DST client modulefunctions in a similar manner as the DST client moduleofto produce the sub-partial tasksand the sub-slice groupingsin accordance with the distribution instructions.

34 168 169 34 102 The DST client modulereceives DST feedback(e.g., sub-partial results), via the interface, from the DST execution units to which the task was offloaded. The DST client moduleprovides the sub-partial results to the DST execution unit, which processes the sub-partial results to produce the partial result(s).

86 98 98 96 86 176 86 178 If the controllerdetermines that the partial taskshould be partially offloaded, it determines what portion of the taskand/or slice groupingshould be processed locally and what should be offloaded. For the portion that is being locally processed, the controllergenerates task control informationas previously discussed. For the portion that is being offloaded, the controllergenerates DST control informationas previously discussed.

34 168 90 90 102 When the DST client modulereceives DST feedback(e.g., sub-partial results) from the DST executions units to which a portion of the task was offloaded, it provides the sub-partial results to the DT execution module. The DT execution moduleprocesses the sub-partial results with the sub-partial results it created to produce the partial result(s).

88 100 104 102 90 102 104 88 98 86 174 88 100 104 The memorymay be further utilized to retrieve one or more of stored slices, stored results, partial resultswhen the DT execution modulestores partial resultsand/or resultsin the memory. For example, when the partial taskincludes a retrieval request, the controlleroutputs the memory controlto the memoryto facilitate retrieval of slicesand/or results.

12 FIG. 1 1 86 174 88 is a schematic block diagram of an example of operation of a distributed storage and task (DST) execution unit storing encoded data slices and executing a task thereon. To store the encoded data slices of a partitionof slice grouping, a controllergenerates write commands as memory control informationsuch that the encoded slices are stored in desired locations (e.g., permanent or temporary) within memory.

86 176 90 176 90 88 90 1 1 15 1 15 Once the encoded slices are stored, the controllerprovides task control informationto a distributed task (DT) execution module. As a first step of executing the task in accordance with the task control information, the DT execution moduleretrieves the encoded slices from memory. The DT execution modulethen reconstructs contiguous data blocks of a data partition. As shown for this example, reconstructed contiguous data blocks of data partitioninclude data blocks-(e.g., d-d).

90 1 With the contiguous data blocks reconstructed, the DT execution moduleperforms the task on the reconstructed contiguous data blocks. For example, the task may be to search the reconstructed contiguous data blocks for a particular word or phrase, identify where in the reconstructed contiguous data blocks the particular word or phrase occurred, and/or count the occurrences of the particular word or phrase on the reconstructed contiguous data blocks. The DST execution unit continues in a similar manner for the encoded data slices of other partitions in slice grouping. Note that with using the unity matrix error encoding scheme previously discussed, if the encoded data slices of contiguous data are uncorrupted, the decoding of them is a relatively straightforward process of extracting the data.

If, however, an encoded data slice of contiguous data is corrupted (or missing), it can be rebuilt by accessing other DST execution units that are storing the other encoded data slices of the set of encoded data slices of the corrupted encoded data slice. In this instance, the DST execution unit having the corrupted encoded data slices retrieves at least three encoded data slices (of contiguous data and of error coding data) in the set from the other DST execution units (recall for this example, the pillar width is 5 and the decode threshold is 3). The DST execution unit decodes the retrieved data slices using the DS error encoding parameters to recapture the corresponding data segment. The DST execution unit then re-encodes the data segment using the DS error encoding parameters to rebuild the corrupted encoded data slice. Once the encoded data slice is rebuilt, the DST execution unit functions as previously described.

13 FIG. 82 24 82 180 182 184 186 188 186 188 is a schematic block diagram of an embodiment of an inbound distributed storage and/or task (DST) processing sectionof a DST client module coupled to DST execution units of a distributed storage and task network (DSTN) module via a network. The inbound DST processing sectionincludes a de-grouping module, a DS (dispersed storage) error decoding module, a data de-partitioning module, a control module, and a distributed task control module. Note that the control moduleand/or the distributed task control modulemay be separate modules from corresponding ones of outbound DST processing section or may be the same modules.

102 82 102 188 82 102 104 102 188 102 104 In an example of operation, the DST execution units have completed execution of corresponding partial tasks on the corresponding slice groupings to produce partial results. The inbound DST processing sectionreceives the partial resultsvia the distributed task control module. The inbound DST processing sectionthen processes the partial resultsto produce a final result, or results. For example, if the task was to find a specific word or phrase within data, the partial resultsindicate where in each of the prescribed portions of the data the corresponding DST execution units found the specific word or phrase. The distributed task control modulecombines the individual partial resultsfor the corresponding portions of the data into a final resultfor the data as a whole.

82 100 180 100 122 182 122 120 In another example of operation, the inbound DST processing sectionis retrieving stored data from the DST execution units (i.e., the DSTN module). In this example, the DST execution units output encoded data slicescorresponding to the data retrieval requests. The de-grouping modulereceives retrieved slicesand de-groups them to produce encoded data slices per data partition. The DS error decoding moduledecodes, in accordance with DS error encoding parameters, the encoded data slices per data partitionto produce data partitions.

184 120 92 186 100 92 190 186 180 182 184 The data de-partitioning modulecombines the data partitionsinto the data. The control modulecontrols the conversion of retrieved slicesinto the datausing control signalsto each of the modules. For instance, the control moduleprovides de-grouping information to the de-grouping module, provides the DS error encoding parameters to the DS error decoding module, and provides de-partitioning information to the data de-partitioning module.

14 FIG. 194 196 is a logic diagram of an example of a method that is executable by distributed storage and task (DST) client module regarding inbound DST processing. The method begins at stepwhere the DST client module receives partial results. The method continues at stepwhere the DST client module retrieves the task corresponding to the partial results. For example, the partial results include header information that identifies the requesting entity, which correlates to the requested task.

198 200 The method continues at stepwhere the DST client module determines result processing information based on the task. For example, if the task were to identify a particular word or phrase within the data, the result processing information would indicate to aggregate the partial results for the corresponding portions of the data to produce the final result. As another example, if the task were to count the occurrences of a particular word or phrase within the data, results of processing the information would indicate to add the partial results to produce the final results. The method continues at stepwhere the DST client module processes the partial results in accordance with the result processing information to produce the final result or results.

15 FIG. 9 FIG. 1 1 5 is a diagram of an example of de-grouping selection processing of an inbound distributed storage and task (DST) processing section of a DST client module. In general, this is an inverse process of the grouping module of the outbound DST processing section of. Accordingly, for each data partition (e.g., partition #), the de-grouping module retrieves the corresponding slice grouping from the DST execution units (EU) (e.g., DST-).

1 1 15 2 16 30 3 31 45 4 5 As shown, DST execution unit #provides a first slice grouping, which includes the first encoded slices of each of the sets of encoded slices (e.g., encoded data slices of contiguous data of data blocks-); DST execution unit #provides a second slice grouping, which includes the second encoded slices of each of the sets of encoded slices (e.g., encoded data slices of contiguous data of data blocks-); DST execution unit #provides a third slice grouping, which includes the third encoded slices of each of the sets of encoded slices (e.g., encoded data slices of contiguous data of data blocks-); DST execution unit #provides a fourth slice grouping, which includes the fourth encoded slices of each of the sets of encoded slices (e.g., first encoded data slices of error coding (EC) data); and DST execution unit #provides a fifth slice grouping, which includes the fifth encoded slices of each of the sets of encoded slices (e.g., first encoded data slices of error coding (EC) data).

100 180 190 122 The de-grouping module de-groups the slice groupings (e.g., received slices) using a de-grouping selectorcontrolled by a control signalas shown in the example to produce a plurality of sets of encoded data slices (e.g., retrieved slices for a partition into sets of slices). Each set corresponding to a data segment of the data partition.

16 FIG. 182 182 202 204 206 208 210 186 is a schematic block diagram of an embodiment of a dispersed storage (DS) error decoding moduleof an inbound distributed storage and task (DST) processing section. The DS error decoding moduleincludes an inverse per slice security processing module, a de-slicing module, an error decoding module, an inverse segment security module, a de-segmenting processing module, and a control module.

202 186 122 190 186 202 122 158 202 122 158 122 158 6 FIG. In an example of operation, the inverse per slice security processing module, when enabled by the control module, unsecures each encoded data slicebased on slice de-security information received as control information(e.g., the compliment of the slice security information discussed with reference to) received from the control module. The slice security information includes data decompression, decryption, de-watermarking, integrity check (e.g., CRC verification, etc.), and/or any other type of digital security. For example, when the inverse per slice security processing moduleis enabled, it verifies integrity information (e.g., a CRC value) of each encoded data slice, it decrypts each verified encoded data slice, and decompresses each decrypted encoded data slice to produce slice encoded data. When the inverse per slice security processing moduleis not enabled, it passes the encoded data slicesas the sliced encoded dataor is bypassed such that the retrieved encoded data slicesare provided as the sliced encoded data.

204 158 156 190 186 204 156 206 156 190 186 154 The de-slicing modulede-slices the sliced encoded datainto encoded data segmentsin accordance with a pillar width of the error correction encoding parameters received as control informationfrom the control module. For example, if the pillar width is five, the de-slicing modulede-slices a set of five encoded data slices into an encoded data segment. The error decoding moduledecodes the encoded data segmentsin accordance with error correction decoding parameters received as control informationfrom the control moduleto produce secure data segments. The error correction decoding parameters include identifying an error correction encoding scheme (e.g., forward error correction algorithm, a Reed-Solomon based algorithm, an information dispersal algorithm, etc.), a pillar width, a decode threshold, a read threshold, a write threshold, etc. For example, the error correction decoding parameters identify a specific error correction encoding scheme, specify a pillar width of five, and specify a decode threshold of three.

208 186 154 190 186 208 154 152 208 154 152 The inverse segment security processing module, when enabled by the control module, unsecures the secured data segmentsbased on segment security information received as control informationfrom the control module. The segment security information includes data decompression, decryption, de-watermarking, integrity check (e.g., CRC, etc.) verification, and/or any other type of digital security. For example, when the inverse segment security processing moduleis enabled, it verifies integrity information (e.g., a CRC value) of each secure data segment, it decrypts each verified secured data segment, and decompresses each decrypted secure data segment to produce a data segment. When the inverse segment security processing moduleis not enabled, it passes the decoded data segmentas the data segmentor is bypassed.

210 152 190 186 210 152 120 120 The de-segment processing modulereceives the data segmentsand receives de-segmenting information as control informationfrom the control module. The de-segmenting information indicates how the de-segment processing moduleis to de-segment the data segmentsinto a data partition. For example, the de-segmenting information indicates how the rows and columns of data segments are to be rearranged to yield the data partition.

17 FIG. 8 FIG. 204 158 190 156 158 204 1 1 2 3 1 d is a diagram of an example of de-slicing and error decoding processing of a dispersed error decoding module. A de-slicing modulereceives at least a decode threshold number of encoded data slicesfor each data segment in accordance with control informationand provides encoded data. In this example, a decode threshold is three. As such, each set of encoded data slicesis shown to have three encoded data slices per data segment. The de-slicing modulemay receive three encoded data slices per data segment because an associated distributed storage and task (DST) client module requested retrieving only three encoded data slices per segment or selected three of the retrieved encoded data slices per data segment. As shown, which is based on the unity matrix encoding previously discussed with reference to, an encoded data slice may be a data-based encoded data slice (e.g., DS_&d) or an error code based encoded data slice (e.g., ES_).

206 156 190 154 1 1 1 1 2 2 16 17 3 31 32 2 7 8 15 30 45 An error decoding moduledecodes the encoded dataof each data segment in accordance with the error correction decoding parameters of control informationto produce secured segments. In this example, data segmentincludes 3 rows with each row being treated as one word for encoding. As such, data segmentincludes three words: wordincluding data blocks dand d, wordincluding data blocks dand d, and wordincluding data blocks dand d. Each of data segments-includes three words where each word includes two data blocks. Data segmentincludes three words where each word includes a single data block (e.g., d, d, and d).

18 FIG. 210 152 1 8 190 120 is a diagram of an example of de-segment processing of an inbound distributed storage and task (DST) processing. In this example, a de-segment processing modulereceives data segments(e.g.,-) and rearranges the data blocks of the data segments into rows and columns in accordance with de-segmenting information of control informationto produce a data partition. Note that the number of rows is based on the decode threshold (e.g., 3 in this specific example) and the number of columns is based on the number and size of the data blocks.

210 120 The de-segmenting moduleconverts the rows and columns of data blocks into the data partition. Note that each data block may be of the same size as other data blocks or of a different size. In addition, the size of each data block may be a few bytes to megabytes of data.

19 FIG. 10 FIG. 92 92 1 212 214 is a diagram of an example of converting slice groups into datawithin an inbound distributed storage and task (DST) processing section. As shown, the datais reconstructed from a plurality of data partitions (-x, where x is an integer greater than 4). Each data partition (or chunk set of data) is decoded and re-grouped using a de-grouping and decoding functionand a de-partition functionfrom slice groupings as previously discussed. For a given data partition, the slice groupings (e.g., at least a decode threshold per data segment of encoded data slices) are received from DST execution units. From data partition to data partition, the ordering of the slice groupings received from the DST execution units may vary as discussed with reference to.

20 FIG. 34 24 34 80 82 86 88 90 34 is a diagram of an example of a distributed storage and/or retrieval within the distributed computing system. The distributed computing system includes a plurality of distributed storage and/or task (DST) processing client modules(one shown) coupled to a distributed storage and/or task processing network (DSTN) module, or multiple DSTN modules, via a network. The DST client moduleincludes an outbound DST processing sectionand an inbound DST processing section. The DSTN module includes a plurality of DST execution units. Each DST execution unit includes a controller, memory, one or more distributed task (DT) execution modules, and a DST client module.

34 92 92 80 92 216 80 24 21 23 FIGS.- 24 FIG. In an example of data storage, the DST client modulehas datathat it desires to store in the DSTN module. The datamay be a file (e.g., video, audio, text, graphics, etc.), a data object, a data block, an update to a file, an update to a data block, etc. In this instance, the outbound DST processing moduleconverts the datainto encoded data slicesas will be further described with reference to. The outbound DST processing modulesends, via the network, to the DST execution units for storage as further described with reference to.

34 92 100 82 24 In an example of data retrieval, the DST client moduleissues a retrieve request to the DST execution units for the desired data. The retrieve request may address each DST executions units storing encoded data slices of the desired data, address a decode threshold number of DST execution units, address a read threshold number of DST execution units, or address some other number of DST execution units. In response to the request, each addressed DST execution unit retrieves its encoded data slicesof the desired data and sends them to the inbound DST processing section, via the network.

82 100 100 82 92 When, for each data segment, the inbound DST processing sectionreceives at least a decode threshold number of encoded data slices, it converts the encoded data slicesinto a data segment. The inbound DST processing sectionaggregates the data segments to produce the retrieved data.

21 FIG. 80 24 80 110 112 114 116 118 is a schematic block diagram of an embodiment of an outbound distributed storage and/or task (DST) processing sectionof a DST client module coupled to a distributed storage and task network (DSTN) module (e.g., a plurality of DST execution units) via a network. The outbound DST processing sectionincludes a data partitioning module, a dispersed storage (DS) error encoding module, a grouping selector module, a control module, and a distributed task control module.

110 92 112 116 110 220 110 In an example of operation, the data partitioning moduleis by-passed such that datais provided directly to the DS error encoding module. The control modulecoordinates the by-passing of the data partitioning moduleby outputting a bypassmessage to the data partitioning module.

112 92 112 160 116 218 92 160 92 160 The DS error encoding modulereceives the datain a serial manner, a parallel manner, and/or a combination thereof. The DS error encoding moduleDS error encodes the data in accordance with control informationfrom the control moduleto produce encoded data slices. The DS error encoding includes segmenting the datainto data segments, segment security processing (e.g., encryption, compression, watermarking, integrity check (e.g., CRC, etc.)), error encoding, slicing, and/or per slice security processing (e.g., encryption, compression, watermarking, integrity check (e.g., CRC, etc.)). The control informationindicates which steps of the DS error encoding are active for the dataand, for active steps, indicates the parameters for the step. For example, the control informationindicates that the error encoding is active and includes error encoding parameters (e.g., pillar width, decode threshold, write threshold, read threshold, type of error encoding, etc.).

114 218 216 118 The grouping selector modulegroups the encoded slicesof the data segments into pillars of slices. The number of pillars corresponds to the pillar width of the DS error encoding parameters. In this example, the distributed task control modulefacilitates the storage request.

22 FIG. 21 FIG. 112 112 142 144 146 148 150 116 160 is a schematic block diagram of an example of a dispersed storage (DS) error encoding modulefor the example of. The DS error encoding moduleincludes a segment processing module, a segment security processing module, an error encoding module, a slicing module, and a per slice security processing module. Each of these modules is coupled to a control moduleto receive control informationtherefrom.

142 92 160 116 142 92 152 In an example of operation, the segment processing modulereceives dataand receives segmenting information as control informationfrom the control module. The segmenting information indicates how the segment processing module is to segment the data. For example, the segmenting information indicates the size of each data segment. The segment processing modulesegments the datainto data segmentsin accordance with the segmenting information.

144 116 152 160 116 144 152 144 152 146 152 146 The segment security processing module, when enabled by the control module, secures the data segmentsbased on segment security information received as control informationfrom the control module. The segment security information includes data compression, encryption, watermarking, integrity check (e.g., CRC, etc.), and/or any other type of digital security. For example, when the segment security processing moduleis enabled, it compresses a data segment, encrypts the compressed data segment, and generates a CRC value for the encrypted data segment to produce a secure data segment. When the segment security processing moduleis not enabled, it passes the data segmentsto the error encoding moduleor is bypassed such that the data segmentsare provided to the error encoding module.

146 160 116 146 The error encoding moduleencodes the secure data segments in accordance with error correction encoding parameters received as control informationfrom the control module. The error correction encoding parameters include identifying an error correction encoding scheme (e.g., forward error correction algorithm, a Reed- Solomon based algorithm, an information dispersal algorithm, etc.), a pillar width, a decode threshold, a read threshold, a write threshold, etc. For example, the error correction encoding parameters identify a specific error correction encoding scheme, specifies a pillar width of five, and specifies a decode threshold of three. From these parameters, the error encoding moduleencodes a data segment to produce an encoded data segment.

148 148 222 The slicing moduleslices the encoded data segment in accordance with a pillar width of the error correction encoding parameters. For example, if the pillar width is five, the slicing module slices an encoded data segment into a set of five encoded data slices. As such, for a plurality of data segments, the slicing moduleoutputs a plurality of sets of encoded data slices as shown within encoding and slicing functionas described.

150 116 160 116 150 150 218 112 The per slice security processing module, when enabled by the control module, secures each encoded data slice based on slice security information received as control informationfrom the control module. The slice security information includes data compression, encryption, watermarking, integrity check (e.g., CRC, etc.), and/or any other type of digital security. For example, when the per slice security processing moduleis enabled, it may compress an encoded data slice, encrypt the compressed encoded data slice, and generate a CRC value for the encrypted encoded data slice to produce a secure encoded data slice tweaking. When the per slice security processing moduleis not enabled, it passes the encoded data slices or is bypassed such that the encoded data slicesare the output of the DS error encoding module.

23 FIG. 92 224 92 is a diagram of an example of converting datainto pillar slice groups utilizing encoding, slicing and pillar grouping functionfor storage in memory of a distributed storage and task network (DSTN) module. As previously discussed the datais encoded and sliced into a plurality of sets of encoded data slices; one set per data segment. The grouping selector module organizes the sets of encoded data slices into pillars of data slices. In this example, the DS error encoding parameters include a pillar width of 5 and a decode threshold of 3. As such, for each data segment, 5 encoded data slices are created.

The grouping selector module takes the first encoded data slice of each of the sets and forms a first pillar, which may be sent to the first DST execution unit. Similarly, the grouping selector module creates the second pillar from the second slices of the sets; the third pillar from the third slices of the sets; the fourth pillar from the fourth slices of the sets; and the fifth pillar from the fifth slices of the set.

24 FIG. 169 86 88 90 34 26 90 34 88 is a schematic block diagram of an embodiment of a distributed storage and/or task (DST) execution unit that includes an interface, a controller, memory, one or more distributed task (DT) execution modules, and a DST client module. A computing coremay be utilized to implement the one or more DT execution modulesand the DST client module. The memoryis of sufficient size to store a significant number of encoded data slices (e.g., thousands of slices to hundreds-of-millions of slices) and may include one or more hard drives and/or one or more solid-state memory devices (e.g., flash memory, DRAM, etc.).

216 169 216 1 88 216 174 86 86 174 169 88 174 86 88 100 169 In an example of storing a pillar of slices, the DST execution unit receives, via interface, a pillar of slices(e.g., pillar #slices). The memorystores the encoded data slicesof the pillar of slices in accordance with memory control informationit receives from the controller. The controller(e.g., a processing module, a CPU, etc.) generates the memory control informationbased on distributed storage information (e.g., user information (e.g., user ID, distributed storage permissions, data access permission, etc.), vault information (e.g., virtual memory assigned to user, user group, etc.), etc.). Similarly, when retrieving slices, the DST execution unit receives, via interface, a slice retrieval request. The memoryretrieves the slice in accordance with memory control informationit receives from the controller. The memoryoutputs the slice, via the interface, to a requesting entity.

25 FIG. 82 92 82 180 182 184 186 188 186 188 is a schematic block diagram of an example of operation of an inbound distributed storage and/or task (DST) processing sectionfor retrieving dispersed error encoded data. The inbound DST processing sectionincludes a de-grouping module, a dispersed storage (DS) error decoding module, a data de-partitioning module, a control module, and a distributed task control module. Note that the control moduleand/or the distributed task control modulemay be separate modules from corresponding ones of an outbound DST processing section or may be the same modules.

82 92 188 180 100 190 186 218 182 190 186 218 92 184 226 190 186 In an example of operation, the inbound DST processing sectionis retrieving stored datafrom the DST execution units (i.e., the DSTN module). In this example, the DST execution units output encoded data slices corresponding to data retrieval requests from the distributed task control module. The de-grouping modulereceives pillars of slicesand de-groups them in accordance with control informationfrom the control moduleto produce sets of encoded data slices. The DS error decoding moduledecodes, in accordance with the DS error encoding parameters received as control informationfrom the control module, each set of encoded data slicesto produce data segments, which are aggregated into retrieved data. The data de-partitioning moduleis by-passed in this operational mode via a bypass signalof control informationfrom the control module.

26 FIG. 182 182 202 204 206 208 210 182 218 228 230 92 is a schematic block diagram of an embodiment of a dispersed storage (DS) error decoding moduleof an inbound distributed storage and task (DST) processing section. The DS error decoding moduleincludes an inverse per slice security processing module, a de-slicing module, an error decoding module, an inverse segment security module, and a de-segmenting processing module. The dispersed error decoding moduleis operable to de-slice and decode encoded slices per data segmentutilizing a de-slicing and decoding functionto produce a plurality of data segments that are de-segmented utilizing a de-segment functionto recover data.

202 186 190 218 190 186 202 218 202 218 218 6 FIG. In an example of operation, the inverse per slice security processing module, when enabled by the control modulevia control information, unsecures each encoded data slicebased on slice de-security information (e.g., the compliment of the slice security information discussed with reference to) received as control informationfrom the control module. The slice de-security information includes data decompression, decryption, de-watermarking, integrity check (e.g., CRC verification, etc.), and/or any other type of digital security. For example, when the inverse per slice security processing moduleis enabled, it verifies integrity information (e.g., a CRC value) of each encoded data slice, it decrypts each verified encoded data slice, and decompresses each decrypted encoded data slice to produce slice encoded data. When the inverse per slice security processing moduleis not enabled, it passes the encoded data slicesas the sliced encoded data or is bypassed such that the retrieved encoded data slicesare provided as the sliced encoded data.

204 190 186 The de-slicing modulede-slices the sliced encoded data into encoded data segments in accordance with a pillar width of the error correction encoding parameters received as control informationfrom a control module. For example, if the pillar width is five, the de-slicing module de-slices a set of five encoded data slices into an encoded data segment. Alternatively, the encoded data segment may include just three encoded data slices (e.g., when the decode threshold is 3).

206 190 186 The error decoding moduledecodes the encoded data segments in accordance with error correction decoding parameters received as control informationfrom the control moduleto produce secure data segments. The error correction decoding parameters include identifying an error correction encoding scheme (e.g., forward error correction algorithm, a Reed-Solomon based algorithm, an information dispersal algorithm, etc.), a pillar width, a decode threshold, a read threshold, a write threshold, etc. For example, the error correction decoding parameters identify a specific error correction encoding scheme, specify a pillar width of five, and specify a decode threshold of three.

208 186 190 186 152 208 152 210 152 92 190 186 The inverse segment security processing module, when enabled by the control module, unsecures the secured data segments based on segment security information received as control informationfrom the control module. The segment security information includes data decompression, decryption, de-watermarking, integrity check (e.g., CRC, etc.) verification, and/or any other type of digital security. For example, when the inverse segment security processing module is enabled, it verifies integrity information (e.g., a CRC value) of each secure data segment, it decrypts each verified secured data segment, and decompresses each decrypted secure data segment to produce a data segment. When the inverse segment security processing moduleis not enabled, it passes the decoded data segmentas the data segment or is bypassed. The de-segmenting processing moduleaggregates the data segmentsinto the datain accordance with control informationfrom the control module.

27 FIG. 1 34 86 90 88 is a schematic block diagram of an example of a distributed storage and task processing network (DSTN) module that includes a plurality of distributed storage and task (DST) execution units (#through #n, where, for example, n is an integer greater than or equal to three). Each of the DST execution units includes a DST client module, a controller, one or more DT (distributed task) execution modules, and memory.

1 1 3 19 FIGS.- 20 26 FIGS.- In this example, the DSTN module stores, in the memory of the DST execution units, a plurality of DS (dispersed storage) encoded data (e.g.,through n, where n is an integer greater than or equal to two) and stores a plurality of DS encoded task codes (e.g.,through k, where k is an integer greater than or equal to two). The DS encoded data may be encoded in accordance with one or more examples described with reference to(e.g., organized in slice groupings) or encoded in accordance with one or more examples described with reference to(e.g., organized in pillar groups). The data that is encoded into the DS encoded data may be of any size and/or of any content. For example, the data may be one or more digital books, a copy of a company's emails, a large-scale Internet search, a video security file, one or more entertainment video files (e.g., television programs, movies, etc.), data files, and/or any other large amount of data (e.g., greater than a few Terabytes).

3 19 FIGS.- 20 26 FIGS.- The tasks that are encoded into the DS encoded task code may be a simple function (e.g., a mathematical function, a logic function, an identify function, a find function, a search engine function, a replace function, etc.), a complex function (e.g., compression, human and/or computer language translation, text-to-voice conversion, voice-to-text conversion, etc.), multiple simple and/or complex functions, one or more algorithms, one or more applications, etc. The tasks may be encoded into the DS encoded task code in accordance with one or more examples described with reference to(e.g., organized in slice groupings) or encoded in accordance with one or more examples described with reference to(e.g., organized in pillar groups).

3 19 FIGS.- 3 19 FIGS.- 20 26 In an example of operation, a DST client module of a computing device or of a DST processing unit issues a DST request to the DSTN module. The DST request may include a request to retrieve stored data, or a portion thereof, may include a request to store data that is included with the DST request, may include a request to perform one or more tasks on stored data, may include a request to perform one or more tasks on data included with the DST request, etc. In the cases where the DST request includes a request to store data or to retrieve data, the client module and/or the DSTN module processes the request as previously discussed with reference to one or more of(e.g., slice groupings) and/or-(e.g., pillar groupings). In the case where the DST request includes a request to perform one or more tasks on data included with the DST request, the DST client module and/or the DSTN module process the DST request as previously discussed with reference to one or more of.

28 39 FIGS.- In the case where the DST request includes a request to perform one or more tasks on stored data, the DST client module and/or the DSTN module processes the DST request as will be described with reference to one or more of. In general, the DST client module identifies data and one or more tasks for the DSTN module to execute upon the identified data. The DST request may be for a one-time execution of the task or for an on-going execution of the task. As an example of the latter, as a company generates daily emails, the DST request may be to daily search new emails for inappropriate content and, if found, record the content, the email sender(s), the email recipient(s), email routing information, notify human resources of the identified email, etc.

28 FIG. 1 2 234 236 234 22 236 22 is a schematic block diagram of an example of a distributed computing system performing tasks on stored data. In this example, two distributed storage and task (DST) client modules-are shown: the first may be associated with a computing device and the second may be associated with a DST processing unit or a high priority computing device (e.g., high priority clearance user, system administrator, etc.). Each DST client module includes a list of stored dataand a list of tasks codes. The list of stored dataincludes one or more entries of data identifying information, where each entry identifies data stored in the DSTN module. The data identifying information (e.g., data ID) includes one or more of a data file name, a data file directory listing, DSTN addressing information of the data, a data object identifier, etc. The list of tasksincludes one or more entries of task code identifying information, when each entry identifies task codes stored in the DSTN module. The task code identifying information (e.g., task ID) includes one or more of a task file name, a task file directory listing, DSTN addressing information of the task, another type of identifier to identify the task, etc.

234 236 As shown, the list of dataand the list of tasksare each smaller in number of entries for the first DST client module than the corresponding lists of the second DST client module. This may occur because the computing device associated with the first DST client module has fewer privileges in the distributed computing system than the device associated with the second DST client module. Alternatively, this may occur because the computing device associated with the first DST client module serves fewer users than the device associated with the second DST client module and is restricted by the distributed computing system accordingly. As yet another alternative, this may occur through no restraints by the distributed computing system, it just occurred because the operator of the computing device associated with the first DST client module has selected fewer data and/or fewer tasks than the operator of the device associated with the second DST client module.

238 240 232 232 22 In an example of operation, the first DST client module selects one or more data entriesand one or more tasksfrom its respective lists (e.g., selected data ID and selected task ID). The first DST client module sends its selections to a task distribution module. The task distribution modulemay be within a stand-alone device of the distributed computing system, may be within the computing device that contains the first DST client module, or may be within the DSTN module.

242 240 238 242 232 242 22 29 39 FIGS.- Regardless of the task distribution module's location, it generates DST allocation informationfrom the selected task IDand the selected data ID. The DST allocation informationincludes data partitioning information, task execution information, and/or intermediate result information. The task distribution modulesends the DST allocation informationto the DSTN module. Note that one or more examples of the DST allocation information will be discussed with reference to one or more of.

22 242 2 1 22 242 22 238 22 22 The DSTN moduleinterprets the DST allocation informationto identify the stored DS encoded data (e.g., DS error encoded data) and to identify the stored DS error encoded task code (e.g., DS error encoded task code). In addition, the DSTN moduleinterprets the DST allocation informationto determine how the data is to be partitioned and how the task is to be partitioned. The DSTN modulealso determines whether the selected DS error encoded dataneeds to be converted from pillar grouping to slice grouping. If so, the DSTN moduleconverts the selected DS error encoded data into slice groupings and stores the slice grouping DS error encoded data by overwriting the pillar grouping DS error encoded data or by storing it in a different location in the memory of the DSTN module(i.e., does not overwrite the pillar grouping DS encoded data).

22 242 22 22 244 244 22 242 22 242 The DSTN modulepartitions the data and the task as indicated in the DST allocation informationand sends the portions to selected DST execution units of the DSTN module. Each of the selected DST execution units performs its partial task(s) on its slice groupings to produce partial results. The DSTN modulecollects the partial results from the selected DST execution units and provides them, as result information, to the task distribution module. The result informationmay be the collected partial results, one or more final results as produced by the DSTN modulefrom processing the partial results in accordance with the DST allocation information, or one or more intermediate results as produced by the DSTN modulefrom processing the partial results in accordance with the DST allocation information.

232 244 104 104 244 244 The task distribution modulereceives the result informationand provides one or more final resultstherefrom to the first DST client module. The final result(s)may be result informationor a result(s) of the task distribution module's processing of the result information.

238 240 232 232 232 232 In concurrence with processing the selected task of the first DST client module, the distributed computing system may process the selected task(s) of the second DST client module on the selected data(s) of the second DST client module. Alternatively, the distributed computing system may process the second DST client module's request subsequent to, or preceding, that of the first DST client module. Regardless of the ordering and/or parallel processing of the DST client module requests, the second DST client module provides its selected dataand selected taskto a task distribution module. If the task distribution moduleis a separate device of the distributed computing system or within the DSTN module, the task distribution modulescoupled to the first and second DST client modules may be the same module. The task distribution moduleprocesses the request of the second DST client module in a similar manner as it processed the request of the first DST client module.

29 FIG. 28 FIG. 232 232 242 248 250 252 246 is a schematic block diagram of an embodiment of a task distribution modulefacilitating the example of. The task distribution moduleincludes a plurality of tables it uses to generate distributed storage and task (DST) allocation informationfor selected data and selected tasks received from a DST client module. The tables include data storage information, task storage information, distributed task (DT) execution module information, and task ⇔ sub-task mapping information.

248 260 262 264 266 1 1 1 1 1 1 The data storage information tableincludes a data identification (ID) field, a data size field, an addressing information field, distributed storage (DS) information, and may further include other information regarding the data, how it is stored, and/or how it can be processed. For example, DS encoded data #has a data ID of 1, a data size of AA (e.g., a byte size of a few Terabytes or more), addressing information of Addr__AA, and DS parameters of 3/5; SEG_; and SLC_. In this example, the addressing information may be a virtual address corresponding to the virtual address of the first storage word (e.g., one or more bytes) of the data and information on how to calculate the other addresses, may be a range of virtual addresses for the storage words of the data, physical addresses of the first storage word or the storage words of the data, may be a list of slice names of the encoded data slices of the data, etc. The DS parameters may include identity of an error encoding scheme, decode threshold/pillar width (e.g., 3/5 for the first data entry), segment security information (e.g., SEG_), per slice security information (e.g., SLC_), and/or any other information regarding how the data was encoded into data slices.

250 268 270 272 274 2 2 2 2 2 2 The task storage information tableincludes a task identification (ID) field, a task size field, an addressing information field, distributed storage (DS) information, and may further include other information regarding the task, how it is stored, and/or how it can be used to process data. For example, DS encoded task #has a task ID of 2, a task size of XY, addressing information of Addr__XY, and DS parameters of 3/5; SEG_; and SLC_. In this example, the addressing information may be a virtual address corresponding to the virtual address of the first storage word (e.g., one or more bytes) of the task and information on how to calculate the other addresses, may be a range of virtual addresses for the storage words of the task, physical addresses of the first storage word or the storage words of the task, may be a list of slices names of the encoded slices of the task code, etc. The DS parameters may include identity of an error encoding scheme, decode threshold/pillar width (e.g., 3/5 for the first data entry), segment security information (e.g., SEG_), per slice security information (e.g., SLC_), and/or any other information regarding how the task was encoded into encoded task slices. Note that the segment and/or the per-slice security information include a type of encryption (if enabled), a type of compression (if enabled), watermarking information (if enabled), and/or an integrity check scheme (if enabled).

246 256 258 256 258 246 1 1 2 The task ⇔ sub-task mapping information tableincludes a task fieldand a sub-task field. The task fieldidentifies a task stored in the memory of a distributed storage and task network (DSTN) module and the corresponding sub-task fieldsindicates whether the task includes sub-tasks and, if so, how many and if any of the sub-tasks are ordered. In this example, the task ⇔ sub-task mapping information tableincludes an entry for each task stored in memory of the DSTN module (e.g., taskthrough task k). In particular, this example indicates that taskincludes 7 sub-tasks; taskdoes not include sub-tasks, and task k includes r number of sub-tasks (where r is an integer greater than or equal to two).

252 276 278 280 276 278 1 1 1 1 2 1 3 280 1 1 The DT execution module tableincludes a DST execution unit ID field, a DT execution module ID field, and a DT execution module capabilities field. The DST execution unit ID fieldincludes the identity of DST units in the DSTN module. The DT execution module ID fieldincludes the identity of each DT execution unit in each DST unit. For example, DST unitincludes three DT executions modules (e.g.,_,_, and_). The DT execution capabilities fieldincludes identity of the capabilities of the corresponding DT execution unit. For example, DT execution module_includes capabilities X, where X includes one or more of MIPS capabilities, processing resources (e.g., quantity and capability of microprocessors, CPUs, digital signal processors, co-processor, microcontrollers, arithmetic logic circuitry, and/or any other analog and/or digital processing circuitry), availability of the processing resources, memory information (e.g., type, size, availability, etc.), and/or any information germane to executing one or more tasks.

232 242 From these tables, the task distribution modulegenerates the DST allocation informationto indicate where the data is stored, how to partition the data, where the task is stored, how to partition the task, which DT execution units should perform which partial task on which data partitions, where and how intermediate results are to be stored, etc. If multiple tasks are being performed on the same data or different data, the task distribution module factors such information into its generation of the DST allocation information.

30 FIG. 318 92 2 1 2 3 1 2 3 is a diagram of a specific example of a distributed computing system performing tasks on stored data as a task flow. In this example, selected datais dataand selected tasks are tasks,, and. Taskcorresponds to analyzing translation of data from one language to another (e.g., human language or computer language); taskcorresponds to finding specific words and/or phrases in the data; and taskcorresponds to finding specific translated words and/or phrases in translated data.

1 1 1 1 2 1 3 1 4 1 3 1 5 1 4 1 6 1 5 1 1 1 7 1 5 1 2 2 3 3 1 3 2 In this example, taskincludes 7 sub-tasks: task_-identify non-words (non-ordered); task_-identify unique words (non-ordered); task_-translate (non-ordered); task_-translate back (ordered after task_); task_-compare to ID errors (ordered after task-); task_-determine non-word translation errors (ordered after task_and_); and task_-determine correct translations (ordered after_and_). The sub-task further indicates whether they are an ordered task (i.e., are dependent on the outcome of another task) or non-order (i.e., are independent of the outcome of another task). Taskdoes not include sub-tasks and taskincludes two sub-tasks: task_translate; and task_find specific word or phrase in translated data.

92 306 282 300 286 302 290 316 92 298 In general, the three tasks collectively are selected to analyze data for translation accuracies, translation errors, translation anomalies, occurrence of specific words or phrases in the data, and occurrence of specific words or phrases on the translated data. Graphically, the datais translatedinto translated data; is analyzed for specific words and/or phrasesto produce a list of specific words and/or phrases; is analyzed for non-words(e.g., not in a reference dictionary) to produce a list of non-words; and is analyzed for unique wordsincluded in the data(i.e., how many different words are included in the data) to produce a list of unique words. Each of these tasks is independent of each other and can therefore be processed in parallel if desired.

282 3 2 304 288 282 308 1 4 284 1 3 284 310 92 294 1 5 310 306 308 1 3 1 4 The translated datais analyzed (e.g., sub-task_) for specific translated words and/or phrasesto produce a list of specific translated words and/or phrases. The translated datais translated back(e.g., sub-task_) into the language of the original data to produce re-translated data. These two tasks are dependent on the translate task (e.g., task_) and thus must be ordered after the translation task, which may be in a pipelined ordering or a serial ordering. The re-translated datais then comparedwith the original datato find words and/or phrases that did not translate (one way and/or the other) properly to produce a list of incorrectly translated words. As such, the comparing task (e.g., sub-task_)is ordered after the translationand re-translation tasks(e.g., sub-tasks_and_).

294 312 290 292 294 314 298 296 The list of words incorrectly translatedis comparedto the list of non-wordsto identify words that were not properly translated because the words are non-words to produce a list of errors due to non-words. In addition, the list of words incorrectly translatedis comparedto the list of unique wordsto identify unique words that were properly translated to produce a list of correctly translated words. The comparison may also identify unique words that were not properly translated to produce a list of unique words that were not properly translated. Note that each list of words (e.g., specific words and/or phrases, non-words, unique words, translated words and/or phrases, etc. ,) may include the word and/or phrase, how many times it is used, where in the data it is used, and/or any other information requested regarding a word and/or phrase.

31 FIG. 30 FIG. 29 FIG. 2 88 1 5 1 1 3 1 5 2 2 3 7 is a schematic block diagram of an example of a distributed storage and task processing network (DSTN) module storing data and task codes for the example of. As shown, DS encoded datais stored as encoded data slices across the memory (e.g., stored in memories) of DST execution units-; the DS encoded task code(of task) and DS encoded taskare stored as encoded task slices across the memory of DST execution units-; and DS encoded task code(of task) is stored as encoded task slices across the memory of DST execution units-. As indicated in the data storage information table and the task storage information table of, the respective data/task has DS parameters of 3/5 for their decode threshold/pillar width; hence spanning the memory of five DST execution units.

32 FIG. 30 FIG. 242 242 320 322 324 320 322 326 328 330 332 324 334 336 338 340 is a diagram of an example of distributed storage and task (DST) allocation informationfor the example of. The DST allocation informationincludes data partitioning information, task execution information, and intermediate result information. The data partitioning informationincludes the data identifier (ID), the number of partitions to split the data into, address information for each data partition, and whether the DS encoded data has to be transformed from pillar grouping to slice grouping. The task execution informationincludes tabular information having a task identification field, a task ordering field, a data partition field ID, and a set of DT execution modulesto use for the distributed task processing per data partition. The intermediate result informationincludes tabular information having a name ID field, an ID of the DST execution unit assigned to process the corresponding intermediate result, a scratch pad storage field, and an intermediate result storage field.

30 FIG. 1 3 2 2 2 2 2 1 2 z Continuing with the example of, where tasks-are to be distributedly performed on data, the data partitioning information includes the ID of data. In addition, the task distribution module determines whether the DS encoded datais in the proper format for distributed computing (e.g., was stored as slice groupings). If not, the task distribution module indicates that the DS encoded dataformat needs to be changed from the pillar grouping format to the slice grouping format, which will be done by the DSTN module. In addition, the task distribution module determines the number of partitions to divide the data into (e.g.,_through_) and addressing information for each partition.

1 1 2 1 2 1 1 2 1 3 1 4 1 5 1 1 1 2 1 3 1 4 1 5 1 2 1 2 1 1 1 1 1 2 1 1 1 2 1 2 z z The task distribution module generates an entry in the task execution information section for each sub-task to be performed. For example, task_(e.g., identify non-words on the data) has no task ordering (i.e., is independent of the results of other sub-tasks), is to be performed on data partitions_through_by DT execution modules_,_,_,_, and_. For instance, DT execution modules_,_,_,_, and_search for non-words in data partitions_through_to produce task_intermediate results (R-, which is a list of non-words). Task_(e.g., identify unique words) has similar task execution information as task_to produce task_intermediate results (R-, which is the list of unique words).

1 3 1 1 2 1 3 1 4 1 5 1 2 1 2 4 1 2 2 2 3 2 4 2 5 2 2 5 2 1 3 1 3 z Task_(e.g., translate) includes task execution information as being non-ordered (i.e., is independent), having DT execution modules_,_,_,_, and_translate data partitions_through_and having DT execution modules_,_,_,_, and_translate data partitions_through_to produce task_intermediate results (R-, which is the translated data). In this example, the data partitions are grouped, where different sets of DT execution modules perform a distributed sub-task (or task) on each data partition group, which allows for further parallel processing.

1 4 1 3 1 3 1 3 1 1 1 2 1 3 1 4 1 5 1 1 3 1 3 1 1 3 4 1 2 2 2 6 1 7 1 7 2 1 3 1 3 5 1 3 1 4 1 4 z Task_(e.g., translate back) is ordered after task_and is to be executed on task_'s intermediate result (e.g., R-_) (e.g., the translated data). DT execution modules_,_,_,_, and_are allocated to translate back task_intermediate result partitions R-_through R-_and DT execution modules_,_,_,_, and_are allocated to translate back task_intermediate result partitions R-_through R-_to produce task-intermediate results (R-, which is the translated back data).

1 5 1 4 1 4 4 1 1 1 2 1 3 1 4 1 5 1 2 1 2 1 4 1 4 1 1 4 1 5 1 5 z z Task_(e.g., compare data and translated data to identify translation errors) is ordered after task_and is to be executed on task_'s intermediate results (R-) and on the data. DT execution modules_,_,_,_, and_are allocated to compare the data partitions (_through_) with partitions of task-intermediate results partitions R-_through R-_to produce task_intermediate results (R-, which is the list words translated incorrectly).

1 6 1 1 1 5 1 1 1 5 1 1 1 5 1 1 2 1 3 1 4 1 5 1 1 1 1 1 1 1 1 1 5 1 5 1 1 5 1 6 1 6 z z Task_(e.g., determine non-word translation errors) is ordered after tasks_and_and is to be executed on tasks_'s and_'s intermediate results (R-and R-). DT execution modules_,_,_,_, and_are allocated to compare the partitions of task_intermediate results (R-_through R-_) with partitions of task-intermediate results partitions (R-_through R-_) to produce task_intermediate results (R-, which is the list translation errors due to non-words).

1 7 1 2 1 5 1 2 1 5 1 1 1 5 1 2 2 2 3 2 4 2 5 2 1 2 1 2 1 1 2 1 5 1 5 1 1 5 1 7 1 7 z z Task_(e.g., determine words correctly translated) is ordered after tasks_and_and is to be executed on tasks_'s and_'s intermediate results (R-and R-). DT execution modules_,_,_,_, and_are allocated to compare the partitions of task_intermediate results (R-_through R-_) with partitions of task-intermediate results partitions (R-_through R-_) to produce task_intermediate results (R-, which is the list of correctly translated words).

2 2 1 2 3 1 4 1 5 1 6 1 7 1 3 1 4 1 5 1 6 1 7 1 2 1 2 2 2 z z Task(e.g., find specific words and/or phrases) has no task ordering (i.e., is independent of the results of other sub-tasks), is to be performed on data partitions_through_by DT execution modules_,_,_,_, and_. For instance, DT execution modules_,_,_,_, and_search for specific words and/or phrases in data partitions_through_to produce taskintermediate results (R, which is a list of specific words and/or phrases).

3 2 1 3 1 3 1 1 3 1 2 2 2 3 2 4 2 5 2 1 2 2 2 3 2 4 2 5 2 1 3 1 1 3 3 2 3 2 z z Task_(e.g., find specific translated words and/or phrases) is ordered after task_(e.g., translate) is to be performed on partitions R-_through R-_by DT execution modules_,_,_,_, and_. For instance, DT execution modules_,_,_,_, and_search for specific translated words and/or phrases in the partitions of the translated data (R-_through R-_) to produce task_intermediate results (R-, which is a list of specific translated words and/or phrases).

1 1 1 1 1 1 1 1 5 For each task, the intermediate result information indicates which DST unit is responsible for overseeing execution of the task and, if needed, processing the partial results generated by the set of allocated DT execution units. In addition, the intermediate result information indicates a scratch pad memory for the task and where the corresponding intermediate results are to be stored. For example, for intermediate result R-(the intermediate result of task_), DST unitis responsible for overseeing execution of the task_and coordinates storage of the intermediate result as encoded intermediate result slices stored in memory of DST execution units-. In general, the scratch pad is for storing non-DS encoded intermediate results and the intermediate result storage is for storing DS encoded intermediate results.

33 38 FIGS.- 30 FIG. 33 FIG. 92 1 90 90 z are schematic block diagrams of the distributed storage and task network (DSTN) module performing the example of. In, the DSTN module accesses the dataand partitions it into a plurality of partitions-in accordance with distributed storage and task network (DST) allocation information. For each data partition, the DSTN identifies a set of its DT (distributed task) execution modulesto perform the task (e.g., identify non-words (i.e., not in a reference dictionary) within the data partition) in accordance with the DST allocation information. From data partition to data partition, the set of DT execution modulesmay be the same, different, or a combination thereof (e.g., some data partitions use the same set while other data partitions use different sets).

1 1 2 1 3 1 4 1 5 1 1 1 102 1 1 2 1 3 1 4 1 5 1 1 1 102 1 1 1 1 102 32 FIG. 32 FIG. For the first data partition, the first set of DT execution modules (e.g.,_,_,_,_, and_per the DST allocation information of) executes task_to produce a first partial resultof non-words found in the first data partition. The second set of DT execution modules (e.g.,_,_,_,_, and_per the DST allocation information of) executes task_to produce a second partial resultof non-words found in the second data partition. The sets of DT execution modules (as per the DST allocation information) perform task_on the data partitions until the “z” set of DT execution modules performs task_on the “zth” data partition to produce a “zth” partial resultof non-words found in the “zth” data partition.

32 FIG. 1 1 1 90 1 1 1 1 1 1 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results to produce the first intermediate result (R-), which is a list of non-words found in the data. For instance, each set of DT execution modulesstores its respective partial result in the scratchpad memory of DST execution unit(which is identified in the DST allocation or may be determined by DST execution unit). A processing module of DST executionis engaged to aggregate the first through “zth” partial results to produce the first intermediate result (e.g., R_). The processing module stores the first intermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

1 1 1 1 1 1 1 1 m DST execution unitengages its DST client module to slice grouping based DS error encode the first intermediate result (e.g., the list of non-words). To begin the encoding, the DST client module determines whether the list of non-words is of a sufficient size to partition (e.g., greater than a Terabyte). If yes, it partitions the first intermediate result (R-) into a plurality of partitions (e.g., R-_through R-_). If the first intermediate result is not of sufficient size to partition, it is not partitioned.

2 1 5 For each partition of the first intermediate result, or for the first intermediate result, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-).

34 FIG. 1 2 92 92 1 1 1 1 2 1 2 z In, the DSTN module is performing task_(e.g., find unique words) on the data. To begin, the DSTN module accesses the dataand partitions it into a plurality of partitions-in accordance with the DST allocation information or it may use the data partitions of task_if the partitioning is the same. For each data partition, the DSTN identifies a set of its DT execution modules to perform task_in accordance with the DST allocation information. From data partition to data partition, the set of DT execution modules may be the same, different, or a combination thereof. For the data partitions, the allocated set of DT execution modules executes task_to produce a partial results (e.g., 1st through “zth”) of unique words found in the data partitions.

32 FIG. 1 102 1 2 1 2 92 1 1 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial resultsof task_to produce the second intermediate result (R-), which is a list of unique words found in the data. The processing module of DST executionis engaged to aggregate the first through “zth” partial results of unique words to produce the second intermediate result. The processing module stores the second intermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

1 1 2 1 2 1 1 2 m DST execution unitengages its DST client module to slice grouping based DS error encode the second intermediate result (e.g., the list of non-words). To begin the encoding, the DST client module determines whether the list of unique words is of a sufficient size to partition (e.g., greater than a Terabyte). If yes, it partitions the second intermediate result (R-) into a plurality of partitions (e.g., R-_through R-_). If the second intermediate result is not of sufficient size to partition, it is not partitioned.

2 1 5 For each partition of the second intermediate result, or for the second intermediate results, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-).

35 FIG. 1 3 92 92 1 1 1 1 3 1 1 2 1 3 1 4 1 5 1 2 1 2 4 1 2 2 2 3 2 4 2 5 2 2 5 2 90 1 3 102 z z In, the DSTN module is performing task_(e.g., translate) on the data. To begin, the DSTN module accesses the dataand partitions it into a plurality of partitions-in accordance with the DST allocation information or it may use the data partitions of task_if the partitioning is the same. For each data partition, the DSTN identifies a set of its DT execution modules to perform task_in accordance with the DST allocation information (e.g., DT execution modules_,_,_,_, and_translate data partitions_through_and DT execution modules_,_,_,_, and_translate data partitions_through_). For the data partitions, the allocated set of DT execution modulesexecutes task_to produce partial results(e.g., 1st through “zth”) of translated data.

32 FIG. 2 1 3 1 3 2 2 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results of task_to produce the third intermediate result (R-), which is translated data. The processing module of DST executionis engaged to aggregate the first through “zth” partial results of translated data to produce the third intermediate result. The processing module stores the third intermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

2 1 3 1 3 1 1 3 2 2 6 y DST execution unitengages its DST client module to slice grouping based DS error encode the third intermediate result (e.g., translated data). To begin the encoding, the DST client module partitions the third intermediate result (R-) into a plurality of partitions (e.g., R-_through R-_). For each partition of the third intermediate result, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-per the DST allocation information).

35 FIG. 1 4 90 1 4 1 1 2 1 3 1 4 1 5 1 1 3 1 1 3 4 1 2 2 2 6 1 7 1 7 2 1 3 5 1 3 1 4 102 z As is further shown in, the DSTN module is performing task_(e.g., retranslate) on the translated data of the third intermediate result. To begin, the DSTN module accesses the translated data (from the scratchpad memory or from the intermediate result memory and decodes it) and partitions it into a plurality of partitions in accordance with the DST allocation information. For each partition of the third intermediate result, the DSTN identifies a set of its DT execution modulesto perform task_in accordance with the DST allocation information (e.g., DT execution modules_,_,_,_, and_are allocated to translate back partitions R-_through R-_and DT execution modules_,_,_,_, and_are allocated to translate back partitions R-_through R-_). For the partitions, the allocated set of DT execution modules executes task_to produce partial results(e.g., 1st through “zth”) of re-translated data.

32 FIG. 3 1 4 1 4 3 3 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results of task_to produce the fourth intermediate result (R-), which is retranslated data. The processing module of DST executionis engaged to aggregate the first through “zth” partial results of retranslated data to produce the fourth intermediate result. The processing module stores the fourth intermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

3 1 4 1 4 1 1 4 2 3 7 z DST execution unitengages its DST client module to slice grouping based DS error encode the fourth intermediate result (e.g., retranslated data). To begin the encoding, the DST client module partitions the fourth intermediate result (R-) into a plurality of partitions (e.g., R-_through R-_). For each partition of the fourth intermediate result, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-per the DST allocation information).

36 FIG. 35 FIG. 1 5 92 92 1 1 In, a distributed storage and task network (DSTN) module is performing task_(e.g., compare) on dataand retranslated data of. To begin, the DSTN module accesses the dataand partitions it into a plurality of partitions in accordance with the DST allocation information or it may use the data partitions of task_if the partitioning is the same. The DSTN module also accesses the retranslated data from the scratchpad memory, or from the intermediate result memory and decodes it, and partitions it into a plurality of partitions in accordance with the DST allocation information. The number of partitions of the retranslated data corresponds to the number of partitions of the data.

1 1 90 1 5 1 1 2 1 3 1 4 1 5 1 1 5 102 For each pair of partitions (e.g., data partitionand retranslated data partition), the DSTN identifies a set of its DT execution modulesto perform task_in accordance with the DST allocation information (e.g., DT execution modules_,_,_,_, and_). For each pair of partitions, the allocated set of DT execution modules executes task_to produce partial results(e.g., 1st through “zth”) of a list of incorrectly translated words and/or phrases.

32 FIG. 1 1 5 1 5 1 1 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results of task_to produce the fifth intermediate result (R-), which is the list of incorrectly translated words and/or phrases. In particular, the processing module of DST executionis engaged to aggregate the first through “zth” partial results of the list of incorrectly translated words and/or phrases to produce the fifth intermediate result. The processing module stores the fifth intermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

1 1 5 1 5 1 1 5 2 1 5 z DST execution unitengages its DST client module to slice grouping based DS error encode the fifth intermediate result. To begin the encoding, the DST client module partitions the fifth intermediate result (R-) into a plurality of partitions (e.g., R-_through R-_). For each partition of the fifth intermediate result, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-per the DST allocation information).

36 FIG. 1 6 1 5 1 1 As is further shown in, the DSTN module is performing task_(e.g., translation errors due to non-words) on the list of incorrectly translated words and/or phrases (e.g., the fifth intermediate result R-) and the list of non-words (e.g., the first intermediate result R-). To begin, the DSTN module accesses the lists and partitions them into a corresponding number of partitions.

1 1 1 1 5 1 90 1 6 1 1 2 1 3 1 4 1 5 1 1 6 102 For each pair of partitions (e.g., partition R-_and partition R-_), the DSTN identifies a set of its DT execution modulesto perform task_in accordance with the DST allocation information (e.g., DT execution modules_,_,_,_, and_). For each pair of partitions, the allocated set of DT execution modules executes task_to produce partial results(e.g., 1st through “zth”) of a list of incorrectly translated words and/or phrases due to non-words.

32 FIG. 2 1 6 1 6 2 2 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results of task_to produce the sixth intermediate result (R-), which is the list of incorrectly translated words and/or phrases due to non-words. In particular, the processing module of DST executionis engaged to aggregate the first through “zth” partial results of the list of incorrectly translated words and/or phrases due to non-words to produce the sixth intermediate result. The processing module stores the sixth intermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

2 1 6 1 6 1 1 6 2 2 6 z DST execution unitengages its DST client module to slice grouping based DS error encode the sixth intermediate result. To begin the encoding, the DST client module partitions the sixth intermediate result (R-) into a plurality of partitions (e.g., R-_through R-_). For each partition of the sixth intermediate result, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-per the DST allocation information).

36 FIG. 1 7 1 5 1 2 As is still further shown in, the DSTN module is performing task_(e.g., correctly translated words and/or phrases) on the list of incorrectly translated words and/or phrases (e.g., the fifth intermediate result R-) and the list of unique words (e.g., the second intermediate result R-). To begin, the DSTN module accesses the lists and partitions them into a corresponding number of partitions.

1 2 1 1 5 1 90 1 7 1 2 2 2 3 2 4 2 5 2 1 7 102 For each pair of partitions (e.g., partition R-_and partition R-_), the DSTN identifies a set of its DT execution modulesto perform task_in accordance with the DST allocation information (e.g., DT execution modules_,_,_,_, and_). For each pair of partitions, the allocated set of DT execution modules executes task_to produce partial results(e.g., 1st through “zth”) of a list of correctly translated words and/or phrases.

32 FIG. 3 1 7 1 7 3 3 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results of task_to produce the seventh intermediate result (R-), which is the list of correctly translated words and/or phrases. In particular, the processing module of DST executionis engaged to aggregate the first through “zth” partial results of the list of correctly translated words and/or phrases to produce the seventh intermediate result. The processing module stores the seventh intermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

3 1 7 1 7 1 1 7 2 3 7 z DST execution unitengages its DST client module to slice grouping based DS error encode the seventh intermediate result. To begin the encoding, the DST client module partitions the seventh intermediate result (R-) into a plurality of partitions (e.g., R-_through R-_). For each partition of the seventh intermediate result, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-per the DST allocation information).

37 FIG. 2 92 1 1 1 90 2 2 102 z In, the distributed storage and task network (DSTN) module is performing task(e.g., find specific words and/or phrases) on the data. To begin, the DSTN module accesses the data and partitions it into a plurality of partitions-in accordance with the DST allocation information or it may use the data partitions of task_if the partitioning is the same. For each data partition, the DSTN identifies a set of its DT execution modulesto perform taskin accordance with the DST allocation information. From data partition to data partition, the set of DT execution modules may be the same, different, or a combination thereof. For the data partitions, the allocated set of DT execution modules executes taskto produce partial results(e.g., 1st through “zth”) of specific words and/or phrases found in the data partitions.

32 FIG. 7 2 2 2 7 2 2 7 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results of taskto produce taskintermediate result (R), which is a list of specific words and/or phrases found in the data. The processing module of DST executionis engaged to aggregate the first through “zth” partial results of specific words and/or phrases to produce the taskintermediate result. The processing module stores the taskintermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

7 2 2 2 2 1 2 2 m DST execution unitengages its DST client module to slice grouping based DS error encode the taskintermediate result. To begin the encoding, the DST client module determines whether the list of specific words and/or phrases is of a sufficient size to partition (e.g., greater than a Terabyte). If yes, it partitions the taskintermediate result (R) into a plurality of partitions (e.g., R_through R_). If the taskintermediate result is not of sufficient size to partition, it is not partitioned.

2 2 2 1 4 7 For each partition of the taskintermediate result, or for the taskintermediate results, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-, and).

38 FIG. 3 1 3 3 90 3 102 In, the distributed storage and task network (DSTN) module is performing task(e.g., find specific translated words and/or phrases) on the translated data (R-). To begin, the DSTN module accesses the translated data (from the scratchpad memory or from the intermediate result memory and decodes it) and partitions it into a plurality of partitions in accordance with the DST allocation information. For each partition, the DSTN identifies a set of its DT execution modules to perform taskin accordance with the DST allocation information. From partition to partition, the set of DT execution modules may be the same, different, or a combination thereof. For the partitions, the allocated set of DT execution modulesexecutes taskto produce partial results(e.g., 1st through “zth”) of specific translated words and/or phrases found in the data partitions.

32 FIG. 5 3 3 3 5 3 3 7 As indicated in the DST allocation information of, DST execution unitis assigned to process the first through “zth” partial results of taskto produce taskintermediate result (R), which is a list of specific translated words and/or phrases found in the translated data. In particular, the processing module of DST executionis engaged to aggregate the first through “zth” partial results of specific translated words and/or phrases to produce the taskintermediate result. The processing module stores the taskintermediate result as non-DS error encoded data in the scratchpad memory or in another section of memory of DST execution unit.

5 3 3 3 3 1 3 3 m DST execution unitengages its DST client module to slice grouping based DS error encode the taskintermediate result. To begin the encoding, the DST client module determines whether the list of specific translated words and/or phrases is of a sufficient size to partition (e.g., greater than a Terabyte). If yes, it partitions the taskintermediate result (R) into a plurality of partitions (e.g., R_through R_). If the taskintermediate result is not of sufficient size to partition, it is not partitioned.

3 3 2 1 4 5 7 For each partition of the taskintermediate result, or for the taskintermediate results, the DST client module uses the DS error encoding parameters of the data (e.g., DS parameters of data, which includes 3/5 decode threshold/pillar width ratio) to produce slice groupings. The slice groupings are stored in the intermediate result memory (e.g., allocated memory in the memories of DST execution units-,, and).

39 FIG. 30 FIG. 104 2 3 1 1 1 1 1 2 1 1 6 1 1 7 104 is a diagram of an example of combining result information into final resultsfor the example of. In this example, the result information includes the list of specific words and/or phrases found in the data (taskintermediate result), the list of specific translated words and/or phrases found in the data (taskintermediate result), the list of non-words found in the data (taskfirst intermediate result R-), the list of unique words found in the data (tasksecond intermediate result R-), the list of translation errors due to non-words (tasksixth intermediate result R-), and the list of correctly translated words and/or phrases (taskseventh intermediate result R-). The task distribution module provides the result information to the requesting DST client module as the results.

40 FIG.A 1 FIG. 1 FIG. 1 FIG. 1 FIG. 1 2 1 22 24 1 2 12 1 16 is a schematic block diagram of an embodiment of a dispersed storage network (DSN) that includes at least two user devices-, a plurality of distributed storage and task (DST) processing units-D, and the distributed storage and task network (DSTN) moduleof. The DSN may further include the networkof. The user devices-may be implemented utilizing the computing deviceof. The DST processing units-D may be implemented utilizing the DST processing unitof.

1 1 1 2 2 1 2 1 2 1 2 1 1 1 The DSN functions to store data as stored data and to retrieve the stored data to reproduce the data. In an example of operation of storing the data, the user deviceselects a DST processing unit of the DST processing units-D based on an identifier of a data object for storage. The selecting may be based on one or more of an address space mapping and performing a deterministic function on the identifier of the data object. For example, the user deviceselects the DST processing unitwhen the identifier of the data object indicates data object-and the address mapping indicates that the data object-is affiliated with DST processing unit. As another example, the user deviceselects the DST processing unitwhen a result of applying the deterministic function to an identifier of another data object-produces an indicator that includes an identifier of the DST processing unit.

2 1 2 2 2 2 1 2 1 2 2 1 2 2 2 1 2 1 2 Having selected the DST processing unit, the user deviceissues a data access messageto the DST processing unit, where the data access messageincludes a store data request. The store data request includes the data object-. Having received the data object-, the DST processing unitcaches the data object-in a local memory of the DST processing unitin accordance with the address space mapping and an available capacity level. For example, the DST processing unitcaches the data object-when the available capacity level compares favorably (e.g., greater than) to a minimum available capacity threshold level and the address space mapping indicates that the data object-is affiliated with the DST processing unit.

2 2 1 2 22 22 2 22 2 The DST processing unitdispersed storage error encodes the data object-to produce a plurality of sets of encoded data slices. Having produced the plurality of sets of encoded data slices, the DST processing unitissues a DSTN message to the DSTN moduleto facilitate storage of the plurality of sets of encoded data slices in the DSTN module. For example, the DST processing unitgenerates a set of write slice requests that includes the plurality of sets of encoded data slices and sends the set of write slice requests to the DSTN moduleas the DSTN message.

2 2 2 1 2 2 2 1 1 1 1 2 1 1 2 1 2 18 1 2 1 FIG. When the available capacity level of the DST processing unitdoes not compare favorably to the minimum available capacity threshold level, the DST processing unitselects at least one locally stored data object for transfer. For example, the DST processing unitselects a data object-N associated with an address at an end of an address range associated with the DST processing unit. Having selected the store data object for transfer, the DST processing unitissues a transfer message-to the DST processing unitthat includes the data object-N. One or more of the DST processing unitand DST processing unitfacilitates updating of the address space mapping to indicate that the data object-N is affiliated with DST processing unitand is no longer affiliated with DST processing unit. Having updated the address space mapping, the one or more of the DST processing units-sends the address space mapping that has been updated to one or more entities of the DSN (e.g., to the DSTN managing unitoffor further distribution, to other DST processing units, to the user devices-).

2 2 2 1 2 2 2 2 2 2 2 1 In an example of operation to retrieve the stored data to reproduce the data, the user deviceidentifies the DST processing unitas affiliated with the data object-for retrieval based on accessing the address space mapping. Having identified the DST processing unit, the user deviceissues a data access messageto the identified DST processing unit. For example, the user deviceissues a retrieve data request to the DST processing unit, where the retrieve data request includes the identifier of the data object-.

2 2 2 1 2 1 2 2 1 22 2 1 2 2 2 2 2 1 2 2 2 1 Having received the retrieve data request from the user device, the DST processing unitobtains the data object-. The obtaining includes at least one of retrieving the data object-from the local memory of DST processing unitand recovering the data object-from the DSTN module. When the data object-is available from the local memory of the DST processing unit, the DST processing unitissues a data access messageto the user device, where the data access message includes the data object-. For example, the DST processing unitsends a retrieve data response to the user device, where the retrieve data response includes the data object-.

2 1 2 2 2 22 2 22 2 1 2 1 When the data object-is not available from the local memory of the DST processing unit, the DST processing unitissues a DSTN messageto the DSTN module. For example, the DST processing unitissues a read threshold number of read slice requests to the DSTN modulewith regards to the data object-, receives read slice responses, and decodes encoded data slices of the received read slice responses to reproduce the data object-.

40 FIG.B 360 is a flowchart illustrating an example of accessing data. The accessing includes one or more of storing the data, transferring the data, and retrieving the data. The method begins or continues, when storing a data object, at stepwhere a requesting entity (e.g., a user device) identifies an access module (e.g., a distributed storage and task (DST) processing unit) for a dispersed storage network (DSN) based on an identifier of the data object for storage. For example, the requesting entity accesses an address based mapping utilizing an identifier of the data object to retrieve an identifier of the access module that is associated with the identifier the data object.

362 364 366 The method continues at stepwhere the requesting entity sends the data object to the access module for storage in the DSN. The method continues at stepwhere the access module stores the data object in the DSN. For example, the access module dispersed storage error encodes the data to produce a plurality of sets of encoded data slices and sends the plurality of sets of encoded data slices to a set of storage units for storage. The method continues at stepwhere the access module facilitates storage of the data object in a cache memory in accordance with the address based mapping. For example, the access module stores the received data object in a memory of the identified access module. As another example, the access module sends the data object to another access module for storage, where the other access module is affiliated with the identifier of the data object.

368 When transferring the data object, the method continues at stepwhere the access module determines whether to transfer one or more data objects from the cache memory. The determining may be based on one or more of an available cache memory storage level, a frequency of access level for the one or more data objects, the available input/output resources, and an elapsed time of storage of the one or more data objects. For example, the access module determines to transfer a first data object from the cache memory when the first data object is associated with a frequency of access level that compares unfavorably (e.g., greater than) to a maximum frequency of access threshold level.

370 When transferring the data object, the method continues at stepwhere the access module identifies a cached data object for transfer and another access module. For example, the access module selects a cached data object associated with an identifier at an end of an address range associated with the access module where the identifier at the end meets a beginning identifier of another address range associated with another access module. Having selected the cached data object, the access module initiates a capacity query to the other access module and receives a favorable query response (e.g., to approve transfer of the identified cached data object).

372 374 The method continues at stepwhere the access module facilitates transfer of the cached data object to the other access module. For example, the access module issues a transfer message to the other access module, where the transfer message includes the identified cached data object and an identifier of the cached data object. Having received the transfer message, the other access module stores the data object in a local cache memory associated with the other access module. Having stored the data object in the local cache memory, the other access module may verify that the data object is available for retrieval from the DSN. Having stored the data object in the local cache memory associated with the other access module, the other access module sends a transfer confirmation message to the access module indicating that the data object has been successfully transferred. Having received the transfer confirmation message, the method continues at stepwhere the access module facilitates updating the address based mapping to disassociate the data object with the access module and associate the data object with the other access module.

376 378 When retrieving the data object from the DSN, the method continues atwhere the requesting entity identifies the other access module based on the identifier of the data object for retrieval. For example, the other access module recovers an identifier of the other access module from the address space mapping using the identifier of the data object. The method continues at stepwhere the requesting entity issues a data object retrieval request to the other access module. The issuing includes generating the data object retrieval request to include the identifier of the data object and sending the data object retrieval request to the other access module.

380 382 The method continues at stepwhere the other access module retrieves the data object from the local cache memory of the other access module when the data object is available from the cache memory of the other access module. Alternatively, the other access module retrieves the data object from the DSN when the data object is unavailable from the local cache memory of the other access module. The method continues at stepwhere the other access module sends the data object to the requesting entity.

41 FIG.A 1 FIG. 1 FIG. 1 FIG. 1 FIG. 1 FIG. 1 FIG. 3 FIG. 1 FIG. 400 402 22 403 12 24 402 403 16 402 404 34 403 406 34 400 404 406 88 16 12 34 is a schematic block diagram of another embodiment of a dispersed storage network (DSN) that includes a content source, an ingestion distributed storage and task (DST) processing unit, the DSTN moduleof, a retrieval DST processing unit, and the computing deviceof. Alternatively, the DSN further includes the networkof. The ingestion DST processing unitand the retrieval DST processing unitmay be implemented utilizing the DST processing unitof. The ingestion DST processing unitincludes a content ingestion moduleand the DST client moduleof. The retrieval DST processing unitincludes a content retrieval moduleand the DST client moduleof. The content source, the content ingestion module, and the content retrieval modulemay be implemented utilizing one or more of the processing moduleof, the DST processing unitof, the computing device, a server, a computing device, and the DST client module.

22 404 408 400 408 The system functions to ingest content for storage as stored content in the DSTN moduleand to retrieve the stored content. The content includes one or more of files and streams. The files and streams include one or more content types. The content types include one or more of multimedia, audio, text, messages, movies, movie trailers, web videos, music recordings, etc. In an example of operation of the ingesting of the content for storage, the content ingestion modulereceives the contentfrom the content source. The contentmay include one or more formats of common source material. Formats include one or more of encoding algorithms and encoding parameters associated with representing the source material in a digital form.

408 404 408 1 2 50 Having received the content, the content ingestion moduletransforms the contentinto one or more formats-F in accordance with a formatting scheme. The formatting scheme includes one or more of a number of formats, a type of format for each of the formats, and parameters associated with each format. For example, a first format includes aMbps standard definition video stream and a second format includes aMbps high-definition video stream.

1 34 402 34 34 34 22 34 3 400 3 For each of the formats-F, the DST client moduleof the ingestion DST processing unitdispersed storage error encodes the format to produce a plurality of sets of encoded data slices. Having generated the plurality of sets of encoded data slices, the DST client modulegenerates a plurality of sets of slice names for the plurality of sets of encoded data slices based on one or more of a content type indicator and a content identifier. Having generated the plurality of sets of slice names, the DST client modulefacilitates storage of the plurality of sets of encoded data slices. For example, the DST client modulegenerates a set of write slice requests and sends the set of write slice requests to the DSTN module, where the set of write slice requests includes the plurality of sets of encoded data slices and the plurality of sets of slice names. For instance, the DST client moduleissues write slice requestsfor a third format from the content ingestion modulefor, where the write slice requestsincludes a set of write slice requests.

34 402 34 3 3 Having facilitated the storage of the plurality of sets of encoded data slices for each format, the DST client moduleof the ingestion DST processing unitfacilitates updating one or more of a dispersed hierarchical index and a DSN directory to associate identifiers of each of the formats and logical DSN addresses of the storage of each of the plurality of sets of encoded data slices. For example, the DST client moduleupdates the dispersed hierarchical index to include a formatidentifier with a source name associated with the plurality of sets of slice names corresponding to the plurality of sets of encoded data slices from encoding of the format.

12 410 403 410 410 406 In an example of operation of retrieving stored content, the computing deviceissues a content requestto the retrieval DST processing unit, where the content requestincludes one or more of a content identifier, a time indicator, and a type indicator of one or more desired formats associated with content of the content identifier. Having received the content request, the content retrieval moduleaccesses one or more of the dispersed hierarchical index and the DSN directory using one or more of a content ID, the time indicator, and the type indicators of the one or more desired formats to identify a logical DSN address for each desired format.

34 403 34 22 412 Having identified the DSN addresses for each of the desired formats, the DST client moduleof the retrieval DST processing unitrecovers at least some of the plurality of sets of encoded data slices using the DSN addresses. For example, the DST client moduleissues a set of read slice requests to the DSTN moduleand receives read slice responses. The set of read slice requests includes a plurality of sets of slice names associated with each of the plurality sets of encoded data slices of each of the desired formats.

412 34 403 414 414 406 416 12 416 414 Having received the read slice responses, the DST client moduleof the retrieval DST processing unitdispersed storage error decodes a decode threshold number of encoded data slices for each set of the plurality of sets of encoded data slices of each plurality of sets of encoded data slices to reproduce the one or more formats. Having recovered the one or more formats, the content retrieval modulesends requested contentto the computing device, where the requested contentincludes the reproduced one or more formats.

41 FIG.B 420 422 is a flowchart illustrating another example of accessing data. The method begins or continues, when ingesting data, at stepwhere an ingestion unit (e.g., a distributed storage and task (DST) processing unit utilized for ingestion) receives content for storage in a set of storage units. The ingestion unit may further receive one or more of an identifier of the content, a content size indicator, a content format indicator, and a content type indicator. The method continues at stepwhere the ingestion unit transforms the content into one or more formats. For example, the ingestion unit re-encodes the content into a plurality of other formats in accordance with a formatting scheme.

424 426 For each format, the method continues at stepwhere the ingestion unit dispersed storage error encodes the format to produce a plurality of sets of encoded data slices. For each format, the method continues at stepwhere the ingestion unit generates a logical address. For example, the ingestion unit produces a unique source name for each format based on one or more of the content identifier, a content type, and a format identifier.

428 430 For each logical address, the method continues at stepwhere the ingestion unit generates a plurality of sets of slice names. For example, the ingestion unit generates each slice name to include an associated unique source name. For each format, the method continues at stepwhere the ingestion unit facilitates storage of the corresponding plurality of sets of encoded data slices in the set of storage units using a corresponding plurality of sets of slice names. For example, the ingestion unit generates a set of write slice requests and sends the set of write slice requests to the set of storage units, where the set of write slice requests includes the corresponding plurality of sets of encoded data slices and the corresponding plurality of sets of slice names.

432 For each format, the method continues at stepwhere the ingestion unit updates one or more of a directory and a dispersed hierarchical index to associate one or more of a content identifier, the format, and the associated logical address. For example, the ingestion unit adds index keys to entries of the dispersed hierarchical index, where the index keys are associated with identifiers of the formats and the entries includes the logical address of the formats.

434 436 The method continues or begins, when retrieving the data, at stepwhere a retrieval unit receives a content request, where the content request includes one or more of type indicators of one or more desired formats, the content ID, and a time identifier. The method continues at stepwhere the retrieval unit accesses one or more of the directory and the dispersed hierarchical index using the type indicators and the content ID to recover an associated logical address for each desired format. For example, the retrieval unit utilizes a type indicator as an index key to access the dispersed hierarchical index to recover a corresponding logical address for the desired format.

438 For each desired format, the method continues at stepwhere the retrieval unit recovers at least some sets of the plurality of sets of encoded data slices associated with the desired format from the set of storage units. For example, the retrieval unit converts the associated logical address into sets of slice names, issues a set of read slice requests that includes the sets of slice names, and receives at least a decode threshold number of encoded data slices for each set of encoded data slices.

440 442 For each desired format, the method continues at stepwhere the retrieval unit decodes the received at least some sets of the plurality of sets of encoded data slices to reproduce the content in the desired format. Alternatively, or in addition to, the retrieval unit restricts decoding of slices to encoded data slices associated with a desired time frame within a video stream in accordance with the time identifier. For each desired format, the method continues at stepwhere the retrieval unit outputs the reproduced content of the desired format to a requesting entity.

42 FIG.A 1 FIG. 1 FIG. 1 FIG. 1 1 24 18 36 1 1 1 2 1 1 1 1 2 1 1 3 1 1 1 1 is a schematic block diagram of another embodiment of a dispersed storage network (DSN) that includes M distributed storage and task (DST) execution unit sets-M implemented at a plurality of N sites-N, the networkof, and the distributed storage and task network (DSTN) managing unitof. Each DST execution unit set includes a set of n DST execution units (alternatively referred to herein as “storage units” or, in the singular, as a “storage unit”). Each DST execution unit may be implemented utilizing the DST execution unitof. As a specific example of the implementation of the M DST execution unit sets-M at the N sites-N, siteincludes a first DST execution unit from each DST execution unit set, siteincludes a second DST execution unit from each DST execution unit set, etc. For instance, the siteincludes DST execution units--,--,--, through M--(set, site, unit). As described below, a scalable approach is provided for detecting and reporting certain failure conditions that a target storage unit may not be able to self-report (e.g., loss of network connectivity, power failure, or a disabling failure of a critical component).

18 18 24 The illustrated DSN functions to establish monitoring resources for monitoring of one or more of the DST execution units and to perform the monitoring of the one or more DST execution units in accordance with the establishing of the monitoring resources. In an example of operation to establish the monitoring resources, the DSTN managing unitobtains DSN configuration information. Alternatively, any other module, unit, or DSN entity of the DSN may perform such steps described for the DSTN managing unit. The obtaining includes at least one of performing a lookup, accessing at least a portion of registry information, receiving the configuration information, initiating a query, and receiving a query response. The DSN configuration information includes one or more of site location information, power source information, configuration information of the network, a number of sites indicator, a number of DST execution unit sets indicator, a mapping of DST execution units to each DST execution unit set, a mapping of each DST execution unit to a site, a rebuilding capability level of a DST execution unit, a foster encoded data slice storage capability level of a DST execution unit, a DST execution unit storage capacity level, and a DST execution unit storage utilization level.

18 Having obtained the DSN configuration information, the DSTN managing unitselects a target DST execution unit to be monitored. The selecting may be based on one or more of identifying a next DST execution unit from a list of DST execution units to be monitored, detecting a new DST execution unit, determining that the DST execution unit is not being monitored, determining that a timeframe has elapsed since a last monitoring of the DST execution unit, interpreting an error message associated with the DST execution unit, and receiving a request.

18 Having selected the target DST execution unit, the DSTN managing unitdetermines a number of monitoring DST execution units to associate with the target DST execution unit. The determining may be based on one or more of a DSN activity level, an expected failure rate level, a monitoring table, and a predetermination. For example, the DSTN managing unit selects the number of the monitoring DST execution units to be 1 when the DSN activity level is greater than a high threshold level. As another example, the DSTN managing unit selects the number of the monitoring DST execution units to be 3 when the DSN activity level is within an expected threshold level of an average DSN activity level and the monitoring table entry indicates to utilize three monitoring units when the DSN activity level is within the expected threshold level of the average DSN activity level.

18 18 18 Having determined the number of monitoring DST execution units, the DSTN managing unitdetermines an estimated level of failure correlation between the target DST execution unit and at least some of the other DST execution units of the plurality of DST execution units based on the DSN configuration information. For example, the DSTN managing unitindicates a higher than average estimated level of failure correlation between the target DST execution unit and another DST execution unit when the DSN configuration information indicates that the target DST execution unit and the other DST execution unit are implemented at a common site. As another example, the DSTN managing unitindicates a lower than average estimated level of failure correlation between the target DST execution unit and the other DST execution unit when the DSN configuration information indicates that the target DST execution unit and the other DST execution unit are implemented at different sites and are part of different DST execution unit sets.

18 18 18 1 3 1 1 2 2 2 2 2 2 Having determined the number of monitoring DST execution units, the DSTN managing unitselects at least some of the other DST execution units based on the estimated level of failure correlation and the number of monitoring DST execution units. For example, the DSTN managing unitrank orders other DST execution units by corresponding estimated levels of failure correlation and selects the number that is associated with a least amount of failure correlation. For instance, the DSTN managing unitselects three DST execution units-N-n,--, and M-N-n when the target DST execution unit is DST execution unit--and the three DST execution units are associated with the least amount of failure correlation with regards to DST execution unit--.

18 18 2 2 2 Having selected the monitoring DST execution units, the DSTN managing unitassigns the selected monitoring DST execution units for the target DST execution unit. For example, the DSTN managing unitissues unit status information to the three selected monitoring DST execution units, where the unit status information includes a monitoring assignment request to monitor DST execution unit--. The monitoring assignment request may include one or more of an identifier of the target DST execution unit, identifiers of the monitoring DST execution units, a frequency of reporting schedule, a threshold for reporting, and a type of monitoring indicator. The type of monitoring indicator indicates one or more types of monitoring including one or more of sending a periodic ping to the target storage unit, receiving a ping response, sending data to the target storage unit, receiving a response to the sending of the data, exchanging security information, obtaining performance information relating to the target storage unit, and initiating a test involving the target storage unit and analyzing responsive test results.

450 450 450 18 3 1 1 2 2 2 450 3 1 1 450 450 18 3 1 1 450 18 450 3 1 1 450 18 3 1 1 450 In an example of operation to perform the monitoring, at least some of the assigned monitoring DST execution units perform a monitoring function in accordance with the type of monitoring indicator to monitor the target DST execution unit and to produce unit status informationthat includes monitoring results. Having produced unit status information, the assigned monitoring DST execution unit sends the unit status informationto the DSTN managing unitin accordance with one or more of a frequency of reporting schedule and the threshold for reporting. For example, the DST execution unit--continually monitors DST execution unit--and produces the unit status information. The DST execution unit--interprets the unit status informationand determines to send the unit status informationto the DSTN managing unitwhen a portion of the unit status information compares unfavorably to a desired threshold level. As another example, the DST execution unit--determines to send the unit status informationto the DSTN managing unitwhen interpreting of the frequency of reporting schedule indicates to send the unit status information. The DST execution unit--sends the unit status informationto the DSTN managing unitwhen the DST execution unit--determines to send the unit status information.

42 FIG.B 452 is a flowchart illustrating an example of monitoring storage units. The method begins or continues at stepwhere one or more processing modules (e.g., of a distributed storage and task network (DSTN) managing unit) selects a target storage unit for monitoring, where a dispersed storage network (DSN) includes a plurality of storage units that includes the storage unit. The selecting may be based on one or more of interpreting a selection list, detecting that no monitoring units are associated with the target storage unit, detecting activation of a new storage unit, and receiving a request.

454 456 The method continues at stepwhere the processing module obtains configuration information for the DSN. For example, the processing module accesses a portion of a system registry. The method continues at stepwhere the processing module determines a number of monitoring storage units to monitor the target storage unit. For example, the processing module selects a higher than average number of monitoring storage units when a DSN activity level is lower than an average DSN activity level. As another example, the processing module selects a lower than average number of monitoring storage units when an estimated storage unit failure rate is lower than an average storage unit failure rate.

458 460 For each of at least some of other storage units of the plurality of storage units, the method continues at stepwhere the processing module determines an estimated failure correlation level with the target storage unit based on the configuration information for the DSN. The method continues at stepwhere the processing module selects monitoring storage units based on the associated estimated failure correlation levels and the determined number of monitoring storage units. For example, the processing module selects monitoring storage units associated with minimal estimated failure correlation levels.

462 The method continues at stepwhere the processing module assigns the monitoring storage units to monitor the target storage unit. For example, the processing module issues a monitoring assignment message to each of the monitoring storage units and to the target storage unit, where the monitoring assignment message indicates a monitoring relationship between the assigned monitoring storage units and the target storage unit.

464 The method continues at stepwhere each monitoring storage unit monitors the target storage unit. For example, the monitoring storage unit, from time to time, initiates a status check message to the target storage unit, receives a status check response from the target storage unit, indicates a potential failure when not receiving a favorable status check response within a response timeframe, gathers statistics, and reports failures and the statistics to one or more of a managing unit, the target storage unit, and at least one of other monitoring storage units.

43 FIG.A 1 FIG. 1 FIG. 1 FIG. 34 24 1 2 1 1 1 2 2 36 n, is a schematic block diagram of another embodiment of a dispersed storage network (DSN) that includes the distributed storage and task (DST) client moduleof, the networkof, and at least two DST execution unit sets-. Each DST execution unit set includes a set of DST execution units-where each set of DST execution units is associated with an address range of the DSN. For example, DST execution unitis associated with an address rangeand DST execution unitis associated with an address range. The address range includes a starting address and an ending address, where the addresses includes one or more of a source name, a file name, and a slice name associated with an encoded data slice. Each DST execution unit may be implemented utilizing the DST execution unitof.

34 34 34 34 The DSN functions to store data as sets of encoded data slices in accordance with a time frame based addressing scheme. In an example of operation of the storing of the data, the DST client modulereceives data for storage associated with a first timeframe. Having received the data, the DST client moduleidentifies a current timeframe. The timeframe may include a repeating range. For example, the repeating range repeats every five minutes. Having identified the current timeframe, the DST client modulegenerates a source name based on the current timeframe such that source names for adjacent time frames are associated with different address space ranges assigned to DST execution unit sets. The generating includes performing a deterministic function on the current timeframe to produce the source name. Alternatively, the DST client modulegenerates a source name that maps to two or more address ranges of two or more DST execution unit sets and is always different than a next source name for a next time frame (e.g., the next time frame maps to different address ranges and DST execution unit sets).

34 34 34 Having generated the source name, the DST client modulegenerates a plurality of sets of slice names, where each slice name includes the source name and other slice name field entries in accordance with the system registry information and the data. Having generated the plurality of sets of slice names, the DST client moduledispersed storage error encodes the data to produce a plurality of sets of encoded data slices. Having generated the plurality of sets of encoded data slices, the DST client modulegenerates one or more sets of write slice requests that includes the plurality of sets of slice names and the plurality of sets of encoded data slices.

34 34 24 1 1 1 1 n Having generated the one or more sets of write slice requests, the DST client moduleidentifies a DST execution unit set based on the source name and an assigned address range of the DST execution unit set, where the source name falls within the assigned address range of the DST execution unit set. Having identified the DST execution unit set, the DST client moduleissues, via the network, write slice requests for rangeto the identified DST execution unit setto facilitate storage of the plurality of sets of encoded data slices in the set of DST execution units-of the DST execution unit set.

43 FIG.B 470 472 is a flowchart illustrating an example of storing data. The method begins or continues at stepwhere a processing module (e.g., of a distributed storage and task (DST) client module) receives data for storage. The method continues at stepwhere the processing module identifies a current timeframe. For example, the processing module obtains a current timestamp from a clock function and identifies the current timeframe based on the current timestamp (e.g., determining whether the current timestamp is within a timeframe interval).

474 The method continues at stepwhere the processing module generates a source name based on the current timeframe, where an address range of a plurality of address ranges associated with a plurality of storage unit sets includes the source name and where the address range is different than a previous address range associated with an immediately previous timeframe. For example, the processing module selects (e.g., round robin, random, next) one constant of a set of constants and performs a deterministic function on the selected constant and the current timeframe to produce an intermediate result. The processing module utilizes the intermediate result as at least some most significant bits of the source name.

476 478 The method continues at stepwhere the processing module identifies a storage unit set of the plurality of storage unit sets, where the storage unit set is associated with the address range that includes the source name. For example, the processing module accesses an address to storage unit set identification list. As another example, the processing module initiates a query and receives a query response. The method continues at stepwhere the processing module issues one or more sets of write slice requests to the identified storage units set to facilitate storage of the data. For example, the processing module generates a plurality of sets of slice names utilizing the source name, dispersed storage error encodes the data to produce a plurality of sets of encoded data slices, generates the one or more sets of write slice requests to include the plurality of sets of encoded data slices and the plurality of sets of slice names, and sends the one or more sets of write slice requests to the identified storage unit set.

44 FIG.A 1 FIG. 1 FIG. 1 FIG. 1 FIG. 1 FIG. 3 FIG. 1 4 490 16 36 20 24 1 4 490 1 1 1 1 1 2 1 2 2 1 2 2 2 36 88 1 1 1 1 1 2 1 2 1 1 2 1 2 1 2 2 2 2 2 2 n. is a schematic block diagram of another embodiment of a dispersed storage network (DSN) that includes a plurality of rebuilding modules-and a distributed storage and task (DST) execution (EX) unit set. Each rebuilding module may be implemented utilizing at least one of the DST processing unitof, the DST execution unitof, and the DST integrity processing unitof. The DSN may further include the networkofto operably couple the rebuilding modules-and the DST execution unit set. The DST execution unit set includes a set of DST execution units-Each DST execution unit includes a group of M memories. For example, DST execution unitincludes memories-,-, through-M, DST execution unitincludes memories-,-, through-M, etc. Each DST execution unit may be implemented utilizing the DST execution unitof. Each memory may be implemented utilizing the memoryof. Each memory is associated with an address range of the DSN. For example, memory-is associated with address range-, memory-associated with address range-, through memory-M is associated with address range-M, and memory-is associated with address range-, memory-associated with address range-, through memory-M is associated with address range-M, etc.

1 1 2 1 2 1 2 1 2 1 The DSN functions to adjust a rate of rebuilding activities, where the rebuilding activities includes detecting storage errors and rebuilding encoded data slices associated with the detected storage errors. In an example of operation, rebuilding moduleselects an address range for execution of rebuilding activities. The selecting includes at least one of interpreting a list, receiving an error message, receiving a rebuilding assignment, and a predetermination. For example, the rebuilding moduleselects address range-when receiving an error message associated with memory-and identifying the address range-as associated with memory-.

1 1 1 2 2 1 2 2 1 2 2 1 2 3 2 1 2 2 1 Having selected the address range for rebuilding activities, the rebuilding modulefacilitates execution of the rebuilding activities for the selected address range. For example, the rebuilding moduleidentifies DST execution unitto be associated with the selected address range, exchanges rebuilding messages range-with DST execution unitto detect the storage errors (e.g., issue list slice requests, receives list slice responses, interprets the list slice responses to identify the storage errors), rebuilds one or more encoded data slices associated with the identified storage errors, and exchanges further rebuilding messages range-with the DST execution unitto facilitate storage of one or more rebuilt encoded data slices in the memory-. Substantially in parallel, the rebuilding modulesandmay also be exchanging rebuilding messages range-with the DST execution unitto facilitate rebuilding activities within the address range-.

1 2 3 2 1 Having facilitated execution of the rebuilding activities for the selected address range, the rebuilding moduledetermines whether continuing execution of the rebuilding activities for the selected address range is associated with favorably impacting overall rebuilding activities associated with the selected address range (e.g., since rebuilding modulesandmay also be performing rebuilding activities with regards to the selected address range-). The determining whether the continuing execution of the rebuilding activities for the selected address range is associated with favorably impacting the overall rebuilding activities includes a series of steps.

1 2 1 2 The series of steps begins with the rebuilding moduleissuing a rebuilding rate request to the identified DST execution unitwith regards to a previous timeframe. The rebuilding rate includes a rate of rebuilding activities per unit of time (e.g., number of scans for detecting storage errors, number of writing rebuilt encoded data slices). The rebuilding modulereceives a rebuilding rate response from the identified DST execution unit, where the response indicates a rate of rebuilding activities associated with the selected address range over the previous timeframe.

1 1 1 2 1 2 1 Having received the rebuilding rate associated with the previous timeframe, the rebuilding moduletemporarily suspends execution of the rebuilding activities for a suspension timeframe. For example, the rebuilding modulediscontinues issuing further list slice requests for further sub-address ranges and discontinues issuing further write rebuilt slice requests. Having temporarily suspended the execution of the rebuilding activities, the rebuilding moduleissues another rebuilding rate request to the identified DST execution unitwhen the suspension timeframe has expired. The rebuilding modulereceives another rebuilding rate response from the identified DST execution unit, where the other response indicates another rate of rebuilding activities associated with the selected address range over the suspension timeframe (e.g., when the rebuilding modulewas not participating in the rebuilding activities).

1 1 1 2 1 Having received the other rebuilding rate of rebuilding activities over the suspension timeframe, the rebuilding moduledetermines whether the other rebuilding rate compares favorably to the rebuilding rate. For example, the rebuilding moduleindicates that the comparison is unfavorable when a difference between the other rebuilding rate and the rebuilding rate is less than a low rebuilding rate threshold level. As such, the rebuilding moduleis not contributing in a meaningful way to the rebuilding activities associated with the selected address range-.

4 1 1 4 4 1 1 In another scenario, where rebuilding moduledetermines whether continuing execution of rebuilding activities for address range-is associated with favorably impacting the overall rebuilding activities, the rebuilding moduleindicates that the comparison is favorable when a difference between the other rebuilding rate and the rebuilding rate is greater than a high rebuilding rate threshold level. As such, the indication indicates that the rebuilding moduleis contributing in a significant way to the rebuilding activities associated with the address range-.

4 1 1 1 When the comparison is favorable, the rebuilding modulefacilitates resuming the execution of rebuilding activities for address range-. When the comparison is unfavorable, the rebuilding moduleselects another address range and facilitates execution of other rebuilding activities for the other selected address range.

44 FIG.B 492 is a flowchart illustrating an example of rebuilding data. The method begins or continues at stepwhere a processing module (e.g., of a rebuilding module) selects an address range for rebuilding activities. The selecting includes at least one of identifying a next address range, interpreting an error message to identify the address range, receiving a request, and predicting that additional rebuilding resources can meaningfully impact execution of rebuilding activities associated with the address range.

494 496 The method continues at stepwhere the processing module facilitates execution of the rebuilding activities for the selected address range (e.g., scanning for storage errors, rebuilding encoded data slices associated with the storage errors, storing rebuilt encoded data slices). The method continues at stepwhere the processing module issues a rebuilding rate request to an identified storage unit for a previous timeframe, where the identified storage unit is associated with the selected address range. For example, the processing module identifies a memory device associated with the selected address range and identifies a storage unit of a plurality of storage units, where the storage unit is associated with the memory device to produce the identified storage unit.

498 500 The method continues at stepwhere the processing module receives a rebuilding rate response from the identified storage unit. The receiving includes the processing module extracting a rebuilding rate for the previous timeframe from the rebuilding rate response. The method continues at stepwhere the processing module temporarily suspends execution of rebuilding activities for the selected address range for a suspension timeframe (e.g., halts the scanning for storage errors, halts the rebuilding of the encoded data slices, and halts the storing of the rebuilt encoded data slices).

502 504 The method continues at stepwhere the processing module issues another rebuilding rate request to the identified storage unit for the suspension timeframe. The method continues at stepwhere the processing module receives another rebuilding rate response from the identified storage unit. The receiving includes extracting another rebuilding rate for the previous timeframe.

506 510 508 508 The method continues at stepwhere the processing module determines whether the other rebuilding rate compares favorably to the rebuilding rate. For example, the processing module indicates that the comparison is favorable when a difference between the other rebuilding rate and the rebuilding rate is greater than a high rebuilding rate threshold level. As another example, the processing module indicates the comparison is unfavorable when the difference between the other rebuilding rate and the rebuilding rate is less than a low rebuilding rate threshold level. The method branches to stepwhen the comparison is not favorable. The method continues to stepwhen the comparison is favorable. The method continues at stepwhere the processing module facilitates resuming the execution of the rebuilding activities for the selected address range (e.g., continue scanning, rebuilding, storing rebuilt encoded data slices).

510 512 The method continues at stepwhere the processing module selects another address range for rebuilding activities when the comparison is unfavorable. The selecting includes one or more of selecting a next address range, receiving an error message, identifying an address range associated with a storage error, receiving a request, selecting another memory device, and identifying the address range associated with the selected other memory device. The method continues at stepwhere the processing module facilitates execution of the rebuilding activities for the other selected address range.

45 FIG.A 1 FIG. 1 FIG. 1 FIG. 1 2 1 1 24 20 36 1 2 1 1 2 1 1 1 1 2 1 1 2 1 2 2 2 2 2 is a schematic block diagram of another embodiment of a dispersed storage network (DSN) that includes at least two distributed storage and task (DST) execution unit sets-implemented at a plurality of N sites-N, a rebuilding-only storage unit DST execution unit R, the networkof, and the DST integrity processing unitof. Each DST execution unit set includes a set of n DST execution units. Each DST execution unit may be implemented utilizing the DST execution unitof. As a specific example of the implementation of the DST execution unit sets-at the N sites-N, siteincludes a first DST execution unit from each DST execution unit set, siteincludes a second DST execution unit from each DST execution unit set, etc. For instance, the siteincludes DST execution units--and--, while the siteincludes DST execution units--and--, etc. (e.g., set, site, unit).

20 1 2 2 2 The DSN functions to rebuild data stored in the sets of DST execution units. In particular, one or more of the DST integrity processing unitand any DST execution unit may include a rebuilding module capable of facilitating the rebuilding of the stored data. For example, DST execution unit Ris capable of rebuilding the stored data. As another example, DST execution unit--is capable of rebuilding the stored data.

20 24 520 520 520 20 1 1 1 1 2 1 2 2 1 2 1 1 2 2 2 2 In an example of operation of the rebuilding of the stored data, the rebuilding module identifies a storage error of an encoded data slice associated with an unavailable memory device. For example, the DST integrity processing unitissues, via the network, rebuilding messagesto one or more of the DST execution units, receives rebuilding messagesthat includes list slice responses, and compares the list slice responses to identify storage errors. The rebuilding messagesincludes one or more of a list slice request, a list slice response, a rebuild an encoded data slice request, an available storage capacity level, DSN topology information, a write foster encoded data slice request, and a migrate foster encoded data slice back to a home storage location request. For instance, the DST integrity processing unitidentifies a storage error associated with slice A-at DST execution unit--, a storage error associated with slice A-at DST execution unit--, a storage error associated with slice B-at DST execution unit--, and a storage error associated with slice B-at DST execution unit--.

1 1 1 1 2 2 2 2 1 1 1 2 1 1 1 2 2 2 2 2 1 Having identified the storage error, the rebuilding module obtains DSN configuration information associated with the encoded data slice of the storage error. Having obtained the DSN configuration, the rebuilding module rebuilds the encoded data slice to produce a foster encoded data slice. For example, the rebuilding module produces a foster slice FA-for slice A-, a foster slice FB-for slice B-, a foster slice FA-for slice A-, and foster slice FB-for slice B-. Having produced the foster encoded data slice, the rebuilding module identifies a plurality of candidate storage locations for the foster encoded data slice based on the DSN configuration information. For example, the rebuilding module identifies the DST execution units--,--,--,--and the DST execution unit Ras the candidate storage locations when the DSN configuration information indicates that each DST execution unit is capable of storing foster encoded data slices.

1 1 1 2 1 1 1 2 2 2 2 2 1 Having identified the candidate storage locations for the foster encoded data slice, the rebuilding module obtains storage status of the candidate storage locations. The storage status indicates an available capacity level of storing a foster encoded data slice. The obtaining includes at least one of interpreting an error message, accessing a storage status list, initiating a query to a DST execution unit, and receiving a query response that includes the storage status. For example, the rebuilding module obtains storage status from DST execution units--,--, and--indicating no available capacity (e.g., since unavailable) to store the foster encoded data slice and obtains storage status from the DST execution units--and Rindicating sufficient capacity to store the foster encoded data slice.

1 1 1 1 1 1 1 1 2 1 1 1 1 1 2 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 Having obtained the storage status of the candidate storage locations, the rebuilding module selects at least one storage location of the candidate storage locations for storing the foster encoded data slice based on one or more of the storage status and the DSN configuration information. For example, the rebuilding module selects DST execution unit Rto store foster slices FA-and FB-in accordance with the DSN configuration information indicating that DST execution unit Ris implemented at a common sitewith DST execution units--and--(e.g., to facilitate migration of foster encoded data slices back to home DST execution units--and--when the DST execution units are available). As another example, the rebuilding module selects DST execution unit--to store foster slice FB-when the configuration information indicates that DST execution unit--is a home DST execution unit associated with slice B-and the storage status of DST execution unit--indicates that the DST execution unit--has sufficient capacity to store foster encoded data slices. As yet another example, the rebuilding module selects DST execution unit--to store foster slice FA-when the DSN configuration information indicates that DST execution unit--is implemented at a common site with home DST execution unit--for slice A-.

1 1 1 2 2 2 2 2 Having selected the at least one storage location, the rebuilding module facilitates temporary storage of the foster encoded data slice in the selected storage location. For example, the rebuilding module facilitates storage of foster slices FA-and FB-in DST execution unit Rand facilitates storage of foster slices FA-and FB-in DST execution unit--. When any of the home DST execution units become available, the rebuilding module may facilitate migration of foster slice from the temporary storage in the selected storage location to the home DST execution unit. Having facilitated the temporary storage of the foster encoded data slice, the rebuilding module may update a storage location list to associate the foster encoded data slice with the at least one candidate storage location.

45 FIG.B 522 524 is a flowchart illustrating another example of rebuilding data. The method begins or continues at stepwhere a processing module (e.g., of a rebuilding module) identifies a storage error associated with an encoded data slice. For example, the processing module identifies an unavailable memory device and identifies an encoded data slice associated with the unavailable memory device as the encoded data slice. The method continues at stepwhere the processing module obtains dispersed storage network (DSN) configuration information associated with the encoded data slice.

526 The method continues at stepwhere the processing module generates a rebuilt encoded data slice as a foster encoded data slice, where the rebuilt encoded data slice corresponds to the encoded data slice associated with the storage error. For example, the processing module recovers a decode threshold number of encoded data slices of a set of encoded data slices associated with the encoded data slice, dispersed storage error decodes the decode threshold number of encoded data slices to reproduce a data segment, and dispersed storage error encodes the reproduced data segment to produce the rebuilt encoded data slice.

528 The method continues at stepwhere the processing module identifies candidate storage locations for the foster encoded data slice based on the DSN configuration information. For example, the processing module identifies a storage unit at a site associated with the identified unavailable memory device. As another example, the processing module identifies another storage unit at an adjacent site to the site associated with the identified unavailable memory device.

530 The method continues at stepwhere the processing module determines storage status of the identified candidates storage locations. For example, the processing module initiates a storage status request and receives a storage status response indicating the storage status of the identified candidate storage locations.

532 The method continues at stepwhere the processing module selects at least one storage location from the candidate storage locations based on the storage status and the DSN configuration information. For example, the processing module selects a storage unit at the site associated with the identified unavailable memory device when the storage unit is available to store the foster encoded data slice and has sufficient memory capacity in accordance with the storage status of the storage unit.

534 The method continues at stepwhere the processing module facilitates temporary storage of the foster encoded data slice at the selected at least one storage location. For example, the processing module stores the foster encoded data slice in a local memory when the storage location is another memory of a common storage unit associated with the processing module. As another example, the processing module issues a write slice request to the storage unit of the selected storage location, where the write slice request includes the foster encoded data slice.

536 The method continues at stepwhere the processing module updates a storage location list to associate the foster encoded data slice with the selected at least one storage location. The updating includes, for each of the selected at least one storage locations, updating one or more of a dispersed hierarchical index and a DSN directory to associate a slice name of the foster encoded data slice with one or more identifiers associated with the selected storage location.

46 FIG.A 4 FIG. 13 FIG. 3 FIG. 1 FIG. 4 FIG. 3 FIG. 112 540 542 544 546 548 182 540 548 84 34 114 80 542 546 544 558 558 544 is a schematic block diagram of another embodiment of a dispersed storage network (DSN) that includes a dispersed storage (DS) error encodingof, a molecule encoder, a molecule synthesizer, a molecule storage, a molecule sequencer, a molecule decoder, and a DS error decodingof. The molecule encoderand the molecule decodermay be implemented utilizing one or more of the processing moduleof, the distributed storage and task (DST) client moduleof, the grouping selectorof, and the outbound DST processingof. The molecule synthesizermay be implemented utilizing an oligonucleotide synthesizer. The molecule sequencermay be implemented utilizing a Deoxyribonucleic acid (DNA) sequencer. The molecule storageincludes an environment to sustain encoded molecules, where degradation is minimized due to one or more of age, radiation, and environmental conditions. Hereafter, the encoded moleculesmay be interchangeably referred to as molecule storage structures. The molecule storagemay include one or more molecule storage units, where a set of molecule storage units substantially partitions one or more sets of molecules.

550 558 544 570 564 550 112 550 552 550 552 560 562 The DSN functions to store dataas the encoded modulesin the molecule storageand to produce recovered datafrom retrieved encoded molecules. In an example of operation of the storing of the data, the DS error encodingreceives the dataand identifier (ID) informationassociated with the data. The identifier informationincludes one or more of a data object ID, a DSN address, a source name, a virtual DSN address, a slice name, storage ID information, and retrieval ID information.

560 562 558 544 544 560 562 560 562 558 544 The storage ID informationand the retrieval ID informationincludes one or more identifiers associated with the encoded moleculesto facilitate storage and retrieval of the encoded molecules. For example, a first storage ID includes an identifier associated with a first molecule that is to be stored in a first molecule storage unit of the set of molecule storage units of the molecule storage, a second storage ID includes an identifier associated with a second molecule that is to be stored in a second molecule storage unit of the set of molecule storage units of the molecule storage, etc. The storage ID informationand the retrieval ID informationmay be encoded in accordance with one or more industry and/or proprietary standards. For instance, the first storage ID is implemented utilizing a unique first Internet protocol (IP) address and the second storage ID is implemented utilizing a unique second IP address, etc. The storage ID informationand the retrieval ID informationmay be maintained in one or more of a database system, a DSN directory, a dispersed hierarchical index, and as encoded moleculesin the molecule storage.

550 552 112 550 1 550 112 1 550 n, Having received the dataand the ID information, the DS error encodingpartitions the datainto a plurality of data segments-S. Having partitioned the data, the DS error encodingdispersed storage error encodes each data segment into a set of encoded data slices-where a decode threshold number of encoded data slices of the set of encoded data slices is needed to recover the data segment and where each slice includes one or more coded blocks. Each coded block includes one or more bytes that is produced as a result of matrix multiplying an encoding matrix of a dispersed storage error coding function by a row of a data matrix that includes the data.

554 112 554 112 112 552 552 550 Having produced a plurality of sets of encoded data slices, the DS error encodingappends a portion of the ID information to one or more of the plurality of encoded data slices. For example, the DS error encodingappends a corresponding slice name to each encoded data slice. As another example, the DS error encodingdispersed storage error encodes the ID informationto produce a set of encoded identifier slices (e.g., for subsequent storage, for inclusion in a node of the dispersed hierarchical index, for inclusion in the DSN directory that associates the ID informationwith the data).

540 554 552 556 556 552 The molecule encodertranslates one or more of the plurality of sets of encoded data slices, the appended portions of the ID information, the ID information, and the set of encoded identifier slices into molecule encoding informationin accordance with a translation approach. The molecule encoding informationincludes encoding patterns for a plurality of molecules (e.g., for DNA molecules, for other molecules). The encoding may include obtaining the translation approach. The obtaining of the translation approach may be based on one or more of the ID information, a size of each slice, a size of the data, available storage facilities, a predetermination, and interpreting a query response.

The translation approach includes at least three approaches. A first approach includes each permutation of a plurality of bits of the encoded data slice field is mapped to a corresponding variant of a molecule structure. For example, mapping to molecules of one or more genes of DNA, or a DNA portion, or junk DNA. A second approach includes each permutation of the plurality of bits of the slice field map to a corresponding variant of two or more molecule structures. For example, in a q-ary encoding approach, where q is greater than or equal to two. A third approach includes a permutation of a plurality of bits of a portion of a coded block of the encoded data slice field is mapped to a corresponding variant of the one or more molecule structures.

554 556 540 540 540 540 540 In an example of operation of the translating of the one or more of the plurality of sets of encoded data slicesto produce the molecule encoding information, the molecule encodergenerates a number based on a selected encoded data slice for translation and identifies a gene based on the number to produce an identified gene. For instance, the molecule encoderconverts data (e.g., a slice portion such as one or more coded blocks) of the encoded data slice of the set of encoded data slices into the number (e.g., equates or performs a number lookup) and uses the number as a gene identifier to identify the gene (e.g., equates or performs a gene lookup). As another instance, the molecule encoderperforms a mathematical function (e.g., a deterministic function such as at least one of a hashing function, a hash based message authentication code, a mask generating function, and a sponge function) on the data of the encoded data slice of the set of encoded data slices to produce the number and uses the number as a gene identifier to identify the gene. As yet another instance, the molecule encoderconverts data of the encoded data slice of the set of encoded data slices into a plurality of numbers (e.g., a number for one or more coded blocks), where the plurality of numbers includes the number, generates a plurality of gene identifiers from the plurality of numbers, and uses the plurality of gene identifiers to identify a plurality of genes, where the plurality of genes includes the gene. As a still further instance, the molecule encodergenerates a preliminary number based on the encoded data slice, determines whether the preliminary number substantially matches a known gene identifier (e.g., attempts a lookup), sets the number to the preliminary number when the preliminary number substantially matches the known gene identifier, or performs a function (e.g., another deterministic function) on the preliminary number to produce the number when the preliminary number does not substantially match the known gene identifier.

540 552 540 540 Having identified the gene, the molecule encodercreates a linking identifier (e.g., the ID informationand/or a slice name that corresponds to the encoded data slice) that links the encoded data slice to the identified gene, where, for the set of encoded data slices, a set of identified genes and a set of linking identifiers are created. As a specific example, when using the plurality of gene identifiers to identify a plurality of genes, the molecule encodercreates the linking identifier that links the encoded data slice to the plurality of identified genes. As another specific example, when performing a function on the preliminary number to produce the number, the molecule encodercreates the linking identifier that links the encoded data slice to the identified gene and identifies the function performed on the preliminary number.

540 556 556 540 542 556 542 540 47 FIGS.A-C Having created the linking identifier, the molecule encodergenerates molecular encoding informationfrom the set of identified genes and the set of linking identifiers, and outputs, the molecular encoding informationvia an interface of the molecule encoder, to the molecule synthesizer, where the molecular encoding informationis used by the molecule synthesizerto create a molecular storage structure for each identified gene of the set of identified genes yielding a set of molecular storage structures. The method of operation of the molecule encoderis discussed in greater detail with reference to.

556 542 556 558 544 560 540 542 544 542 558 546 542 558 546 Having produced the molecule encoding information, the molecule synthesizerprocesses the molecule encoding informationto fabricate a collection of encoded moleculesfor storage in the molecule storagein accordance with the storage ID information. For example, for a third encoded data slice of a second set of encoded data slices, the molecule encodertranslates the third encoded data slice into a third molecule encoding of the molecule encoding information, and the molecule synthesizerfabricates the third molecule encoding into a third encoded molecule for storage in a third storage unit of the molecule storage, where the third storage unit is associated with a third storage ID of the storage ID information. Alternatively, the molecule synthesizersends, via a direct path, the encoded moleculesto the molecule sequencer. For example, the molecule synthesizerreceives a test request and sends the encoded moleculesto the molecule sequencer.

558 544 550 558 544 546 564 566 548 568 182 570 112 540 548 182 With the encoded moleculesstored in the molecule storage, the DSN may verify storage of the set of identified genes in the set of molecular structures (e.g., to verify storage of the dataas the encoded moleculesin the molecule storage). In an example of operation of the verifying, the molecule sequenceraccesses at least a test decode threshold number of molecular storage structures (e.g., retrieved encoded molecules) of the set of molecular storage structures to retrieve at least a decode threshold number of identified genes (e.g., molecule decoding information) of the set of identified genes. The molecule decoderrecovers at least a test decode threshold number of test encoded data slices (e.g., recovered slices) from the at least the test decode threshold number of identified genes. The DS error decodingreconstructs a data segment from the at least a test decode threshold number of test encoded data slices to produce a reconstructed data segment (e.g., a portion of recovered data). At least one of the DS error encoding, the molecule encoder, the molecule decoder, and the DS error decodingindicates a level of verifiable storage when the reconstructed data segment substantially matches the data segment. The level includes at least one of an estimated level of data retrieval reliability, an estimated level of data storage availability, a number of molecular storage structures accessed, a number of identified genes, and a number of test encoded data slices.

550 570 182 564 562 182 550 562 562 544 564 544 562 544 564 546 562 544 546 554 562 In an example of operation of the recovering of the dataas the recovered data, the DS error decodingfacilitates retrieval of some of the encoded molecules associated with the data to produce the retrieved encoded moleculesin accordance with the retrieval ID information. For example, the DS error decodingidentifies the data for retrieval (e.g., by translating a name of the datainto the retrieval ID informationutilizing the DSN directory) and outputs the retrieval ID informationto the molecule storageto facilitate extracting of the retrieved encoded moleculesfrom molecule storage. Having received the retrieval ID information, the molecule storageprovides the retrieved encoded moleculesto the molecule sequencerin accordance with the retrieval ID information. For example, the molecule storageprovides at least a decode threshold number of molecules of the set of molecules to the molecule sequenceras the retrieved encoded moleculeswhen the retrieval ID informationidentifies a set of storage units associated with storage of the set of molecules.

564 546 564 566 566 548 546 564 566 566 548 566 Having received the retrieved encoded molecules, the molecule sequenceranalyzes the retrieved encoded moleculesin accordance with a sequencing scheme associated with the collection of encoded molecules to produce molecule decoding informationand sends the molecule decoding informationto the molecule decoder. For example, the molecule sequenceranalyzes the retrieved encoded moleculesin accordance with a DNA sequencing scheme to produce the molecule decoding informationand sends the molecule decoding informationto the molecule decoder, where the molecule decoding informationincludes identified DNA structures (e.g., one or more genes) and components.

566 548 566 568 554 568 182 1 1 k Having received the molecule decoding information, the molecule decoderdecodes the molecule decoding informationto produce the recovered slicesof the plurality of sets of encoded data slicesin accordance with the translation approach and sends the recovered slicesto the DS error decoding, where at least a decode threshold number of encoded data slices are recovered for each set of encoded data slices. The decoding may further include interpreting slice names (e.g., from the linking identifiers) associated with the recovered slices to facilitate grouping of recovered slices-for each data segment-S.

568 182 1 182 1 570 182 572 182 572 Having received the recovered slices, the DS error decodingdispersed storage error decodes a decode threshold number of recovered slices per set of encoded data slices to produce a plurality of recovered data segments-S. The DS error decodingaggregates the plurality of recovered data segments-S to produce the recovered data. The DS error decodingmay further dispersed storage error decode another decode threshold number of recovered slices of the set of encoded identifier slices to reproduce the portion of the ID information as recovered ID information. Alternatively, or in addition to, the DS error decodingmay de-append the portion of the ID information from one or more of the recovered slices to produce the recovered ID information(e.g., extracts the linking identifiers).

46 FIG.B 580 is a flowchart illustrating an example of accessing data via molecule configuration. The method begins or continues, when storing the data, at stepwhere an encoder generates a plurality of data segments based on data for storage and data identification information. The generating includes one or more of partitioning the data in accordance with a partitioning scheme to produce the data segments and appending a portion of the data identification information to one or more of the data segments.

582 The method continues at stepwhere the encoder encodes each of the plurality of data segments to produce a plurality of sets of encoded data slices in accordance with a dispersed storage error coding function. The encoding may further include combining the data identification information with one or more of the encoded data slices. The combining includes at least one of appending and interleaving. Alternatively, or in addition to, the encoder encodes the data identification information using the dispersed storage error coding function to produce a set of encoded identifier slices.

584 The method continues at stepwhere a molecule encoder translates the plurality of sets of encoded data slices into molecule encoding information in accordance with a translation approach. For example, the molecule encoder obtains the translation approach and translates each encoded data slice of each set of encoded data slices into molecule encoding information for a molecule of a set of molecules.

586 The method continues at stepwhere a molecule synthesizer processes the molecule encoding information to fabricate a collection of encoded molecules. For example, the molecule synthesizer processes the molecule encoding information to fabricate a set of encoded molecules for each set of encoded data slices. As another example, the molecule synthesizer processes the molecule encoding information to fabricate a set of encoded molecules for two or more sets of encoded data slices. As yet another example, the molecule synthesizer processes the molecule encoding information to fabricate a single molecule for each set of encoded data slices, where a portion of the single molecule represents an encoded data slice of the set of encoded data slices.

588 The method continues at stepwhere a molecule storage stores the collection of encoded molecules in accordance with storage identification information. For example, the molecule storage interprets the data identification information to produce the storage identification information and utilizes the storage identification information to store the collection of encoded molecules in one or more storage facilities of the molecule storage.

590 The method continues, when retrieving the data, at stepwhere a decoder facilitates retrieval of at least some of the encoded molecules of the collection of molecules in accordance with retrieval identification information. For example, the decoder receives the retrieval identification information and sends the retrieval identification information to the molecule storage.

592 The method continues at stepwhere a molecule sequencer analyzes the retrieved encoded molecules to produce molecule decoding information in accordance with a sequencing scheme associated with the collection of encoded molecules. For example, the molecule sequencer obtains the sequencing scheme based on the collection of encoded molecules by performing a lookup based on the retrieval identification information.

594 596 The method continues at stepwhere a molecule decoder decodes the molecule decoding information to produce recovered slices in accordance with the translation approach. For example, the molecule decoder obtains a mapping of the translation approach, identifies patterns of the molecule decoding information based on a comparison to the mapping of the translation approach, and translates the identified patterns into blocks of encoded data slices as the recovered slices. The method continues at stepwhere the decoder decodes, utilizing a dispersed storage error coding function, a decode threshold number of recovered encoded data slices per set of encoded data slices to produce recovered data and recovered data identification information.

46 FIG.C 1 39 46 FIGS.-,A 46 FIG.C 600 is a flowchart illustrating an example of generating molecular encoding information for data storage. In particular, a method is presented for use in conjunction with one or more functions and features described in conjunction with-B, and also. The method begins at stepwhere a processing module of a computing device of one or more computing devices, for each encoded data slice of a set of encoded data slices, generates a number based on encoded data slice. A data segment is dispersed storage error encoded into the set of encode data slices. A decode threshold number of encoded data slices of the set of encoded data slices is needed to recover the data segment in accordance with a dispersed storage error coding function.

602 The method continues at stepwhere the processing module identifies a gene based on the number to produce an identified gene. As a specific example, the processing module converts data of the encoded data slice of the set of encoded data slices into the number and uses the number as a gene identifier to identify the gene. As another specific example, the processing module performs a mathematical function on data of the encoded data slice of the set of encoded data slices to produce the number and uses the number as a gene identifier to identify the gene. As yet another specific example, the processing module converts data of the encoded data slice of the set of encoded data slices into a plurality of numbers, where the plurality of numbers includes the number, generates a plurality of gene identifiers from the plurality of numbers, and uses the plurality of gene identifiers to identify a plurality of genes, where the plurality of genes includes the gene. As a further example, the processing module generates a preliminary number based on the encoded data slice, determines whether the preliminary number substantially matches a known gene identifier, sets the number to the preliminary number when the preliminary number substantially matches the known gene identifier, and performs a function on the preliminary number to produce the number when the preliminary number does not substantially match the known gene identifier.

604 606 The method continues at stepwhere the processing module creates a linking identifier that links the encoded data slice to the identified gene, where, for the set of encoded data slices, a set of identified genes and a set of linking identifiers are created. As a specific example, when using the plurality of gene identifiers to identify a plurality of genes, the processing module creates the linking identifier that links the encoded data slice to the plurality of identified genes. As another specific example, when performing a function on the preliminary number to produce the number, the processing module creates the linking identifier that links the encoded data slice to the identified gene and identifies the function performed on the preliminary number. The method continues at stepwhere the processing module generates molecular encoding information from the set of identified genes and the set of linking identifiers, where the molecular encoding information is used to create a molecular storage structure for each identified gene of the set of identified genes yielding a set of molecular storage structures.

608 610 612 614 The method continues at step, when verifying storage of the set of identified genes in the set of molecular structures, where the processing module accesses at least a test decode threshold number of molecular storage structures of the set of molecular storage structures to retrieve at least a decode threshold number of identified genes of the set of identified genes. The method continues at stepwhere the processing module recovers at least a test decode threshold number of test encoded data slices from the at least the test decode threshold number of identified genes. The method continues at stepwhere the processing module reconstructs a data segment from the at least a test decode threshold number of test encoded data slices to produce a reconstructed data segment. When the reconstructed data segment substantially matches the data segment, the method continues at stepwhere the processing module indicates a level of verifiable storage.

The method described above in conjunction with the processing module can alternatively be performed by other modules of the dispersed storage network or by other devices. In addition, at least one memory section of a computer readable storage medium that stores operational instructions can, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), cause the one or more computing devices to perform any or all of the method steps described above.

47 FIG.A 3 FIG. 540 1 620 620 84 540 556 556 n is a schematic block diagram of an embodiment of a molecule encoderthat includes a set of translators-and a formatting module. Each translator and the formatting modulemay be implemented utilizing a processing module (e.g., the processing moduleof). The molecule encoderfunctions to translate a plurality of sets of encoded data slices into molecule encoding informationin accordance with a translation approach, where the translation approach includes translation without regards to previously stored data. The molecule encoding informationincludes encoding patterns for a plurality of molecules.

1 1 1 1 2 2 2 n In an example of operation, each translator maps a bit pattern of a portion of a corresponding encoded data slice of a set of encoded data slices-to a set of genes. The mapping includes one or more of performing a lookup, initiating a query, receiving a query response, and applying a deterministic function. A number of portions of the encoded data slice may be based on a slice size, where a number of permutations of each portion is less than a number of permutations of an available number of genes per molecule. For example, the translatormaps the encoded data sliceto a gene identifier (ID) set, the translatormaps the encoded data sliceto a gene ID set, etc.

1 1 1 1 2 2 1 2 Having mapped the bit pattern of the portion of the corresponding encoded data slice to the set of genes, the translator obtains identification information (e.g., a slice name) for the encoded data slice. The obtaining includes one or more of receiving the identification information, extracting the identification information from the encoded data slice, and subsequently extracting the identification information from a reproduced data segment. For example, the translatorobtains identification (ID) informationfor the encoded data sliceas ID info, the translatorobtains ID informationfor the encoded data sliceas ID info, etc.

1 620 556 n, 46 FIG.A Having received the sets of gene identifiers and identification information from the set of translators-the formatting moduletranslates each gene identifier set and corresponding ID information into a portion of the molecule encoding informationin accordance with the translation approach of. For example, the formatting module maps a combination of genes and identifier information of each gene identifier set to produce a corresponding molecule encoding, where the mapping is in accordance with a mapping of gene combinations and identifier information to molecule encodings.

47 FIG.B 47 FIG.A 540 1 1 622 622 540 556 556 n n, is a schematic block diagram of another embodiment of a molecule encoderthat includes the set of translators-of, a set of correlators-and a formatting module. Each correlator and the formatting modulemay be implemented utilizing a processing module. The molecule encoderfunctions to translate a plurality of sets of encoded data slices into molecule encoding informationin accordance with a translation approach, where the translation approach includes translation with regards to previously stored data. The molecule encoding informationincludes encoding patterns for a plurality of molecules.

1 1 1 1 2 2 2 n In an example of operation, each translator maps a bit pattern of a portion of a corresponding encoded data slice of a set of encoded data slices-to a set of genes. For example, the translatormaps the encoded data sliceto a gene identifier (ID) set, the translatormaps the encoded data sliceto a gene ID set, etc.

1 1 1 1 2 2 1 2 Having mapped the bit pattern of the portion of the corresponding encoded data slice to the set of genes, the translator obtains identification information (e.g., a slice name) for the encoded data slice. For example, the translatorobtains identification (ID) informationfor the encoded data sliceas ID info, the translatorobtains ID informationfor the encoded data sliceas ID info, etc.

624 624 Having received a gene identifier set and corresponding identification information, each correlator identifies a similar gene ID set associated with the stored data. The identifying includes comparing the gene ID set to stored gene ID set information(e.g., a directory or list of previously stored gene ID sets) and identifying a closest matching gene ID set (e.g., highest percentage of matching to similar genes). The identifying may further include obtaining the stored gene ID set information. The obtaining includes at least one of retrieving from a directory, retrieving from a dispersed hierarchical index, and retrieving from a molecule storage facility.

1 1 1 Having identified the closest matching gene ID set, the correlator determines a difference gene ID set based on the gene ID set and the closest matching gene ID set. For example, the correlator identifies a gene as a distance gene between a gene of the gene ID set an identified gene of the closest matching gene ID set based on a numerical difference between gene identifiers of the gene and the identified gene of the closest matching gene ID set. For example, correlatorcorrelates the gene ID setto stored gene ID set information to produce a difference gene ID set.

1 1 1 1 Having determined the difference gene ID set, each correlator updates corresponding ID information to produce updated ID information. The updating includes generating a pointer to the closest matching gene set and appending the pointer to the ID information to produce the updated ID information. For example, the correlatoridentifies the closest matching gene identifier set, obtains storage location information for the identified closest matching gene identifier set, appends the location information to the identifier informationto produce updated identifier information, and sends the updated ID informationto the formatting module.

1 622 556 622 n, 46 FIG.A Having received the sets of difference gene sets and updated identification information from the set of correlator's-the formatting moduletranslates each difference gene identifier set and updated ID information into a portion of the molecule encoding informationin accordance with the translation approach of. For example, the formatting modulemaps a combination of difference genes and the updated identifier information of each difference gene identifier set to produce a corresponding molecule encoding, where the mapping is in accordance with a mapping of gene combinations and identifier information to molecule encodings.

47 FIG.C 626 628 is a flowchart illustrating an example of generating molecule encoding information. The method begins or continues at stepwhere a processing module (e.g., of a molecule encoder) obtains a set of encoded data slices, where a data segment is dispersed storage error encoded to produce the set of encoded data slices. For each encoded data slice, the method continues at stepwhere the processing module maps a bit pattern of a portion of the encoded data slice to a set of genes. For example, the processing module applies a deterministic function to the bit pattern to produce and identifier of a set of genes.

630 632 For each encoded data slice, the method continues at stepwhere the processing module obtains identification information. For example, the processing module extracts the identification information from one or more of the set of encoded data slices. The method continues at stepwhere the processing module obtains similar gene identification (ID) set information. The obtaining includes at least one of receiving and performing a lookup based on one or more of the identification information and identifiers of one or more of the sets of genes. For example, the processing module performs a deterministic function on the set of genes to produce a search factor and compares the search factor to factors associated with stored gene sets.

634 The method continues at stepwhere the processing module applies a correlation function to the set of genes and one or more portions of the similar gene ID set information to produce one or more correlation values. For example, the processing module identifies a gene ID set of the similar gene ID set information that is substantially the same as the set of genes. As another example, the processing module identifies another gene ID set of the similar gene ID set information that is less than 10% different than the set of genes.

636 The method continues at stepwhere the processing module selects a portion of the similar gene ID set information based on the one or more correlation values. The selecting includes identifying favorable correlation values (e.g., correlation is greater than a correlation threshold level), rank ordering the favorable correlation values, selecting a most favorable correlation value, and identifying the portion of the similar gene ID set information that corresponds to the selected most favorable correlation value.

638 The method continues at stepwhere the processing module determines a difference set of genes based on the similar gene ID set information and the set of genes. For example, the processing module calculates differences between gene identifiers of the similar gene ID information of the set of genes to produce the difference set of genes. Alternatively, or in addition to, the processing module updates the ID information to include addressing information of the selected portion of the similar gene ID set information. The method may further include translating each of the difference set of genes and updated ID information to produce molecule encoding information.

48 FIG.A 1 FIG. 1 FIG. 3 FIG. 1 FIG. 1 24 34 88 88 1 36 n m. , B, C are schematic block diagrams of another embodiment of a dispersed storage network (DSN) illustrating an example of maintaining critical information. The DSN includes a set of distributed storage and task (DST) execution units-and the networkof. Each DST execution unit includes the DST client moduleofand the memoryof. The memoryincludes a plurality of memory devices-Each DST execution unit may be implemented utilizing the DST execution unitof.

The DSN functions to maintain storage of critical information, where the critical information is required for operation of one or more of the DST execution units to perform further functions of the DSN (e.g., accessing data). The critical information includes one or more of BIOS software, bootstrap software, operating system software, important application software, a high priority encoded data slice, configuration information (e.g., number of memories, type of memory, input/output parameters, DSN address range assignment, storage error address range scanning assignment), system registry information, access list information, security information (an encryption key, an encryption algorithm indicator, a signed certificate), slice storage location information (slice name, a memory device ID, memory offset), and an index node file.

48 FIG.A 1 1 34 1 34 m illustrates initial steps of the example of the maintaining the critical information, where for each memory device-of the DST execution unit, the DST client moduleof the DST execution unitidentifies critical information stored in the memory device. The identifying includes at least one of searching by a critical information type, receiving a list, initiating a query, and receiving a query response. Having identified the critical information, the DST client moduleaggregates the identified critical information from each memory device to produce aggregated critical information.

34 650 650 34 Having produced the aggregated critical information, the DST client moduledispersed storage error encodes the aggregated critical information to produce one or more sets of encoded critical slices. Having produced the one or more sets of encoded critical slices, the DST client moduleselects a set of storage locations in accordance with a storage location selection scheme. The selecting includes obtaining the selection scheme based on retrieving from a list, utilizing a predetermination, interpreting a portion of a system registry, initiating a query, and receiving a query response.

1 34 1 1 2 m n The storage location selection scheme includes at least one of storing within a common computing device (e.g., DST execution unit), storing within computing devices at a common present site, and storing within computing devices of one or more sites. For example, the DST client moduleselects a third memory device of the set of memory devices-of the DST execution unitas a first storage location and selects remaining DST execution units-of the set of DST execution units as remaining storage locations of the set of storage locations.

34 650 34 652 1 652 650 2 34 2 2 n. Having selected the set of storage locations, the DST client modulefacilitates storage of the one or more sets of encoded critical slicesat the selected set of storage locations. For example, the DST client modulewrites a first encoded critical slicein the third memory device of the DST execution unitand sends remaining encoded critical slicesof the encoded critical slice setto the DST execution units-For instance, the DST client moduleissues a write slice request to DST execution unit, where the write slice request includes encoded critical slice.

48 FIG.B 34 1 34 2 88 34 illustrates further steps of the example of the maintaining the critical information, where the DST client moduleidentifies unavailable critical information associated with a memory device of the DST execution unit. The identifying includes at least one of detecting a failure of the memory device, detecting removal of the memory device, interpreting an error message, and detecting a storage error. For example, the DST client moduledetects a failure of memory deviceof the memory. Having identified the failed memory device, the DST client moduleidentifies the unavailable critical information that is associated with the identified failed memory device. The identifying may be based on one or more of a lookup, accessing a directory, interpreting a portion of system registry information, interpreting an error message, and receiving a request.

34 34 2 2 Having identified the unavailable critical information, the DST client moduledetermines whether to remedy the unavailable critical information. The determining may be based on one or more of detecting activation of a replacement memory, detecting operation of the previously identified failed memory, and receiving a request. For example, the DST client moduledetects replacement of the memory devicewith a replacement memory device.

34 34 34 652 34 652 88 2 652 n, When remedying the unavailable critical information, the DST client moduleidentifies the set of storage locations in accordance with the storage location selection scheme. For example, the DST client moduleaccesses BIOS information to retrieve the set of storage locations. Having identified the set of storage locations, the DST client modulefacilitates retrieval of at least a decode threshold number of encoded critical slicesfor each set of encoded critical slices corresponding to the critical information. For example, the DST client moduleretrieves one or more encoded critical slicesfrom the memory, issues read slice requests to the remaining DST execution units-and receives read slice responses that includes encoded critical slices.

48 FIG.C 34 34 34 2 2 illustrates still further steps of the example of the maintaining the critical information, where, for each of the at least a decode threshold number of encoded critical slices, the DST client moduledispersed storage error decodes the at least a decode threshold number of encoded critical slices to reproduce the aggregated critical information. Having reproduced the aggregated critical information, the DST client moduleextracts the critical information from the aggregated critical information. For example, the DST client moduleextracts the critical informationthat is associated with the memory deviceof the unavailable information.

34 34 2 34 34 2 2 Having extracted the critical information, the DST client moduleidentifies a storage location for the extracted critical information. The identifying includes at least one of identifying the memory device that is now available, identifying the replacement memory device, and identifying another suitable memory device. For example, the DST client moduleidentifies a memory deviceas the replacement memory device. Having identified storage location, the DST client modulefacilitates storage of the extracted critical information in the identified storage location. For example, the DST client modulewrites the critical informationto the memory device.

48 FIG.D 656 658 is a flowchart illustrating an example of maintaining critical information. The method begins or continues, when storing critical information, at stepwhere a processing module (e.g., of a distributed storage and task (DST) client module) identifies the critical information stored in one or more memory devices of a computing device. The identifying includes at least one of searching, comparing stored data to a list, initiating a query, and receiving a query response. The method continues at stepwhere the processing module obtains identified critical information from the one or more memory devices. The obtaining includes at least one of reading, issuing a read request, and receiving a read response.

660 The method continues at stepwhere the processing module dispersed storage error encodes the identifying critical information to produce one or more sets of encoded critical slices. For example, the processing module aggregates the identifying critical information to produce aggregated critical information and dispersed storage error encodes the aggregated critical information to produce the one or more sets of encoded critical slices.

662 664 The method continues at stepwhere the processing module selects a set of storage units in accordance with a storage location selection scheme. The selection may be based on one or more of storage location availability, a predetermination, a system registry information, and bios information. The method continues at stepwhere the processing module facilitates storage of the one or more sets of encoded critical slices in the selected set of storage locations. The facilitating includes writing an encoded critical slice to a memory device and issuing a write slice request to a storage unit, where the write slice request includes another encoded critical slice.

666 668 The method continues, when restoring the critical information, at stepwhere the processing module identifies unavailable critical information of a corresponding memory device. The identifying includes one or more of interpreting a test result, receiving an error message, detecting removal of a memory device, and detecting a failure of a memory device. The method continues at stepwhere the processing module determines whether to remedy the unavailable critical information. For example, the processing module indicates to remedy when detecting that a replacement memory device has been configured to replace the corresponding memory device. As another example, the processing module indicates to remedy when receiving a request to remedy. As yet another example, the processing module indicates to remedy when detecting that the corresponding memory device is now available.

670 672 When remedying the unavailable critical information, the method continues at stepwhere the processing module identifies the set of storage locations. For example, the processing module interprets a directory entry. As another example, the processing module interprets retrieval of a portion of BIOS information. The method continues at stepwhere the processing module facilitates retrieval of at least a decode threshold number of encoded critical slices per set of encoded critical slices corresponding to the unavailable critical information. For example, the processing module utilizes the identified set of storage locations to read from a local memory, issues read slice requests to storage units, and receives read slice responses from at least some of the storage units.

674 The method continues at stepwhere, for each set, the processing module dispersed storage error decodes each of at least a decode threshold number of encoded critical slices to produce rebuilt critical information. For example, the processing module decodes each decode threshold number of encoded critical slices to reproduce one or more data segments and aggregates the one or more data segments to produce the rebuilt critical information corresponding to the unavailable critical information.

676 The method continues at stepwhere the processing module identifies a storage location for the rebuilt critical information. For example, the processing module interprets a directory entry to identify the storage location. As another example, the processing module accesses the bios information to identify the storage location. As yet another example, the processing module identifies a replacement memory device as the storage location for the rebuilt critical information.

678 The method continues at stepwhere the processing module facilitates storage of the rebuilt critical information in the identified storage location. The facilitating includes, when storing in a local memory, writing the rebuilt critical information to the local memory. When utilizing a storage unit, the facilitating includes sending the critical information to the storage unit.

1 2 1 2 2 1 As may be used herein, the terms “substantially” and “approximately” provides an industry-accepted tolerance for its corresponding term and/or relativity between items. Such an industry-accepted tolerance ranges from less than one percent to fifty percent and corresponds to, but is not limited to, component values, integrated circuit process variations, temperature variations, rise and fall times, and/or thermal noise. Such relativity between items ranges from a difference of a few percent to magnitude differences. As may also be used herein, the term(s) “operably coupled to”, “coupled to”, and/or “coupling” includes direct coupling between items and/or indirect coupling between items via an intervening item (e.g., an item includes, but is not limited to, a component, an element, a circuit, and/or a module) where, for indirect coupling, the intervening item does not modify the information of a signal but may adjust its current level, voltage level, and/or power level. As may further be used herein, inferred coupling (i.e., where one element is coupled to another element by inference) includes direct and indirect coupling between two items in the same manner as “coupled to”. As may even further be used herein, the term “operable to” or “operably coupled to” indicates that an item includes one or more of power connections, input(s), output(s), etc., to perform, when activated, one or more its corresponding functions and may further include inferred coupling to one or more other items. As may still further be used herein, the term “associated with”, includes direct and/or indirect coupling of separate items and/or one item being embedded within another item. As may be used herein, the term “compares favorably”, indicates that a comparison between two or more items, signals, etc., provides a desired relationship. For example, when the desired relationship is that signalhas a greater magnitude than signal, a favorable comparison may be achieved when the magnitude of signalis greater than that of signalor when the magnitude of signalis less than that of signal.

As may also be used herein, the terms “processing module”, “processing circuit”, and/or “processing unit” may be a single processing device or a plurality of processing devices. Such a processing device may be a microprocessor, micro-controller, digital signal processor, microcomputer, central processing unit, field programmable gate array, programmable logic device, state machine, logic circuitry, analog circuitry, digital circuitry, and/or any device that manipulates signals (analog and/or digital) based on hard coding of the circuitry and/or operational instructions. The processing module, module, processing circuit, and/or processing unit may be, or further include, memory and/or an integrated memory element, which may be a single memory device, a plurality of memory devices, and/or embedded circuitry of another processing module, module, processing circuit, and/or processing unit. Such a memory device may be a read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, and/or any device that stores digital information. Note that if the processing module, module, processing circuit, and/or processing unit includes more than one processing device, the processing devices may be centrally located (e.g., directly coupled together via a wired and/or wireless bus structure) or may be distributedly located (e.g., cloud computing via indirect coupling via a local area network and/or a wide area network). Further note that if the processing module, module, processing circuit, and/or processing unit implements one or more of its functions via a state machine, analog circuitry, digital circuitry, and/or logic circuitry, the memory and/or memory element storing the corresponding operational instructions may be embedded within, or external to, the circuitry comprising the state machine, analog circuitry, digital circuitry, and/or logic circuitry. Still further note that, the memory element may store, and the processing module, module, processing circuit, and/or processing unit executes, hard coded and/or operational instructions corresponding to at least some of the steps and/or functions illustrated in one or more of the Figures. Such a memory device or memory element can be included in an article of manufacture.

The present invention has been described above with the aid of method steps illustrating the performance of specified functions and relationships thereof. The boundaries and sequence of these functional building blocks and method steps have been arbitrarily defined herein for convenience of description. Alternate boundaries and sequences can be defined so long as the specified functions and relationships are appropriately performed. Any such alternate boundaries or sequences are thus within the scope and spirit of the claimed invention. Further, the boundaries of these functional building blocks have been arbitrarily defined for convenience of description. Alternate boundaries could be defined as long as the certain significant functions are appropriately performed. Similarly, flow diagram blocks may also have been arbitrarily defined herein to illustrate certain significant functionality. To the extent used, the flow diagram block boundaries and sequence could have been defined otherwise and still perform the certain significant functionality. Such alternate definitions of both functional building blocks and flow diagram blocks and sequences are thus within the scope and spirit of the claimed invention. One of average skill in the art will also recognize that the functional building blocks, and other illustrative blocks, modules and components herein, can be implemented as illustrated or by discrete components, application specific integrated circuits, processors executing appropriate software and the like or any combination thereof.

The present invention may have also been described, at least in part, in terms of one or more embodiments. An embodiment of the present invention is used herein to illustrate the present invention, an aspect thereof, a feature thereof, a concept thereof, and/or an example thereof. A physical embodiment of an apparatus, an article of manufacture, a machine, and/or of a process that embodies the present invention may include one or more of the aspects, features, concepts, examples, etc., described with reference to one or more of the embodiments discussed herein. Further, from figure to figure, the embodiments may incorporate the same or similarly named functions, steps, modules, etc., that may use the same or different reference numbers and, as such, the functions, steps, modules, etc., may be the same or similar functions, steps, modules, etc., or different ones.

Unless specifically stated to the contra, signals to, from, and/or between elements in a figure of any of the figures presented herein may be analog or digital, continuous time or discrete time, and single-ended or differential. For instance, if a signal path is shown as a single-ended path, it also represents a differential signal path. Similarly, if a signal path is shown as a differential path, it also represents a single-ended signal path. While one or more particular architectures are described herein, other architectures can likewise be implemented that use one or more data buses not expressly shown, direct connectivity between elements, and/or indirect coupling between other elements as recognized by one of average skill in the art.

The term “module” is used in the description of the various embodiments of the present invention. A module includes a processing module, a functional block, hardware, and/or software stored on memory for performing one or more functions as may be described herein. Note that, if the module is implemented via hardware, the hardware may operate independently and/or in conjunction software and/or firmware. As used herein, a module may contain one or more sub-modules, each of which may be one or more modules.

While particular combinations of various functions and features of the present invention have been expressly described herein, other combinations of these features and functions are likewise possible. The present invention is not limited by the particular examples disclosed herein and expressly incorporates these other combinations.