Firmware-Based Out-Of-Order Command Scheduler

This invention was made with government support under prime contract number 14 D 0423/19 F 0004 awarded by the United States Government. The government has certain rights in the invention.

The present disclosure relates to computing systems, and more particularly, to firmware-based out-of-order command schedulers.

A computing system includes processing resources that are configured to perform one or more tasks, such as processes, threads, or other sequences of instructions. The tasks can be executed according to the capabilities of the processing resources and/or the relative execution priorities assigned to the tasks. For example, several tasks can be executed by a single central processing unit (CPU, or processor) in series, with one task beginning after the completion of another. In a multiprocessor system, certain tasks can be executed by multiple CPUs concurrently. In many systems, a scheduling process, also referred to as a scheduler, manages utilization of the processing resources. There remain a number of non-trivial issues with respect to schedulers.

Although the following detailed description will proceed with reference being made to illustrative examples, many alternatives, modifications, and variations thereof will be apparent in light of this disclosure.

Techniques for scheduling commands in a computing device are disclosed. In an example, a scheduling method includes receiving command data, where the command data includes a command to be executed on a processor of the computing device, and an execution start time for executing the command. The method further includes storing the command data in a data storage of the computing device; reading, responsive to determining that the execution start time is within an execution window, the command data from the data storage; and causing the command data to be output to a processor for execution. In some examples, the method includes assigning a label to the command data, where the label corresponds to an address in the data storage, and where the command data is stored at, and read from, the address in the data storage. In some examples, the method includes storing, responsive to determining that the execution start time is not within the execution window, the label and the execution start time in a fast scheduler first-in-first-out buffer (FIFO) and a fast scheduler memory, and further reading into a fast scheduler memory, responsive to determining that the execution start time is within the execution window, the label and the execution start time from the fast scheduler. In this manner, commands that arrive before the execution start time are temporarily stored in the long-term storage until the execution start time is within the execution window where they are then transferred to the fast scheduler. As will be described in further detail below, the commands stored in the fast scheduler (or in a buffer, such as a long-term storage) can be executed out of the order in which they were received according to the corresponding execution start times. The use of labels facilitates efficient use of memory for temporarily storing the commands within the scheduler while they are awaiting execution by the processor.

As noted above, there remain a number of non-trivial issues with respect to command schedulers. For instance, a conventional scheduler processes tasks in the order received, which may not be the most efficient use of the computing resources or may otherwise be inefficient for a given application. For example, if a first-received command is scheduled to execute in five seconds and a second-received command is scheduled to execute in one second, and if the commands are executed in the order received, then the second-received command will not execute until the first-received command completes execution, thus incurring at least a five-second execution delay. Executing tasks in the order received can limit the ability of the computing device to efficiently execute multiple concurrent tasks, tasks with varying resource requirements, and tasks from multiple users, and can also cause delays in the execution of time critical commands. Some software-based schedulers, such as those integrated into the operating system of a computing system, may have significant overhead that reduces throughput and increases latency, which can be less useful for real-time or high-traffic systems. Therefore, a need remains for a command scheduler that addresses these issues.

is a block diagram of a computing deviceconfigured with a firmware-based out-of-order scheduler, in accordance with an example of the present disclosure. The devicecan be implemented, for example, as a field programmable gate array (FPGA)-based or an application specific integrated circuit (ASIC)-based computing device, or in another integrated circuit-based device or system. For example, the deviceincludes a processor, a data storage, and a clock. The devicereceives one or more commandsfrom a user and/or other system. Each of the commandscan be associated with an execution start time, which designates the real or system time at which the command is to be executed by the processor, which generates an output. The system time is maintained by the clock. The firmware-based out-of-order schedulerincludes a label generator, a data receiver, a long-term storage, and a fast scheduler. The label generator, the data receiver, the long-term storage, and the fast schedulercan execute continuously and concurrently with each other, which allows pipelined processing of the commandsthrough the scheduler.

For example, the schedulerprovides the ability for multiple users to submit the commandsat the same time, or individual users to submit multiple commands, while helping to ensure that all of the commandsare executed according to the associated execution start times. Specifically, the schedulerreceives the commandsand the corresponding execution start times and arranges the commandsto be executed by the processorat, or about, the designated execution start time rather than merely in the order that the commandswere received. For example, if the schedulerreceives a first command having a first execution start time for executing the first command prior to receiving a second command having a second execution start time for executing the second command, and if the second execution start time is before the first execution start time, the schedulercan cause the processorto execute the second command before executing the first command, despite having received the second command after the first command. Thus, depending on the execution start times, the schedulercan cause the processorto execute the commandsin a different order than they were received, such that the commands are more likely to be executed at or near the execution start times and not otherwise delayed by other commands. Having such an out-of-order scheduler based in firmware, according to some examples, increases the throughput of the commandsand reduces execution latency when resolving commands.

The scheduler, or portions thereof including the label generator, the data receiver, the long-term storage, the fast scheduler, or any combination of these, can be implemented in firmware, which is also referred to herein as a firmware-based scheduler. Such firmware can include machine-executable instructions that allow the hardware components of the system to function and communicate with any other component of the system, including other hardware, firmware, software, or memory/storage components. For example, the scheduler, or portions thereof, can be implemented as a set of machine-executable instructions that are stored in a non-volatile memory. The non-volatile memory can include, for example, read-only memory (ROM), electrically-programmable ROM (EPROM), electrically-erasable and programmable ROM (EEPROM), flash memory, or another type of device or computer memory that can retain stored information without power. The machine-executable instructions can be coded, for example, in a hardware description language (HDL) or other low-level language, such as machine code. The machine-executable instructions can be accessed and executed by hardware components of the computing device, such as the processor(e.g., a central processing unit (CPU) or other co-processor), below and independently of any software-based components, such as a software-based operating system. For example, the processorcan execute the machine-executable instructions of the firmware-based schedulerindependently of any other machine-executable instructions, including those of the operating system. Such a firmware-based scheduler can thus operate more efficiently and with less latency than a software-based scheduler implemented within the operating system. The machine-executable instructions of the schedulercan be permanently incorporated into the computing deviceat manufacture, or can be updatable or otherwise overwritten in the non-volatile memory after manufacture, but otherwise remain static. The processoris configured to execute the machine-executable instructions of the schedulerstored in the non-volatile memory.

In more detail, the data receiverreceives all incoming commands, which are associated with a corresponding execution start time. The execution start time for each of the commandsspecifies a target system (or real) time at which the respective command is to be executed by the processor. Upon startup or reset of the device, the label generatorgenerates a set of labels, which are used to mark and store incoming commandsin the data storageand/or in the long-term storageof the scheduler. The labelscan be, for example, a number of any bit size. In some examples, the labelsrepresent vectors (pointers) into addresses within the data storagewhere the commandsare temporarily stored prior to execution. For example, a label “0001” can represent memory address “0001” in the data storage.

As the schedulerreceives the commands, the schedulerassigns a unique label from the set of labelsto each of the commandsand schedules the commandsfor execution using the labels. A benefit of using labels as representatives of the command data is that the size of the label data may be smaller than the size of the command data, and thus the labelsmay be more efficient for the schedulerto store and process. However, in some examples, the command data can be used instead of a label when the size of the command data is relatively small (e.g., approximately the same size as the label data). The schedulercan reuse or recycle each of the labelsfor other commandsafter the command associated with the label has been output for execution by the processor.

When the labels are used, the schedulerassigns an unused label from the set of labelsto the incoming commandsand stores the assigned label in the data storage. Otherwise, when the labels are not used, the schedulerstores the command data in the data storage. In any event, the schedulerdetermines whether the execution start time of each of the incoming commandsis within an execution window of the fast scheduler. The fast schedulerschedules the commandsfor execution if the execution start time is within the execution window. The execution window begins when the execution start time is within (less than or equal to) a configurable threshold time of the system time, as determined by the clock. For example, if the threshold time is one second and if the execution start time is within one second of the system time, then the execution window begins. If the execution start time is within the execution window, the fast schedulerreads the command data associated with the label from the data storageand sends the command data to the processorfor immediate execution. If the execution start time is not within the execution window, the schedulerstores the label and the execution start time corresponding to the command data in the long-term storagefor deferred execution closer to the execution start time.

For the deferred command data, the schedulercontinuously scans through the execution start times for all of the labels(or command data) stored in the long-term storage. When the execution start time of one of the labelsenters the execution window of the fast scheduler, the schedulertransfers the label to the fast schedulerfor execution. The fast schedulerreads each label and corresponding execution start time of the command data from the long-term storage. The fast schedulerthen arranges the labelsin the order that the labels will be output to the processoraccording to the execution start times, where each label represents an address in the data storage. Each address represents a different execution start time within the execution window of the fast scheduler. If the address is already being used for another command, then the fast schedulerassigns the label to the next available address in the long-term storage. When the execution start time of the command data is within the execution window of the fast scheduler, the scheduleroutputs the command data associated with the label to the processorfor execution.

The long-term storagepermits the commandsto be scheduled at various future times outside of the execution window, which shifts with the system time. The amount of the long-term storagecan be sized based on how far into the future the commandsare to be scheduled and the number of commandsthat are expected to be stored in that time frame. The fast schedulerfacilitates execution of the commandsat the corresponding execution start time. By using different addresses in memory to represent different execution start times, the labels, and thus the corresponding command data, are automatically reordered when arranged in the long-term storageby the fast scheduler. As noted above, the use of short, unique labels as a substitute for large amounts of command data increases the efficiency of the scheduler. However, the command data can be stored in the long-term storageinstead of a label when the size of the command data is relatively small.

is a flow diagram of a methodof scheduling commands, in accordance with an example of the present disclosure. The methodcan be implemented on, for example, the deviceof. Upon start/reset, the methodincludes generatinga set of labels and storing the labels in a first-in-first-out (FIFO) buffer for later use. Each of the labels represents an address in the data storage.

The methodfurther includes receivingdata input representing command data and an execution start time, assigning a label from the FIFO to the command data, and storing the command data, including the command and the execution start time, in the data storage. The label assigned to the command data is, for example, the next available and unused label in the set of labels stored in the FIFO. The command data is stored into the data storageat the address corresponding to the label assigned to the command data. For example, if the label assigned to the data input is “0001” then the command data is stored at address ‘0001’ in the data storage.

The methodfurther includes storingthe label assigned to the command data and the execution start time in the long-term storage. If the execution start time is within the execution window of the fast scheduler at, then the methodincludes storingthe label and the execution start time to a fast scheduler buffer FIFO, and then in a fast scheduler memory (e.g., a block RAM (BRAM) or a dynamic RAM (DRAM)). Otherwise, the execution start time is periodically compared to the execution window of the fast scheduleruntil the execution start time is within the execution window.

Note that when the command data includes several commands with different execution start times, the respective commands may be executed in a different order than received, depending on the execution start times. For example, if receivingthe command data includes receiving first and second command data, where the first command data includes a first command and a first execution start time for executing the first command, and the second command data includes a second command and a second execution start time for executing the second command, and where the first command data being received prior to receiving the second command data, then different labels are assigned to the first and second command data and the labels are storedin the long-term storage.

The methodfurther includes readingthe label from the fast scheduler memory, and reading the command data from the data storagebased on the label. For example, as noted above, the label represents an address in the data storagewhere the corresponding command data was previously stored in step. Thus, the command data can be read from the data storageat the address specified by the label read out of the fast scheduler during the execution window. The label is recycled for use with future command data received at step. The methodfurther includes outputtingthe command data to the processorfor execution. Note that the order of execution can be different from the order in which the commands were received by the scheduler, based on the execution start times of each command rather than on the order in which the commands were received.

is a flow diagram of a portionof the methodof, in accordance with an example of the present disclosure. Long-term storage begins once a label and execution start time are received at step. As noted above, the methodincludes checkingif the execution start time is within the execution window of the fast scheduler. If it is, then the methodincludes checkingto determine whether the fast scheduler memory and buffer FIFO is full. If the fast scheduler memory and buffer FIFO is not full, then the label and the execution start time are storedin the fast scheduler buffer FIFO. Otherwise, the execution start time is canceledbecause the command will not be able to execute before the execution start time, and the label is recycled.

If the newly received execution start time is not within the execution window of the fast scheduler, then the methodincludes storingthe label and the execution start time in a long-term storage FIFO along with the corresponding label. The methodfurther includes readinga label out of the long-term storage FIFO during each processor clock cycle that a new execution start time is not received. If the execution start time associated with the label is currently within the execution window at step, then the label and execution start time are sent to the fast scheduler FIFO. Otherwise, the label is stored again in the long-term FIFO at step, and further processed as discussed above until the execution start time is within the execution window.

is a flow diagram of a methodof processing the fast scheduler memory, in accordance with an example of the present disclosure. The methodincludes readingthe label from the long-term storage. The methodfurther includes looking upthe address specified by the label and corresponding execution start time in the fast scheduler memory. If the address in the fast scheduler memory associated with the label is empty at step, then the methodincludes writingthe label received from the long-term storageinto the address of the fast scheduler memory. If the address in the fast scheduler memory is not empty at step, then the methodincludes incrementingthe address by one along with a pause time, which represents an amount of time the label has to wait to be written or otherwise stored into the fast scheduler buffer FIFO. This continues until an unused address is found in the fast scheduler memory or until the pause time reaches a maximum threshold at step. If the pause time exceeds the maximum threshold, then the execution start time (and the associated command) is canceled and the corresponding label is recycled. In such cases, the methodincludes readingthe next execution start time and label from the long-term storage.

Meanwhile, the system time is used to read through the addresses in the fast scheduler memory, and when an address with a label stored there at stepis found, the methodincludes readingthe label from the fast scheduler memory and reading the command data from the data storagebased on the label, such as at stepof. The methodfurther includes clearingthe label from the fast scheduler memory after the data has been read out at step.

is a flow diagram of a methodof processing data in the data storage, in accordance with an example of the present disclosure. The methodincludes two parallel tasksand. Taskincludes receivingnew data and the associated label, and storingthe data at the address in the data storage specified by the label. Taskincludes receivinga label from the fast memory and readingthe data from the address in the data storage specified by the label, and then recycling the label for future use.

is a flow diagram of a methodof out-of-order scheduling when labels, such as described above, are not used, in accordance with an example of the present disclosure. The methodcan be used, for example, when the data being scheduled is small enough that labels are not needed to increase efficiency of the scheduler. Instead of generating labels for incoming data, the methodincludes receiving and storingcommand data and associated execution start times where labels would be in the long-term storage and the fast scheduler. The methodfurther includes storingthe command data and the execution start time in the long-term storage. If the execution start time is within the execution windowof the fast scheduler, then the methodincludes sending the command data and the execution start time to a fast scheduler memory. Otherwise, the execution start time is periodically compared to the execution window of the fast scheduler until the execution start time is within the execution window. The methodfurther includes readingthe command data from the fast scheduler memory. The methodfurther includes outputtingthe command data to the processor for execution.

The following examples pertain to further examples, from which numerous permutations and configurations will be apparent.

Example 1 provides a method of scheduling commands for execution in a computing device. The method comprises receiving, by at least one of a field gate programmable array (FPGA) and an application-specific integrated circuit (ASIC), command data including a command and an execution start time for executing the command; storing, by the at least one of the FPGA and the ASIC, the command data in a data storage; reading, by the at least one of the FPGA and the ASIC and responsive to determining that the execution start time is within an execution window, the command data from the data storage; and causing, by the at least one of the FPGA and the ASIC, the command data to be output to a processor for execution.

Example 2 includes the subject matter of Example 1, further comprising assigning a label to the command data, wherein the label corresponds to an address in the data storage, and wherein the command data is stored at, and read from, the address in the data storage.

Example 3 includes the subject matter of Example 2, further comprising storing, responsive to determining that the execution start time is not within the execution window, the label and the execution start time in a fast scheduler buffer first-in-first-out (FIFO) and a fast scheduler memory; and reading, responsive to determining that the execution start time is within the execution window, the label and the execution start time from the fast scheduler memory.

Example 4 includes the subject matter of Example 3, further comprising incrementing, responsive to determining that that the address associated with the label is not empty, the address associated with the label by one while a pause time is less than or equal to a maximum threshold, wherein the pause time represents an amount of time the label had to wait to be stored into the data storage.

Example 5 includes the subject matter of any one of Examples 3 and 4, further comprising canceling the execution start time responsive to determining that the fast scheduler buffer FIFO and the fast scheduler memory is full.

Example 6 includes the subject matter of any one of Examples 3-5, further comprising clearing data from the address associated with the label responsive to reading the label and the execution start time from the fast scheduler memory.

Example 7 includes the subject matter of any one of Examples 1-6, wherein the execution window is a period of time beginning when the execution start time is less than or equal to a configurable threshold time of a system time.

Example 8 includes the subject matter of any one of Examples 1-7, wherein: receiving the command data includes receiving first and second command data, the first command data including a first command and a first execution start time for executing the first command, and the second command data including a second command and a second execution start time for executing the second command, the first command data being received prior to receiving the second command data; storing the command data includes storing the first and second command data in the data storage; reading the command data from the data storage includes reading, responsive to determining that the second execution start time is within a first execution window, the second command data from the data storage, causing the second command data to be output to the processor for execution, reading, responsive to determining that the first execution start time is within a second execution window, the first command data from the data storage, and causing the first command data to be output to the processor for execution, such that the second command data is executed by the processor prior to the first command data being executed by the processor.

Example 9 provides a computing device comprising at least one processor; a data storage operatively coupled to the at least one processor; and a non-volatile memory storing machine-executable instructions of a firmware-based scheduler configured to receive command data, the command data including a command and an execution start time for executing the command, store the command data in a data storage, read, responsive to determining that the execution start time is within an execution window, the command data from the data storage, and cause the command data to be output to the at least one processor for execution.

Example 10 includes the subject matter of Example 9, wherein the firmware-based scheduler is further configured to assign a label to the command data, wherein the label corresponds to an address in the data storage, and wherein the command data is stored at, and read from, the address in the data storage.

Example 11 includes the subject matter of Example 10, wherein the firmware-based scheduler is further configured to store, responsive to determining that the execution start time is not within the execution window, the label and the execution start time in a fast scheduler buffer first-in-first-out (FIFO) and a fast scheduler memory; and read, responsive to determining that the execution start time is within the execution window, the label and the execution start time from the fast scheduler memory.

Example 12 includes the subject matter of Example 11, wherein the firmware-based scheduler is further configured to increment, responsive to determining that that the address associated with the label is not empty, the address associated with the label by one while a pause time is less than or equal to a maximum threshold, wherein the pause time represents an amount of time the label had to wait to be stored into the data storage.

Example 13 includes the subject matter of any one of Examples 11 and 12, wherein the firmware-based scheduler is further configured to cancel the execution start time responsive to determining that the fast scheduler buffer FIFO and a fast scheduler memory is full.

Example 14 includes the subject matter of any one of Examples 11-13, wherein the firmware-based scheduler is further configured to clear data from the address associated with the label responsive to reading the label and the execution start time from the fast scheduler memory.

Example 15 includes the subject matter of any one of Examples 9-14, wherein the execution window is a period of time beginning when the execution start time is less than or equal to a configurable threshold time of a system time.

Example 16 includes the subject matter of any one of Examples 9-15, further comprising at least one of a field gate programmable array (FPGA) and an application-specific integrated circuit (ASIC) each configured to execute at least a portion of the machine-executable instructions of the firmware-based scheduler.

Example 17 provides a firmware-based scheduler for scheduling commands in a computing system, the firmware-based scheduler configured to receive command data, the command data including a command and an execution start time for executing the command; store the command data in a data storage; read, responsive to determining that the execution start time is within an execution window, the command data from the data storage, the execution window being a period of time beginning when the execution start time is less than or equal to a configurable threshold time of a system time; and cause the command data to be output to at least one processor for execution.

Example 18 includes the subject matter of Example 17, further configured to assign a label to the command data, wherein the label corresponds to an address in the data storage, and wherein the command data is stored at, and read from, the address in the data storage.

Example 19 includes the subject matter of Example 18, further configured to store, responsive to determining that the execution start time is not within the execution window, the label and the execution start time in a fast scheduler buffer first-in-first-out (FIFO) and a fast scheduler memory; and read, responsive to determining that the execution start time is within the execution window, the label and the execution start time from the fast scheduler memory.

Example 20 includes the subject matter of any one of Examples 18 and 19, further configured to increment, responsive to determining that that the address associated with the label is not empty, the address associated with the label by one while a pause time is less than or equal to a maximum threshold, wherein the pause time represents an amount of time the label had to wait to be stored into the data storage.

Numerous specific details have been set forth herein to provide a thorough understanding of the examples. It will be understood, however, that other examples may be practiced without these specific details, or otherwise with a different set of details. It will be further appreciated that the specific structural and functional details disclosed herein are representative of examples and are not necessarily intended to limit the scope of the present disclosure. In addition, although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described herein. Rather, the specific features and acts described herein are disclosed as example forms of implementing the claims. Furthermore, examples described herein may include other elements and components not specifically described, such as electrical connections, signal transmitters and receivers, processors, or other suitable components for operation of the modular antenna.

The terms and expressions which have been employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described (or portions thereof), and it is recognized that various modifications are possible within the scope of the claims. Accordingly, the claims are intended to cover all such equivalents. Various features, aspects, and examples have been described herein. The features, aspects, and examples are susceptible to combination with one another as well as to variation and modification, as will be appreciated in light of this disclosure. The present disclosure should, therefore, be considered to encompass such combinations, variations, and modifications. It is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims appended hereto. Future filed applications claiming priority to this application may claim the disclosed subject matter in a different manner and may generally include any set of one or more elements as variously disclosed or otherwise demonstrated herein.