Systems and methods for high bandwidth memory with unidirectional data flow

This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63/530,106 filed Aug. 1, 2023, the disclosure of which is hereby incorporated herein by reference.

High-speed computing can be performed using computing packages in which computing dies control read and write operations to be performed by high bandwidth memory. However, the bandwidth of data transmissions for high-bandwidth memory read and write operations are currently impeded by the time required for the high-bandwidth memory to transition between read transmissions and write transmissions.

The present application relates to systems and methods for providing high bandwidth connections between memory and computing units. In accordance with aspects of the disclosure, memory units can be configured to perform concurrent read and write operations in parallel to one another. The memory units can also be configured to alter the relative bandwidths that are available for the read and write operations. For example, bi-directional transmission interfaces of a memory unit can be assigned to operate as uni-directional interfaces that are a part of either a data input path or a data output path.

In accordance with aspects of the disclosure, a system for computing and memory communication may include a computing unit, and a high-bandwidth memory (HBM), having a plurality of data transmission interfaces. The HBM may be configured to assign a first subset of the data transmission interfaces to operate as a data input path for receiving data corresponding to write operations, and to assign a second subset of the data transmission interfaces to operate as a data output path for transmitting data corresponding to read operations. The HBM may be further configured to communicate with the computing unit so as to perform a read operation and a write operation in parallel.

In accordance with other aspects of the disclosure, the HBM may be further configured so that the data output path and the data input path each operate as unidirectional transmission paths.

In accordance with still other aspects of the disclosure, the plurality of data transmission interfaces may be bi-directional data interfaces, and wherein the HBM is configured to assign the first subset of data transmission interfaces and the second subset of the data transmission interfaces based on one or more inputs received from the computing unit.

In accordance with yet other aspects of the disclosure, the HBM may be further configured to assign the plurality of data transmission interfaces to the first subset and the second subset based on a determined relative ratio between a number of data transmission interfaces that are in the first subset and the second subset. The HBM may be further configured to re-assign one or more data transmission interfaces of the plurality of data transmission interfaces with respect to being in either the first subset or second subset. Re-assigning the one or more data transmission interfaces may result in altering a relative ratio of data transmission interfaces that are in the first subset and the second subset.

In accordance with other aspects of the disclosure, the HBM may be further configured to have a write-address input for receiving transmissions identifying one or more addresses for write operations and a read-address input for receiving transmissions identifying one or more addresses for read operations. The HBM may be further configured to identify a coherency conflict in connection with one or more write operations and one or more read operations. The HBM may be further configured to perform the one or more write operations and read operations in a determined order based on identifying the coherency conflict. The HBM may be further configured to transmit coherency-related data to the computing unit.

In accordance with still other aspects of the disclosure, a method for computing and memory communication may include: assigning, by a high-bandwidth memory (HBM), a first subset of data transmission interfaces to operate as a data input path for receiving data corresponding to write operations and a second subset of data transmission interfaces to operate as a data output path for transmitting data corresponding to read operations: receiving, by the HBM, a first transmission identifying a read operation to be performed and a second transmission identifying a write operation to be performed; and performing, by the HBM, the write operation using the first subset of data transmission interfaces and the read operation using the second subset of data transmission interfaces. In addition, the write operation and the read operation may be performed by the HBM in parallel.

In accordance with aspects of the disclosure the plurality of data transmission interfaces may be bi-directional data interfaces, and wherein the HBM is configured to assign the first subset of data transmission interfaces and the second subset of the data transmission interfaces based on one or more inputs received from the computing unit. In addition, the first subset and second subset may be assigned in accordance with a determined relative ratio.

In accordance with other aspects of the disclosure, the method may include re-assigning one or more data transmission interfaces with respect to being in the first subset or second subset. Re-assigning the one or more data transmission interfaces may result in altering a relative ratio of data transmission interfaces that are in the first subset and the second subset.

In accordance with still other aspects of the disclosure, the HBM may receive the first transmission via a read-address input and the second transmission via a write-address input. The method may further include identifying a coherency conflict in connection with one or more write operations and one or more read operations. In addition, the method may further include performing, by the HBM, the one or more write operations and read operations in a determined order based on the coherency conflict. The method may also include transmitting, by the HBM, coherency-related data to the computing unit.

The technology relates to high bandwidth processing using memory units that are configured to allow for independent read and write operations. In accordance with aspects of the disclosure, memory units can be configured to perform concurrent read and write operations in parallel to one another. The disclosed system may also perform coherency checks to determine if potential conflicts exist between the read and write operations. The memory units can also be configured to alter the relative bandwidths that are available for the read and write operations.

1 FIG. 100 101 101 110 130 130 131 110 111 110 111 101 130 110 110 110 130 154 158 114 118 110 134 138 130 is a block diagramof a systemin accordance with aspects of the disclosure. Systemincludes a high-bandwidth memory (HBM)and a computing unit. The computing unitis configured to perform high bandwidth processing using one or more processors. HBMcontains memoryfor storing data, and HBMmay perform read operations and write operations in connection with storing and accessing data within memory. In connection with the processing that is performed by system, computing unitmay communicate with HBMand command HBMto perform particular read and write operations. Communication between HBMand computing unitmay be performed via a plurality of transmission paths. These transmission paths-may be connected to transmission interfaces-of HBMand to transmission interfaces-of computing unit.

130 134 135 134 154 110 114 135 155 110 115 110 114 115 112 112 112 112 The commands to perform read and write operations include identification of one or more memory addresses for which the operation is to be performed. For example, computing unitcontains a read-address output, which is configured to transmit signals that identify memory addresses for commands to perform read operations. In addition, write-address outputis configured to transmit signals that identify memory addresses for commands to perform write operations. Read-address outputtransmits signals that travel along read-address pathand are received by HBMat read-address input. Write-address outputtransmits signals that are transmitted along write-address pathand are received by HBMat write-address input. Upon HBMreceiving signals at read-address inputand/or write-address input, those signals can be sent to address-processing controller. Address-processing controllercan take the form of a programmable processor, hardware, or some combination of firmware and software. Address-processing controllercan be configured to identify one or more memory addresses that are contained within the received signals. Through the address-processing controller, the HBM can be said to be configured to perform operations, such as reading, writing, or assigning pins in a data path for input/output, as discussed herein.

114 112 110 111 110 117 115 112 110 116 116 117 116 117 If the signal is received by read-address input, then address-processing controlleris configured to have HBMperform a read operation, so that data from the identified memory addresses are read from memory. The data read from the identified address can then be transmitted by HBMvia one or more data transmission interfaces. If the signal is received by write-address input, then address-processing controlleris configured to have HBMto perform a write operation, so that data received from one or more data transmission interfacesare written to the memory address that have been identified from the received signal. Data transmission interfaces,may include, for example, a plurality of pins and one or more data buses, wherein the pins can be assigned to operate as either an input or an output in accordance with a data bus. In addition, data transmission interfaces,may be configured to transmit data in accordance with universal chiplet interconnect express (UCIe) as well as through using optical data paths.

116 117 110 116 117 110 101 116 156 117 157 116 130 110 117 130 110 In accordance with aspects of the disclosure, the plurality of data transmission interfaces,of HBMcan be assigned to operate as part of a data input path or as part of a data output path. For example, data transmission interfaces,may take the form of a plurality of pins that are capable of bi-directional transmissions, however HBMmay assign the pins to operate either as a data output path or a data input path. For system, a first subset of data transmission interfaceshave been assigned to operate as interfaces for a data input path, while a second subset of data transmission interfaceshave been assigned to operate as interfaces for a data output path. Thus, data transmission interfacesare available to receive data from computing unitin connection with a write operation that is being performed by HBM. In addition, data transmission interfacesare available to transmit data to computing unitin connection with a read operation that is being performed by HBM.

116 117 110 110 154 110 117 110 110 115 110 110 130 116 110 117 By assigning data transmission interfacesandto perform different operations independently of one another, HBMcan be configured to perform read operations and write operations in parallel. For example, HBMmay receive a signal via read-address input, and perform a read operation in accordance with the received signal. As part of the read operation, HBMmay transmit data that has been read from identified memory addresses via a data output path that corresponds to data transmission interfaces. In addition, while HBMis performing the read operation, HBMmay also receive a signal at write-address input, which identifies an address to which a write operation is to be performed. HBMmay perform this write operation prior to the termination of the read operation, in that HBMmay receive data from computing unitat data transmission interfaceswhile HBMis still transmitting data in accordance with the read operation from data transmission interfaces.

101 110 110 101 130 Systemmay also be configured to maintain data coherency with respect to the read operations and write operations that are to be performed by HBM. Data coherency can be compromised if the read and write operations that are being performed by HBMare in conflict with one another. For example, a requested write operation may conflict with a requested read operation if at least a portion of the data to be accessed in connection with the requested read operation will be overwritten by the requested write operation. Systemmay be configured to prevent conflicts between the read and write operations and prevent particular read and write operations from accessing the same memory region. Computing unitmay be configured to perform a coherency check for read and write requests.

130 139 130 110 139 130 134 135 139 130 130 101 For example, computing unitmay have a coherency controllerthat is configured to identify conflicts between read and write operations that are to be performed. If computing unitis preparing to transmit requests for HBMto perform read and write operations, coherency controllermay determine whether the read and write operations access overlapping memory addresses or are otherwise in conflict with one another. If no conflict is identified, computing unitmay transmit requests for the read and write operations via read-address outputand write-address outputin an unspecified order, including by transmitting the read and write operations in parallel to one another. However, if a conflict is identified by coherency controller, computing unitmay determine an appropriate order for the read and write operations to occur. For example, if a write operation would overwrite data that is to be read by a pending read operation, computing unitmay delay sending a request for a write operation until after a particular set of read operations have already been performed. Similarly, a request for a read operation may be delayed until after a write operation has been performed. Accordingly, systemmay be configured to allow for independent and concurrent read and write operations to be performed, while also specifying an order of read and write operations when a potential conflict is found to exist.

110 112 110 139 112 130 HBMmay also be configured to perform coherency checks so as to avoid potential conflicts. For example, address-processing controllermay be configured to identify if received read and write operations are in conflict with one another and determine if the read and write operations are to be performed in a particular order, so as to avoid the conflict. The coherency check by HBMmay be performed in addition to the one performed by coherency controller, or address-processing controllermay perform the coherency check in lieu of one being performed by computing unit.

110 116 117 101 110 116 117 100 110 116 156 117 157 110 100 110 101 116 156 117 157 100 116 156 117 157 116 117 116 117 1 FIG. 1 FIG.B In addition, HBMmay be configured to re-assign data transmission interfaces,, so as to switch the interfaces between being a part of the data input path or data output path. This re-assigning can allow systemto control the number of interfaces that are a part of the data input path and data output path. For example, HBMmay have any number of total data transmission interfaces,. In block diagramof, HBMhas been configured to so that a quarter of these data interfaces have been assigned to be data transmission interfaces, which operate as a part of the data input path. The remaining three-quarters of the data interfaces have been assigned to be data transmission interfaces, which operate as a part of the data output path. Thus, HBMhas been configured so as to have a greater bandwidth for transmitting data in connection with read operations than the bandwidth that is available for performing write operations. However,is a block diagram′ in which HBMof systemhas been re-configured so that it now has a greater number of data transmission interfacesthat are operating as part of data input pathcompared to the number of data transmission interfacesthat are operating as a part of the data output path. For example, in block diagram′, two-thirds of the total data interfaces have been assigned to be data transmission interfaces, which operate as part of data input path, while the remaining third of data interfaces have been assigned to be data transmission interfaces, which operate as part of data output path. The ratio between the data transmission interfacesand the data transmission interfacesmay take any relative value that is available for a given total of data transmission interfaces,.

119 110 116 117 156 157 119 116 117 116 117 131 110 156 157 138 130 116 117 110 118 119 119 116 117 116 117 110 Transmission controllerof HBMmay be configured to assign and re-assign data transmission interfaces,as either a part of data input pathor data output path. For example, transmission controllermay determine whether each data transmission interface,is to operate as uni-directional input or a uni-directional output, and may then assign each data transmission interface,with either an input or output designation. Additionally, computing unitmay be configured to send signals to HBMthat identify the data transmission interfaces that are to be a part of data input pathand data output path. For example, interfaceof computing unitmay be configured to transmit one or more signals that contain assignment data identifying the total number of data transmission interfaces,that can operate as either a data input or a data output. HBMmay receive the one or more signals at interface, and may transmit the assignment data to transmission controller. Transmission controllermay then assign transmission interfaces,to operate as part of either the data input path or the data output path in accordance with the assignment data. This assignment data may include specifying a particular ratio of data transmission interfaces,that are to be assigned to the data input and data output paths. Thus, HBMmay be configured to alter and control the relative bandwidth of its read operations and its write operations.

111 101 111 200 201 210 212 214 214 116 117 119 112 212 210 130 101 119 112 130 201 214 212 119 214 2 FIG. 1 FIGS.A-B The memoryof systemmay be configured as a plurality of memories. For example, memorymay include a plurality of memory dies that are configured as a memory stack, so as to transmit data between each of the memory dies and a base die.is a block diagramof a systemin which HBMincludes a base dieand a stack of memory dies. Memory diesare configured within the stack so as to receive and transmit data from data transmission interfacesand, respectively. In addition, memory dies can be configured to be in communication with transmission controllerand address-processing controllerof base die. Communication between HBMand computing unitcan occur in the same manner as described herein in connection with systemof. However, transmission controller, address-processing controller, and computing deviceof systemmay each be configured to have read and write operations be performed in connection with specific sets of one or more memory dies. In addition, components of base die, such as address-processing controller, may be distributed across memory dies.

101 110 130 190 110 130 190 110 130 300 301 110 390 130 391 1 FIGS.A-B 3 FIG. Systemofinclude an HBMand computing uniton a substrate, which may be a part of a single device. The transmission between HBMand computing unitmay take the form of an electrical connection within substrate. However, in accordance with aspects of the disclosure, HBMand computing unitmay reside in separate devices. For example,is a block diagramof a systemin which HBMresides on a first-device substrateand computing unitresides on a second-device substrate.

184 188 301 110 115 118 164 168 130 134 138 174 178 184 188 110 130 210 130 110 210 101 201 301 130 2 FIG. In addition, transmission paths-may take the form of optical paths, in which data is transmitted over optical cables. In system, the transmission interfaces for HBMmay include transmission interfaces-that are electrically connected to optical interfaces-, while the transmission interfaces for computing unitmay include transmission interfaces-that are electrically connected to optical interfaces-. The length of optical paths-may be 10 meters or more. Accordingly, HBMand computing unitmay reside on devices that are located on devices that reside on different racks of servers. Similarly, HBMand computing unitofmay each reside on different substrates, including being within separate devices. In addition, the HBMandof systems,, andmay take the form of any high-bandwidth memory, such as dynamic random access memory devices. The HBM may also be configured to transmit data via peripheral component interconnect express (PCIe) connections. In addition, the computing unitmay be one of a variety of different types of processors, e.g., a CPU, GPU, an FPGA, an ASIC such as an TPU, etc.

4 FIG. 1 3 FIGS.- 4 FIG. 400 101 301 400 110 130 400 400 401 408 401 408 is a flow diagramin accordance with aspects of the disclosure. The components of systemsanddescribed above may perform one or more of the operations described in flow diagram. For example, HBMof, in communication with computing unit, may be configured to perform operations in accordance with flow diagram. While flow diagramofpresents blocks-in a particular order, one or more operations associated with blocks-may be performed in another order in accordance with aspects of the disclosure. Some operations may also be removed and other operations added without deviating from aspects of the disclosure provided herein.

401 402 In accordance with block, an HBM may assign data transmission interfaces to operate as part of either a data input path or a data output path. The assignment of the data transmission interfaces may be in accordance with predetermined settings, including settings that the HBM received from a computing unit. The HBM may identify requests for read and write operations, such as those transmitted by a computing unit (block).

403 404 405 As described herein, the requests for read and write operations may be transmitted concurrently with one another. In accordance with block, a coherency check may be performed to identify any potential conflicts between the read and write operations. If a conflict is found to exist, the read and write operations are not coherent, and an analysis of the read and write operations is conducted to identify the proper order in which the read and write operations are to be performed (block). The HBM may proceed to perform the read and write operations in the identified order (block). If no conflict exists, so that the read and write operations are identified as being coherent, HBM may perform the read and write operations without requiring that they be performed in a particular order. Thus, read and write operations may be concurrently conducted in parallel to one another, with the read operation data being output by the data transmission interfaces that have been assigned to the data output path, and the write operation data being input by the data transmission interfaces that have been assigned to the data input path.

407 402 In accordance with block, a determination can be made whether to terminate the HBM's transmission mode. This may be based on the HBM completing the requested read and write operations and a determination being made that the HBM should enter into a non-transmission mode. The non-transmission mode may be based on diagnostics that are to be performed on the HBM, but may also be based on the HBM receiving a command to re-assign one or more of the data transmission interfaces. If HBM is not to terminate its current transmission mode, the system may proceed to identify additional read and write operations in accordance with block

407 408 401 402 If HBM receives a command to terminate its transmission mode (block), it may determine if the HBM has also been commanded to re-assign one or more data transmission interfaces (block). If it is determined that one or more of the data transmission interfaces are to be re-assigned, the HBM can perform the re-assignment in accordance with block, and the system can then proceed to again identify read and write operations in accordance with block. As discussed above, the re-assignment of the data transmission interfaces can result in a change in the relative number of data transmission interfaces that are included in the data input path and the data output path. Thus, HBM is capable of altering the relative bandwidth of its data input and data output paths.

Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as “such as,” “including” and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. Further, the same reference numbers in different drawings can identify the same or similar elements.